Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) Report Number : ICRR0020287 1. Project Data Project ID Project Name P114174 HT EFA FTI Catalytic Fund Country Practice Area(Lead) Haiti Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-97009 31-Oct-2015 22,000,000.00 Bank Approval Date Closing Date (Actual) 11-Dec-2009 31-Oct-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 22,000,000.00 22,000,000.00 Revised Commitment 21,999,969.10 21,999,969.10 Actual 21,999,969.10 21,999,969.10 Sector(s) Primary Education(78%):Health(14%):Public Administration - Education(8%) Theme(s) Education for all(82%):Child health(9%):Other human development(9%) Prepared by Reviewed by ICR Review Coordinator Group Sheri Lee Bastien Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The objective of the project, according to the multi-donor trust fund (MDTF) grant agreement (GA) dated June 10, 2010, was "to assist the Recipient in implementing its National Education for All Strategy (SNA/EPT) through the improvement of: (a) access to primary education for poor children aged 6-12; and (b) equity, quality and governance in the Recipient’s education sector." b. Were the project objectives/key associated outcome targets revised during implementation? Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) No c. Components Component 1: Improved Access and Equity of Primary Education (Cost at Appraisal: US$12.5M; Actual Expenditure: US$17.26M). Component 1 was initially to co-finance an existing tuition waiver program implemented by the Ministry of Education and Vocational Training (MENFP) and several donors. The program provided per-student subsidies to cover costs of tuition, fees, and learning materials for poor students who were not enrolled in school to attend non-public primary schools free of charge. Component 2: Improved School Performance in Primary Education (Cost at Appraisal: US$7.7M; Actual Expenditure: US$3.66M). Component 2 consisted of two sub-components and was to improve school performance in primary education through investments in addressing nutrition and health of school children, as well as the provision of school supplies and materials. Sub-component 2.A: Nutrition and Health Program. This sub-component was to finance a school feeding program and the distribution of albendazole for de-worming alongside the implementation of a School Health Promotion policy that included a comprehensive package of health and nutrition interventions, including a focus on the school environment. Schools selected for the tuition waiver program were to be prioritized to maximize synergies. Sub-component 2.B: School Kits to Public Schools. This sub-component was to finance the provision of school supplies and materials, including a container for proper storage, to all public primary schools in Haiti. Component 3: Capacity-Building to Improve Governance of MENFP (Cost at Appraisal: US$1.7M; Actual Expenditure: US$1.07M). Component 3 consisted of four sub-components and was to strengthen the institutional capacity of the MENFP to improve service delivery and overall education governance. Sub-component 3.A: School Construction Unit (Direction de la Genie Scolaire (DGS)). This sub-component was to modernize and strengthen the institutional capacity of the School Construction Unit, a department within the MENFP, with respect to the construction and rehabilitation of classrooms. Sub-component 3.B: Enhanced Statistical and Information System. This sub-component was to enable the MENFP Planning Department (DPCE) to identify and prioritize government interventions relating to the supply and demand side of the education sector. Sub-component 3.C: National Education Partnership Office. This sub-component was to finance the creation of the National Education Partnership Office (Office National du Partenariat en Education – ONAPE) to provide an institutional structure for policy dialogue, strategic planning, and operational collaboration. Sub-component 3.D: Strengthening the central and regional level structures of the MENFP. This sub-component was to support the MENFP to develop and implement a comprehensive capacity building plan using the "EFA-FTI Guidelines for Capacity Building Assessment" and also provide resources to carry out work related to Catalytic Fund (CF) project coordination. Revised components: In March 2010, following a major earthquake in January of that year, a Technical Note was approved that proposed the following reallocations. Component 1 was to provide support for restarting educational services in earthquake-affected areas. This component was to focus on the provision of compensation to the teachers in target areas within three earthquake-affected departments, as a temporary measure. Component 1.B was to continue to focus on the nutrition and health of school children. Component 1 at appraisal was to cost $US12.5M, and was revised to $US 16M. Component 2 was to focus on a nutrition and health program, including financing of school canteens. In addition, sub-component 2.B was to support the development of educational tools adapted to the teaching situation as well as ongoing training of teachers post-earthquake. Component 2 at appraisal was to cost $US7.7M, and was revised to $US 5 M. Component 3 was to focus on the reconstruction of the capacity of the Ministry of Education, and include purchasing of emergency equipment and technical assistance. Component 3 at appraisal was to cost $US 7.7M and was revised to $US 1M. In September 2011, a Technical Note was approved, resuming the focus of improving access and equity in primary education. The main change was the reallocation from the temporary, one-year teacher compensation program to the Tuition Waiver Program. Component 1 was to continue to provide compensation to teachers in target areas, as well as interventions to improve the nutrition and health of school children, but in addition provide 235,000 school children with school fee subsidies in eight departments. The cost of Component 1 was revised to $US 15M. Components 2 and 3 remained unchanged. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost and financing The Education for All Fast-Track Initiative Catalytic Fund Project for Haiti was funded by the Haiti Education for All (EFA) Multi-donor Trust Fund (MDTF). The original total commitment to the project was US$22.00 million. The ICR (page iv) notes a decrease in cumulative disbursements reported in Implementation Status Report number 3 due to the refund of unused, direct payment funds (US$4,675,000 – January 5, 2012) from the teacher compensation component. An initial amount of US$11.735 million had been Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) disbursed to ensure that funds were available after the earthquake. The unused funds (US$4.675 million) were subsequently used at a later date for the Tuition Waver Program. According to the ICR (page 23), the amounts are still being finalized; however, disbursement was reported at near 100% (US$21.99 million) at project closing. Borrower contribution No Recipient contribution was expected or made. Dates The project did not have an approved project appraisal document (PAD). The PAD for the original, pre-earthquake project was drafted in September 2008 for EFA-FTI funding, but according to the ICR (p. 5), approval was delayed due to a contention by UNICEF that it was the appropriate agency to implement education projects in a fragile state such as Haiti. The Multi-Donor Trust Fund (MDTF), which ultimately funded the project, was not approved until December 11, 2009, and ultimately the Bank was designated to be the Supervising Entity (SE) for the project. In addition to a memo from Robert Prouty (Head of FTI Secretariat) dated March 17, 2010, two Technical Notes (Notes Techniques) dated March 2010 and September 2011 were submitted and approved that document revised project activities after the January 12, 2010 earthquake. These were not appended to the ICR, but were provided by the project team upon request. The project closed as scheduled on October 31, 2015. 3. Relevance of Objectives & Design a. Relevance of Objectives Relevance of the project’s objectives is rated Modest. Haiti is classified as a "fragile state" due to prolonged political instability and economic crisis and a low standard of living. Haiti’s education sector at appraisal faced several challenges, according to the PAD (p. 8), including low and inequitable access, poor levels of education quality, and weak institutional capacity. Other challenges noted by the ICR (p. 1) include the high proportion of children enrolled in private schools requiring fees and a substantial proportion of children that are malnourished. Educational infrastructure was also damaged by four cyclones in 2008. After project appraisal, on January 12th, 2010, an earthquake struck Haiti that resulted in hundreds of thousands of fatalities, a large number of displaced persons, and sustained damage to infrastructure; for instance, the Ministry of Education buildings and school buildings were destroyed in several areas of the country. In March 2010, in the aftermath of the earthquake, the Government of Haiti established an interim action plan to restore services in the country, including access to education. The ICR (p. 1) reports that the project had to be redesigned to address the urgent needs of the emergency situation post-earthquake. The objective of this project to improve access was aligned with the SNA/EPT, the sector plan under implementation at the time of the ICR, and the Bank’s Country Partnership Framework (FY2016-2019). The PDOs in particular related to access, equity, and quality remain relevant and consistent with the Bank’s Country Assistance Strategy (CAS) (FY2016-2019). Given the high poverty rate and the large number of out- of-school children, the focus on access and equity was relevant to the context. However, the ICR reports that the objectives of quality and governance were of lesser focus and not as relevant to the country context, particularly in the post-earthquake period where the greatest need was to restore access to schools for students most in need. In the final months before project closing, when the emergency period had passed, the ICR states that quality and governance received greater attention in ongoing dialogue and priority-setting, which is evidenced in the Country Partnership Framework for Haiti (FY2016-FY2019). The ICR notes (p. 12) that in retrospect, the project would have been strengthened by a narrower scope of its objectives at the outset or through timely restructuring to focus only on access and equity. Because of the limited relevance of the quality and governance objectives to country conditions throughout most of the project's lifetime, Relevance of Objectives is rated Modest. Rating Modest b. Relevance of Design The project’s design was modestly relevant to achievement of its objectives. The project included activities in Objectives 1 and 2 that would plausibly lead to improved access and equity by addressing cost, which is one of the main barriers in the Haitian context. The teacher compensation grants were an appropriate temporary measure in the post-earthquake context to ensure that schools would remain open. When Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) the emergency phase had passed, the remaining funds were to be transferred to the tuition waiver program, which the ICR states is a program with evidence of effectiveness. As part of Objective 3, the student nutrition and health-related activities were also relevant to the context, as there were many children with nutrition deficits and in need of de-worming, and the planned activities would therefore likely improve the learning capacity of school children. The school canteen was reportedly needed to increase the likelihood that students would attend school, but also to ensure poorer students’ capacity to learn, thereby addressing the objectives of improved equity and quality. However, as noted elsewhere, quality was poorly operationalized, and there were too few activities both planned and implemented that would have contributed to improvements in quality The project activities included in Objective 4 that aimed to build capacity to improve governance of the MENFP were appropriate to improve service delivery and overall governance in the education sector. The project would have benefitted from an articulated causal chain at the outset demonstrating how each of the components would achieve improvements in access, equity, quality and governance. Rating Modest 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective Improved access to primary education for poor children aged 6-12 Rationale The ICR indicates that there was "no formal revision" to the PDO and uses the indicators and targets that were set at first supervision, which included: a) number of schools participating in the teacher compensation program (3,000); b) number of children participating in school health and nutrition programs (50,000); and c) number of children benefitting from primary education enrollment subsidies (83,000). The indicator on children benefitting from primary education enrollment subsidies was added in 2011 to capture a reallocation of funds to the tuition waiver program. The ICR (p. 22) notes that there is a discrepancy with the Borrower’s ICR, which uses the original targets from the 2009 PAD. The PAD was never formally approved, and therefore the ICR uses the original targets established at supervision. Outputs: No original target values were established in project documents related to the number of schools participating in the teacher compensation program or number of children benefitting from primary education enrollment subsidies. Targets were set at the first supervision mission. • One-time compensation grants were made to 2,824 private schools to participate in the teacher compensation program, essentially meeting the formally revised target of 3,000. • 93,624 children benefited from primary education enrollment subsidies, exceeding the formally revised target of 83,000. Outcomes: A thorough assessment of the objective to improve access is difficult, as the project did not include adequate outcome indicators to convincingly demonstrate the impact of the project. Ideally, the project would have included data on increases in enrollment rates and decreased repetition specific to project schools, among other relevant indicators. The ICR (p.13) reports that according to field interviews, the grants to schools enabled the retention of teachers and covered other eligible expenditures that the schools could not have afforded in the aftermath of the earthquake. Annex 11 (in French) references beneficiary testimonial evidence underscoring the value of the Tuition Waiver Program. The ICR (p.14) notes that additional evidence for improved access for both boys and girls through the provision of tuition waivers can be found using data from the Haiti national school census 2011/12. In addition, the ICR cites the World Bank Haiti Poverty Assessment as an indication that overall primary school enrollment increased significantly in Haiti during 2011-2012. Given the random selection of schools and the focus on the 2011/12 school year (which is when the project funded tuition waivers), it is plausible that similar results would have been obtained for the project. Finally, the ICR notes on page 14 that schools whose students received tuition waivers had an average of 88 more students in grades 1-4 (grades which benefitted from funding by the project in 2011/12), than schools that applied to the program but were not selected, with a roughly even gender split. However, it is important to note the challenge of attributing specific outcomes or trends to this project in the absence of more detailed information concerning access. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) Rating Substantial PHREVISEDTBL Objective 2 Objective Improved equity in the education sector Rationale Outputs: No original output target values were established in project documents related to improved equity of primary education. The outputs are therefore those discussed above, under the first objective. Outcomes: A robust assessment of the objective to improve equity is challenging given the limited data provided by the ICR, which should have included indicators that demonstrate decreased inequity in access disaggregated by sex, socio-economic status/income quintile, and/or rural/urban residence, among other potentially relevant measures. The ICR does indicate a 10% reduction in students repeating grades 1-6 and a 9% reduction of overage students in grades 1-4, with an equivalent reduction for boys and girls, an indication of gender equity. The ICR also rightly points out that poverty is widespread in Haiti (78% pre-earthquake), making it likely that project resources would reach the most poor and vulnerable students. It is also relevant to mention that the Tuition Waiver Program was designed with a mechanism to concentrate the allocation of resources to the poor by providing subsidies that were sufficiently low so as to not attract the interest and applications of students at more well-resourced schools, thereby ensuring that funds reached those most in need. The teacher compensation grant was similarly structured to be sufficiently modest to avoid attracting the interest of teachers at more well-off schools, with applicant schools having slightly worse (on average) student-teacher ratios and repetition rates. The ICR (p. 15) references a comparison of schools that applied to the program with non-applicant schools, confirming that the Tuition Waiver Program benefited average or slightly worse-than-average schools with regard to resources. The ICR (p. 15) also mentions that with respect to equity, school selection took place according to a "poverty map," which enabled the project to be successful in ensuring the desired geographic distribution of the project and its reach into rural areas where poverty rates tend to be much higher. As with the access objective, the ICR would have been strengthened with a discussion of attribution, as there are likely other factors related to improved equity, such as other donor-led projects. Rating Substantial PHREVISEDTBL Objective 3 Objective Improved quality in the education sector Rationale Outputs: No original target values were established in project documents related to the number of children participating in the school health and nutrition program. • 62,000 children participated in the school health and nutrition program for a 110-day school year, exceeding the revised target of 50,000. • A contract was signed with UNESCO for curricular and pedagogical support to the MENFP in September 2013. The project also aimed to achieve quality improvements through its support to an early childhood development (ECD) action plan and curriculum development, as evidenced by the output measures of a signed contract with UNESCO for curriculum development and pedagogical support. The ICR states that the small curriculum development and ECD activities were not implemented successfully due to contract delays and lack of policy guidance, with only half of the activities implemented, including the development of an ECD strategy, a workshop on the implementation of the ECD strategy, and the purchase of equipment to support the ECD unit at MENFP. The action plan Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) called for the building of ECD classrooms; however, the proposal did not comply with a requirement that implementation be carried out by a third party, and therefore this activity did not take place. The ICR (p.9) also reports that the action plan called for an inventory of local educational materials to ascertain the need for pre-school materials; however, this inventory was not conducted. The last Implementation Status Report (ISR) indicates that the implementation of an Early Childhood Development (ECD) action plan is still in progress. Outcomes: The ICR does not provide concrete information on the extent to which these outputs resulted in improvements in the quality of education. This makes it difficult to assess whether or not the intended outcome of this objective was achieved. More robust M&E would have ensured that relevant indicators such as primary school transition or completion rates were tracked, or included other potentially relevant indicators such as the number of instructional hours, relevance of the curriculum, and others. A system for assessing student learning and evidence through improved test scores would have been most appropriate to generate credible evidence on achievement of this objective. There are no indicators related to quality in the project’s results framework, making it difficult to assess the achievement of this objective. The ICR (p. 16) cites several proxy indicators from the project and other sources. For instance, it notes that design features of the Tuition Waiver Program had potential impacts on educational quality through its requirement that schools hire qualified teachers and provide each student with at least three textbooks, and that students be accepted between the ages of 6 and 8 into the first grade with support through to the sixth grade. The ICR also mentions, as earlier cited, a 10% reduction in students repeating grades 1-6 and a 9% reduction of overage students in grades 1-4, with an equivalent reduction for boys and girls, indicating improvements in quality. It is acknowledged that a reduction in the families’ financial burden was likely a contributing factor in reduced rates of repetition and overage students. The ICR (p. 16) acknowledges that these achievements, as evidence of improved educational quality, are modest. Other sources of evidence that are mentioned include field interviews indicating that children participating in the program are more likely to attend school, have a greater readiness to learn, and have a higher equality educational experience. Without additional data to confirm and triangulate, this assertions are not confirmable. Rating Modest PHREVISEDTBL Objective 4 Objective Improved governance in the education sector Rationale Outputs: No original output target values were established in project documents related to capacity to improve governance of the MENFP. The following activities were undertaken: • Provision of vehicles and equipment to the Ministry of Education and Vocational Training • Support for the establishment of an ECD policy • Funding to the Project Coordination Unit to implement the Catalytic Fund project Outcomes: The project did not achieve the intended target of implementing an institutional strengthening action plan. In spite of the four sub-components initially envisaged to build capacity to improve governance of the MENFP, there were no outcome indicators for governance and only one specified intermediate indicator, and this was not met within the lifespan of the project. The ICR (p. 16) describes this objective as "minor and indirect." This highlights the inadequacy of the project’s design and the weakness of the M&E system to capture the impact of the operation. The ICR (p. 17) reports that in spite of this, there were some improvements in governance, and the project employed third-party verification of its main components. It is noted that the project helped the MENFP build capacity and governance through the purchase of vehicles and equipment to help carry out monitoring activities. A beneficiary survey was conducted of 9 of the 10 directors of regional education departments (DDEs) concerning the impact of the resources provided by the project, which indicate that without the resources provided by the project, eight of nine respondents would not have been able to carry out their main duties, in particular with respect to responsibilities that required transportation and computers. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) Rating Modest 5. Efficiency The ICR (p. 17 and annex 3) reports an economic rate of return analysis based on the estimated value of an additional year of education and the likelihood that the project made that additional year possible. An internal rate of return of 11.8 percent is estimated for the three components, including teacher compensation, the Tuition Waiver Program, and the school canteen. The net present value (NPV) is US $16.6 million, and the ICR concludes that value for money in the project was achieved. The economic analysis for the estimated gains to educational attainment from tuition waivers distinguishes between students who are classified as extensive (subsidized students who would not otherwise have been in school) and intensive students (subsidized students who would have been in school and less likely to drop out), based on a conservative estimate that only 10% of students are extensive and 90% are intensive. Other assumptions in the economic analysis include coefficients from the Mincer equation, where total gains from the project (not expressed in monetary terms) are multiplied by the years of education coefficient from the Mincer equation (0.162) and subsequently multiplied by the current per capita income in Haiti. The ICR reports that the project was substantially efficient, as it met the per-pupil cost standard laid out by EFA-1 and EFA-2, which cost US$0.66 per meal and US$90 per student per year, respectively, with a reasonable cost for administration of the programs. Nearly all of the funds were disbursed, and there were not extensions of the project. The ICR reports that the project also contributed to substantial improvements in internal efficiency through reduced repetition and potentially reduced dropouts by overage children, which reduces overall costs in the education system. The project was successful at identifying and rejecting false applications for school grants. In addition, according to the ICR (p. 8), the Bank, together with the MENFP, worked to improve the transparency and timeliness of verification, and at project close there were remaining funds that were used to fund one tuition waiver tranche for 2014/15, with a second tranche provided by an unidentified development partner. With respect to implementation challenges, several issues were identified by the ICR that significantly hindered project efficiency, including staff turnover and the lack of a financial management specialist, a lack of bookkeeping during 2014-2015, and inadequate information on disbursements by source of funds and category. In addition, the ICR notes slow selection processes for the school canteen contracts, as well as ineffective use of available resources for curriculum development resulting in a necessary reallocation of funds to the Tuition Waiver Program. However, once reallocated, the program was implemented rapidly, although schools did experience significant delays in receiving the second tranche of the tuition waiver due to difficulties in verifying student attendance at recipient schools. Other approaches that could have led to similar outcomes in the project are not presented or discussed, making it difficult to ascertain whether or not an alternative strategy could have led to a more efficient use of resources. A sustainability analysis would also have been beneficial, and a more robust analysis on the return of investments to health of school children to improve quality of learning and cognitive capacity would be important for future decision making and policy. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 100.00 ICR Estimate  11.80 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) 6. Outcome The ICR (p. 2) focuses on the "major intent" of the CF project, improved access and equity, in line with the post-earthquake APNRD (Action Plan for National Recovery and Development of Haiti). It observes that three relatively small subcomponents (less than 15 percent of total planned project costs) also aimed at improved education quality and governance, and that therefore the quality and governance objectives should be assessed proportionately to their importance in the Project as defined by these planned costs. The ICR (p. 2) also states that the grant agreement intended to cover additional activities through the MDTF, including school reconstruction and teacher training, that were to contribute to the quality and governance objectives; however, these activities were to be financed by other donors, and this financing never materialized. The ICR argues (p. 17) that while the project must be held accountable for the "weak performance of a few activities, it is also necessary to weigh their lesser importance to the overall project." IEG/OPCS guidelines, however, state that all objectives are to be considered equally important unless the grant agreement or PAD explicitly state otherwise, and relative importance of objectives is not measured by the proportion of resources allocated to them. There is no formal statement in the grant agreement of PAD for this project indicating that any objectives are to be considered more or less important than others. This Review follows those guidelines and assesses Outcome with the four objectives equally weighted. Relevance of Objectives is rated Modest, as two of the four objectives were not strongly relevant to country context throughout most of the project's lifetime. Relevance of design is rated Modest due to a lack of activities that would plausibly contribute to improved quality of education. Achievement of the objectives to improve access and equity of primary education are both rated Substantial. Achievement of the objectives to improve quality and governance in primary education are both rated Modest. Efficiency is rated Modest. Taken together, these ratings are indicative of significant shortcomings in the project’s preparation and implementation, leading to an Outcome rating of Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating The ICR reports (p. 19) that the outcomes have already been banked by: a) keeping schools open and running in the aftermath of the earthquake; b) improved attendance and readiness to learn of primary school students; and c) improved access to schools through the tuition waiver program. However, the sustainability of some of these activities is uncertain due to the need for continued financing from the Government of Haiti. The ICR (p .11) reports that some programs, including the tuition waiver and school health and nutrition programs, continue to be supported by a second Education for All (EFA-2) project (2011-2017) and will be followed up by a new IDA operation that is under preparation. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry Although the Bank, the GPE and the LEG contributed to the development of an appropriate project design given country context, the ICR (p. 19) acknowledges that the Bank did not follow established protocol for project approval. There was no formally approved PAD, and the project was not appropriately designated as an "emergency" project, which would have ensured a more timely process for project approval and provision of resources. Instead, the project was approved as a Trust Fund Program (TFP) as part of the EFA MDTF. As a consequence, the project was defined in focus and scope by a six-page Technical Note (Note Technique) that, according to the ICR (p. 19), did not appear to be a part of either the MDTF negotiations package or the TFP. This contributed to delays that could have been minimized by following the Bank’s approval procedures. An alternative option the Bank could have pursued to reduce delays, according to the ICR, was to designate the funds for EFA-2 rather than channel the funding through the MDTF. Taking into account these issues, as well as shortcomings with both relevance of objectives and relevance of design and the lack of a formal Results Framework, Quality-at-Entry is rated Moderately Unsatisfactory. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Given the complexity of the context, the ICR (p. 20) reports that the Bank provided close implementation support for the majority of planned activities and demonstrated flexibility where needed, for instance concerning the teacher compensation component that needed to be adjusted when only half of the allocated funds were disbursed. However, as noted in the ICR (p. 9), there was no midterm review of the project, and it was never designated "at-risk" by any of the Implementation Status Reports (ISRs), even though the overall ratings for progress towards achievement of PDOs were downgraded in April 2013 from Satisfactory to Moderately Satisfactory and in November 2013 from Moderately Satisfactory to Moderately Unsatisfactory. A substantial shortcoming was the failure to formally restructure the project once the emergency phase subsided post-earthquake, which would have provided an important opportunity to strengthen the Results Framework. It is not clear how the quality of the Bank’s supervision affected the small curriculum development and ECD activities, which were not implemented fully. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The ICR (p. 20) reports that the Government/MENFP worked closely with the Local Education Group (LEG) and the Bank to develop the original project proposal, which built on the experience of the EFA-1 project but was only modestly relevant to country context. The Government was responsive to the earthquake, as evidenced by the APNRD (Action Plan for National Recovery and Development of Haiti) and the Technical Note (Note Technique) developed by the MENFP in order to rapidly assist the education sector. According to the ICR, there were no issues with compliance to covenants, and the grant was nearly fully disbursed ($21.99 million). However, high turnover of ministers at MENFP (five during the project's lifetime) produced implementation challenges. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance Although the Project Coordination Unit (PCU) administered the project with close to full disbursement, the ICR (p. 20) identifies a number of substantial shortcomings related to M&E, with ongoing challenges associated with interpreting the relative importance of indicators. In addition, there were issues with financial management, as evidenced by an unsatisfactory FM rating at project close (see Section 11b). The high turnover of MENFP ministers during the course of the project produced challenges with financial management, procurement, and the final accounting process. Moreover, there was ineffective use of resources available for curriculum development, and as a result, most of the resources allocated for this activity had to be reallocated to the TWP (2014/15). Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) a. M&E Design The project lacked a results framework (RF) at inception. This is a substantial shortcoming that makes it difficult to evaluate achievement of objectives and ultimately attribution of observed outcomes to the project. The absence of an articulated design also resulted in a missed opportunity to monitor project progress, inform decisions around project restructuring as necessary, and more generally, learn from project implementation. The ICR (p. 10) reports that an RF was developed after the earthquake that focused on output measures for the PDO and intermediate indicators. As reported in the ICR (p. 10), the PDO indicators related primarily to access to education and measured numbers of beneficiaries (operationalized as schools participating in the teacher compensation program, primary students receiving tuition waivers, and number of children benefitting from the school health and nutrition program). The indicator targets primarily reflected the resources allocated for the activity divided by the benefit amount either per school or per student. The ICR acknowledges the limitations of the RF and the inadequacy of the indicators to provide evidence concerning the concrete impact of the school grants and tuition waver program (TWP) on improved educational access, equity, and quality for students. Indeed, access, equity, quality and governance in the education sector are not properly defined or operationalized in project documents. Data were not disaggregated by sex, socio-economic status, rural/urban residence, or other factors important to assess the achievement of the equity objective. There were no outcome indicators related to governance. With respect to quality, relevant indicators or data might have taken into consideration teacher-student ratios, textbook-student ratios, issues related to inclusiveness, effective teaching and learning, safe environment (construction/latrines), local participation, effective school management, or language of instruction, all of which are acknowledged in project documents to be of importance to the Haitian context. According to the ICR (p.10), weaknesses in the project’s M&E design were addressed by employing third parties to verify the legitimacy of applications for teacher compensation, implementation of the school health and nutrition program, and implementation of the TWP. In the absence of baseline data, the project could have made more strategic use of national level surveys (though these may not be available in a fragile state such as Haiti) or other data sources to supplement project level data. b. M&E Implementation The M&E data collected in the project tracked the implementation of three project components: teacher compensation, the TWP, and school nutrition and health programs. A call center was reported to improve M&E with real-time data during 2014/15, helping to identify and remedy issues with the program, in particular relating to quality of food services and improved governance of the program. The teacher compensation component included verification by a third party to determine school eligibility, which assisted in identifying and screening out bogus applications. However, as noted in the ICR, there were substantial weaknesses in M&E that prevented the project from assessing higher level impacts (including concrete outcomes) and conducting a robust analysis of returns to project investments. There was inconsistent reporting of objectives and indicators across Implementation Status Reports (ISRs). Specifically, none of the ISRs, with the exception of the final ISR, mention the objectives related to quality and governance in the education sector. Other inconsistencies include the number of children benefiting from primary education enrollment subsidies (reported as an end target of 83,000 in the last ISR, while the ICR uses an end target of 93,624). c. M&E Utilization The ICR (p. 10) reports that M&E data were used to track progress and guide decisions concerning reallocations of funding. However, the M&E system did not generate data and analysis that might have allowed for course correction through formal restructuring or for a more comprehensive analysis of returns to project investments. As the ICR notes, a tracking system of cohorts of students from first to sixth grades who received tuition waivers would have provided credible evidence of achievement of the project's stated objectives. M&E Quality Rating Modest 11. Other Issues a. Safeguards According to the ICR, the project was classified as an Environmental Category ‘C’ project. No environmental or social safeguard policies were triggered, given that there were no planned construction activities or plans that would have potentially negative environmental or Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) social impacts. b. Fiduciary Compliance Financial management (FM): The teacher compensation component was contracted to a payment agency, Fonkoze. The ICR reports that Fonkoze’s performance in making payments to the schools (US$2,500 school grants) and uncovering and rejecting false applications for funding, in spite of challenges associated with determining school eligibility, was satisfactory. The Project Coordination Unit (PCU) suffered the loss of two lead specialists who resigned in September 2015, creating challenges for FM, procurement, and the final accounting process. The Bank provided support to the PCU to address a number of issues identified in the ICR in an effort to improve the Unsatisfactory rating for FM in the final archived Implementation Status Report, including a category 5 disbursement condition that rejected a proposal by the Ministry of Education and Vocational Training (MENFP) for classroom construction, a UNESCO refund, and a review of an unqualified audit in 2014 despite a lack of access to accounting records. However, the FM rating remained Unsatisfactory at closing due to the following shortcomings identified in the ICR: a) absence of an FM specialist; b) lack of bookkeeping during the 2014-2015 period; and c) unavailability of information on disbursements by source of funds and category expenditure. The ICR (page 11) also notes that at the time of the report, progress had been made to address these shortcomings in FM, including hiring of a new FM specialist. Procurement: The main challenges identified with procurement include the need for improved planning around the selection process for contracts and the delivery of services. The ICR notes that the UNESCO contract was a single-source selection that took more than 18 months of negotiations to finalize. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Two of the four PDOs were not substantially relevant to the country context. Design was modest given the limited planned Outcome Moderately Satisfactory Moderately Unsatisfactory activities to improve education quality. Achievement of two PDOs was downgraded due to lack of evidence of achievement. Efficiency was modest. Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Unsatisfactory Moderately Unsatisfactory --- There were significant shortcomings in M&E, financial Borrower Performance Moderately Satisfactory Moderately Unsatisfactory management, and procurement, as well as some implementation inefficiencies. Quality of ICR Substantial --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT EFA FTI Catalytic Fund (P114174) Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons offered by IEG include: • Sound M&E design in place at the outset of a project, with indicators and targets that are specific, realistic and measurable, is necessary to ensure appropriate and adequate assessment of progress and achievement of stated objectives. Mid-term reviews are also key to ensuring that projects remain relevant and on-track and that results are measurable. • Additional Bank supervision, careful adherence to Bank procedures, and rigor in Implementation Status Reports in projects that are identified as ‘potential problem projects’ or ‘at risk’ and which take place in a disaster or emergency context is essential, in case it is necessary to restructure and for effective monitoring and implementation. • Clear documentation regarding implementation processes is important for assessment of progress, in this case concerning the selection, vetting and monitoring process of grant compensation applications by an externally contracted agent. 14. Assessment Recommended? Yes Please explain This project may hold lessons for other fragile states and/or disaster/emergency contexts. 15. Comments on Quality of ICR The ICR effectively highlights the complexities of the Haitian education sector context both pre- and post- earthquake. It also includes candid reflections on the project's implementation challenges and the shortcomings in the quality of evidence available to demonstrate achievement of project objectives. Evidence from outside the project's formal results framework is used to assess achievement of objectives. The formal economic analysis is based on sound assumptions. The ICR provides an assessment of the challenges of donor coordination in a post- disaster context. There are however, several shortcomings in the ICR. It does not critically discuss attribution of observed outcomes to the project's interventions. The ICR could have provided a clearer explanation of the timeline of the project, including the reallocations of funds, as well as the evolution of the indicators and targets. Finally, the ICR argues that the objectives of the project should not be weighted equally, even in the absence of this assertion in project documents, which is not in keeping with IEG/OPCS guidelines. Despite these shortcomings, however, the overall quality of the ICR is substantial. a. Quality of ICR Rating Substantial