Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Report Number: ICRR0021697 1. Project Data Project ID Project Name P126357 NI 2nd Support to the Education Sector Country Practice Area(Lead) Nicaragua Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-50360,IDA-59060 30-Jun-2016 28,151,374.20 Bank Approval Date Closing Date (Actual) 17-Jan-2012 30-Sep-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 25,000,000.00 0.00 Revised Commitment 29,999,996.65 0.00 Actual 28,151,374.20 0.00 Prepared by Reviewed by ICR Review Coordinator Group Eduardo Fernandez Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) The Project Development Objective (PDO) as stated in the Project Appraisal Document (PAD) was to: “(a) improve the students’ retention rate in Primary Education Schools located in Participating Municipalities; and (b) strengthen the Ministry of Education's (MINED’s) education management capacity” (PAD, p. 5). The Financing Agreement provided an identical formulation (FA, p. 5). Throughout implementation, there were no changes to the PDO or the overall intended outcomes (ICR, p. 10). However, the PDO outcome indicators were revised in terms of wording, and in some cases, baselines and targets were updated. The wording revisions improved the formulation of indicators by bringing them into alignment with international practice, and hence, they do not represent substantial changes in the meaning of the indicators. The project team adjusted baselines and targets to reflect more up-to-date information (namely, 2012 school year data that was unavailable at project appraisal). Although the target for PDO Indicator #1 was revised downwards, a split rating is deemed unnecessary for purposes of this review since the project met both the original and revised target. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-Sep-2015 c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the Schools' Learning Environment (appraisal cost US$9 million; actual cost US$10.28 million). This component sought to increase the quality of learning conditions for students in 40 selected municipalities through: (i) the provision of solidarity packages (“Paquetes Solidarios”: 14 notebooks, 6 pencils, 2 pens, 1 eraser, 1 ruler, and a pair of shoes) to eligible children; (ii) the purchase and distribution of basic furniture for beneficiary schools; and (iii) the development, printing, and distribution of learning and teaching materials for primary schools, including materials for indigenous groups whose primary language is not Spanish. According to the PAD (p. 6), the component also planned to finance a qualitative assessment and an impact evaluation of the impact of the solidarity packages on student retention. The project had originally envisaged to distribute solidarity packages according to a needs-based criterion in rural areas of the targeted municipalities. However, the government rejected this approach and, at the first restructuring, decided to expand the distribution to all students and fully fund the “Paquetes Solidarios” program (see Section 2e, First Restructuring) (Source: TTL interview). With respect to the selection of schools for refurbishment, the project relied on data from a 2012 census of school infrastructure to identify those schools that did not meet minimum furniture requirements (Source: TTL interview). Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Component 2: Improving the Quality of Teaching Methods (appraisal cost US$8 million; actual cost US$14.83 million). Under this component, five sub-components sought to improve the quality of teaching and learning in schools located in the 40 selected municipalities. 1. The first sub-component financed pre-service training for multi-grade primary school teachers in Teacher Education Institutions (TEIs) and minor rehabilitation works and acquisition of supplies for TEIs. Multi-grade schools are in rural areas and, for this reason, the project chose secondary school graduates with good grades who lived in these areas, wanted to pursue a teaching career, and had made a commitment to return to their communities to teach upon graduation (Source: TTL interview). 2. The second sub-component financed in-service training for primary school teachers. Specifically, it financed a monthly Evaluation, Programing, and Educational Training Workshop (TEPCE) for all primary teachers and TEPCE coordinators. The teacher selection process followed a cascade approach by which coordinators were trained in each of the 40 municipalities. These trained coordinators were then supposed to conduct training with the remaining teachers in their schools (Source: TTL interview). 3. The third sub-component financed activities to improve multi-grade and bilingual instruction. Specifically, it financed a multi-grade instruction course on reading, writing, and mathematics; courses on intercultural bilingual education techniques and methodologies for primary education teachers in regions with indigenous populations; and an assessment of Nicaragua’s existent multi-grade teaching program. 4. The fourth sub-component financed the design and implementation of national and international standardized tests to measure learning outcomes of primary education students. 5. The fifth sub-component financed the design, implementation, and evaluation of an accelerated remedial school pilot program for over-aged students in primary schools located in rural areas. Component 3: Strengthening MINED's Education Management Capacity (appraisal cost US$7 million; actual cost US$9.03 million). This component sought to strengthen MINED’s education management capacity through three sub-components. 1. The first subcomponent supported MINED in improving its planning and statistical information systems through the provision of technical assistance and training to MINED staff and the acquisition of specific software and equipment to strengthen MINED’s planning, monitoring, and statistical capacity. 2. The second subcomponent aimed to improve MINED’s capacity to plan and manage school infrastructure and its ability to prepare pre-investment infrastructure studies. Specifically, the subcomponent financed activities to support: (i) the development of a school infrastructure inventory data system and a national geo-referenced database of public schools; and (ii) the elaboration of technical assessments on the conditions of the public-school infrastructure located in the participating municipalities. 3. The third subcomponent financed training and technical assistance for MINED staff, minor maintenance and improvement activities in MINED’s facilities, and the acquisition of software, hardware and small furniture to support MINED in the administration, financial management, auditing, and procurement aspects of the project. Targeting Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) The project targeted Nicaragua’s poorest municipalities with the worst primary education performance as measured by repetition and dropout rates (PAD, p. 6). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost The total project cost was US$28.15 million (ICR, p.41) as compared to an original amount of US$32.95 million (PAD, p. 12). The original amount envisaged US$25 million of IDA financing and US$7.95 million of Borrower financing. Financing The original project amount (US$32.95 million) was to be financed through an IDA credit in the amount of SDR 15.8 million (US$25 million equivalent) and government funds in the amount of US$7.95 million to finance the salaries of the new teachers to be hired and trained during the project implementation period (PAD, paragraph 43). In September 2016, the project received additional financing in the amount of SDR 3.6 million (US$ 5 million equivalent) through an IDA Credit. The original IDA credit disbursed US$23.5 million owing to exchange rate fluctuations, and the additional IDA credit was fully disbursed. The total disbursed amount between the two credits was US$28.5 million. The ICR did not report on whether the Borrower disbursed the full contribution committed at appraisal (US$7.95 million). According to the TTL interview, the Borrower contribution materialized and financed the salaries of teachers hired and the distribution of “Paquetes Solidarios”. The Government has not yet provided data on the exact amount contributed. Dates The project was approved in January 17, 2012, became effective on April 10, 2012, conducted a mid-term review on April 24, 2015, and closed on September 30, 2018. The original closing date was June 30, 2016. The project had three restructurings (September 2015, June 2017, and December 2017) and one additional financing (September 2016). First Restructuring The first restructuring (September 2015) updated baseline data, adjusted targets, and revised the wording and descriptions to improve clarity of four PDO indicators in the results framework (indicators 1,2, 4, and 5). The change in the Results Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Framework also: (i) introduced two new sub-indicators for PDOs 1 and 2; and (ii) revised the wording and targets for eight intermediate indicators and introduced two additional ones. In addition, the restructuring made changes to the scope of the activities of all three components: (i) component 1 was revised to reflect the government strategy to expand and fully fund the “Paquetes Solidarios” program; (ii) component 2 was revised to incorporate training and textbooks on the government’s Learning Strategy with an Early Development-based Approach (ESAEDI); and (iii) component 3 was revised to adjust the number of pre-investment studies (PIS) to reflect a revised government infrastructure strategy that reduced the number of classrooms rehabilitated but ensured that the rehabilitated classrooms had an appropriate functional environment because they were paired with complementary investments (water and sanitation, security, recreational facilities, and electricity). Thus, the overall number of PIS was reduced, while their comprehensiveness increased. The restructuring introduced changes to the implementation arrangements by: (i) revising the structure (names and quantities) of the safeguards and fiduciary positions for the project; (ii) incorporating into the Legal Agreement every single fiduciary position overseeing the project; and (iii) increasing the number of fiduciary staff needed to oversee the project from 23 to 27 (Restructuring Paper 2015, p. III-IV). Finally, the restructuring extended the closing date by six months (from June to December 2016) to ensure the completion of the development of MINED’s new Planning and Monitoring System (PMS), adjusted the implementation schedule accordingly, and changed the financing plans to accommodate a decrease in the dollar value of the Credit as a result of a lower exchange rate of the US dollar (USD) in relation to Special Drawing Rights (SDR) (Restructuring Paper 2015, p. III). Additional Financing The additional financing (September 2016) was requested by the government to (a) cover a financing gap due the depreciation of SDRs relative to the USD; (b) scale up the refurbishing of schools; and (c) extend the implementation of in- service teacher training for an additional year. The additional financing supported: (i) operational and recurrent costs; (ii) the development and implementation of an action plan to improve learning / teaching practices based on the outcomes of the student learning assessments; (iii) the deployment of the first phase of MINED’s PMS; (iv) the development and maintenance of the infrastructure inventory system; and (iv) the completion of required PIS. With respect to the results framework, the additional financing changed target dates for outcome indicators #4 and #5 and revised the formulation of the former. Likewise, targets and dates of achievement were also revised for intermediate indicators. Finally, the additional financing extended the closing date by one year (from December 2016 to December 2017). Second Restructuring The second restructuring (June 2017) extended the closing date by five months (December 2017 to May 2018) to allow time for the completion of the bidding process related to the acquisition of planning and monitoring software under component 3 Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) and the provision of equipment and consumables to schools’ centers to carry out teacher trainings under component 2 (Restructuring Paper, June 2017). Third Restructuring The third restructuring (December 2017) reallocated US$2 million from component 1 to component 3 of the project to accommodate exchange rate fluctuations and the need for increased expenditures under component 3. The fund reallocation to component 3 supported: implementation of a statistical data management system; improvements of the Ministry’s internal communications network; and strengthening of MINED’s school infrastructure planning capacity. Additionally, the restructuring extended the closing date by four months from May 2018 to September 2018 (Restructuring Paper, December 2017). 3. Relevance of Objectives Rationale Relevance objectives is rated High. The PDOs were relevant to the context of the education sector in the country. At appraisal, low retention and completion rates characterized the Nicaraguan primary education system. Despite having universal enrollment in first grade, fewer than 75 percent of children completed primary school (1-6 grades) because of elevated dropout and repetition rates. Moreover, the primary education level was characterized by a large share of overage students (54 percent of primary students) and marked regional disparities. For instance, while the overall dropout rate for primary education was 12 percent in 2008, it reached 29 per cent in some municipalities of the South Caribbean Coast Autonomous Region. Against this backdrop, MINED made universal completion of six grades of primary education a key priority through an initiative called “The Battle for Sixth Grade." Under this initiative, the World Bank provided funding to improve student retention rates in primary education schools in selected municipalities and to strengthen MINED’s education management capacity. At the time of appraisal, the PDOs were aligned with the Government’s Education Sector Strategy (ESS) 2011-2014 that included achieving universal primary education and strengthening the institutional capacity of the education sector among its priorities. Likewise, at appraisal, the PDOs were aligned with the World Bank’s Country Strategy (FY12-FY17), whose Strategic Area 1 focused on raising welfare by improving access to quality basic services such as education. At the time of project closing (September 2018), the PDOs remained highly relevant to government priorities as laid out in the ESS 2017-2021. The ICR reported that the ESS 2017-2021 emphasized equitable access to quality education and MINED’s institutional development among its priorities. Finally, at closing, the PDOs were aligned with Pillar 1 (“Investing Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) in Human Capital for Disadvantaged Groups”) of the World Bank Country Strategy Partnership Framework (FY18-FY22). This pillar focused, among other things, on improving access to and the quality of the education system. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve the students’ retention rate in Primary Education Schools located in Participating Municipalities Rationale The theory of change underlying this objective assumed that supply-side challenges such as inadequate schools and lack of materials for learning and teaching drive low student retention rates in primary education schools. Based on this assumption, the project invested in (i) the refurbishment and furnishing of schools, and (ii) the development, printing, and distribution of learning and teaching materials. Additionally, the theory of change for this objective assumed that low retention rates are driven by demand-side factors such as inability to continue in school because of lack of resources or the need to work and the perception that the benefits and quality of schooling are low. To address these demand-side challenges, the project financed the provision of backpacks with school materials and shoes (“Paquete Solidario”). The ICR noted that empirical evidence indicates that similar demand-side interventions such as the provision of school uniforms have a significant impact on increasing student attendance, retention, and ultimately learning. To improve the perceived value of education and make primary education students less likely to drop out, the project financed improvements in the quality of teaching methods, with a special focus on multi-grade and bilingual education through pre-and in-service trainings and the use of learning assessment to inform improvements in instruction. Finally, the project also sought to increase retention rates through the piloting of a new school modality to enable over-age students to continue and finalize their education. Outputs • 427,917 students in targeted municipalities received “Paquetes Solidarios." Target: 280,000 (exceeded). • A qualitative study of the “Paquetes Solidarios” program was carried out. Target: Yes (achieved). Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) • 2,393 primary schools in targeted municipalities were adequately equipped with furniture (February 2018). Target: 1,500 (exceeded). • The percentage of primary school students with a language and literature textbook increased from 75 percent (March 2012) to 100 percent (September 2018). Target: 95 percent (exceeded). • The percentage of primary school students with a math textbook increased from 56 percent (March 2012) to 100 percent (September 2018). Target: 95 percent (exceeded). • 81,500 books were distributed in six native languages in the Autonomous Regions Educational System (SEAR) (September 2018). Target: 81,500 (achieved). • 103,897 1st and 2nd grade students received ESAEDI textbooks in targeted municipalities (September 2018). Target: 98,000 (exceeded). • 1,568 teacher trainees enrolled in the multi-grade pre-service training program under the project (June 2017). Original and revised target: 1,656 (mostly achieved) and 1,541 (achieved). The original target was revised downwards at the first restructuring (September 2015) to account for the total number of teachers enrolled within the duration of the project. • The percentage of graduates from the multi-grade teacher training program who taught for at least one year in targeted areas under the project reached 91 percent (February 2018). Target: 80 percent. The target was also exceeded for other municipalities (90 percent against an 80 percent target) and SEAR municipalities (92 percent against a 70 percent target). • The average number of primary education teachers from targeted municipalities who participated in in-service training through TEPCE workshops during a given school year under the project reached 8,128 (September 2018). Original and revised target: 6,200 (exceeded) and 7,000 (exceeded). • The average number of primary education teachers from other municipalities who participated in in-service training through TEPCE workshops reached 4,707 (September 2018). Original and revised target: 6,400 (partially achieved) and 4,200 (achieved). • The average number of primary education teachers from SEAR municipalities who participated in in-service training through TEPCE workshops reached 3,424 (September 2018). Original and revised target: 600 (exceeded) and 2,000 (exceeded). • The systematization of the multi-grade modality was officially published by MINED (September 2018). Target: Yes (achieved). • The number of classrooms in targeted municipalities offering the pilot of the accelerated primary school program under the project reached 15 (September 2018). Target: 15 (achieved). • Although not included in the project’s results framework, the ICR reported that the share of multi-grade schools offering all six grade levels of primary education increased by 10 percentage points in targeted municipalities during the timeframe of the project (as comparted to a nationwide 4 percentage points improvement). Moreover, the ICR reported that the project expanded the multi-grade primary teacher workforce by 44 percent in targeted municipalities (by adding 1,568 new multi-grade teachers to the already 3,262 existing teachers) (ICR, paragraph 31). Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Outcomes • The survival rate through 5th and 6th grade increased from baselines 35 percent and 34.7 percent in 2012 to 52.6 percent and 47.3 percent in 2017 (17.6 percentage points increase for 5th grade and 6th grade). Source: TTL interview. • The survival rate through 4th grade in targeted municipalities increased from a baseline of 46 percent (March 2012) to 75.8 percent (February 2019) (29.8 percentage points increase) (PDO Indicator #1). Target: 50 percent (exceeded). The original target was 53 percent (exceeded). The country average survival rate through grade 4 increased by 19 percentage points during the life of the project (ICR, p. 13) as compared to a 29.8 percentage points increase in the targeted municipalities. According to the PAD (p. 6), 4th grade is where most of the dropout and repetition occurred and, for this reason, the project specifically measured impact on survival through the 4th grade. • The survival rate through 3rd grade in targeted municipalities increased from a baseline of 54.4 percent (March 2012) to 81.9 percent (February 2019) (27.5 percentage points increase). Target: 69 percent (exceeded). The country average survival rate through grade 3 increased by 16 percentage points during the life of the project (ICR, p. 13), as compared to a 27.5 percentage point increase in the targeted municipalities. • The survival rate through 2nd grade in targeted municipalities increased from a baseline of 66 percent (March 2012) to 85.2 percent (February 2019) (19 percentage points increase). Target: 72 percent (exceeded). The country average survival rate through grade 2 increased by 12 percentage points during the life of the project (ICR, p. 13), as compared to a 19 percentage point increase in the targeted municipalities. • The project could not collect reliable data on within-year retention rates (PDO Indicator #2) and, for this reason, results for this indicator were not reported (ICR, p. 27). Although the project was not able to produce data on this indicator, it can be plausibly argued that within-year retention rates would improve as a consequence of increased enrollment and survival rates and a decrease in the share of overage students. As mentioned previously, survival rates increased, and primary education enrollment in targeted municipalities increased by 5 percentage points while the country average remained constant (ICR, p. 14). With respect to the share of overage primary education students, targeted municipalities experienced a 5 percentage point decrease by the end of the project (as compared to a 7 percentage point nationwide reduction). Finally, the project delivered interventions to lessen the burden of education and to improve its relevance and quality. These interventions can be reasonably expected to increase within-year retention rates among the most at-risk students. • The percentage of parents who agreed or mostly agreed with the statement “the ‘paquete solidario’ (school kit) was a key reason why I was able to keep my child at school this year” reached 94.3 percent by the end of the project (September 2018). Target: 50 percent (exceeded). Rating Substantial Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) OBJECTIVE 2 Objective Strengthen MINED’s education management capacity Rationale The theory of change underlying this objective postulated that MINED’s education management capacity could be improved through: (i) the development of a system for assessing learning at the primary level; (ii) the elaboration of PIS and the development of a system to inventory school infrastructure; (iii) the development and use of a planning and monitoring system; and (iv) training for MINED technical staff. The system for assessing learning would improve decision making by providing data aimed at informing teacher practices in the classroom. The system to inventory school infrastructure and the PIS would increase MINED’s capacity to build and manage school infrastructure. The development of a new planning, monitoring and statistical system would integrate MINED’s information systems and facilitate the use of information for decision making and service delivery. Finally, training would enable MINED technical staff to better operate the new systems. Overall, it can be plausibly argued that these outputs would increase the availability and quality of data and, thus, improve MINED’s education management capacity. Outputs / Outcomes • The project supported the development of a system for learning assessments at the primary level (September 2018) (PDO Indicator #3). Target: Yes (achieved). The ICR reported that the project supported the analysis for two rounds of national assessments. The data from the learning assessments was analyzed and reported, including disaggregation by gender, urban / rural, and geographical location. Target: Yes (achieved). The results of the learning assessment were used as an input for the formulation of the 2017-2021 ESS (ICR, paragraph 35). • The number of teachers who participated in the dissemination events of national learning assessment results under the project reached 6,145. Target: 6,000 (achieved). • The project supported the development and use of a new planning and monitoring system (PDO Indicator #4). Target: 100% completed or Yes (achieved). • The number of official annual statistical reports published under the project reached 1 (September 2018). Original and revised target: 4 (not achieved) and 1 (achieved). • The project supported the development and use by MINED of a school infrastructure inventory system (PDO Indicator #5). Target: 100% completed or Yes (achieved). Because of the project, MINED was able to elaborate its first ever national inventory of school buildings nationwide and design infrastructure rehabilitation plans based on detailed school-level information collected by specialists on the ground. • The number of PIS developed under the project reached 214 (September 2018). Original and revised target: 586 (partially achieved) and 212 (achieved). As noted earlier, the overall number of PIS was reduced to accommodate a change in the government’s infrastructure strategy (see Section 2e). Out of these 214 pre-investment studies, 88 transitioned to actual infrastructure investments supported by a Global Partnership for Education-financed project Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) (Education Sector Strategy Support - P133557) and a European Union-financed project (“Programa de Apoyo al Sector Educacion en Nicaragua," PROSEN). Rating Substantial OVERALL EFF TBL OLD Rationale Achievement of both objectives is rated substantial, and thus overall efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency The project conducted and ex-ante and ex-post cost benefit analyses. Appraisal At appraisal, the project carried out an economic analysis that covered the benefits and costs associated with Component 1 (36 percent of IDA credit at appraisal). This component included the provision of “Paquetes Solidarios” to students in need, the refurbishing of beneficiary schools, and the development, printing, and distribution of learning and teaching materials for primary education schools. The analysis assumed that these outputs would reduce primary education dropout and repetition rates, thus leading to an average 0.7 years increased educational attainment for the target population that would translate into increased lifelong earnings. The analysis noted that it relied on labor market data included in a 2009 household survey to estimate the earnings premium; however, it lacked detailed information on how the estimation was carried out (PAD, p. 70). With respect to costs, the analysis estimated the following: (i) cost of providing education per primary education student; (ii) cost of providing education to an increased number of students (owing to increased retention rates); and (iii) the cost of providing the “Paquetes Solidarios” program, refurbishing beneficiary schools, and developing learning and teaching materials for primary education. The economic analysis calculated an internal rate of return of 11 percent considering a time horizon of 4 years. The analysis conducted a sensitivity test by reducing the expected increase in average educational attainment, and the results of the analysis still showed a 7 percent internal rate of return. Given the defined scope (only component 1), the analysis argued that the calculation should be Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) considered a lower bound since it only focused on educational attainment, thus not factoring into the analysis other benefits such as the improved quality of education or the positive externalities associated to higher educational attainment. The analysis lacked information on the discount rate used for the internal rate of return estimation. Closing At closing, the project conducted an ex-post economic analysis that covered components 1 and 2 (68% IDA credit at appraisal). The analysis considered the benefits associated with the “Paquetes Solidarios” and teacher training in multi-grade schools and assumed that these two interventions would lead to increased education quality and increased educational attainment. Costs factored into the analysis included: “Paquetes Solidarios” program, teacher training, refurbishment of schools, and provision of learning materials. Using several reasonable assumptions, the analysis estimated a 16.8 percent rate of return (8 percent discount rate and 38-year time horizon). Finally, the analysis carried out several sensitivity tests, and under all the scenarios the estimated benefits of the project outweighed the costs. Implementation Efficiency Overall, the project was implemented efficiently with only moderate shortcomings. In this sense, it is worth noting that the 18- month extension in the original closing date was not driven mainly by implementation inefficiencies but rather by the need to ensure that that the activities whose scope was increased were carried out fully. Two implementation inefficiencies caused delays: changes in World Bank procurement systems, and staff turnover associated with MINED’s reorganization in 2014. The World Bank team and the government adequately addressed these challenges, and implementation pace recovered. Overall, efficiency is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 36.00 Appraisal  11.00  Not Applicable 68.00 ICR Estimate  16.80  Not Applicable Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated High, as the objectives remained fully aligned at closing with country priorities (ESS 2017-2021) and the World Bank Country Strategy Partnership Framework (FY18-FY22). Efficacy is rated Substantial, as the project almost fully achieved its objectives. Efficiency is rated Substantial in view of favorable returns and largely efficient implementation. Given these ratings, shortcomings are considered minor, and the overall outcome is therefore rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome This IEG validation identified two risks to development outcome. Sustainability of institutional capacity gains. MINED’s education management capacity has improved through increased availability of quality data. This increased capacity is unlikely to be reversed, since the project has left arrangements in place to ensure sustainability. To illustrate: the project has supported the elaboration of manuals on how to routinely update the infrastructure inventory system. Moreover, capacity is likely to be augmented in the future through the ongoing Alliance for Education Quality Project (P161029) supported by the World Bank. Overall, this risk is considered low. Sustainability of results achieved. The political and economic crisis that broke out in April 2018 has negatively affected employment and incomes, and this could lead to reduced enrollment and attendance rates and increased dropout rates. These negative effects will likely reverse the gains in survival rates achieved by the project. Overall, this risk is considered substantial. 8. Assessment of Bank Performance a.Quality-at-Entry The PDOs were aligned with country priorities and World Bank strategies at appraisal (see Section 3), and project design included a plausible results chain. The design of the project was also adequately coordinated with other projects, such as Education Sector Strategy Support (P133557) funded by the Global Partnership for Education and the “Programa de Apoyo al Sector Educacion en Nicaragua” (PROSEN) funded by the European Union. Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Despite these positive aspects, quality at entry had moderate shortcomings. First, although by and large the project included an adequate identification of risk and mitigation measures at appraisal, some of these mitigation measures, particularly the ones related to procurement, proved insufficient and prompted further World Bank and government action throughout implementation to address problems. Second, the targeting approach for “Paquetes Solidarios” lacked clarity, and the ICR reported that it proved operationally cumbersome. Moreover, the ICR (p. 19) reported that it proved strategically problematic, since it created an underlying sense of “fracturing” and “deepening inequities” among teachers who participated in pre and in-service trainings with different levels of support (e.g. teachers from targeted municipalities received per diems to participate in trainings) Likewise, the “Paquetes Solidarios” initially funded by the project were different from those later distributed by the Government (Source: TTL Interview). Considering these shortcomings, quality at entry is rated moderately satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The World Bank team was adequately staffed, had in-country presence, and reported implementation progress regularly, candidly, and comprehensively through Implementation and Supervision Reports and Aides Memoire. By and large, the team adequately supervised the project and effectively addressed implementation bottlenecks. For instance, throughout implementation, the team faced several problems related to fiduciary compliance that affected implementation pace (see Section 10b). To address these issues, the World Bank team and the government built a comprehensive roadmap to reestablish fiduciary controls. For the most part, these efforts removed were effective, and for this reason, World Bank management upgraded its financial management rating to “moderately satisfactory” for the remainder of the project. However, supervision shortcomings can also be noted, such as a lack of full compliance with OP/BO 4.10 (Indigenous Peoples) that was discovered at project closing and that explains the World Bank management downgrade of safeguards performance to moderately unsatisfactory (ISR Sequence 14) (see Section 10a). Considering this shortcoming, quality of supervision is rated moderately satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) a. M&E Design The PDO clearly articulated the outcomes sought by the project: (i) improve student retention rates in primary education schools located in participating municipalities, and (ii) strengthen MINED’s education management capacity. The PDO indicators encompassed all outcomes of the PDO statement and, together with the intermediate indicators, were adequate for validating the theory of change, which was well reflected in the results framework. At the time of appraisal, the results framework included baselines and targets based on the information, structures, and procedures available within MINED. To collect data on the outcome and intermediate indicators, the M&E design envisaged reliance on the following sources: (i) the project’s management and information system; (ii) the annual collection of enrollment, repetition, dropout, and retention data; (iii) targeted monitoring information collected in the 40 municipalities; (iv) periodic student learning assessments; and (v) an assessment of the “Paquetes Solidarios” program including a qualitative and quasi-experimental impact evaluation (PAD, p. 14). The data collection arrangements involved several MINED directorates and were to be carried out under the oversight of a Project Coordinator and a Division of Monitoring and Evaluation that was to be created within MINED with project support. b. M&E Implementation By and large, the project successfully implemented the M&E system and systematically collected data for the indicators included in the results framework throughout the life of the project. This is evidenced in World Bank management assessments rating M&E as moderately satisfactory or better throughout the project life. Moreover, throughout implementation, the team proactively adjusted baselines and targets when new and more accurate data became available. Finally, the project provided critical support for the implementation of a Division of Monitoring and Evaluation within MINED’s General Directorate for Planning. This Division has greatly enhanced MINED’s capacity and, because its functioning is embedded within MINED, the capacity created will remain after project closing. Within the context of this overall successful implementation, one main shortcoming can be noted. In 2014, the government created an independent office (“Sistema Nacional de Información y Estadística” - SINFO) that took responsibility for data collection on some key education indicators that were originally under MINED. Data collection procedures implemented by SINFO were unreliable, thus yielding data that was not adequate for calculating PDO Indicator #2 (within-year retention rate). This institutional rearrangement within the government was outside the control of MINED and the project’s M&E mechanisms. Because of SINFO’s procedures for collecting and publishing data, the project team detected the unreliability of the calculation only at closing (Source: TTL Interview). Despite this shortcoming, there was adequate data from other indicators to assess the achievement of the project's first PDO. c. M&E Utilization Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) The ICR presented evidence and examples indicating that the World Bank and the government routinely used monitoring and evaluation data to assess progress and inform decision-making. Several examples can be noted. First, the results of the periodic student learning assessments were used as an input for the formulation of the 2017-2021 Education Sector Strategy. Second, the overall M&E findings were used to inform subsequent interventions such as the World Bank financed Alliance for Education Quality Project (P161029). Third, data on intermediate indicators was closely monitored to ensure that funds and efforts were allocated to the most dispersed and vulnerable municipalities. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Environmental Safeguards At appraisal, the bulk of the project's planned physical activities involved refurbishing schools with furniture and school supplies and small repairs to MINED offices, as well as limited renovations in teacher training institutions (PAD, p. 19). Given the nature of the small works to be completed, the project was rated Category “C” as per OP/BP 4.01, and thus did not require an environmental assessment. During implementation, the government changed its infrastructure strategy by deciding to increase the number of complementary investments (water and sanitation, security, recreational facilities, and electricity) per school at the expense of reducing the number of rehabilitated classrooms. This change was reflected in more comprehensive PIS that required higher environmental considerations. To address these considerations, the project leveraged procedures and technical assistance that were ongoing for another World Bank-financed project supporting pre-school and lower secondary education (PEASE Project P133557). The ICR (p. 22) reported that all PIS financed by the project met adequate safeguards standards. Social Safeguards The project triggered OP/BO 4.10 (Indigenous Peoples), as indigenous peoples were among the main beneficiaries of the project. To respond to their needs, the Borrower prepared, consulted, and disclosed an Indigenous Peoples’ Plan (IPP) on March 2012. Out of the 15 actions outlined in the IPP, the project implemented 11. Of the 4 remaining actions, three were partially implemented and later transferred to another World Bank project for implementation (Alliance for Education Quality Project - P161029). Finally, the project dropped the implementation of the Accelerated Remedial School Program owing to the poor results obtained from the implementation of the pilot. Another shortcoming was that the project failed to put in place a Grievance Redress Mechanism, thus making the project not fully compliant with OP 4.10. The ICR (p. 57) reported that the PIS developed by MINED under the project included social safeguards considerations. Implementation Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Status Reports (ISRs) indicate that the OP/BO 4.10 (Indigenous Peoples) safeguard was monitored and rated moderately satisfactory or better by World Bank management until the last ISR Sequence (#14), when the rating was downgraded to moderately unsatisfactory. Neither the ICR nor the ISR included a discussion of the rationale for the downgrade. The World Bank team clarified that the downgrade was motivated by the lack of full compliance with OP/BO 4.10 (Indigenous People) (Source: TTL interview). b. Fiduciary Compliance Financial Management At appraisal, the project team carried out a financial management (FM) assessment to determine FM implementation risk and help establish adequate FM arrangements for the operation. The assessment rated FM risk as “Substantial” and identified actions to ensure that the project would be implemented within a sound fiduciary environment in compliance with World Bank requirements (PAD, p. 17). The project placed responsibility for FM under MINED’s Administrative and Financial General Division (DGAF), which already had experience managing World Bank funds. In addition, the project augmented the fiduciary capacity of the DGAF through the hiring of a senior financial management specialist, two accountants, and six controllers, and the training of key staff. As per management assessments, FM was rated satisfactory throughout ISR Sequences 1-4 and moderately satisfactory in ISR Sequence 5. In 2014, management downgraded FM to moderately unsatisfactory (ISR Sequence 6) following a weakening caused by MINED’s reorganization. The reorganization caused high staff turnover, with several legally mandated positions in the DGAF remaining vacant. In addition, because of the reorganization, the DGAF experienced delays with the submission of payments to consultants and Statements of Expenditure to the World Bank. Because of these shortcomings, external audit reports issued qualified opinions (ICR, p. 58). Following the reorganization, the DGAF substantially addressed these issues, and for this reason, World Bank management upgraded its financial management rating to “moderately satisfactory” for the remainder of the project. The ICR reported that the final external audit would be completed by June 30, 2019. Procurement The project's procurement arrangements and performance were adequate throughout most of the implementation period. ISRs indicate that procurement was rated moderately satisfactory or better by World Bank management until ISR Sequences #6 and #7, when it was downgraded to unsatisfactory. The downgrade was driven by delays associated with the heavy workload placed under MINED’s unit responsible for procurement (which was also handling other projects) and MINED’s 2014 reorganization. The World Bank and the government took proactive steps to resolve these delays. By ISR Sequence #8, World Bank management upgraded its procurement rating to moderately satisfactory or better for the remainder of the project's life. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) c. Unintended impacts (Positive or Negative) The project's activities aimed at strengthening the quality of multi-grade education have resulted in positioning Nicaragua as an international benchmark of multi-grade modalities for countries with high rural and dispersed populations. Throughout the project, MINED hosted and was invited to several international conferences to showcase its experience with multi-grade education (ICR, p. 18). d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Although PDO indicator #2 was not measured, this was mainly because of Quality of M&E Modest Substantial factors outside the control of the project. The other PDO indicators were sufficient for assessing results. Quality of ICR --- Substantial 12. Lessons The ICR identified three lessons (p. 25). Among these, IEG highlights and adapts the following: When targeting strategies are not aligned with government preferences, project implementation and efficacy can suffer. In this project, which used a geographic targeting approach, this type of misalignment arguably created a “sense of fracturing” between beneficiaries and non-beneficiaries. Specifically, the targeting approach for “Paquetes Solidarios” was not aligned with government preferences and, for this reason, it was abandoned at the first restructuring when the government decided to expand and fully fund “Paquetes Solidarios” nationally. Yet, the geographic targeting approach continued for other project activities such as aiding teachers for participating in TECPE workshops. This created a sense of fracture between “teachers from targeted municipalities” (who received Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) support for transportation costs) and “teachers from non-participating municipalities” (who did not receive support for transportation costs). Choosing PDO indicators that rely on data collection methods outside the control of the project is risky, especially if adequate mitigation measures are not in place. This project was unable to measure PDO Indicator #2 (within-year retention rate), as the data collection and calculation methods were affected by factors outside the control of the project. Projects that use PDO indicators whose data collection methods (and/or implementation) are outside the span of the control of the project can mitigate this risk by identifying at appraisal (i) the likelihood of changes in data collection methods and (ii) backup indicators. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain To validate data on within-year retention rates when it becomes available and assess sustainability of the multi-grade teacher training approach (the approach rests on the assumption that teachers will return and remain in their communities after finishing pre-service training). 14. Comments on Quality of ICR The ICR provided a thorough description of the project with a well-written and coherent narrative that broke down the PDOs into two distinct objectives. Following guidelines, the ICR included a schematic figure describing the project result’s chain (ICR, p. 8) that helped the reader quickly understand the intervention logic. Some minor shortcomings can be noted. First, more information on the targeting approaches for “Paquetes Solidarios” and pre-service and in-service teacher training could have been included. Second, the description of the restructuring was not thorough and was sometimes confusing. These shortcomings were relatively minor, and the overall quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI 2nd Support to the Education Sector (P126357) Page 20 of 20