Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) Report Number : ICRR0021523 1. Project Data Project ID Project Name P131331 Enhancing Education Development Project Country Practice Area(Lead) Maldives Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H8410 30-Jun-2018 9,159,128.58 Bank Approval Date Closing Date (Actual) 30-Apr-2013 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 0.00 Revised Commitment 9,569,403.03 0.00 Actual 9,159,128.58 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Project Appraisal Document (PAD) (p.ii) and the Financing Agreement of June 17, 2013 (p. 4), the objective of the project was “to enhance and strengthen strategic dimensions of education access and quality.” Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project included three components: Component one: National Level Education Development (appraisal estimate US$8.55 million, actual US$7.81 million): This component was to promote strategic initiatives at the country level to strengthen and develop the education system, including building technical capacity within the Ministry of Education (MoE), especially the Education Supervision and Quality Improvement Division (ESQID), to undertake rigorous, state-of-the-art national assessments and to deliver short courses. Also, this component was to support the National Institute of Education (NIE) to conduct advanced certificate or diploma level courses in national assessments of learning outcomes. The component was also to build the capacity of selected policy makers and education specialists within the MoE, especially in the ESQID, the Policy Planning and Research Section (PPRS), and the NIE, to use the results and findings from national assessments for strategic policy and management decisions. Furthermore, this component was to finance the management and leadership development of staff in the education sector as well as enrich the learning environment of 45 strategically selected schools through the provision of information technology (IT) and science equipment and the expansion of classroom facilities in overcrowded large schools. Component two: School Level Education Development (appraisal estimate US$1.15 million, actual US$0.83 million): This component was to promote strategic initiatives to improve the performance of schools through supporting the MoE to train and strengthen the capacity of Senior Management Teams of schools and School Boards to take school-based management forward efficiently, modernizing quality assurance for school improvement, and strengthening school-based professional development (SBPD) for teachers and other school staff. Furthermore, this component was to support a system of School Quality Assurance and Teacher Development Awards (QATDA). Under this component, all schools were to receive an annual award of between MVR 6,000-7,500 (about US$ 400-500), depending on school size, to implement the findings and recommendation of quality assurance (QA) self-evaluations to improve student learning and soft skills, and to implement SBPD programs to enhance teacher performance. Component three: Project Coordination, Monitoring and Evaluation, and Program Development (appraisal estimate US$1.30 million, actual US$1.58 million): This component was to support the coordination of project components, undertake monitoring and evaluation of results and outcomes, and support the MoE to undertake program development in key areas for the future of the education system. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) Project Cost: The project was estimated to cost US$11 million. Actual cost was US$10.22 million. Financing: The project was financed by a US$10 million grant, of which US$9.57 million was disbursed. US$ 0.3 million was cancelled, and the remaining difference between planned and actual financing was due to exchange rate losses. Borrower Contribution: The Borrower was to contribute US$1 million, of which US$650,000 was used (according to the Bank team, February 11, 2019). This amount was not included in the ICR's reporting of total project costs. Dates: The project closed on its original closing date. There were no restructurings or extensions. 3. Relevance of Objectives Rationale At the time of project appraisal, despite Maldives’ achievement in the education sector (net primary enrollment rate at 94 percent, net lower secondary enrollment rate at 81 percent), the country was experiencing several challenges. First, there was a sharp drop in enrollment at the higher secondary education level due to the limited number of schools offering education in grades 11 to 12. Out of 220 schools in the Maldives, only five schools offered higher secondary education. The Maldives also lagged in higher education compared to other small island nations. The quality of education was another challenge that the education system faced at all levels. The MoE prepared an education sector development program, which was still relevant at the time of the ICR, to address these challenges. The education development program also aimed to address the low quality of education through several strategic initiatives such as the development of a system of regular National Assessments of Learning Outcomes (NOLA), a QA system, and pre-service teacher education and continuing teacher development programs. The objective of the project was in line with the Bank’s most recent Country Partnership Framework (2016- 2019), which identified three priority areas: to promote growth, reduce poverty, and increase shared prosperity in a sustainable manner. Priority area one aims to provide employability and economic opportunities for Maldivians. The Bank’s systematic country diagnostic, conducted under the Country Partnership Framework, found that shortcomings in the education system have limited participation of the local population in the existing job market and entrepreneurial activity, hindering growth and poverty reduction. Rating High Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To enhance and strengthen strategic dimensions of education access Rationale The project’s theory of change linked improving teaching-learning facilities with enhancing and strengthening strategic dimensions of education access. Outputs • The learning environment was enriched in 53 secondary schools, surpassing the target of 45 schools. Coverage included all 38 secondary schools in the country, plus some larger secondary feeder schools. For the expansion of Advanced-Level education (that is, higher secondary education) at atoll education centers, the following improvements were made: construction of 42 new classrooms and 36 toilets, and purchase of new classroom furniture, computer systems, equipment for science laboratories, and state-of-the-art video conference facilities. Outcomes • The survival rate in higher secondary education (grades 1 to 12) increased from 17 percent in 2013 to 32 percent (29 percent for boys and 35 percent for girls) in 2018, surpassing the target of 27 percent. • The ICR (p. 12) stated that with the expansion of physical facilities in the outer islands, the Gross Enrollment Rate (GER) in higher secondary education increased from 26.8 percent in 2013 to 36.8 percent in 2017. The Bank team stated (February 11, 2019) that the increase in the GER can be attributed to the project, as the main reason for the relatively low initial GER at the higher secondary level was the limited number of schools that offered grades 11-12. The expansion of physical facilities in the outer islands had a positive impact on the GER. • Approximately 66,000 students benefitted from the project, surpassing the target of 50,000 students. 49 percent of beneficiaries were female, surpassing the target of 48 percent. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) Objective To enhance and strengthen strategic dimensions of education quality Rationale The project’s theory of change linked providing research equipment, conducting curriculum reforms, providing training/research opportunities for faculty and students, supporting scholarships for staff, developing leadership training and opportunities, and supporting the creation of institutional strategic plans with enhanced quality of education. Outputs • A system of NALO was developed and operationalized in grades four and seven for English, Mathematics, and Dhivelhi, with the goal of informing education policy and program development. Two NALOs were implemented, in 2015 and 2017. According to the ICR, because there was a curriculum change between these two rounds of the NALO, their results could not be compared. According to the Bank team (February 11, 2019), however, these results were "fully utilized," defined as: i) data is being analyzed and results are being reported to education policymakers and/or the public; ii) results are being reported for at least one of the following student subgroups: gender, urban/rural, geographic region; and iii) the assessment exercise is being repeated at least once every 5 years for the same subject area(s) and grade(s). It is noteworthy that the ICR did not report information on the findings of these learning assessments; those findings would have been useful in discussing the extent to which education quality was enhanced and strengthened. • All atolls established a system of QA, and self-evaluations were conducted and their findings incorporated in the implementation of school improvement plans, surpassing the target of 80 percent. A Master of Education degree in educational assessment was developed, and 16 persons completed the course in 2018. These graduates were assigned to the National Network on School Assessment to assist schools in conducting quality assessments in classroom and nationwide NALOs. • 132 staff were trained in short-term and long-term courses in education management and administration, exceeding the target of 100 staff. The percentage of principals and members of Senior Management Teams (SMT) formally trained in their respective fields increased from 40 percent prior to the project to 85 percent at completion. • In order to improve the quality of teaching and raise school standards, training was provided on School- Based Management (SBM), covering school leadership and management skills, training for school boards, and training of trainers for teacher resource center coordinators. A total of 586 participants (175 principals, 207 SMT members, and 204 school board members) completed this training. The ICR (p. 14) stated that the training resulted in the development of school-specific plans to strengthen SBM. • A system for SBPD was initiated in all schools to improve the capacity of professional development coordinators and teacher resource center coordinators, surpassing the target of 80 percent. A total of 3,658 teachers benefited from this training program. QADTAs were introduced to strengthen the link between QA, school self-evaluations, and school improvement plans. 193 schools received QATDAs. • Technical advice was provided for the development of a framework for multi-grade teaching (MGT). This framework included policy guidelines, a curriculum framework, and a communications strategy to educate Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) stakeholders on the advantages of MGT. Plans were developed for the training of teachers and preparation of teaching and learning materials, as well as monitoring and feedback mechanisms to be used for MGT. MGT was established in 44 schools, of which 10 were financed by the project. Under the project, by the end of 2016, 85 MGT teachers out of a total of 139 were trained. The rest were trained by the government. • A vocational education strategic implementation model was developed and implemented in 2016, resulting in the introduction of the first vocational stream in all schools across the country. • The NIE was provided with technical assistance to establish a delivery mechanism for inclusive education. 571 teachers were trained in inclusive education. • A master plan for higher education (2017-2022) was developed and launched to guide institutional and sectoral development of higher education. This plan focused on five main areas: desired size and structure, governance, quality, economic and social relevance, and budget and financing of higher education. A Strategic Action Plan, including costing for the master plan, was completed and implemented in 2017. The Maldives Qualification Authority (MQA) was supported in developing international benchmarks, standards, and review procedures for external quality reviews and guidelines for internal quality assurance in higher education institutions. Also, training was provided for higher education institutions on how to conduct internal reviews. Seven higher education institutions in 2017 and five in 2018 were audited, and reports were shared with stakeholders. Furthermore, a management information system for MQA was developed and integrated with a similar system developed for the Department for Higher Education. Outcomes • All atolls are conducting self-evaluations, and findings are incorporated in the implementation of School Improvement Plans, surpassing the target of 80 percent. In addition, 91 percent of schools in all atolls are engaged in self-evaluation activities. In 2017, a School Improvement and Quality Assurance and Accountability Framework (SIQAAF) was prepared and implemented, and analytic reports were prepared for 168 schools that completed external reviews in 2016 and 2017. • According to the ICR (p. 13), the NALO results showed that girls outperformed boys in Dhivehi, English, and mathematics in grades four and seven across the country. Also, the results of NALO and school-self evaluations were reported to have been used to make improvements, including through individual school improvement plans. However, the ICR did not provide detailed data on learning outcomes. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The achievement of both objectives was Substantial. Project-supported expansion and improvement in secondary education facilities clearly improved access to secondary education. The ICR (pp.13-14) provided Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) significant information on a wide range of trainings that were provided to teachers and education administrators, and on the use of data and analysis for quality improvements in the sector, that plausibly led to strengthening of education quality. However, those data and analysis themselves were not provided in the ICR. Given that NALO results are available, for example, even though they are not comparable across the two administered rounds, it would have been useful for the ICR to have reported this information to augment assessment of improvements in education quality. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency The PAD (p. 17) conducted a cost-benefit analysis that assumed 4,534 additional secondary education graduates over the project’s implementation period. The benefit stream was calculated over a 20-year time frame. The annual growth rate of wages was close to the annual growth rate of the Gross Domestic Product. A discount rate of 12 percent was applied. A 55 percent labor force participation rate for secondary education graduates was used as the base case scenario. Furthermore, it was assumed that secondary education graduates would experience an eight percent quality premium over the existing wage rate due to project activities that were to improve the quality of education. The Net Present Value (NPV) of the project was estimated at US$22 million, and an Internal Rate of Return (IRR) was estimated at 30 percent. The ICR (p. 17) made the same assumptions as the PAD. Benefits were assumed to result from an increased number of higher secondary school completers and graduates, and from improved quality of the education system, which would translate into higher labor productivity and lifetime earnings. A sensitivity analysis found that the NPV ranged between US$17.6 million and US$21.3 million, depending on different scenarios. The IRR ranged from 27.3 percent in a low-case scenario to 30.7 percent in a high-case scenario. Operational Efficiency The project did not experience any major delays and closed at its original closing date. Administrative efficiency was demonstrated by strong coordination across atolls (ICR, p. 20). Minor procurement issues were resolved quickly (see Section 10b), indicating that the project’s financial resources were used efficiently. Taking everything together, the project’s overall efficiency is rated Substantial. Efficiency Rating Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 100.00 Appraisal  30.00 Not Applicable 100.00 ICR Estimate  27.30 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the objective is High given its alignment with country context and with the Bank’s most recent Country Partnership Framework. Efficacy and Efficiency are rated Substantial. The overall outcome rating is therefore Satisfactory, consistent with only minor shortcomings in the project's preparation and implementation. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (p. 23) stated that the government continues to be committed to implementing its education reform program, which will support the sustainability of the outcomes achieved under this project. The Bank is preparing a follow-on “Enhancing Employability through Human Capital and Entrepreneurship Development” project that focuses on the building of tertiary skills for youth in preparation for private sector employment in key areas such as tourism and IT services. The achievements made under his project will support the implementation of the new project. However, the ICR (p. 23) stated that the education sector faces the challenge of high per capita costs of education service delivery due to the widely spread islands with small populations. Even with the development of video conferencing that can connect the atolls, one program participant stated that “without proper/additional training and resources, the program is never going to be sustainable" (ICR, p. 23). 8. Assessment of Bank Performance Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) a. Quality-at-Entry According to the ICR (p. 19), the Bank collaborated with relevant stakeholders from the education sector to define the objective of the project and respective activities. The ICR (p. 22) stated that the project design took lessons learned from a previous education project in Sri Lanka into account and remained highly relevant throughout implementation. According to the Bank team (February 11, 2019), these lessons included: the importance of a project focusing on a single sector and having one counterpart ministry; the benefit of adopting a layered approach, focusing on both national-level initiatives and school-level initiatives, to have a greater impact on the entire education system; and the importance of having a strong coordination unit that works with different agencies in the MoE. The Bank identified relevant risk factors, including weak capacity within school communities to implement new initiatives and the MoE's unfamiliarity with the Bank’s financial management and procurement policies. Risks related to delivery monitoring and sustainability were also identified as Substantial. It was thought that monitoring activities in the scattered islands of the Maldives could stretch the capacity of the MoE, and results of monitoring and evaluation activities might not be adequately linked to improved management and teaching practices at the school level. In order to mitigate these risks, the Bank provided financial management and procurement trainings and built capacity within school communities for the implementation of these new initiatives. These mitigation measures were adequate. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR (p. 22), the Bank team consisted of team members with an adequate skill mix and benefited from the continuity of the same task team leader over the project's lifetime. The Bank conducted regular supervision missions at least twice a year and was proactive and responsive to government requests, including the government deciding to use its own funds for the construction of some school buildings. The ICR (p. 23) stated that the Bank provided timely financial management, procurement, and environmental safeguard support and conducted relevant and timely trainings to strengthen capacity. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) a. M&E Design The objective of the project was clearly specified. The project’s theory of change was sound and reflected in the Results Framework (RF). The number of selected indicators (four PDO indicators and three intermediate outcome indicators) was small, but they captured achievement of project activities and were logically linked to the intended outcomes; it would have been useful, however, for the RF to have included more direct measures of improvement in education quality. The indicators included in the RF were specific, measurable, and relevant. The PPRS of the MoE was responsible for the monitoring of project activities. Other central level MoE agencies were to monitor the implementation of activities under their mandates and responsibilities at the atoll and school level. The PPRS was to be supported by the Coordination Support Unit (CSU). b. M&E Implementation According to the ICR (p. 20), M&E activities were conducted throughout project implementation on three different levels: national, atoll, and school. Research and evaluation activities were contracted out to independent private higher education institutions and individual consultants. The M&E officer within the CSU was responsible for collecting data and preparing M&E progress reports. The Bank team stated (February 11, 2019) that the data was found to be reliable and of good quality. Evaluations included beneficiary feedback surveys, in particular for school-level development activities. According to the ICR (p. 21), monitoring activities followed a cascade model, with atoll education officials and teacher resource center coordinators assisting the national MoE agencies in monitoring the activities in the atolls and islands. The CSU shared with the Bank its M&E results on a monthly basis. Furthermore, the ICR (p. 21) stated that the CSU worked with the National Institute of Education to prepare reports by teacher resource center coordinators to receive feedback by beneficiaries and assess the impact of school self- evaluation and school-based professional development activities on improving school performance. According to the Bank team (February 11, 2019), the indicator targets could have been revised to be more ambitious, since all of them were achieved or surpassed. The Bank team stated that sustainability of M&E functions and processes after project closing is not an ongoing concern. c. M&E Utilization M&E data were used to inform decision making. For example, the ICR (p. 21) stated that the outcomes of the NALO were used in education sector analysis and sector plans to define literacy and numeracy standards. However, without reporting on the specific NALO results, it is difficult to assess the ways in which these results were used. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as environmental assessment category B and triggered the Bank’s safeguard policy OP/BP 4.10 (Environmental Assessment). According to the ICR (p. 21), an environmental and social management framework was developed to support the safeguard process during project implementation and to mitigate potential impacts from project activities, especially in regards to physical facilities. The ICR (p. 22) stated that the project’s compliance with the Bank’s safeguard policy was rated moderately Unsatisfactory at the time of the mid-term review due to lack of staffing and changes in project implementation structure. An environmental officer was not assigned to the project, and construction activities were managed by engineering consultants. The rating was upgraded to Satisfactory once mitigation measures were implemented. These mitigation measures included conducting a rapid environmental screening and field verification of construction project sites. b. Fiduciary Compliance Procurement According to the ICR (p. 22), the project’s overall procurement rating was Satisfactory throughout implementation. The project experienced some delays and cancellation of contracts due to contract variations requested by the MoE. However, these delays were addressed, and the ten construction activities were completed on time. The Bank introduced a new online procurement system during project implementation that produced some delays, as staff had to be trained on the new system. However, all goods and consultancy contracts were completed on time. Financial Management The ICR (p. 22) stated that the project’s financial management performance was Satisfactory throughout project implementation. Quarterly interim unaudited financial reports were submitted in a timely manner, the external auditor’s opinions were unqualified, and no major systems and control issues were identified by the auditors during the 2017 and 2018 audits. c. Unintended impacts (Positive or Negative) Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Bank Performance Satisfactory Satisfactory --- Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR (p. 23-24) provided valuable lessons, adapted here by IEG: • In order to ensure positive project outcomes and sustainability of knowledge built among education officials for future activities, it is critical to provide continuous capacity building activities during project implementation. In this project, atoll- and island-level education officials were trained on an ongoing basis to achieve technical and management support for schools to implement education development activities. • When a project introduces a new system midway through implementation, it is important to find a way to avoid disruption of project implementation. In this project, a new procurement system was introduced in the middle of project implementation, resulting in several challenges due to lack of capacity to implement the new system, inflexibility of the system, and slow internet speed. These challenges were mitigated through training and capacity development. • Activities to strengthen school-based management contribute to ownership and sustainability of improvements. This project provided training for senior management teams in school leadership and management skills, training for school boards, and training of trainers for teacher resource center coordinators, all aimed at strengthening school-based management. The ICR (p. 14) quoted a participant as saying, “understanding on how to make our school improvement plans was a huge step for us to make plans, and to implement and monitor them for the betterment of our schools.” 13. Assessment Recommended? No Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhancing Education Development Project (P131331) 14. Comments on Quality of ICR The ICR provided a strong overview of project preparation and implementation and included an adequate economic analysis. The ICR was concise and internally consistent. However, it did not provide results from the two NALO assessments; even though the two rounds of the NALO may not have been comparable, their results would have provided an additional, useful dimension to the assessment of the quality objective. The lessons learned included in the ICR could also have provided additional detail. The Borrower’s note was of notably high quality and provided important augmentation and clarification to many of the points made in the ICR. . a. Quality of ICR Rating Substantial Page 13 of 13