Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) Report Number : ICRR0020855 1. Project Data Project ID Project Name P105036 JO - ERfKE II Country Practice Area(Lead) Jordan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-77020 31-Dec-2015 408,000,000.00 Bank Approval Date Closing Date (Actual) 19-May-2009 31-Dec-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 60,000,000.00 0.00 Revised Commitment 60,000,000.00 0.00 Actual 60,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The stated objective of the project was to assist the Borrower in providing students enrolled in pre-tertiary education institutions with increased levels of skills to participate in the knowledge economy (Loan Agreement, 12/30/2009). The PAD’s statement of objectives is consistent with the Loan Agreement (PAD, p. 5). Note: The PAD did not provide a definition of skills relevant to the knowledge economy, but it was understood from Vision Forum 2002 and from clarifications provided by the project team (9/5/2017) that these are defined as skills relevant to current and future needs of an educated population, workforce, and the Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) market, to stimulate economic development, with a special focus on science and mathematics, and that such skills would be strengthened through curriculum reform and review of educational assessments. The PAD explained the context of the project as an operation aimed at supporting the second phase of the Education Reform for the Knowledge Economy Program (ERFKE), a 10-year multi-donor sector program designed to deliver on the 2002 Vision Forum for the Future of Education in Jordan (PAD, p. 5). The scope of the operation was at the national level. A Level I restructuring on 10/24/2013 revised key outcome indicators and targets, and, therefore, this review undertakes a split rating. At the time of restructuring, 38% of the loan had been disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Oct-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project’s restructuring on 10/24/2013 did not revise the original components listed below: Note on Actual Costs by Component: The ICR does not provide actual component costs. The project team clarified (9/5/2017) that, beyond Bank financing, information on parallel donor financing was difficult to obtain, as many donor agencies did not disclose financial data. However, the project team confirmed that financing of the main planned activities was actually met and/or exceeded, as also visible in project outputs, except for some funding gaps in Component 4 (Special Focus Programs, including for children with special needs/disabled children, early childhood education, and vocational education). Also, the government did not finance the school maintenance program, citing financial constraints. Component 1: Establishment of a National School-Based Development System (Appraisal US$13.3 million). This component aimed at supporting the creation of an effective, school-based development process to deliver quality education to improve students’ abilities, skills, attitudes, and values associated with a knowledge-based economy. Specific activities included a process of self-evaluation by each school and production of the school’s own development plan, including teacher professional development (PAD, p. 6). Component 2: Monitoring and Evaluation and Organizational Development (Appraisal, US$13.6 million). This component built upon the investments of the first Education Reform Project (ERFKE I) in policy, planning, monitoring, and evaluation, to ensure that related activities inform and support the Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) adoption of a school-centered approach to the delivery of education services. It consisted of two sub- components: (i) Policy Development, Strategic Planning, Monitoring and Evaluation; and (ii) Organizational Development. Activities included institutional capacity strengthening of the Ministry of Education (MOE), strengthening the School Information System, aligning the Education Management Information System (EMIS) with the Geographic Information System and external databases, and improving the organizational culture, structure, and processes of MOE. Component 3: Development of Teaching and Learning (Appraisal US$49.3 million). There were two sub-components, on (i) Teacher Policies, Training and Professional Development, and (ii) Curriculum, Assessment, and Learning Resources, aimed at reviewing and developing teacher utilization and professional development practices, and fine-tuning curricula and student assessments. Component 4: Development of Special Focus Programs (Appraisal US$20.6 million). This component aimed at improving inclusive access to learning for all children in three subsectors: (i) Early Childhood Education; (ii) Special Education; and (iii) Vocational Education. Component 5: Improvement of the Physical Learning Environment (Appraisal US$310.8 million). The aim of this component was to improve the provision of quality education facilities in environmentally friendly and efficiently operated quality physical learning environments. It consisted of three sub-components: (i) alignment of MOE standards with international design standards and education reform requirements; (ii) construction and equipment of new schools and provision of extensions and equipment to existing schools; and (iii) establishment of a maintenance and management system for school buildings. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs and Financing. The Second Education Reform for the Knowledge Economy Project (ERFKE II) was a multi-donor-financed project designed as a specific investment loan. While the total estimated cost was shown to be US$408 million at appraisal, with a stated financing gap of US$153.9 million, the project team clarified (9/5/2017) that the government highlighted a financing gap at appraisal as a means to mobilize additional resources for the education sector. Accordingly, an amount of US$31.68 million was mobilized from the European Commission. At project closing, the actual total project cost was US$283.4 million. Financing by various agencies was as follows: • IBRD financed US$60 million, mostly for the construction, extension and equipment of schools (US$57.3 million). The total loan of US$60 million was disbursed. • The Borrower contribution was estimated at US$57.6 million at appraisal, with an actual contribution of US$55.79 million. • The Canadian Int. Dev. Agency - CIDA: Appraisal US$17 million; Actual US$18.56 million; • Japan Int. Cooperation Agency - JICA: Appraisal US$2.50 million; Actual 0; • USAID: Appraisal US$117 million; Actual US$117.40 million; • The European Commission provided 30,000,000 EUR or US$31.68 million. Dates. The project was appraised on 02/20/2009 and became effective on 01/22/2010. A mid-term Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) review was undertaken on 11/12/2012. A level I restructuring on 10/24/2013 revised outcome targets and indicators; and a level II restructuring on 03/19/2015 extended the closing date of 12/31/2015 by one year to allow the completion of civil works. The project closed on 12/31/2016. 3. Relevance of Objectives & Design a. Relevance of Objectives At the 2013 restructuring, and as stated previously. the statement of objectives remained unchanged, but key outcome targets were revised. Under both original and revised objectives, relevance of objectives is rated Substantial due to the importance of knowledge economy skills to the country’s needs and vision, and because of consistency with the objectives of the Bank Country Assistance Strategy for the Hashemite Kingdom of Jordan for the period 2006-2010, particularly to its first cluster on strengthening the investment environment and building human resources for a value-added, skill-intensive and knowledge-based economy (ICR, p.1). The PAD highlighted the importance of the project in focusing on quality education and developing human resource linkages between education, skill development, and employment (PAD, p. 5). The objectives remain relevant, as enhanced and relevant skills constitute a long-term need for the country, and, as clarified by the project team (9/5/2017), the objectives are consistent with the Bank’s Country Partnership Framework at project closing (2017-22), which aims at improving the quality and equity of service delivery, as well as better employment opportunities. The new education strategy in Jordan -- Education for Prosperity: Delivering Results, 2016-2025 -- directly builds on the framework and achievements of ERFKE II (ICR, p. 7, and TTL clarifications 9/5/2017), and before project closing, MOE started working with the Bank on the design of a follow-on project, in the context of the government’s new strategy (ICR, p. 7). Rating Revised Rating Substantial Substantial b. Relevance of Design Under both original and revised objectives, relevance of design is rated Substantial, as the design linked multiple institutional and technical interventions in a mutually reinforcing and complementary manner, with all ultimately contributing to the desired outcome of increased skills that are necessary for a knowledge-based economy. The project results chain presented the underlying logic and linked planned activities to outputs and intermediate outcomes to outcomes. Improved teaching curricula, teachers’ professional development and merit-based selection, school decentralization, strengthened capacities in planning, monitoring and evaluation, and improved and expanded physical learning environments would all contribute to better learning outcomes. Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) The previous first phase of ERFKE I focused on management at the central level, while ERFKE II moved down to the school level to ensure that learning gains were enhanced at the classroom level (PAD, p. 2). Project interventions are recognized in international literature as important elements of successful education reform (ICR, p. 10). At the same time, the ICR noted that there was complexity, as each donor was responsible for supporting MOE’s delivery of the respective donor-supported components (ICR, p. 8). Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Provide students enrolled in pre-tertiary education institutions with increased levels of skills to participate in the knowledge economy. Rationale Outputs: The main outputs included the following: Establishment of a National School-Based Development System (Component 1) • The project rolled out the School and Directorate Development Program. Plans were developed locally by the schools themselves in all of the country’s 3,554 schools (exceeding the target of 3,000 schools) and 42 Field Directorates (exceeding the target of 35 directorates). • The project established an Education Quality and Accountability Unit within MOE, staffed by 51 assessors. Monitoring and Evaluation and Organizational Development (Component 2) • 2,425 non-school directorate staff participated in professional development programs to support the delivery of the school improvement program (exceeding the target of 1,200 staff). • A new and enhanced open EMIS was made operational and rolled out in all schools. • Studies and analytical work were completed to guide policy development: School Rationalization Evaluation; Program for International Student Assessment National Report; Trends in International Mathematics and Science Study; Classroom Observation for teacher application of “Student-Centered Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) Active Learning and Teaching”; Gender Gap in Student Achievement in Jordan; and Mapping of Student Assessments in Jordan. Development of Teaching and Learning (Component 3) • 90% of teacher policies were revised to support application of national teacher standards (short of the target of 100%). • A teacher induction training program was developed and implemented. • 96% of newly appointed teachers completed training in a final year in the Education Training Center (exceeding the target of 80%). • A new selection and recruitment process, with merit-based criteria, was established for new teachers. • All subjects for Grades 4-11 were reviewed and fine-tuned to ensure alignment with knowledge economy skills (exceeding the target of Grades 5-10). • Information and communication technology was incorporated into the teaching and learning process. • The Tawjihi (High School) examination was revised in harmonization with the goals of the ERFKE curriculum. • The national test (NAFKE) was revised for technical soundness and alignment with the ERFKE curriculum. Development of Special Focus Programs (Component 4) • An Early Childhood Education Framework was developed for the expansion and quality improvement of kindergartens (KGs). • 94% of KG classrooms met MOE quality assurance standards through training of kindergarten teachers and parents (exceeding the target of 75%). • 99% of KG teachers and supervisors successfully completed a training program for early childhood education (essentially reaching the target of 100%). • 61% of eligible children enrolled in a final year KG2 in all authorities (meeting the target of 60%). • 4,333 parents of KG students enrolled in volunteer programs in a final year (short of the target of 5,000 parents). • A Policy Framework for Special Education was approved, and teachers trained. • Training outcomes for vocational training were developed. • 65% of vocational education textbooks were reviewed and adopted for Grades 11 and 12 to align with market requirements (short of the target of 75%). • 19,650 students enrolled in special education programs for disabled students (exceeding the target of 18,500 students). • 8,313 students enrolled in special education programs for gifted students (short of the target of 8,750 Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) students), and 8,313 students having access to services relevant to their individual needs (exceeding the target of 6,500 students) • 1,791 disabled students had access to services relevant to their individual needs (exceeding the target of 850). • Equipment and workshops for vocational training were upgraded (office equipment, computers, welding and general machines, central heating, and automotive workshops). Improvement of the Physical Learning Environment (Component 5) • 985,553 students had access to learning in a safe and well-managed physical environment (exceeding the target of 945,000), although the ICR notes that, while the numbers have increased, the proportion of students with access to a safe physical learning environment has decreased due to the influx of Syrian refugee children (ICR, p. xiii). • 26 new schools and 42 school extensions were constructed. • Furniture, and laboratory, science, and information technology equipment were provided. Outcomes This ICR Review assesses the achievement of the objective over the life of the project. According to the ICR, learning improvements were confirmed by increases in test scores in the national assessments aligned with knowledge economy skills, known as the National Assessment for the Knowledge Economy (NAFKE). In 2016, an average increase of 3 score points for Math, Science, and Arabic in Grades 5, 9 & 11 was reported in NAFKE 2014 as compared with the NAFKE baseline of 2008, attaining the 3-score points target. The enrollment rate in the secondary cycle increased from a baseline of 61.5% in 2008 to 72.6% in 2015-16, exceeding the target of 63%. The enrollment rate in the basic cycle increased from a baseline of 97.6% in 2008 to 97.9% in 2015-16, slightly exceeding the maintenance target of 97.6%. These rates did not include refugee students (ICR, p. vii; and confirmed by project team clarifications, 9/5/2017). At the broader institutional development level, the ICR notes that the project produced systemic changes that are desirable in mid- and long-term education reform, such as decentralized school management and local education directorates; establishment of an Education Quality and Accountability Unit and an operational management information system; merit-based selection of teachers; enhancement of early child education; and expansion and improvement of the physical learning environment, and that such strengthening would be expected over time to favor a more equitable distribution of positive learning outcomes amongst the youth population (ICR, pp. 10-11). The information provided by the ICR supports the conclusion that the project achieved its intended Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) outcomes as defined by the original objective. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective Provide students enrolled in pre-tertiary education institutions with increased levels of skills to participate in the knowledge economy. (The objective was not revised, but key indicators and outcome targets were changed.) Revised Rationale Outputs The outputs shown above under Objective 1 also apply to the revised targets. Outcomes The 2013 restructuring set the NAFKE 2011 scores as the new baseline (replacing the NAFKE 2008 scores), and revised the target at 1 score point increase from the new baseline. The project achieved an average increase of 3 score points for Math, Science, and Arabic in Grades 5, 9 & 11 between the NAFKE 2011 and NAFKE 2014 assessments, surpassing the revised target. The revised target for enrollment rate in the secondary cycle, 78.2%, was not achieved (72.6% actual achievement in 2016). The target for the enrollment rate in the basic cycle, which was revised upwards to 98%, was substantively achieved (97.9% actual achievement in 2016). The information provided by the ICR supports the conclusion that the project achieved its intended outcomes under the revised targets. Revised Rating Substantial PHREVISEDTBL 5. Efficiency General. The PAD presented general arguments that education generates positive labor market outcomes (PAD, p. 13), and a literature review supporting the choice of policies that the project intended to put in place. It stated that the annual rate of return to investment in a year of schooling in Jordan was above 9%, and that returns to schooling rose faster beyond secondary education. The PAD stated that, to achieve the goals of education reform under ERFKE II, Jordan must spend close to US$408 million over the next 6 years, and that additional recurrent costs would be manageable at less than 1% (PAD, p. 14) while noting that the share of Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) recurrent educational expenditure dedicated to personnel and salaries was high at 89% (PAD, p. 14). According to the ICR, school construction provided additional learning spaces for both Jordanian nationals and refugees, with the additional result that there would be less need for rental school space than would have been the case had the project not built additional classrooms (ICR, p. 10). Implementation. There were significant shortcomings in the efficiency of implementation, and, under both the original and revised outcome targets, four ISRs in 2011, 2012, and 2015 rated implementation progress as moderately unsatisfactory (ICR, p. 6). Related shortcomings included the following: • The 2013 Restructuring Paper stated that unit costs of school construction were considerably higher than the estimates made during appraisal (Restructuring Paper, 10/3/2013, p. 7), and that, as a result, available funds only allowed the construction of 26 new schools, instead of the planned 36 schools (ICR, p. 3). The ICR states that ERFKE II offers a warning on cost estimations (ICR, p. 14), as cost estimates in 2009 were underestimated for civil works by 24% and overestimated for goods by 40%, and that in the 2013 restructuring, estimates continued to be unbalanced. • MOE had procurement shortcomings that led to repetitive re-bidding (ICR, p. 7). A Level II restructuring on 3/19/2015 extended the closing date of the project by another year, from 12/31/2015 to 12/31/2016 (ICR, p. iv), to allow more time for the completion of school construction and extensions (ICR, p. 4). • Continuity was affected by frequent changes in political leadership, and this affected the implementation of some of the project teacher policies (ICR, p. 13); high turnover in senior and technical positions presented a challenge during the implementation stage (Borrower’s comments, ICR, p. 15). • The maintenance system for infrastructure, which was planned to be implemented by the Borrower, was not implemented due to government financial constraints (TTL clarifications, 9/5/2017), thereby increasing the risk to school construction sustainability (ICR, p. 3 and p. 12). • According to the ICR (p. 6), MOE miscalculated the effect of the Syrian refugee crisis, anticipating a rapid return of refugees, and therefore MOE diverted its attention and resources toward dealing with the short- term effects of the refugee crisis (providing space in schools for the new arrivals), and away from long-term efforts related to the main project purpose (reforming systems designed to improve learning quality). The influx of refugee children into the schooling system caused a considerable stress to the education sector. While the government admirably went to considerable lengths to welcome the newcomers, the impact was Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) visible in the variable rate of implementation of different project components (ICR, p. 6). Also, in the early years of project implementation, and in view of fiscal constraints, the government allocated less capital funding to the project in its annual national budgets than had been required by the project. Funding was subsequently mobilized by the Ministry of Planning and International Cooperation from its own budget. This contributed to a slow start in implementation and delays. The government’s budget transfers to the project were further skewed by requirements to allocate funds to specific components because the various bilateral funding agreements set national funding targets. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under both the original objectives and revised outcome targets, relevance of objectives is rated Substantial due to the importance of knowledge economy skills to the country’s needs and vision, and because of consistency with Bank strategies. Under both the original and revised outcome targets, relevance of design is rated Substantial, as the design linked multiple institutional and technical interventions in a mutually reinforcing and complementary manner, with all ultimately contributing to the desired outcome of increased relevant skills. The project results chain presented an underlying logic and linked planned activities to outputs and intermediate outcomes to outcomes. Under both the original and revised outcome targets, the objective to increase the level of skills to participate in the knowledge economy in pre-tertiary education students was achieved and is rated Substantial. Efficiency is rated Modest because of significant shortcomings in the efficiency of implementation. In conclusion, the information provided by the ICR supports the conclusion that the project achieved its intended outcomes, but with efficiency concerns. These ratings are indicative of moderate shortcomings in the project’s preparation and implementation, and therefore an Outcome rating of Moderately Satisfactory under both the original and revised outcome targets. According to IEG/OPCS guidelines, when a project’s objectives or its key associated outcome targets are Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) revised, the final outcome is determined by the weight of Bank disbursements under each set of objectives (38% under the original objectives, and 62% under the revised objectives). However, since the outcome ratings are the same for both, a formal calculation is unnecessary, and the overall outcome is rated as Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating At the technical level, the risk appears to be moderate, as systemic changes related to education quality improvements and skills development, decentralized school management, robust management information systems, merit-based selection of teachers, improved early child education, and improved physical learning environment are likely to continue to generate future benefits (ICR, p. 12). The persistent constrained fiscal situation, however, could put future funding of related programs at risk. Already, the maintenance system originally planned to be implemented by the Borrower did not materialize, thereby increasing the risk of infrastructure sustainability (ICR, p. 12). External factors such as the Syrian refugee crisis may disrupt the performance of the education sector. The ICR notes that attention and resources were diverted from systems aiming at improving learning quality towards instead dealing with school space for refugee children (ICR, p. 6). Whereas ERFKE II aimed to eliminate double-shift schools, the need to enroll the rapid influx of refugee children in fact resulted in an increase of double-shift schools. There was an increase in school violence (ICR, p. 6) that may negatively affect the safety of learning environments. It would be reasonable to assume that the above issues are not likely to be resolved in the near term. In conclusion, the overall risk that development outcomes may not be maintained is considered to be substantial. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The information provided by the ICR shows that the World Bank’s preparatory technical work was extensive, and was set in a well-established political framework, with lessons drawn from ERFKE I (ICR, p. 13). The strategic relevance for developing skills pertinent to the knowledge economy was high, and ERFKE II was Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) shaped by good practices and by the government’s strategy, notably the National Agenda 2006– 2015 and MOE Strategic Plan 2008–2013, whose focus was on the quality of education at all levels. The priorities were well identified, and the components that built around these priorities corresponded to education policy widely accepted as good practice, and pertinent to the Jordanian context (ICR, p. 4). The selection of ERFKE II focus areas was informed by a review of the ERFKE I experience, which took the form of a series of commissioned studies on school planning, teacher utilization, education sector finance, vocational education, decentralization, and school-based management (ICR, p. 4). Hence, ERFKE II was designed with consideration to lessons learned from the preceding project, mainly: (a) focus on teachers as the key agents of change; (b) focus on the school as the locus of change while continuing to support the enhancement of the institutional framework and capacity building at the central and field directorate levels; (c) building of capacity at the district and school levels; (d) having MOE serve as the facilitator of change; and (e) adoption of a participatory approach to the reform process. The selection of the development objectives was appropriate and measurable, as the country had its own national assessment program, and it participated regularly in related international surveys (ICR, p. 13). Implementation and monitoring arrangements were adequately prepared, and provided an important role to MOE’s Development Coordination Unit, which pre-existed under the previous project. Project preparation and implementation arrangements were consistent with the Bank’s fiduciary role. Environmental aspects were adequately addressed. Preparatory work, however, had some shortcomings. The mis-estimation of costs was substantial. Project appraisal underestimated important risks, such as the impact of various bilateral funding agreements on budget transfers, the government’s financial constraints, availability of additional financing, vulnerability to external shocks, and disruptions due to frequent changes in sectoral leadership and turnover in technical positions. The above issues have negatively impacted the efficiency rating. In conclusion, given adequate Bank performance in project identification, facilitation of preparation, and appraisal, but with moderate shortcomings, Bank performance in ensuring quality at entry is rated Moderately Satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision As the project was funded by multiple donor agencies, each donor was effectively responsible for supporting MOE’s delivery of the specific interventions that the donors were financing, although the Bank was assigned overall responsibility for coordinating supervision. The Bank and the donors developed a sound and positive approach to joint supervision (ICR, p. 13) and good reporting. The Bank was effective in convening multiple donors, and the joint-donor supervision offered a comprehensive, harmonized, and efficient approach to project oversight, and benefitted from complementarities in expertise (ICR, p. 5). The Bank provided intellectual leadership, with a focus on political dialogue, and also provided stability, as there was one team leader throughout project implementation (ICR, p. 13). The Bank team's focus on development effectiveness was demonstrated by its proactiveness in resolving threats, including two project restructurings in 2013 and 2015 to amend the results framework and to allow sufficient time to complete project activities. When a lack of coordination between MOE and the Ministry of Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) Public Works and Housing was observed during the first two years of implementation, the Bank team was instrumental in bringing both ministries together. This resulted in safeguards and fiduciary issues being resolved, and implementation expedited (ICR, p. 7). Over time, the respective teams of both ministries discussed, clarified, and agreed on their respective roles and responsibilities in a formal way. The Bank recommended the employment of an independent consultant by the Ministry of Public Works and Housing to carry out quarterly environmental reporting, and to prepare monthly site-by-site status reports, thus improving oversight (ICR, p. 7). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The ICR notes that the government established and supported a strong policy framework that facilitated project design and implementation, and that the government showed a strong commitment to maintaining project implementation in the face of the enormous stress placed on the school system with the arrival of large numbers of refugee children (ICR, p. 13). The government supported stakeholder consultations and implementation arrangements, and provided counterpart funding. However, frequent changes in political leadership, with 11 ministers of education appointed during the life of the project (ICR, p. 5), made policy continuity uncertain, and this uncertainty affected the implementation of some of the teachers’ policies (ICR, p. 13). Also, there were some issues in actual versus planned allocations, and in fund transfer delays (ICR, p. 6), although such issues were understandable in view of significant fiscal constraints. Given adequate performance, but with moderate shortcomings, government performance is rated Moderately Satisfactory. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The main implementing agency was MOE and its Development Coordination Unit. The Ministry of Public Works and Housing shared responsibility with MOE in school construction. Both ministries were responsive and highly committed to the project (ICR, p. 14). The Development Coordination Unit, the monitoring unit, and the component implementation teams were also committed and collaborated effectively (ICR, p. 13). MOE’s Development Coordination Unit was diligent and maintained strong donor coordination (ICR, p. 14). The Unit showed resilience under frequent political turnover and in the context of the challenging impact of the refugee crisis on the education sector. Fiduciary arrangements and M&E performance were adequate. Some shortcomings were noted in MOE’s procurement functions leading to repetitive re-bidding. Also, the Ministry of Public Works and Housing experienced shortcomings in planning and design, and in contract management (ICR, p. 7). Given adequate performance, but with moderate shortcomings, the Implementing Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) Agencies' Performance is rated Moderately Satisfactory. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E design was sound and its framework was comprehensive, covering all project components (ICR, p. 6). The choice of the development objectives was appropriate, and as importantly, progress towards the objectives was measurable, as Jordan had a well-established national assessment program (ICR, p. 13). Hence, there was ownership of M&E arrangements. Some baselines for intermediate level indicators remained to be determined after appraisal through related studies. b. M&E Implementation The required information was collected and reported regularly (ICR, p. 6). New information from the information system and from studies conducted by the National Center for Human Resources Development provided a strong rationale to refine some indicators and/or targets, and during the mid-term review of November 2012, it was decided to fully revise the results framework, where changes included the revision, addition, or dropping of some intermediate outcome indicators. More importantly, the 2013 restructuring changed the baseline from the NAFKE 2008 scores to the NAFKE 2011 scores by specifying that the change in scores would compare “year 6” with “year 4” of the project (ICR, p. 2), while concurrently revising the target to an average increase of 1 score point for Math, Science, and Arabic. The targets for the increase in enrollment rates for both the basic and secondary cycles were revised upwards. Under ERFKE II, the Open Education Management Information System was developed and became operational. Coordination improved between MOE and the National Center for Human Resource Development by way of a Memorandum of Understanding that led to analytical work and studies to inform MOE’s decision-making (ICR, p. 6). c. M&E Utilization M&E information was effectively used to measure the intended development outcomes, and to fine-tune the results framework to better measure such outcomes. The ICR does not provide information on whether Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) shifts in the project direction were needed or whether M&E findings were used for such shifts. At the same time, the ICR notes that the experience of ERFKE II and its lessons were guiding the proposed follow-on education project assisted by the Bank. It also states that the National Strategy for Human Resources Development (2016-20125) -- the government’s new ten-year education strategy approved in 2016 -- builds directly upon the framework of ERFKE II, most notably for the national school-based development system resulting in a sustainable management system at the local level, with parents involved in their schools' development; establishment of early childhood development quality standards; merit-based teacher recruitment; further development of in-service and pre-service teacher training; and the development of MOE accountability. In conclusion, the M&E system as designed and implemented was suitable to assess the achievement of objectives, to demonstrate the links in the results chain, and to serve future operations. Hence, M&E Quality is rated High. M&E Quality Rating High 11. Other Issues a. Safeguards The project was classified as Environmental Category B. It triggered Environmental Assessment (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12), as the project included new construction as well as extension and rehabilitation of existing schools, but with minor adverse environmental impact expected (PAD, p. 17). The concerned municipalities were consulted. The Environmental Management Plan, which was used under ERFKE I, was updated by MOE, disclosed, and adopted by ERFKE II. Also, Jordan had national regulations governing environment, public health, and safety for any new construction, and these regulations were based on the guidelines of the International Federation of Consulting Engineers (PAD, p. 18). According to the ICR, the Bank provided a special focus on safeguard compliance because more than 90% of Bank funding was allocated to the improvement of the physical learning environment (ICR, pp. 6-7). The ICR states that oversight was further strengthened through the assignment of an independent consultant by the Ministry of Public Works and Housing to carry out quarterly environmental reporting, and to prepare monthly site-by-site status reports (ICR, p. 7). However, the ICR does not provide sufficient or explicit information about compliance with safeguard policies, nor on potential impacts and mitigation activities. The project team stated that no environmental or social impacts were recorded during the project, and there was no resettlement (TTL clarification, 9/5/2017). In conclusion, there was compliance, but with insufficient reporting by the ICR. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) b. Fiduciary Compliance Financial Management. Financial management arrangements were adequate and benefited from the experience gained under ERFKE I, notably on Bank guidelines and procedures (PAD, p. 15). The external auditor issued unqualified audit opinions on all audited financial statements of the project (ICR, p. 7). Management letters did not report any significant weaknesses in the project’s internal controls, with one exception: the 2012 management letter reported that MOE had paid bonuses from the government’s contribution to employees directly and indirectly working on ERFKE II. MOE subsequently recovered all unwarranted payments from employees not directly involved in the project. There were short delays in the submission of the project’s interim financial reports due to coordination gaps between MOE’s Development Coordination Unit and the General Budget Department. Although the use of accounting software was prevalent in government institutions, the General Budget Department used Excel spreadsheets to manage project contracts, resulting in errors that were subsequently corrected (ICR, p. 7). Procurement. Procurement was undertaken according to Bank guidelines and the government public procurement, but there were shortcomings in MOE’s procurement processes related to weak drafting of technical specifications, which led to repetitive re-bidding (ICR, p. 7). Also, the Ministry of Public Works and Housing showed weaknesses in planning and contract management (ICR, p. 7). c. Unintended impacts (Positive or Negative) According to the ICR, the project helped the government in maintaining a resilient schooling system in the context of considerable stress resulting from the arrival of large numbers of refugee children from Syria (ICR, p. 10). d. Other N/A. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory The ICRR rates the risk as Substantial because of fiscal Risk to Development constraints and the impact of Modest Substantial Outcome the refugee crisis on the education sector. These risks are likely to continue in the Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) near term. The ICRR rates the quality of supervision as Satisfactory, and the Quality at Entry as Moderately Bank Performance Satisfactory Moderately Satisfactory Satisfactory because of moderate shortcomings in costing and risk assessment. Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (pp. 14-15) offers several useful lessons, including the following lessons adapted by IEG: • A bottom-up change built around the school community is more effective than top-down change. The project showed that the sustainability of a national school-based development system relies on the system’s development at the local level, rather than on a series of isolated interventions. When donors supported local players and built their capacity, rather than substituting for them, sustainability was enhanced. Local community engagement, such as having parents involved in the school development improvement plan, countersigning school expenditures, and meeting regularly with school staff, resulted in a sense of ownership that contributed to sustainability. • Close and early attention to costing inputs during the design process reduce the risks of financial shortfalls. Under the project, unbalanced estimates for infrastructure resulted in the curtailment of the number of schools that were constructed. • When priorities are clearly established within a national policy framework, a programmatic approach that brings donors together to cooperate on a single platform is more effective than separate interventions. This was demonstrated under the project through harmonization and cooperation among the donors, facilitated by the Bank’s effective convening role in a multi-donor partnership. 14. Assessment Recommended? No Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JO - ERfKE II (P105036) 15. Comments on Quality of ICR The ICR is clear and concise, with a direct performance storyline. It provides quality analysis and candor. The ICR is results-oriented. Its discussion of the results framework and institutional development aspects is thorough. The quality of the evidence is robust and well presented, which provides a basis for drawing conclusions. The report offers useful lessons derived from project experience. The report is consistent with the guidelines, but with gaps in reporting and explaining actual component costs, and insufficient information on compliance with environmental safeguards. a. Quality of ICR Rating Substantial Page 18 of 18