Document of The World Bank Report No: ICR00004001 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-77020) ON A LOAN IN THE AMOUNT OF US$60.0 MILLION TO THE HASHEMITE KINGDOM OF JORDAN FOR A SECOND EDUCATION REFORM FOR THE KNOWLEDGE ECONOMY PROJECT June 29, 2017 Education Global Practice Middle East and North Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2016) Currency Unit = Jordanian Dinar (JOD) JOD 1.00 = US$ 1.41 US$ 1.00 = JOD 0.71 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CAS Country Assistance Strategy CIDA Canadian International Development Association DCU Development Coordination Unit ERFKE I Education Reform for the Knowledge Economy (P075829) ERFKE II Second Education Reform for the Knowledge Economy Project (P105036) ETC Education Training Center GOJ Government of Jordan IBRD International Bank for Reconstruction and Development ICR Implementation Completion Report IFC International Finance Corporation ISR Implementation Status and Results Report JICA Japanese International Cooperation Agency KG1 Kindergarten Level 1 (four-year-olds) KG2 Kindergarten Level 2 (five-year-olds) MOE Ministry of Education MOPWH Ministry of Public Works and Housing NAFKE National Assessment for the Knowledge Economy NCHRD National Center for Human Resources Development PAD Project Appraisal Document PISA Program for International Skills Assessment PDO Project Development Objective PPF Project Preparation Facility QRTA Queen Rania Teacher Academy SDDP School Directorate and Development Plan TIMSS Trends in Mathematics and Science Study USAID United States Agency for International Development Senior Practice Director: Jaime Saavedra Practice Manager: Safaa El-Kogali Project Team Leader: Juan Manuel Moreno ICR Author: Jeffrey Waite ii JORDAN Second Education Reform for the Knowledge Economy CONTENTS Data Sheet A. Basic Information ................................................................................................... iv B. Key Dates ............................................................................................................... iv C. Ratings Summary ................................................................................................... iv D. Sector and Theme Codes ........................................................................................ v E. Bank Staff ............................................................................................................... v F. Results Framework Analysis .................................................................................. vi G. Ratings of Project Performance in ISRs .............................................................. xiv H. Restructuring ....................................................................................................... xiv I. Disbursement Graph ............................................................................................. xv Implementation Completion and Results Report 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 4 3. Assessment of Outcomes ............................................................................................ 8 4. Assessment of Risk to Development Outcome......................................................... 12 5. Assessment of Bank and Borrower Performance ..................................................... 13 6. Lessons Learned ....................................................................................................... 14 7. Comments on Issues Raised by Borrower ................................................................ 15 Annexes Annex 1. Project Costs and Financing .......................................................................... 16 Annex 2. Outputs by Component ................................................................................. 18 Annex 3. Economic and Financial Analysis ................................................................. 20 Annex 4. Bank Lending and Implementation Supervision Processes .......................... 25 Annex 5. Borrower's Implementation Completion Report ........................................... 27 Annex 6. List of Supporting Documents ...................................................................... 36 Annex 7. PDO Indicator No. 1: Calculation of Actual Value Achieved ...................... 38 iii A. Basic Information Second Education Country: Jordan Project Name: Reform for the Knowledge Economy Project ID: P105036 Loan Number: IBRD-77020 ICR Date: 11/07/2016 ICR Type: Core ICR The Hashemite Lending Instrument: SIL Borrower: Kingdom of Jordan Original Total US$60.0 million Disbursed Amount: US$60.0 million Commitment: Environmental Category: B Implementing Agencies: Ministry of Education Cofinanciers and Other External Partners: CIDA, JICA, USAID; European Commission B. Key Dates Process Date Process Original Date Actual Date Concept Review: 05/27/2008 Effectiveness: 03/30/2010 01/22/2010 Appraisal: 02/20/2009 Mid-term Review: 11/18/2012 11/12/2012 10/24/2013 & Approval: 05/19/2009 Restructuring: -- 03/19/2015 Closing: 12/31/2015 12/31/2016 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Moderately satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Moderately satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Moderately satisfactory Performance: Performance: iv C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Major Sector/Sector Public Administration Public administration - Education 20 20 Education Vocational training 10 10 Secondary Education 25 25 Primary Education 25 25 Early Childhood Education 20 20 Major Theme/Theme/Sub Theme Human Development and Gender Education 102 102 Access to Education 27 27 Education Financing 3 3 Science and Technology 24 24 Standards, Curriculum and Textbooks 24 24 Teachers 24 24 E. Bank Staff Positions At ICR At Approval Vice President: Hafez Ghanem Daniela Gressani Country Director: Ferid Belhaj Hedi Larbi Practice Safaa El-Kogali Mourad Ezzine Manager/Manager: Project Team Leader: Juan Manuel Moreno Peter Buckland ICR Team Leader: Juan Manuel Moreno ICR Primary Author: Jeffrey Waite v F. Results Framework Analysis Project Development Objective (from Project Appraisal Document) ERFKE II had as its development objective that students enrolled in [all streams of] pre- tertiary education in Jordan have increased levels of skills necessary for participation in the knowledge economy.1 Revised Project Development Objective (as approved by original approving authority) No revision was made to the development objective. 1 PAD, section IV.A. (para. 48), as well as PAD, Annexes 4 and 10, specifies “all streams of pre-tertiary education.” PAD, Datasheet, PAD, section II.C (para. 16) and Loan Agreement, Schedule 1, speak only of “pre-tertiary education.” vi (a) PDO Indicators Formally Original Actual Revised Baseline Target Value Target Indicator Value Values Achieved Values (06/01/08) [PAD] [ICR] [MTR] (01/22/10) (12/31/16) (10/24/13) Increase in scores on national assessments NAFKE An average An average Average aligned with knowledge economy skills 20082 increase of increase increase, 3 score (between between points for year 4 & NAFKE Math, year 6)3 of 2011 and Science and 1 score 2014, of Arabic point for 3 score Math, points4 for Science and Math, Arabic Science and Arabic in Grades 5, 9 & 11 Increase in enrollment rate5: 97.6% 97.6% 98.0% 97.9% NER basic cycle (98.8%)6 [2015/16] Increase in enrollment rate: 61.5% 63.0% 78.2% 72.6% NER secondary cycle (76.0%)7 [2015/16] 2 The October 2013 Restructuring Paper (Report No. RES11824), by specifying that the change in scores would compare “year 6” scores with “year 4” scores, effectively changed the baseline from the NAFKE 2008 scores to the NAFKE 2011 scores. The NAFKE 2008 baseline values set out in the PAD (p. 31) were used until the April 2016 ISR; the NAFKE 2011 test score results were used as the new baseline from the October 2016 ISR. 3 Whereas the October 2013 Restructuring Paper includes the words “between year 4 and year 6” (i.e. between the fourth and the sixth year of the project) in the definition of the target set out in the revised Results Framework, these words are absent from the revised Results Framework as it appears in the Notice of Reallocation of Loan Proceeds and Revised Performance Monitoring Indicators, dated October 24, 2013, and countersigned by the Borrower’s Representative. 4 For calculation of “increase in scores on national assessments” between NAFKE 2011 and NAFKE 2014, see Annex 10, table (a). 5 Enrollment rates do not include Syrian refugee students. 6 In PAD Annex 3, the Results Framework (p. 27) gave the original target as 97.6%, whereas the Arrangements for Results Monitoring (p. 31) gave it as 98.8%. 7 In its end-of-project reporting, MOE corrected the 2009/10 baseline secondary enrollment rate to 76.0%. vii (b) Intermediate Outcome Indicators Baseline Original Formally Actual Value Target Revised Value Indicator: Establishment of a National Values Target Achieved School-Based Development System [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Development and implementation of Nil Final Unchanged Yes, imple- enabling policies, guidelines and procedures mented8 A single School Evaluation Instrument, Nil Final Unchanged Yes, being focused on ERFKE outcomes, agreed and used9 being used for school self-evaluation and for public and professional accountability Stakeholder views on extent to which Nil Final Dropped -- decision-making authority and associated resources are being utilized to enable implementation of school improvement plans Stakeholder views on extent to which Nil Final Dropped -- mechanisms for professional and public accountability, linked to the school improvement cycle, have been established and are functioning School principals’ and teachers’ views on Nil Final Dropped -- quality of field directorate and central administration support toward achievement of school development plans External stakeholder assessment of graduate Nil Final Dropped -- knowledge economy skills Number (percentage) of schools that are 0 3,318 3,000 3,554 implementing improvement plans10 (0%) (100%) Number of field directorates that are -- -- 35 42 implementing improvement plans Number of schools having received financial -- -- 3,000 3,554 support from MOE to continue implementing School Improvement Plans11 Number of field directorates having received -- -- 35 42 financial support from MOE to continue implementing Field Improvement Plans 8 School Directorate and Development Program, along with enabling policies, guidelines and procedures. 9 Instrument developed and used as part of School Directorate and Development Program. 10 Excluding the six (6) directorates piloted under the SJE Project. 11 Number of schools having received financial support in a final year (i.e. non-cumulative number). viii Baseline OriginalFormally Actual Value Target Revised Value Indicator: Policy, Planning and Values Target Achieved Organizational Development [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Review and realignment of mandate and Nil Review and Unchanged Review and responsibilities of MOE organizational realignment realignment structures relevant to the school completed completed12 improvement process Number of non-school staff participating in Nil 1,100 1,200 2,425 on-going professional development programs to support the delivery of the school improvement program13 Review and development of mechanisms for Nil 80% of Dropped -- professional and public accountability schools Stakeholder perceptions of extent to which Nil Final Unchanged Baseline implemented studies are responsive to the study M&E framework and the approved conducted in implementation plan is carried out and 2013; no disseminated follow-up Stakeholder perceptions of relevance of Nil Final Unchanged study M&E reports for informing policy and conducted planning Extent to which SIS/EMIS is producing data 10 32 Unchanged 21 for continuous monitoring of the 32 key indicators indicators indicators14 performance indicators 12 MOE’s new organizational management structure published in Official Gazette No. 5775, dated September 4, 2016. 13 Includes directorates’ staff only. 14 New set of indicators adopted as part of the change from the original EMIS to the new Open EMIS. ix Baseline Original Formally Actual Value Target Revised Value Indicator: Teaching and Learning Values Target Achieved Resource Development [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Teacher policies revised to support Nil 100% of Unchanged 90% of application of national teacher standards target target policies policies revised revised Percentage of newly appointed teachers Nil 80% 40% 96%15 completing post-recruitment initial training in the Education Training Center Percentage of new teachers appointed using Nil 80% Dropped -- a competency-based model Percentage of subjects by grade reviewed Nil All subjects All subjects All subjects and fine-tuned to ensure alignment with for Grades for Gradesfor Grades knowledge economy skills 11-12 5-10 4-11 reviewed & reviewed &reviewed & fine-tuned fine-tunedfine-tuned Tawjihi examinations reviewed and Nil Harmonized Unchanged New plan arrangements put in place for harmonization examina- endorsed, to with the goals of the ERFKE curriculum tions in be imple- place mented 2017/18 Extent of e-Learning/ICT utilization in the 67% 73% Unchanged Data result teaching and learning process (by subject & (average) (average) pending16 grade), as revealed by nationally representative classroom observation studies Number (percentage) of schools (or TBD TBD Dropped -- classrooms) meeting basic standards for learning tools and resources Content and design of NAFKE is reviewed Nil Review Unchanged Review for overall technical soundness and completed completed alignment with the ERFKE curriculum, and & revisions & revisions arrangements put in place for any necessary in place in place17 revisions 15 Percentage of teachers completing training in a final year (i.e. non-cumulative number). 16 Pending NCHRD analysis of TIMSS 2015 data. 17 NAFKE instrument was reviewed, and a revised instrument piloted and then used for the 2014 assessment. x Baseline Original Formally Actual Value Target Revised Value Values Target Achieved Indicator: Early Child Education [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Percentage of eligible children enrolled in 52% 60% Unchanged 61%18 KG2 in all authorities Percentage of KG teachers and supervisors 93% 100% Unchanged 99% successfully completing prescribed training program for early childhood education19 Number of parents of KG students enrolled 2,000 3,500 5,000 4,33320 in volunteer programs Percentage of KG classrooms that meet TBD TBD 75% 94% MOE quality assurance standards21 Baseline Original Formally Actual Value Target Revised Value Values Target Achieved Indicator: Vocational Education [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Percentage of VET textbooks aligned with -- -- 75% 65% market requirements and adopted for Grades 11 & 12 Percentage of programs aligned with 0% 100% of Dropped -- vocational & technical guidelines/standards programs for curriculum, equipment and facilities Completion rates for vocational & technical N/A 100% of Dropped -- programs programs Increase in employer satisfaction with the Baseline 20% Dropped -- skills and abilities of labor market entrants survey increase in holding vocational education certificates overall awarded by the MOE satisfaction 18 Percentage of children enrolled in a final year (i.e. non-cumulative number). 19 National Curriculum: Working with Young Children. 20 Number of parents enrolled in a final year (i.e. non-cumulative number). 21 At the time of appraisal, Quality assurance standards were being developed and were expected to be completed in October 2009. xi Baseline Original Formally Actual Value Target Revised Value Values Target Achieved Indicator: Special Education [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Revision and approval of policies to support Nil Policies Unchanged Policies special education programs & services imple- imple- mented mented22 Number of teachers trained and qualified Nil 300 50 5023 according to national standards for gifted students Number of teachers trained and qualified Nil 300 50 6024 according to national standards for disabled students Number of students enrolled in special 3,875 6,500 8,750 8,313 education programs for gifted students Number of students enrolled in special 13,894 15,500 18,500 19,650 education programs for disabled students Number of gifted students with access to 3,875 6,500 Unchanged 8,313 services relevant to their individual needs Number of disabled students with access to 23825 850 Unchanged 1,791 services relevant to their individual needs 22 Pending legislative action. 23 Number of teachers trained and qualified in a final year (i.e. non-cumulative number). 24 Number of teachers trained and qualified in a final year (i.e. non-cumulative number). 25 Number with access to wheelchair or hearing aid. xii Baseline Original Formally Actual Value Target Revised Value Indicator: Physical Learning Values Target Achieved Environments [PAD] Values [ICR] [MTR] (06/01/08) (01/22/10) (10/24/13) (12/31/16) Number of students with access to learning 859,867 920,000 945,000 985,55327 in a safe and well-managed physical environment26 Decrease in number of underutilized schools Nil 20 10 132 in all field directorates (from 1,891 schools) Decrease in number of overcrowded schools Nil 20 10 -13328 in all field directorates (from 1,244 schools) Number of additional classrooms Nil 1,000 583 624 Number of additional laboratory spaces for Nil 120 12 9 ~ 13 ICT Number of additional laboratory spaces for Nil 120 31 30 ~ 34 science Policies for continuous building maintenance Nil Policies in Unchanged Policies in place place & reviewed, reviewed but no decision to implement 26 MOE schools. 27 While numbers have increased, the proportion of students with access to safe physical learning environments has decreased due to the influx of Syrian refugee children and the resultant need to have recourse to rented spaces. 28 Read: number of overcrowded schools increased from 1,244 to 1,377, due to the influx of Syrian refugee children. xiii G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 05/06/2010 Satisfactory Satisfactory 0.00 2 01/08/2011 Moderately Satisfactory Moderately Satisfactory 0.10 Moderately 3 07/05/2011 Moderately Satisfactory 0.10 Unsatisfactory Moderately 4 01/28/2012 Moderately Satisfactory 2.14 Unsatisfactory 5 03/19/2012 Moderately Satisfactory Moderately Satisfactory 2.64 6 07/04/2012 Moderately Satisfactory Moderately Satisfactory 7.10 7 01/06/2013 Moderately Satisfactory Moderately Satisfactory 15.34 8 09/07/2013 Moderately Satisfactory Satisfactory 22.62 9 02/03/2014 Moderately Satisfactory Satisfactory 26.05 10 07/30/2014 Moderately Satisfactory Moderately Satisfactory 29.66 Moderately 11 01/23/2015 Moderately Satisfactory 33.29 Unsatisfactory Moderately 12 07/17/2015 Moderately Satisfactory 39.17 Unsatisfactory 13 10/23/2015 Moderately Satisfactory Moderately Satisfactory 42.07 14 04/14/2016 Moderately Satisfactory Moderately Satisfactory 51.07 15 10/10/2016 Moderately Satisfactory Moderately Satisfactory 59.85 H. Restructuring ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in US$ Millions 10/22/2013 N/A MS S 22.62 Revision of indicator values 03/19/2015 N/A MS MU 34.61 Extension of closing date xiv I. Disbursement Profile Value Date Disbursed Amount (US$) Cumulative Amount (US$) 21-Jun-2010 $ 100,000.00 $ 100,000.00 10-Feb-2011 $ 2,206.06 $ 102,206.06 29-Sep-2011 $ 80,150.01 $ 182,356.07 07-Nov-2011 $ 1,000,000.00 $ 1,182,356.07 24-Jan-2012 $ 395,949.00 $ 1,578,305.07 24-Jan-2012 $ 560,823.02 $ 2,139,128.09 08-Feb-2012 $ 500,000.00 $ 2,639,128.09 05-Apr-2012 $ 889,032.26 $ 3,528,160.35 30-Apr-2012 $ 69,302.97 $ 3,597,463.32 08-May-2012 $ 1,314,936.83 $ 4,912,400.15 29-May-2012 $ 1,500,000.00 $ 6,412,400.15 18-Jun-2012 $ 689,789.71 $ 7,102,189.86 10-Jul-2012 $ 1,333,975.57 $ 8,436,165.43 30-Aug-2012 $ 1,468,644.15 $ 9,904,809.58 13-Nov-2012 $ 42,285.13 $ 9,947,094.71 14-Nov-2012 $ 1,433,653.70 $ 11,380,748.41 29-Nov-2012 $ 1,428,474.14 $ 12,809,222.55 17-Dec-2012 $ 1,082,420.98 $ 13,891,643.53 27-Dec-2012 $ 1,449,755.19 $ 15,341,398.72 19-Feb-2013 $ 662,141.07 $ 16,003,539.79 07-Mar-2013 $ 1,469,817.72 $ 17,473,357.51 12-Mar-2013 $ 47,656.56 $ 17,521,014.07 19-Mar-2013 $ 241,586.75 $ 17,762,600.82 26-Mar-2013 $ 683,311.69 $ 18,445,912.51 xv 29-Apr-2013 $ 1,378,549.72 $ 19,824,462.23 10-Jun-2013 $ 1,282,294.87 $ 21,106,757.10 23-Jul-2013 $ 161,758.40 $ 21,268,515.50 06-Aug-2013 $ 1,347,681.66 $ 22,616,197.16 24-Oct-2013 $ 1,212,635.20 $ 23,828,832.36 05-Nov-2013 $ 500,000.00 $ 24,328,832.36 26-Nov-2013 $ 198,783.28 $ 24,527,615.64 05-Dec-2013 $ 138,066.64 $ 24,665,682.28 30-Dec-2013 $ 1,384,104.09 $ 26,049,786.37 24-Feb-2014 $ 1,064,954.25 $ 27,114,740.62 07-Jul-2014 $ 980,284.68 $ 28,095,025.30 14-Jul-2014 $ 575,562.61 $ 28,670,587.91 24-Jul-2014 $ 990,876.05 $ 29,661,463.96 18-Sep-2014 $ 778,248.95 $ 30,439,712.91 20-Nov-2014 $ 1,238,061.36 $ 31,677,774.27 01-Dec-2014 $ 219,948.21 $ 31,897,722.48 22-Dec-2014 $ 1,324,468.67 $ 33,222,191.15 21-Jan-2015 $ 63,986.55 $ 33,286,177.70 02-Mar-2015 $ 1,321,252.63 $ 34,607,430.33 27-Apr-2015 $ 1,441,105.77 $ 36,048,536.10 11-May-2015 $ 261,121.78 $ 36,309,657.88 11-May-2015 $ 1,076,930.60 $ 37,386,588.48 02-Jun-2015 $ 1,366,478.30 $ 38,753,066.78 16-Jul-2015 $ 411,996.95 $ 39,165,063.73 17-Aug-2015 $ 205,619.31 $ 39,370,683.04 17-Aug-2015 $ 1,470,772.09 $ 40,841,455.13 15-Oct-2015 $ 1,229,356.27 $ 42,070,811.40 09-Nov-2015 $ 947,822.36 $ 43,018,633.76 09-Dec-2015 $ 563,870.07 $ 43,582,503.83 21-Dec-2015 $ 1,468,366.45 $ 45,050,870.28 28-Dec-2015 $ 1,441,295.91 $ 46,492,166.19 04-Jan-2016 $ 1,388,880.50 $ 47,881,046.69 19-Jan-2016 $ 1,422,819.80 $ 49,303,866.49 02-Feb-2016 $ 406,525.08 $ 49,710,391.57 21-Mar-2016 $ 1,356,793.21 $ 51,067,184.78 18-Apr-2016 $ 1,387,930.94 $ 52,455,115.72 09-May-2016 $ 482,070.74 $ 52,937,186.46 23-May-2016 $ 1,090,502.56 $ 54,027,689.02 27-Jun-2016 $ 4,000,000.00 $ 58,027,689.02 28-Jun-2016 $ 1,200,000.00 $ 59,227,689.02 30-Jun-2016 $ 622,310.98 $ 59,850,000.00 xvi 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Jordan’s education system made significant advances during the years 1999/2000 to 2005/06. During that period, net enrollment rates in basic education increased from 89 percent to 97 percent. Transition rates to secondary education increased from 63 percent to 79 percent, thereby putting substantial pressure on the secondary schooling institutional infrastructure. The percentage of 5-year-olds enrolled in kindergarten level 2 (KG2) rose from 47 percent in 1999/2000 to 51 percent in 2005/06 in 2005/06. The number of privately owned and operated schools increased by 12 percent in KG1-KG2, 19 percent in basic education, and 17 percent in secondary education. However, enrollments in privately owned and operated schools amount to less than 10 percent of all enrollments in basic and secondary education. 2. Over the same period, based on the 1999 to the 2007 editions of the Trends in International Mathematics and Science Study (TIMSS), eighth-grade students in Jordan had an increase in science score of around one-third of a standard deviation (32 scale points) and remained around the same for mathematics score (a 1-point decrease). While the gain in science skills could not be attributed solely to the impact of the Education Reform for the Knowledge Economy (ERFKE I) program, which ran from 2003 to 2009 and to which the World Bank contributed, it nonetheless suggests that Jordan had established positive momentum in student performance in this curriculum area. 3. The Second Education Reform for the Knowledge Economy (ERFKE II) program was aligned with the World Bank’s Country Assistance Strategy (CAS) for the Hashemite Kingdom of Jordan for the period 2006–2010. The strategy was organized in four cross-sectoral clusters, the first of which was “strengthening the investment environment and building human resources for a value-added, skill-intensive and knowledge-based economy.” With respect to the education sector, ERFKE II built on the achievements made under ERFKE I and continued to provide support to the Government of Jordan (GOJ) in the implementation of the second phase of their ten-year reform program using a sectoral approach. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 4. ERFKE II aimed to ensure that “students enrolled in [all streams of] pre-tertiary education in Jordan will have increased levels of skills necessary for participation in the knowledge economy.”29 5. ERFKE II had two key indicators:  Increase in scores on national assessments aligned with knowledge economy skills (Target: an average increase of three (3) score points for mathematics, science and Arabic)  Increase in enrollment rates (Target: maintain the current high net enrollment rate [97.6 percent] for the basic cycle, and increase the net enrollment rate to 63.0 percent for the secondary cycle) 29 PAD, section IV.A. (para. 48), as well as PAD, Annexes 4 and 10, specifies “all streams of pre-tertiary education.” PAD, Datasheet, PAD, section II.C (para. 16) and Loan Agreement, Schedule 1, speak only of “pre-tertiary education.” 1 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and Reasons/Justification 6. The PDO was not revised in the course of the implementation period. 7. However, the first of the two key indicators—national assessment scores—was revised in 2013 in light of the National Assessment for the Knowledge Economy (NAFKE) assessment tool redesign that aimed to better measure knowledge economy skills. The October 2013 Restructuring Paper, effectively changed the baseline from the NAFKE 2008 scores to the NAFKE 2011 scores by specifying that the change in scores would compare “year 6” of the project (i.e. 2013) with “year 4” of the project (i.e. 2015).30 The NAFKE 2008 baseline values set out in the PAD were used until the April 2016 Implementation Status and Results Report (ISR); the NAFKE 2011 test score results were used as the new baseline from the October 2016 ISR onwards.31 Whereas the October 2013 Restructuring Paper includes the words “between year 4 and year 6” in the definition of the target set out in the revised Results Framework, these words are absent from the revised Results Framework as it appears in the Notice of Reallocation of Loan Proceeds and Revised Performance Monitoring Indicators, dated October 24, 2013, and countersigned by the Borrower’s representative. 8. More generally, at the time the results framework was finalized in 2009, several baseline and target values were yet to be identified or clarified. By 2013, information generated by the education management information system and studies conducted by the National Center for Human Resources Development (NCHRD) enabled indicator values to be better defined. The resulting changes covered (a) the revision of nine indicators to refine their description and/or targets; (b) the dropping of seven indicators that could not be monitored due to absence of baseline, unclear description, or relevance; and (c) the addition of one indicator and three sub-indicators.32 1.4 Main Beneficiaries 9. According to the PAD, ERFKE II aimed to “provide students enrolled in pre-tertiary education institutions in Jordan with increased levels of skills to participate in the knowledge economy.”33 This set of students remained the primary target beneficiary group throughout project implementation. 10. Secondary target beneficiary groups appear to include those acquiring new skills and thereby greater capacity to deliver better educational outcomes: classroom teachers, school administrators, and GOJ officials. 1.5 Original Components 11. Component 1: Establishment of a National School-Based Development System. To support the creation of an effective, school-based development process as the main vehicle to deliver to Jordan’s young people a quality education focused on developing the abilities, skills, attitudes and values associated with a knowledge-based economy. 30 Report No. RES11824. 31 PAD, p. 31. 32 For details of changes in the results framework, see Datasheet, section F. 33 PAD, p. 5. 2 12. Component 2: Monitoring and Evaluation and Organizational Development. To build upon ERFKE I investments related to policy, planning, monitoring and evaluation, and to ensure that outputs from these activities fully support and inform the adoption of a school-centered approach to the delivery of education services. Two sub-components: (i) Policy Development, Strategic Planning, Monitoring and Evaluation; and (ii) Organizational Development. 13. Component 3: Development of Teaching and Learning. To review and develop teacher employment, utilization and professional development policies and practices, fine-tune a limited range of curriculum and student assessment arrangements, and ensure support for associated new developments. Two sub-components: (i) Teacher Policies, Training and Professional Development, and (ii) Curriculum, Assessment and Learning Resources. 14. Component 4: Development of Special Focus Program. To improve inclusive access to learning for all children through special focus on three critical subsectors: (i) Early Childhood Education, (ii) Special Education and (iii) Vocational Education. 15. Component 5: Improvement of Physical Learning Environments. To improve the provision of quality education facilities in a cost effective and sustainable manner so that students have access to environmentally friendly and efficiently operated quality physical learning environments. Three sub-components: (i) Alignment of Ministry of Education (MOE) standards with international design standards, and the requirement of education reform; (ii) Construction and equipment of new schools and provision of extensions and equipment to existing schools; and (iii) Establishing a maintenance and management system for school buildings. 16. Since almost the entire amount of IBRD funding was allocated to the improvement of physical learning environments (Component 5), the World Bank team’s compliance and reporting efforts focused on this component. The World Bank was to finance the construction of 36 new schools and the extension of a further 120 schools, with furniture and science/IT labs (plus equipment for vocational education streams), but not the establishment of a maintenance system (for which no funding was allocated). 1.6 Revised Components 17. No formal revision was made to project components, with the exception of the reduction– -from 36 to 26–-in the number of schools to be built under Component 5, made at the time of the 2013 restructuring.34 1.7 Other significant changes 18. Funding allocations. On October 24, 2013, as part of the project restructuring prompted by the midterm supervision review, US$5.0 million of the IBRD loan amount was reallocated to the Works category from all other categories, including the total unallocated amount, with the exception of the front-end fee. This reallocation was made in recognition of the higher unit cost of school construction than originally estimated, the inability to close the funding gap and initial delays in the school construction schedule. At the same time, the number of schools to be built under the project was reduced from 36 to 26. 34 See ‘Funding Allocations’, section 1.7. 3 19. Implementation schedule. On March 19, 2015, a Level II restructuring extended the period of project implementation by rescheduling the closing date from December 31, 2015, to December 31, 2016, to allow sufficient time to complete the last four new schools and three school extensions—infrastructure that was considered critical in light of the pressure on educational space coming from the inflow of Syrian refugee children. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 20. Overall, ERFKE II was shaped by GOJ strategy, most notably the National Agenda 2006– 2015 and the derivative MOE Strategic Plan 2008–2013. The focus in these documents was on the quality of education, at the kindergarten, basic education, and secondary education levels, with the aim of developing human resources while assuring efficiency and accountability. 21. Following on closely from ERFKE I, which closed on June 30, 2009, ERFKE II was designed with consideration to lessons learned from the preceding project: (a) focus on teachers as the key agents of change; (b) focus on the school as the locus of change while continuing to support the enhancement of the institutional framework and capacity building at the central and field directorate levels; (c) build capacity at the district and school levels; (d) have the MOE serve as the facilitator of change rather than the implementer of change; and (e) adopt a participatory approach to the reform process.35 22. The selection of ERFKE II focus areas—system management, policy and strategic planning capacity, teacher policy and learning quality, equitable access to physical facilities, and the sub-sectoral focus on early child education and vocational education—was informed primarily by a review of the ERFKE I experience, which took the form of a series of commissioned studies on school planning, teacher utilization, education sector finance, vocational education, decentralization, and school-based management. 23. When the project was appraised, a financing gap of US$153.9 million was identified (or almost 40 percent of the total ERFKE II project cost).36 While the European Union later contributed around 20 percent of the gap amount, further financing contributions were not forthcoming. This left the program underfunded, at least until the World Bank’s 2013 restructuring reduced the volume of school construction and reallocated loan resources to this component.37 2.2 Implementation 24. The World Bank had played a leading role in the development of ERFKE I, which, by the time it closed, had brought together nine donors. 38 Of the more than US$380 million donor- mobilized financing, the World Bank, with US$120 million, had been the largest donor in ERFKE I. ERFKE II was expected to follow the ERFKE I model of implementation, and there was initial 35 World Bank. Education Reform for the Knowledge Economy Project: Implementation Completion and Results Report (Report No. ICR00001208). Washington, D.C.: World Bank, 2009. 36 See Annex 1, table (b). 37 See ‘Funding Allocations’, section 1.7. 38 Most notably, Global Affairs Canada (formerly Canadian International Development Agency), European Union, and United States Agency for International Development. 4 informal agreement among donors that the World Bank would supervise the full project, irrespective of source of funds. However, after the first two supervision missions, donors agreed on joint biannual supervision missions, led by the World Bank. In addition to supervising their respective components (by assigning the task to staff that were not also involved in implementation of that component), each donor also reviewed and contributed to the full aide-mémoire and messaging, based on their areas of expertise; the project therefore benefitted fully from these complementarities in expertise. The MOE’s Development Coordination Unit (DCU) found this joint-donor supervision model exemplary, on the grounds that it offered a comprehensive, integrated, harmonized, coherent, and efficient approach to program oversight. 25. The project was restructured twice. On October 24, 2013, the results framework was revised, and on March 19, 2015, the number of schools was reduced and the project closing date extended. 39, 40 26. Whereas ERFKE I was implemented in a context of economic prosperity and GOJ stability, ERFKE II was implemented against the backdrop of a global financial crisis (which led the Ministry of Finance to curb public spending), an “Arab Spring”, a refugee influx, and GOJ instability (with eleven ministers of education during the life of the project). 27. ERFKE II implementation was particularly affected by the Syrian refugee crisis. In 2004, Jordan had a population of 5.1 million. By 2015, as shown in Table 1, it had increased to 9.5 million, with almost a tenth of those being Syrian refugees who had arrived in the previous five years. While the GOJ went to considerable lengths to welcome the newcomers, it is clear that the sudden, and unexpected, influx of refugee children into the schooling system was disruptive to teaching and learning. Table 1: Syrian refugees as a proportion of the population aged less than 20 Housing & Population Census 201541 Number Percentage Total population 9,531,712 100.0 Of which, age less than 20 4,221,908 44.3 Of which, refugees 1,305,350 13.7 Of which, Syrian refugees 953,289 10.0 Of which, Syrian refugees arrived within 937,388 9.8 previous 5 years 28. By 2015, as shown in Table 2, 12.7 percent of the school population were refugees, and the recent arrival of such large numbers of refugee children placed considerable pressure on the public school system. Table 2: Refugees as a proportion of the population aged 4 and above enrolled in basic education Housing & Population Census 2015 Number Percentage Total population aged 4 and above 2,481,260 100.0 enrolled in basic education42 39 See section 1.3. 40 See section 1.7. 41 Source: Jordan Department of Statistics, Population and Housing Census 2015 (http://www.dos.gov.jo/dos_home_e/main/population/census2015/index.htm) 42 Kindergarten, basic, vocational, and secondary education. 5 Of which, refugees aged 4 and above 314,491 12.7 enrolled in basic education 29. The MOE miscalculated the effect of the Syrian refugee crisis early in the project, anticipating a quick end to conflict in Syria and a rapid return of refugees. Attention and resources were diverted toward dealing with the short-term effects of the refugee crisis (e.g., providing space in schools for the new arrivals), and away from long-term efforts (e.g., reforming systems designed to improve learning quality). This impact can be seen in the variable rate of implementation of different components: (i) whereas ERFKE II aimed to eliminate double-shift schools before moving to eliminate rented school space, the need to enroll the rapid influx of refugee children in fact resulted in an increase of double-shift schools; (ii) the disruption in schooling likely had a negative impact on the quality of teaching and learning, as reflected in the scores of Jordanian students in the national tests; and (iii) the goal of ERFKE II to create safer learning environments was countered by an increase in school violence (especially in boys-only schools), most likely also the result of disruption due to the refugee crisis. 30. In the early years of project implementation, the GOJ, in response to fiscal constraints, allocated less capital funding to the project in its annual national budgets than had been required by the project implementation plan; however, the Ministry of Planning and International Cooperation was able to allocate sufficient funds to cover school construction through transfers within its own budget envelope. This contributed to a slow start in implementation (particularly in school construction—component 5). The GOJ’s budget transfers to the project were further skewed by requirements to allocate funds to specific components simply because the various bilateral funding agreements (notably Canada and the European Commission) set national funding targets. 31. Three ISRs rates ‘implementation progress’ as ‘satisfactory’, eight as ‘moderately satisfactory’ and four as ‘moderately unsatisfactory’.43 2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 32. The M&E framework is quite comprehensive, inasmuch as all components are covered.44 Information on top-level and intermediate-level indicators was collected and reported regularly and in a comprehensive way. ERFKE II saw the development and operationalization of the OpenEMIS system, as well as improved coordination between the National Center for Human Resource Development (NCHRD) and MOE by way of a memorandum of understanding that led to analyses and studies designed to inform MOE decision-making. 2.4 Safeguard and Fiduciary Compliance 33. Since over 90 percent of the IBRD funding—US$54.4 million of the US$60.0 million loan—was allocated to the improvement of physical learning environments (component 5), the World Bank’s safeguard and fiduciary compliance efforts focused on this component. GOJ responsibility for executing component 5 was shared between the MOE and the Ministry of Public Works and Housing (MOPWH). The MOE was responsible for securing land titles, clearing land, and defining requirements (for example, number of classrooms, laboratories, and guidelines that 43 Satisfactory in May 2010, September 2013, and February 2014; moderately satisfactory in January 2011, March 2012, July 2012, January 2013, July 2014, October 2015, April 2016, and October 2016; moderately unsatisfactory in July 2011, January 2012, January 2015, and July 2015. 44 For details, see Datasheet, section F. 6 emphasize improvement of learning environment), then for equipment, teacher supply, and student movement, and finally for maintenance after the first two years’ operation. The MOPWH was responsible for studies, tendering and oversight of design phase and construction phase (including new furniture), plus two years of maintenance. 34. Although the first two years saw a lack of coordination between MOE and MOPWH, coordination improved over time as the teams came together to discuss, clarify, and agree on respective roles and responsibilities in a formal way. With ERFKE I, school construction was under MOE’s full purview; with ERFKE II, responsibilities were shared between the two GOJ ministries. The World Bank was instrumental in bringing these two ministries together, which resulted in safeguards and fiduciary issues being resolved and implementation expedited. In particular, oversight improved as the result of the MOPWH employing an independent consultant to carry out quarterly environmental reporting, in response to a World Bank recommendation, and the MOPWH’s monthly site-by-site status report to the MOE. 35. MOE’s procurement shortcomings were mainly related to weak drafting of technical specifications leading to repetitive re-bidding, and to the (self-imposed) requirement to comply with both the Bank guidelines and the government public procurement. MOPWH, on its side, had weak capacity in planning, in design control and management, and in financial and contract management, as well in planning. 36. The project’s interim financial reports were consistently submitted a few days late, due to poor coordination between the MOE’s DCU and the General Budget Department. In addition, despite the generalized use of accounting software with the GOJ, the General Budget Department used Excel spreadsheets to manage project contracts; this resulted in errors such that the minister resorted to hiring a finance manager to review all project commitments and correct records. 37. The external auditor issued unqualified audit opinions on all audited financial statements of the project, and management letters did not report any significant weaknesses in the project’s internal controls with one exception. The 2012 management letter reported that MOE had paid bonuses, from the Government contribution, to employees directly and indirectly working on ERFKE II. Despite obtaining proper authorization and approvals before paying those bonuses, the MOE’s due diligence was insufficient to ensure only those working on ERFKE II were being paid. The letter was officially transmitted to the MOE and the minister recovered all inappropriate payments. 2.5 Post-completion Operation/Next Phase 38. Before ERFKE II closed, the MOE was in discussions with the World Bank on the design of a follow-on project, in the context of the GOJ’s new education strategy.45 Just as ERFKE II had been developed with a view to lessons learned from ERFKE I, so too the new operation is being built on ERFKE II experiences. Indeed, the National Human Resources Development Strategy— the GOJ’s new ten-year education strategy approved in 2016—builds directly upon the framework of ERFKE II, most notably:  The establishment of a national school-based development system resulted in a sustainable management system at the local level, with parents involved in their school’s development 45 National Committee for Human Resource Development. Education for Prosperity: Delivering Results. A National Strategy for Human Resource Development, 2016-2025. Amman: NCHRD, 2016. 7 improvement plan, countersigning school expenditures, and meeting regularly with school staff to discuss students’ results;  Increase in KG enrollments and establishment of early childhood development quality standards;  Introduction of a meritocratic teacher recruitment process, and the further development of in-service and pre-service teacher training; and  The creation of a MOE accountability unit at MOE. 3. Assessment of Outcomes 39. The October 22, 2013 restructuring, involving changes to key indicators, requires the project to be evaluated in two phases—the first pre-restructuring and the second post-restructuring. In addition, the three dimensions—relevance, PDO achievement, and efficiency—are rated separately for each of the two phases. At the time of the 2013 restructuring, just over one third of the loan amount had been disbursed; considering this distribution of the loan amount across the two phases, greater weight is given to the ratings for the second phase 3.1 Relevance of Objectives, Design, and Implementation 40. The preparatory technical work carried out by all parties—GOJ, World Bank, and other donors—was extensive and set in an established political framework. The priorities identified, and the components built around each of these priorities, corresponded to education policy widely accepted as good practice, and were pertinent to the Jordanian context. 41. The PDO—better acquisition of relevant knowledge and skills by a larger number of young people—was one commonly used in education projects and, given the fact that Jordan had its own national assessment program and participated regularly in international surveys, measurable. 42. Project design was comprehensive, and therefore, to some extent, complex. It was at the same time broad and flexible enough to, as it turned out, adapt to the unexpected arrival of large numbers of refugee students—particularly in its ability, through the school construction program (Component 5), to provide additional space for these new arrivals. Implementation arrangements, where each donor was effectively responsible for supporting MOE’s delivery of the various components and where the World Bank nevertheless had overall supervisory responsibility, were also complex. That said, the World Bank and donors quickly adjusted to the situation and, in fact, developed a sound and positive approach to joint supervision. 43. The relevance of objectives, design, and implementation of the project is rated as substantial for both phases (pre-restructuring and post-restructuring). 3.2 Achievement of Project Development Objectives 44. ERFKE II aimed to ensure that “students enrolled in all streams of pre-tertiary education in Jordan will have increased levels of skills necessary for participation in the knowledge economy.” 45. This overall objective was partially met, as measured through the two revised key indicators: 8  Increase in scores on national assessments aligned with knowledge economy skills (Revised target: an average increase of one (1) score point for mathematics, science and Arabic)  Increase in enrollment rates (Revised target: maintain the current high net enrollment rate [98.0%] for the basic cycle, and increase the net enrollment rate to 78.2% for the secondary cycle) 46. For the full set of achievements at the implementation level, see both Annex 2 and Datasheet Section F, table (d). 47. The first key indicator—on learning achievement—relies on the NAFKE measurement tool, which was developed as an instrument that more closely fitted the national education system that the international benchmarking tools, such as PISA and TIMSS, which Jordan regularly participates in. 48. As noted above, this indicator was modified at the time of the 2013 midterm review, in light of the NAFKE student assessment redesign that aimed to better measure knowledge economy skills, with the result that the NAFKE 2008 scores were effectively replaced as the baseline with the NAFKE 2011 scores. Using the revised baseline (that is, the 2011 scores) yields an average increase across all grades and subjects of the NAFKE of 3 points from 2011 to 2014, and thus the revised target (of 1 point) was surpassed.46 49. The basic enrollment rate PDO target, as revised, was substantively achieved (97.9 percent against a revised target of 98.0 percent), whilst the secondary education enrollment rate target, as revised, was not achieved (72.6 percent against a revised target of 78.2 percent).47 50. In assessing project efficacy, greater weight is given to the learning achievement indicator than to the enrollment indicator, since the former reflects students’ level of acquisition of knowledge and skills and therefore relates directly to the PDO. The enrollment rate indicator, while secondary, is important inasmuch as it represents the denominator over which learning achievement is measured. 51. The achievement of the PDO is rated as modest for the first (pre-restructuring) phase, given that only one NAFKE round of testing was carried out in this first period of project implementation; the rating is substantial for the second (post-restructuring) phase, in line with the increase in skills as measured from the revised learning achievement indicator baseline. 3.3 Efficiency 52. Rather than set up an economic analysis for the project, the PAD presented an economic literature review supporting the choice of policies that the project was to put in place. In particular, the PAD noted that “the Government's goal [for ERFKE II] [was] to expand access to, and enrollment in, secondary education and improve the quality of basic and secondary schooling, through investments in proven inputs, school-based management and teacher development, early childhood education, among other interventions, thus giving students the opportunity to realize 46 For details, see Datasheet, section F, table (a) and Annex 7. 47 Secondary gross enrollment rates fell steadily from 97.5 percent in 2008 to 89.9 percent in 2013, then rose suddenly to 97.3 percent in 2014; secondary enrollments grew constantly from 700,342 in 2008 to 749,134 in 2004, an increase of 7.0 percent (Source: data.uis.unesco.org). 9 higher returns to schooling,” 48 that “the ERFKE II program [would] contribute to increased student test scores on national assessments aligned with knowledge economy skills, increased completion and decreased dropout rates, and increases in enrollment rates,” 49 and that “there [was] also a growing consensus that the returns to simply increasing years of attainment are negligible if the schooling associated with that attainment does not enhance the cognitive skills demanded by the labor market.”50 53. The design of the project did not provide for a cost-effectiveness analysis or a cost-benefit analysis, and so these types of economic analysis have not been undertaken for this ICR. As the project stands, the PDO indicators–-student learning achievement and net enrollment rates–-refer to the national level rather than to the project level. Moreover, no control group was identified at the time the project was designed and launch; the absence of a valid counterfactual means neither effectiveness nor benefits (whether monetized or otherwise) can be measured. 54. In the absence of a cost-effectiveness or cost-benefit analysis, project efficiency has to be assessed in broad terms. 51 First and foremost, the project was fully implemented and thereby succeeded in helping the Government maintain a stable and resilient schooling system, despite the considerable stress and disruption resulting from the arrival of large numbers of refugee children. This was, in and of itself, a major achievement. 55. Second, the ERFKE II project’s school construction component provided additional learning spaces for both Jordanian nationals and refugees, with the additional result that there was likely less need for rental school space than would have been the case had the project not built additional classrooms. 56. Third, a comparison of school construction costs shows that the ERFKE II project’s unit cost were in a reasonable range, higher than normal MOE/MOPWH cost but lower than a recent USAID project’s cost. 57. Fourth, during the period of project implementation, enrollment in basic education grew by 7.0 percent, and enrollment in secondary education by 19.9 percent, from 2010 through 2014; the drop-out rate in the course of the lower secondary cycle fell steadily from 8.56 percent in 2010 to 4.47 percent in 2013, while the survival rate rose every year from 91.44 in 2010 to 95.53 in 2013.52 58. Fifth, the ERFKE II project also produced the kinds of systemic changes that are widely recognized in scientific literature as critical elements of successful mid- and long-term education reform: decentralized school management (component 1), a comprehensive and robust management information system (component 2), merit-based selection of teachers (component 3), and increased investment in quality early child education (component 4).53 These policies can be 48 PAD, p. 13. 49 PAD, p. 14. 50 PAD, p. 82. 51 For details, see Annex 3. 52 Source: UNESCO Institute of Statistics (http://data.uis.unesco.org/) 53 See relevant literature: decentralized school management (Carr-Hill et al. 2015); information systems (Hamid 2014); teacher selection (Jacob et al. 2016); early child education (Engle et al. 2011, Heckman 2016, Nores et al. 2010). 10 expected in time to favor a more equitable distribution of positive learning outcomes amongst the youth population. 59. Finally, the World Bank’s contribution through ERFKE II came not only in the form of school construction (which component was assigned to the World Bank because of its greater financial and operational ‘agility’ in comparison with other donors), but also, and equally importantly, in its leadership of the multi-donor ‘programmatic’ project. The World Bank’s ability to convene many donors was critical both to the project’s solid design and its stable implementation. 60. Project efficiency is therefore rated as substantial for both phases (pre-restructuring and post-restructuring). 3.4 Justification of Overall Outcome Rating Rating: Moderately satisfactory 61. The overall ‘moderately satisfactory’ rating reflects the mix of ratings for relevance, efficacy (PDO achievement) and efficiency, weighted across the two phases, as summarized in Table 3.54 Table 3: Calculation of overall outcome rating First Phase Second Phase Relevance Substantial Substantial Efficacy (PDO achievement) Modest Substantial Efficiency Substantial Substantial Overall Outcome Moderately unsatisfactory Satisfactory Overall Outcome Rating Value 3 5 Loan Disbursement Rate 38% 62% Overall Outcome Rating Share 1.1 3.1 (Weighted by Disbursement) Overall Outcome Rating (Project) Moderately satisfactory (Value 1.1 + 3.1 = 4.2) 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 62. The 2015 census shows some gaps between girls’ and boys’ enrollment at both basic and secondary levels.55 UNESCO Institute of Statistics data shows however, for 2014, a gender parity index that slightly favors girls over boys, for both net and gross enrollment rates.56 Table 4: School enrollment by gender and by level 54 See sections 3.1 (relevance), 3.2 (achievement of PDOs) and 3.3 (efficiency) for explanation of ratings. 55 Source: Source: Jordan Department of Statistics, Population and Housing Census 2015 (http://web.dos.gov.jo/sectors/social/social-census/) 56 Source: UNESCO Institute of Statistics (http://data.uis.unesco.org/) 11 Census 2015 Male Female Total F/M Ratio Basic 1,025,076 982,463 2,007,539 95.8% Secondary 136,907 140,866 277,773 102.9% Table 5: Gender parity index by enrollment rate type and by level Gender Parity Net Enrollment Rate Gross Enrollment Rate Index 2014 Basic 1.01 1.01 Secondary 1.05 1.06 63. A study of the gender gap in learning achievement, completed in 2014, concluded that female students scored higher on average than male students, in NAFKE, PISA and TIMSS tests, but noted nevertheless that students in Jordan score below the average in the international studies, especially in mathematics and sciences.57 (b) Institutional Change/Strengthening 64. The project’s most notable results included: (a) the establishment of a school-based development process in all schools and local education directorates; (b) the establishment, within the MOE, of the Education Quality and Accountability Unit; and (c) the operationalization of the Education Management Information System.58 4. Assessment of Risk to Development Outcome Rating: Moderate 65. Systemic changes–-decentralized school management (component 1), a comprehensive and robust management information system (component 2), merit-based selection of teachers (component 3), and increased investment in quality early child education (component 4)–-that the project laid down are widely recognized as critical elements of successful long-term education reform and, therefore, one can be fairly confident that they will bear fruit in the medium- to long- term, provided that external factors, most notably the Syrian refugee crisis, are resolved.59 66. Under school construction (component 5), the maintenance system, originally planned to be implemented by the Borrower, did not eventuate, thereby increasing the risk to the sustainability of school construction. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 57 Monitoring and Evaluation Partnership Project. Gender Gap in Student Achievement in Jordan – Study Project. Amman: NCHRD & World Education, 2014. Unpublished. 58 See Annex 2 for full set of project results. 59 See section 2.2. 12 67. As noted earlier, the World Bank’s preparatory technical work was extensive and set in an established political framework, with lessons drawn from ERFKE I. The priorities identified, and the components built around each of these priorities, corresponded to education policy widely accepted as good practice and were pertinent to the Jordanian context. The PDO was one commonly used for education projects; in addition, it was, given the fact that Jordan had its own national assessment program and participated regularly in international surveys, measurable. (b) Quality of Supervision Rating: Satisfactory 68. As noted earlier, the World Bank’s project design was comprehensive, and therefore, to some extent, complex. Implementation arrangements, where each donor was effectively responsible for supporting MOE’s delivery of the various components and where the World Bank nevertheless had overall supervisory responsibility, were also complex. That said, the World Bank and donors quickly adjusted to the situation and, in fact, developed a sound and positive approach to joint supervision. As the ‘general chief supervising agency’, the World Bank provided intellectual leadership, with a focus on political dialogue. The World Bank, moreover, provided important stability, with just one team leader in place throughout project implementation. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory 69. The ‘satisfactory’ rating reflects the comprehensiveness of project preparation, and the continuity and diligence of project supervision. These qualities are all the more impressive since the project was both inherently complicated, with stand-alone components supported by different donors, and executed in a complicated context, with changing political oversight and extraordinary pressure from the refugee crisis.60 5.2 Borrower Performance (a) Government Performance Rating: Moderately satisfactory 70. The GOJ had established a strong policy framework that served project design and implementation well, and had a strong commitment to maintaining project implementation in the face of the enormous stress placed on the school system with the arrival of large numbers of refugee children. However, frequent changes in political leadership (with eleven ministers of education during the life of the project) made policy continuity uncertain. Taken together, these factors allowed for project implementation to continue, although some policies—such as those for teachers—could not be implemented as planned. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 71. As in the World Bank team, the MOE’s DCU was led by a single official throughout the period of project preparation, which provided stability in an oft-changing political context. For the most part, component implementation teams were committed and collaborated well with the DCU, as well as with the monitoring unit. Given the frequent changes in ministers, the DCU was 60 See section 2.2 above. 13 instrumental in keeping project implementation moving along, and in maintaining a strong donor coordination. 72. The MOPWH, the implementing agency responsible for school construction, was responsive and highly committed to the project, in spite of a slow start in school construction.61 (c) Justification of Rating for Overall Borrower Performance Rating: Moderately satisfactory 73. The ‘moderately satisfactory’ rating reflects the mix of frequent changes on the political side and considerable continuity, diligence and commitment on the technical side. 6. Lessons Learned 74. On policy, ERFKE II shows, most notably in the establishment of a national school-based development system (component 1), that sustainability relies on the ‘system’ being put in place at the local level, rather than on a series of isolated interventions. In particular, when donors supported local players and built their capacity, rather than substituted for them, sustainability was more likely to occur. Moreover, community engagement was essential: having parents involved in the school development improvement plan, countersigning school expenditures, and meeting regularly with school staff to discuss students’ results, resulted in a sense of ownership that contributed to sustainability. In this regard, the bottom-up change that was built around the school community was more effective than the top-down change that was driven by the MOE. 75. ERFKE II highlights the importance of a well-designed communication strategy. For example, in the case of the development of teaching and learning (component 3), the political will that favored a curriculum promoting learning concepts over memorization, as well as an open and tolerant citizenship, was weakened by criticism against removal of some religious texts from Arabic language textbooks. A communications strategy, developed at the outset to explain the reform, would have set proposed curriculum changes in a wider context and may therefore have resulted in a fuller buy-in from society at large. 76. On project design, ERFKE II offers a warning on cost estimations and financing gaps. At closing, the project revealed that cost estimates of 2009 were underestimated for civil works by 24 percent and overestimated for goods by 40 percent. Even at restructuring of 2013, the project continued to be unbalanced in estimates and had underestimated civil works by 12 percent and overestimated goods by 30 percent. It is important early on in the design process to pay particular attention to costing inputs, to identify the risk associated with any resulting gap, and then to move quickly either to secure additional funding before project launch or to revise project dimensions to ensure that they fit within already committed funding. 77. ERFKE II shows that, where clear policy priorities exist within a national policy framework, a programmatic approach that brings donors together to cooperate on a single platform is quite appropriate. The project also highlights the advantage that the World Bank’s analytical and operational strengths bring to convening multi-donor partnerships and attracting additional financing to support Government policy reforms; this became particularly important with the influx 61 See section 2.4 above. 14 of new donor financing that was added in response to the influx of refugee children into the Jordanian school system. 78. Finally, on implementation, the establishment of a strong and resilient implementation unit, along with continuity in its leadership, within the responsible GOJ department enables project execution to continue even in the face of considerable political instability. Continuity within the membership of the World Bank team, including in the team leader position, was equally important in providing stability through turbulent times. 7. Comments on Issues Raised by Borrower 79. In its report, the Borrower noted that ERFKE II was a program supporting all aspects of the educational reform. Project design was built on the successful experience of the first phase of the reform. However, unpredictable challenges faced the program and affected the achievement of the targeted indicators and the desired improvement of education quality, namely the Syrian crises and the 2008 financial crisis. At the same time, since the qualitative aspects that ERFKE II focused on required a high level of capacities for implementation, the high turn over in senior and technical positions presented a challenge during the implementation stage. 80. Among the lessons drawn from the ERFKE II experience, the Borrower noted the following:  Teacher competency is the key factor in maintaining education quality, as it positively influences students’ performance level in reading, mathematics and science. A sustainable model of pre-service teacher training is therefore required.  Early childhood education must be expanded through innovative approaches that go beyond addition school construction (for example, greater private sector involvement). 15 Annex 1. Project Costs and Financing (a) Total Project Cost at Appraisal, with IBRD Share, by Component (in US$ Million equivalent) Project Appraisal IBRD Appraisal Project as Components Estimate Estimate Percentage of (US$ millions) (US$ millions) Program Component 1 13.3 0.0 0.0 Component 2 13.6 3.0 22.1 Component 3 49.3 0.0 0.0 Component 4 20.6 2.6 12.6 Component 5 310.8 54.4 17.5 Total Financing 407.6 60.0 14.7 (b) IBRD Project Cost by Component (in US$ Million equivalent) Actual as Appraisal Estimate Actual Amount Components Percentage of (US$ millions) (US$ millions) Estimate Component 1 0.0 0.0 -- Component 262 3.0 0.0 0.0 Component 3 0.0 0.0 -- Component 4 2.6 2.7 103.8 Component 5 54.4 57.3 105.3 Total Financing 60.0 60.0 100.0 62 The World Bank contributed to the financing of Component 2 with a US$56,219 grant from its Innovation Fund. 16 (c) Project Financing, by Source of Funds (in US$ Million equivalent) Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (US$ (US$ Appraisal millions) millions)63 Borrower Investment 57.60 55.79 96.9 International Bank for Investment 60.00 60.00 100.0 Reconstruction and Development CANADA: Canadian International Investment 17.00 18.56 109.2 Development Agency (CIDA) JAPAN: Japan International Investment 2.50 0.00 0.0 Cooperation Agency (JICA) USA: Agency for International Investment 117.00 117.40 100.3 Development (USAID) Other agencies64 Budget 153.90 31.68 20.6 support TOTAL 408.00 283.43 69.5 63 End-of-project estimates provided by MOE; conversion from original currencies based on rate as at December 31, 2016. 64 European Commission provided 30,000,000 EUR in budget support. 17 Annex 2. Outputs by Component 1. Component 1: Establishment of a National School-based Development System  A school-based development process has been established (supported by Global Affairs Canada): The School and Directorate Development Program (SDDP) was rolled out in all of the Kingdom’s 3,554 schools and 42 Field Directorates. In addition, the SDDP was the basis for establishing the Education Quality and Accountability Unit within the MOE’s structure.  An Education Quality and Accountability Unit has been established at MOE to ensure continuous improvement and quality of education services. The unit is currently staffed with 51 assessors who received training on the accountability framework. 2. Component 2: Monitoring & Evaluation and Organizational Development  The new, enhanced OpenEMIS is operational: with technical support from the UNESCO and financial support from the World Bank and EU, the OpenEMIS was rolled out in all schools in September 2016. This is considered a significant shift that brings back control and ownership to the MOE of its own data and systems for improved decision-making.  The enhanced monitoring and evaluation function and collaboration between NCHRD and MOE has contributed to producing key pieces of analytical work that have played a significant role in policy development, including: the School Rationalization Evaluation (2011); Program for International Student Assessment (PISA) National Report (2015); Trends in International Mathematics and Science Study (TIMSS); Classroom Observation (to assess teachers’ application of “Student-Centered Active Learning and Teaching”); Gender Gap in Student Achievement in Jordan Study Report (2015); and Mapping of Student Assessments in Jordan (2014). 3. Component 3: Development of Teaching and Learning  With financial support from Global Affairs Canada and USAID, a new teacher induction training program was developed and implemented, and the role of supervisors formally transformed into a more advisory and continuous professional development role.  A new selection and recruitment process of new teachers is in place: In 2015, the MOE radically changed the selection and recruitment of new teachers. New teachers are now selected on new merit-based criteria, which is a major step towards improving the quality of teachers entering the profession.  All subjects for grades 5-10 were reviewed to ensure alignment with knowledge economy skills. All textbooks from grades 4-12 were revised.  The Tawjihi examination, as well as the national test, NAFKE, were revised to include knowledge economy skills. 4. Component 4: Development of Special Focus Program Development  An Early Childhood Education (ECE) Framework was implemented to provide policy options for the expansion of KGs and improve the quality of delivered education services: The gross enrollment rate in KG2 reached 61.4 percent in 2015/16. Training was delivered to all kindergarten teachers and to parents through the Parent Involvement Program. To date, 93.9 percent of KGs meet new KG-quality standards that were developed under ERFKE II. More importantly, there is now a policy directive to move toward making KG part of compulsory education in Jordan. 18  A Policy Framework for Special Education was approved, related teacher training materials were developed, and teachers were trained. In 2015/16, 8,131 gifted students and 19,650 children with disabilities were enrolled in special education programs.  In vocational education, MOE reviewed and developed the general and specific outcomes of all subjects. In addition, the World Bank loan financed the equipment and upgrading of workshops (office machines, computers, welding and general machine, central heating, and automotive workshops). 5. Component 5: Improvement of Physical Learning Environments  The school learning environment has improved: The construction of new schools and extensions by the World Bank and USAID contributed to improving access to a safe environment to 985,553 students all over the country. The World Bank loan financed 28 new schools and 42 extensions (along with the provision of furniture and lab, science and IT equipment). a. The school network optimization plan was approved and implementation started in 2014, involving the merging of small schools and the provision of transportation to students affected. This has allowed a reduction in the number of underutilized schools by 132. 19 Annex 3. Economic and Financial Analysis 1. The economic analysis comprises three levels of assessment detailing (1) unit cost analysis for school construction, (2) the impact of refugee education and (3) financial savings from displacing school rental fees. It attempts to assess the cost efficiency and potential impact of ERFKE II in broader terms in the absence of cost-effectiveness analysis or cost-benefit analyses. Additionally, certain caveats to this analysis must be noted: (1) in the absence of bill of quantities data, the average unit cost per square meter is used to compare different projects. This might be problematic as it prevents a transparent assessment of the itemized classes of the construction project, which could explain quality differentials across buildings.; (2) a wider body of research is referred where there is a lack of evidence on certain issues in the context of Jordan; (3) the counterfactual scenario is established under certain assumptions in the absence of control groups. 2. Unit cost comparison. This section compares the average unit cost per square meter and per school across projects that are funded by the World Bank, USAID and MOE. In the absence of data on specific bill of quantities (BOQ) by project and division of work, the average cost per square meter and per school is used as the only indicators of cost comparison. The total cost of ERFKE II project is US$48.5 million for building new schools and US$13.9 for school extensions. In 2016, the project delivered 24 new schools and 42 new extensions. The average cost is also calculated for schools built by the Jordan School Construction and Rehabilitation Project (JSP) Phase II that is funded by USAID. This project started in 2006 and targeted constructing 28 new schools and rehabilitating 100 existing schools with a budget of US$50 million. The average cost cannot be calculated for MOE funded schools due to lack of data on the actual expenditure on construction. Instead, Al Zarqa’a Al Hadaeqyyeah school in Al-Zarqa is selected as an example for cost comparison. 3. The cost per square meter of ERFKE II new schools and extensions is lower than the average for JSP new schools and rehabilitated schools funded by USAID, but higher than MOE funded schools. The average cost per square meter of ERFKE II new schools is US$579, which is 22 percent lower than USAID funded JSP new schools (figure 1). The average cost per square meter of ERFKE II extensions is US$615, which is 9 percent lower than USAID funded JSP rehabilitated schools. The average cost per school of ERFKE II is almost 60 percent lower than the average for JSP new schools (figure 2). ERFKE II demonstrates cost efficiency compared to USAID funded schools. The cost per square meter of ERFKE II schools is higher than the MOE funded school. This increase in price is reasonable due to the high quality of ERFKE II schools that are built with higher quality of materials, higher grade level of contractors and are equipped with school facilities, such as, laboratories and workshops, which are not available in many MOE schools. 20 Figure 1. Average cost per square meter (m2) by type of school65 900 791 800 674 700 615 579 600 500 380 USD 400 300 200 100 0 JSP average for JSP average for ERFKEII average ERFKEII average Al Zarqa'a Al new schools rehabiliated for extensions for new schools Hadaeqyeah (USAID funding) schools (USAID (MOE funding) funding) Figure 2. Average cost per school by type of schools66  5,000 4,367   4,000 USD thousands  3,000  2,000 1,836  1,634   1,000 434  419   ‐ JSP average ERFKEII Al Zarqa'a Al JSP average ERFKEII for new average for Hadaeqyeah for average for schools new schools (MOE rehabiliated extensions (USAID funding) schools funding) (USAID funding) 4. Impact on the schooling of refugees. This section aims to analyze the impact of newly built schools and extensions that are financed by ERFKE II on enrollment of refugees in the education system. Due to data limitations and the absence of a control group in the design of the project, it is difficult to quantify the exact impact of the project on enrollment. However, the potential impact can be estimated in broader terms. The influx of Syrian refugees has resulted in a significant increase in the demand for schooling in Jordan. According to the 2015 Population and Housing Census, the Syrian crisis has resulted in the influx of around 1.1 million refugees aged 4 years and above in Jordan. The majority of refugees are located in the northern governorates (Amman, Irbid, Zarqa and Mafraq). Overall, there are 28 percent of refugees that are enrolled in schools, 49 percent ever enrolled in schools and 23 percent never enrolled in schools. 5. The ERFKE II project has eased the pressure of getting Syrian refugee students to schools, especially under the circumstance that the current school system has already been challenged by overcrowded classrooms in urban areas. In 2014, there were 180 schools with a class size of over 40 (figure 3). The large number of Syrian refugees has exacerbated the issue of overcrowded classrooms and schools in Jordan, predominantly in urban areas in northern governorates. If the refugees that are currently not in school go to the existing MOE schools, the average class size will be estimated to increase by 10 to 40 percent in Amman, Irbid, Zarqa and Mafraq. The overcrowded schools and classrooms will be likely to cause negative impact on students’ learning outcomes. 65 Source: Jordan ERFKE II – Schools Co-Financed (2016 procurement document); USAID. JSP A Transformational Change: Evaluation of the Jordan School Construction and Rehabilitation Project. Amman: USAID, 2013. Unpublished. 66 Source: idem. 21 And poor learning outcomes are associated with great cost of the society and economy, as proved by global evidence.67 Therefore, by constructing and providing physical space as a fundamental resource of learning and getting refugees into school, ERFKE II has played an important role in alleviating the pressure from the current school system in Jordan and prevented the potential loss in learning outcomes from overcrowded schools and classes. Figure 3. Number of refugees aged 4 and above and number of schools with average class size greater than 40 by governorate68 500,000 76 80 Never enrolled Number of schools with average class size Number of refugees aged 4 and above Ever enrolled 70 400,000 60 51 Currently enrolled 50 greater than 40 300,000 Number of schools with average class size greater than 40 40 200,000 30 20 13 20 100,000 6 4 2 3 3 10 1 1 0 0 0 6. The ERFKE II project has also contributed to the long-term individual private returns by educating more refugees. There are currently 520,000 refugees aged 4 and above who have never been to school. Given that the scale of refugee influx has surpassed what the government estimated, the current school system in Jordan is not able to meet the rapid growth in the demand of schooling. According to the 2013 Household Expenditures & Income Survey, the average annual income from employment for someone with no education is only US$3,676, while for someone with secondary education is US$5,949 (figure 4). This means the society will lose approximately US$1.2 billion in terms of the accumulation of private returns if the school system fails to educate these refugees, compared to getting them into secondary schools. This is only the estimation calculated based on wages - the total cost would be even higher if it the vast amount of social cost were to be taken account. 67 Source: OECD. The High Cost of Low Educational Performance. Paris: OECD, 2010. (https://www.oecd.org/pisa/44417824.pdf) 68 Source: Number of refugees – Jordan Department of Statistics, Population and Housing Census 2015; number of schools with average class size greater than 40 - Jordan PER, with 2014 data. 22 Figure 4. Average income from employment of the head of the household, 201369  25,000 21,599   20,000  15,000 11,991  USD 10,074  10,196   10,000 7,056  7,333  5,229  5,334  5,949  3,676  4,684   5,000  ‐ 7. Savings from renting schools. ERFKE II has reduced the financial cost of renting small schools, and contributed to better quality of learning environment in the school system in Jordan. Due to the population growth and the increase in the demand for schooling, the government has to resort to rent small inappropriate buildings in the absence of land for construction, especially in populated urban areas. For example, there are 94 rented schools in Mafraq with less than 100 students, compared to only 80 MOE owned small schools (figure 5). These rented small schools are usually double-shifted where the Jordanian students attend school in the morning shift, refugee students attend the afternoon shift. Most rented small schools are reported to have very limited space for physical education and extra-curricular activities, and are difficult to make adjustments or expand. The exact of financial and operating cost cannot be calculated due to lack of data on government expenditure on renting schools. However, given the large number of rented small schools, the cost is likely to be high. Although, detailed cost comparison cannot be conducted between building and renting schools in Jordan, the long-term advantage of having spacious, constructed school buildings with quality facilities for permanent use is apparent. 70 ERFKE II contributes to the overall improvement in learning environment by creating more than 27,000 student space and easing the burden of the government from renting small buildings as temporary use of schools. 69 Source: Jordan Department of Statistics, Household Expenditure and Income Survey (http://web.dos.gov.jo/sectors/social/social-census/) 70 Woolner, P. Hall, E. Higgins, S. McCaughey, C. and K. Wall. 2007. A Sound Foundation? What We Know about the Impact of Environments on Learning and the Implications for Building Schools for the Future. Oxford Review of Education 33(1): 47-70. 23 Figure 5. Number of rented schools with fewer than 100 students by governorate, 201471 100 94 90 80 70 65 60 50 39 37 40 35 30 23 20 19 17 17 20 15 10 2 0 8. This improvement in the learning environment for children could potentially lead to improved learning outcomes if enacted with the proper reforms to the education system. According to the 2016 World Bank Education Expenditure Review in Jordan, rented buildings are negatively correlated with student learning outcomes. The study chose tawjihi pass rates as the indicator of learning outcomes, and regressed it on different school characteristics. It has found that rented schools are associated with significantly lower tawjihi pass rates compared to MOE owned schools. Based on the evidence from this study and the fact that the ERFKE II constructed schools have decreased the number of rented schools, one can assume that ERFKE II will potentially contribute to better learning outcomes. 71 Source: World Bank. Hashemite Kingdom of Jordan – Education Public Expenditure Review (P148394). Washington, D.C.: World Bank, 2016. Unpublished. 24 Annex 4. Bank Lending and Implementation Supervision Processes (a) Task Team members Name Title Unit Lending ACHSIEN, Afifa Alia Senior Program Assistant MNSHD Senior Social Protection ALKHOJA, Ghassan N. GSP05 Specialist Senior Social Protection BODOR, Andras GSP06 Economist BROWN, Hyacinth D. Manager WFALS BUCKLAND, Francis Peter Consultant GED01 ENGLISH, Linda K. Consultant GEDDR FARES, Lina Senior Procurement Specialist GGO05 Senior Financial Management MASANQUE, Josephine OPSPF Specialist MWENDA, Kenneth K. Program Manager SECVA PEARSON, J. Roger Consultant GEDDR PHAM, Dung-Kim Senior Operations Officer GED07 SAYED, Haneen Ismail Program Leader MNC02 WANG, Lianqin Senior Education Specialist GED05 Supervision/ICR ACHSIEN, Afifa Alia Senior Program Assistant MNSHD Senior Social Protection ALKHOJA, Ghassan N. GSP05 Specialist BAKUR N. QUOTA, Manal Education Specialist GED05 CHAMPION, Amy Operations Analyst GHN05 DUPETY, Daniel Consultant GED05 FAN, Jiayue Consultant GED05 FARES, Lina Senior Procurement Specialist GGO05 GAVACH, Céline Senior Operations Officer GED07 HALABI, Samira Education Specialist GED05 KAZEM, Amira Senior Operations Officer GED05 Senior Financial Management MAZAHREH, Jad Raji GGO23 Specialist MIYAJIMA, Tomomi Education Specialist GED05 MORENO OLMEDILLA, Juan Lead Education Specialist GED05 Manuel PEZZANI, Karine Senior Operations Officer GED05 SAYED, Haneen Ismail Program Leader MNC02 SETLUR, Banu Senior Environmental Specialist GEN05 SHAQOUR, Ghada Abdel Consultant GSU11 Rahman TRUCANO, Michael Senior Education Specialist GEDGE WAITE, Jeffrey Adviser GEDDR 25 (b) Fixed and Variable Costs Costs (Bank Budget Only) Stage of Project Fixed Costs Variable Costs Total Costs Cycle (US$’000) % (US$’000) % (US$’000) Lending 2008 55.75 45 67.35 55 123.10 2009 218.23 46 253.30 54 471.53 Sub-Total Lending 594.63 Supervision/ICR 2010 54.44 80 13.89 20 71.82 2011 89.37 76 28.35 24 117.72 2012 66.68 61 43.07 39 109.75 2013 93.44 58 67.47 42 160.91 2014 82.05 59 56.81 41 138.87 2015 95.05 74 32.72 26 127.77 2016 117.71 71 48.81 29 166.52 2017 124.45 72 44.99 28 169.45 Sub-Total 1,059.30 Supervision/ICR TOTAL 1,653.93 26 Annex 5. Borrower's Implementation Completion Report Introduction The Education Reform for Knowledge Economy Program (ERFKE) is a multi-donor sector program designed to deliver the “2002 Vision Forum for the Future of Education in Jordan.” This Vision states that “the Hashemite Kingdom of Jordan has the quality competitive human resource development systems that provide all people with lifelong learning experiences relevant to their current and future needs in order to respond to and stimulate sustained economic development through an educated population and an educated workforce.” The second phase of the Education Reform for the Knowledge Economy Program (ERFKE II) was officially launched on March 25, 2010 to continue to build on the attainments made during the first phase of the Program. Moreover, ERFKE II follows the same implementation arrangements that have proven to be successful in ERFKE I, which ran from (2003-2009) and closed in June 2009. ERFKE II focuses on schools as the locus of change as well as on the need to enhance the capacity building at the central and field levels, and to further deepen the impact of the qualitative aspects of the first phase of the reform. The Education Sector Education in Jordan has always been an essential component for the success, health, prosperity and security of its people. As a country with few natural resources, Jordan strives to take its place as a modern country on the world map, where education takes on even more significance. The prime inspiration has been the Royal Vision of His Majesty King Abdullah II Ibn Al Hussein as it emphasizes creativity, excellence, and the activation of partnerships. This Vision sets education as the government’s top priority and the keystone of national development. The education system consists of the following three cycles: (a) pre-school education, which lasts for two years; (b) basic cycle which starts from grade 1 to grade 10; (c) secondary cycle which last for two years, and consists of the academic path and the vocational path. The Achievements of the Program’s Development Objective: The program development objective of ERFKE II is to provide students enrolled in pre-tertiary education institutions in Jordan with increased levels of skills to participate in the knowledge economy. This was monitored through the following key indicators: Increase in scores in the national assessments aligned with knowledge economy skills. The National Assessment for Knowledge Economy (NAFKE) Study and the assessment tool was developed to assess the achievement of the PDO. The Study measured the change in the students’ achievement between (2011-2014). The results of NAFKE 2014 revealed that there are significant improvements in all subjects and grades compared to the baseline study conducted in 2011 as follows: 27 NAFKE 2011, which is considered the baseline showed decline in the results compared to the previous cycle conducted in 2009 (completion status of the ERFKE I), which referred to several factors detected by NAFKE Decline Factors Study 2011. Increase in the enrollment rates. By the beginning of ERFKE II in 2009, the enrollment rates reached 97.6 percent and 76 percent for the basic cycle and the secondary cycle respectively. By the completion of the Program and based on the data of the scholastic year 2015/2016, the MOE maintained the high enrollment rate for the basic cycle while it went down to 72.6 percent for the secondary cycle. This was seriously addressed by the MOE to seek the reasons behind this substantial decrease. Therefore, the MOE conducted a study on the enrollment rate for the scholastic year 2013/2014. The findings of the study showed that this decrease is attributed to following factors: (a) early marriage among females, and joining labor market among males; (b) far distance between the school and places of residence in some cases; (c) decrease in the Syrian students' access to the secondary cycle was one of the factors that led to the decrease in the enrollment rate due to their low academic achievement in the basic cycle. ERFKE II Components and Key Achievements: Component One: Establishment of a National School-based Development System This component was supported through the School and Directorate Development Program (SDDP) 2011-2016, which was rolled out all over the country. The MOE schools and field directorates are enabled to identify their needs with the participation of parents, teachers, and community members using self assessment tools and developing their improvement plans accordingly. The substantial change in the role of supervisors from a supervisory role to an advisory, supportive and enabling role is considered one of the key achievements as it drives up the quality of education in Jordan. Another tangible achievement was the institutionalization of the accountability function at the MOE mandated by the newly established Education Quality and Accountability Unit. This Unit has a quality assurance function as part of a cycle of continuous improvement, as well as a key role in raising the educational performance towards the achievement of the goals laid out in ERFKE II. The Education Quality and Accountability Unit was established through the World Bank grant. The unit is staffed with 51 assessors who recieved training on the accountability framework, and another group of assessors will receive training in 2017. 28 Component Two: Monitoring & Evaluation and Organizational Development This Component aims basically at building the capacity of the MOE in policy and planning to respond to the reform needs. The MOE organizational structure was reviewed to further strengthening this function at the central level. Unfortunately, the new structure faced difficulties to become operational due to several reasons, among which is the high turnover of senior leadership. In order to develop the road map for the new education strategy and future reform plan, and with wide participation of national and international education experts, the MOE held the Educational Development Conference during August 1 and 2, 2015. During this Conference, various key educational issues were discussed and the recommendations and outcomes were submitted to the Royal Committee working on developing the National Strategy for Human Resource Development. A Communication Strategy 2012-2016 was developed to promote the activities of ERFKE II in general and SDDP in specific at all management levels of the MOE, educational councils, local communities, relevant donors and the media. The communication activities were conducted in close coordination with the SDDP which represents the key transformational change in the management of the system. Within the system improvement, an enhanced education management information system was developed as the centralized source of information for monitoring results, improving overall accuracy and reliability of its education statistics and indicators. The OpenEMIS was launched officially on August 29, 2016 as a data system and planning tool to inform the decision- making process pertainng to educational sector based on the revised package of indicators. The Monitoring and evaluation framework was developed at earlier stage of project implementaion with identified suggested studies to be conducted by the National Center for Human Resources Development (NCHRD) as an external body, as well as the internal monitoring and evaluation activities to be conducted by the MOE.72 The annual plan for the M&E activities was endorsed by the Steering Committee that comprises the MOE, the NCHRD and all concerned donors and organizations in order to coordinate all monitoring and evaluation activities and share the results with all projects and donors. The monitoring and evaluation reports are dissiminated and uploaded on the MOE website. On the other hand, the external monitoring and evaluation plan included in addition to ERFKE II related components studies, NAFKE and the international assessment studies TIMSS and PISA. The following are the key assessment studies implemented by the NCHRD:  School Rationalization 2011: The School Rationalization Evaluation Study aimed to determine the characteristics of the MOE schools in 2010-201, and to define the key factors contributing to over-crowdedness and underutilization. The Study's findings revealed that 36.5 percent of the MOE schools are overcrowded, while 55 percent of the schools are underutilized, which negatively affects the students’ achivements and the teaching practices. Accordingly, the MOE revised its construction policies and decided to build central schools , and to merge schools 72 The NCHRD carried out the activities with the support provided through the USAID-funded Monitoring and Evaluation Project, which aimed to build the capacity of the NCHRD to undertake the studies with improved quality. 29 with students less than 50 students, as a result, the MOE merged 26 schools in 15 field directorates exceeding the targeted indicator of the project.  Program for International Skills Assessment: This study aims to assess the performance of 15-year-old students in mathematics, science, and reading. Jordan participated in this study in 2006, 2009, 2012, and 2015. The results for PISA 2015 showed that students’ performance in the three domains of mathematics, science and reading is below the international average.  Trends in Mathematics and Science Study: This study aims at allowing educational systems to compare their students' educational achievement with the international achievement, as well as learning from the experiences of other countries in designing effective educational policies. The NCHRD announced the results of TIMSS 2015 which showed decline in the students’ performance in Math and Science compared with the results of TIMSS 2011.  Classroom Observation: This study highlighted the teachers’ application of Student- Centered Active Learning and Teaching methodologies in Jordanian schools. The baseline study was conducted in 2011, and was re-conducted in 2014 to compare the findings with those of the baseline study. The table below shows the comparison of teachers' performance in using Student- Centered Active Learning and Teaching 2011-2014. Year 2011 2014 (percent) (percent) Teachers’ Rank in Using SCALT High Performers 16.2 24.3 Middle Performers 63.9 61.1 Poor Performers 19.9 14.7  Gender Gap in Student Achievement in Jordan Study Report 2014: Through this study, the MOE aimed to obtain empirical evidence of the gender gap and its related causes in order to address the student and school-related factors that might contribute to overall education inequality in Jordan. The results confirmed that males and females are both performing below the expected average in all subjects or grades across all participating nations. However, the gender achievement gap is evident and females have been outperforming males in TIMSS, NAFKE, PISA, and national testing in all subjects and grades for several assessment cycles.  Mapping of Student Assessments in Jordan 2014: The purpose of this study was to map out all major student assessments in Jordan so as to avoid test fatigue within the system. The study revealed that Jordan ranks in the “middle” in terms of testing frequency, types of students' assessments, grades levels, subject domains, as well as testing instruments, rubrics development and administration. Although there are multiple and diverse assessments in Jordan, there has been no or little attempt to integrate data from these assessments. Furthermore, there has been no integrated analysis by linking multiple assessment results at student, class or school level to identify common problems and conduct higher-order analysis, which means that the utilization of all assessment data and results in Jordan is insufficient in general. Component Three: Development of Teaching and Learning The MOE developed the Teacher Policy Framework which embraces a number of new policy concepts, such as the Teachers’ Professional Standards, and the Teachers’ Continuous Professional 30 Development, and the recruitement and selection of newly appointed teachers on competative basis. The MOE developed and delivered initial training (Induction) programs for the newly appointed teachers before they join the teaching profession to cover the lack of pre-service training at higher education institutions. In the last two years, the MOE was able to provide the induction training for most of the newly appointed teachers exceeding the targets of ERFKE II. During the past two years, the MOE selected new teachers based on a new merit based criteria, which consists of a competitive exam and an interview, with the aim of improving the quality of teachers entering the teaching profession. Throughout the implementation period of ERFKE II, several training projects and initiatives were implemented such as the USAID-funded Education Reform Support Program that provided training-of-trainers for 475 trainers, and training on the improved teaching and learning practices that targeted 12000 educators. Furthermore, the five-year Early Grade Reading and Math Program Initiative was launched in 2015, with the aim of improving the learning outcomes for reading in Arabic and math in grades KG-3 in Jordan. Through the financial support provided by the Canadian government, Queen Rania Teacher Academy delivered professional development activities for MOE teachers and leaders in different areas such as the Expansion of the Networks Model, the Application of a Hybrid Model of Novice Teachers Training Program, Instructional Leadership, and Network Training Programs. The MOE developed the General Framework for Curriculum and Assessment, and the General and Specific Outcomes of all subjects and grades under ERFKE I. The MOE focused in the second phase on the review and improvement of the developed textbook and teachers guides. All textbooks and teachers’ guides for all subjects and grades were developed and reviewed based on the identified criteria. The Information and Communication Technology Strategy Framework of the MOE was developed and approved in October 2014 to guide all ICT activities and projects via the identified strategic goals, and to guide the education vision towards the integration of the ICT tools in the teaching and administration processes. Consequently, the MOE drafted the implementation plan for the MOE's ICT Strategy in cooperation with the Jordanian Education Initiative, and local experts. Component Four: Development of Special Focus Program Development During ERFKE, the Directorate of Early Childhood was established and the Early Childhood Policy Framework was developed to provide policy options for the expansion of KGs and improve the quality of delivered education and services. The enrollment rate for children in the kindergarten went up from 56 percent to 61.4 percent, 1395 KGs were built, training was delivered to all kindergarten teachers and the activities of the Parent Involvement Program were implemented. In spite of the fragmentation at the technical-vocational education and training sector, the MOE made tangible achievements in this area as the Board of Education decided to reduce the Industry branch specializations from 25 to 10, and transfer some specializations to the Vocational Training Center. The Board also decided to transfer Home Economics to the applied secondary education. Moreover, the MOE reviewed and developed the general and specific outcomes of all subjects of vocational education, and the general and specific outcomes of the entire vocational education specializations (industrial education, agricultural education, and hospitality and tourism education). Moreover, the following vocational workshops were upgraded and equipped through World Bank 31 support: Office machines & Computers workshops, Welding and General Machines workshops, Central Heating workshops and Automotive workshops. The Policy Framework of Special Education was developed and approved. Furthermore, the MOE developed comprehensive teachers’ training materials for classroom and learning recourses teachers’ with the aim of improving the quality of instruction provided to children with disabilities. 73 Under the Project of the Establishment of Special School for Hearing Impaired Students, supported by the Korean International Cooperation Agency, the construction works of the specialized deaf school started and around 50 teachers received in-service training on the applicable teaching strategies, diagnosis evaluation, special education curriculum, and teaching materials for students with hearing impairment in Jordan. Component Five: Improvement of Physical Learning Environments The World Bank focused on financing construction of new classrooms to ensure better learning environment. Due to the lack of funding, the scope of this component was reduced and the Government decided not to proceed with any further loans to avoid exceeding the debt ceiling. On the other hand, the movement of the construction function to the Ministry of Public Works and Housing slowed down the implementation, specifically during the first years of implementation. This delay has led to an increase in the unit cost for construction compared to the estimated one. Accordingly, a restructuring of ERFKE II was undertaken at the mid-term stage of the Program. The following construction activities were undertaken:  World Bank Loan: The construciton of 28 schools is completed and 26 schools out of 28 are currently operational, while the remaining two schools are expected to be operatational by the beginning of the schoolastic year 2017/2018. In addition, work is still undergoing in one extention, while the 41 extention are completed. A delay in the implemeting construction plan occured due to the transfer of the construction function from the MOE to be centralized at the Ministry of Public Works and Housing. This delay resulted in the extention of the World Bank Loan Agreement. This, in turn, raised the need for more coordination with the Ministry of Public Works and Housing, in addition to other obstacles met during the construction works in the sites.  European Union grant/Budget Support Funding: New six schools are constructed through the EU grant, one school is operational and the remaining schools are expected to be operational by the beginning of the schoolastic year 2017/2018.  KfW-financed Debt Swap Agreement: Three schools are included, one school is completed and the remaining two schools are under construction  USAID Grant: 28 new schools are completed, furnished and operational, 98 school extensions are rehabilitated and 2 school extensions are canceled. A new school construction program has been initiated with the USAID for the construction of 25 new schools and expansion for 100 schools and expansion of 300 KGs. 73 With TA provide by UNICEF. 32 In addition to the original scope of this componet, the MOE started the development of a new Geographic Information System to enable the MOE to implement rathionalized plans for school construction and human resources and to identify accurate needs. On the other hand, it is essential to sustain the investment made in the infrastructure projects and the expected ones as a response to the Syria crises and the accomodation of Syrian students in the MOE public schools. Therefore, it essential to develop the manitenance system to provide the remedial actions that ensure that the schools meet the identified standards. Key Challenges faced the Implementation of ERFKE II: ERFKE II is an inclusive reform Program that supported all aspects of the educational reform. The design of the Project was basically built on the success experience of the first phase of the reform. However, unpredictable challenges faced the program and affected the achievement of the targeted indicators and the desired improvement of education quality; namely the Syrian crises and the finacial crises. On the other hand, the nature of the qualitative aspects that ERFKE II focused on requires a high level of capacities for implementation, which was also faced as a challenge during the implemntaion stage as a result of high turn over in senior and technical positions. However, during the life span of ERFKE II, the MOE faced the following key challenges:  Economic Crisis: The Government decided to extend the implementation period of ERFKE II up to ten years as a result of the country’s foreign borrowing policy in response to the global financial crisis and to avoid any added pressure on the debt ceilings. This policy has led to substantial financial gaps in many areas, especially in information and communication technology and vocational education as well as the remaining needs of school construction.  Syrian Crisis: The Syrain crisis started in 2011, and so, Jordan had to receive thousands of refugees, most of them are at the school age. This crisis added an enormous burden on the educational services provided by the MOE in terms of maintaining the quality of education and delivery of services to all students all over the country. The MOE accommodated around 125,000 Syrian students in the public system, which in turn has affected the learning environment and the achievement of the related indicators such as the increased percentage of overcrowded school which was planned to be minimized.74  Senior Leadership: The change in the senior leadership positions had a negative impact on the MOE performance as eleven ministers of education were appointed during ERFKE II (2009- 2013). However, the sustanability of the Development Coordination Unit as well as the management level–-managing directors and directors–-minimized the impact of this change.  School Mapping: The current school mapping process is not efficiently addressing the issue of overcrowded and underutilized schools. There is a critical need for better determined action planning and more rationalized use of the existing infrastructure that support the MOE in terms of human and physical resources utilization. However, the Geographic Information System is expected to be operational soon, which would support the MOE in developing the necessary plans for the school construction and mapping in the preparation for the new reform plan. 74 This figure subject to further increase for the second semester as a result of the MOE policy to extend the registration to the mid of March, and to the campaign activity. 33  Utilization of information and communication technology: Inspite of all the considerable efforts that have been achieved during ERFKE I in training teachers to develop basic information and communication technology knowledge and practical skills, the utilization of information and communication technology in the learning and teaching process was below the expected levels.. The assessment of information and communication technology in education indicates low utilization rate because of connectivity issues, access to computers and relevance of e-contents, in addition to the absence of a clear information and communication technology strategy and vision for the role of information and communication technology in education.  E-TVET: The reform in the technical-vocational education and training sector is still limited, due to the lack of alignment among the concerned stakeholders and the lack of funding. The relationship with the private sector and communities did not meet the MOE’s ambitious levels represented in the provision of inclusive training for teachers and students as well as upgrading the levels of graduates to respond to the requirements of the labor market and employers satisfaction. Lessons learned:  The role of the Development Coordination Unit is represented in coordinating all ERFKE II activities among various stakeholder and donors, as well as providing efficient support to the relevant MOE directorates. This task is especially challenging in a context where organizational and ministerial changes are frequent. The Development Coordination Unit played a vital role in keeping the momentum of the project and ensuring coherence in its implementation as evidently witnesed by all donors and stakeholder. On the other hand, the level of cooperation between the World Bank and other donors’ agencies is considered smooth and highly supportive.  The competency of teachers is the key factor to maintain the quality of education, and it positively influences students’ performance level in the key fields such as reading, math and science. The absence of the universities role in pre-service training program yielded the MOE to develop and deliver the Induction Training Program, which is stilll not considered the best cost effective model, as the MOE needs to provide more focus on the in-service teachers’ training.  The MOE needs to develop new innovative approaches for the expansion of the ECE that was acheived during ERFKE II. The expansion limited to the school construction will not lead to the targeted increase in the enrollment rate due to the lack of available space in the areas with high population intensity that need KGs. Alternatively, the enhancement of the role of the private sector in this area would support the MOE to overcome this challenge.  Sustainability of the SDDP program: Maintain the achievements of the reform results is highly vital, especailly for the transformational change and decentralization of the education system. Therefore, it is essential to maintain the developed SDDP Program and secure the necessary funding, as well as enhanging the partnership with the community, and encourage the donors and the non-governmental organizations to use this model to respond to the current needs. Additionally, it is essentail to utilize the SDDP model to provide support and improvements for the schools and field directorates impacted by the Syria crises.  The Quality of Education and Teacher Policy: The MOE should start profound discussions with the faculties of educational sciences at universities for the establishment of a sustainable pre- service training program. The MOE is currently providing the Induction Program for the newly appointed teachers, though at a tight scale due to the lack of pre-service training programs, which 34 is the responsibility of the higher education institutions. Such program is expected to respond to the aspirations of the m, and to positively influence the students’ performance level in the key fields such as reading, math and science. On the other side, the MOE should conduct deep analysis for the recent results of TIMSS and PISA in order to determine the factors that lead to the low results scored in the last cycle of these studies, as well as developing a roadmap for improving the students’ performance in the upcoming cycles.  Strengthening the capacity of the MOE in policy and planning was one of the key objectives to ensure evidence based decions. The MOE adopted several evidence based policies that require further indepth analyis and communication with other stakeholders to avoide decline in enrollement and quality of education, and also to build consensus with community and stakeholders to support these policies (technical-vocational education and training policies, Tawjehi, etc).  Capacity building for the MOE staff: The capacity of the MOE staff at the three management levels needs to be enhanced through providing quality training to improve their skills in the areas of leadership, communication, policy and planning, which would enhance their perfomance and capacities. 35 Annex 6. List of Supporting Documents Ababneh, Abdalla et al. National Assessment for Knowledge Economy Skills: Decline Factors in 2011. Amman: National Center for Human Resource Development, 2013. Carr-Hill, Roy et al. The effects of school-based decision making on educational outcomes in low and middle income contexts: a systematic review. New Delhi, London, Washington, D.C.: International Initiative for Impact Evaluation, 2015. Engle, Patrice L et al. “Strategies for Reducing Inequalities and Improving Developmental Outcomes for Young Children in Low-Income and Middle-Income Countries” in Lancet 378 (9799): 1339–5, 2011. Hamid, Husein Abdul. SABER : What matters for most education management information systems - a framework paper . SABER Working Paper Series no. 7. Washington, D.C.: World Bank, 2014 Heckman, James J. “Skill Formation and the Economics of Investing in Disadvantaged Children” in Science 312 (5782): 1900–02, 2006. Jacob, Brian et al. Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools. NBER Working Paper Series. Cambridge, MA: National Bureau of Economic Research, 2016. Ministry of Education. Education Reform for Knowledge Economy, Second Phase (ERFKE II) 2010-2016, Implementation Completion Report, Borrower’s Contribution. Amman: Ministry of Education, 2017. Unpublished. Monitoring and Evaluation Partnership Project. Gender Gap in Student Achievement in Jordan – Study Project. Amman: NCHRD & World Education, 2014. Unpublished. National Committee for Human Resource Development. Education for Prosperity: Delivering Results. A National Strategy for Human Resource Development, 2016-2025. Amman: NCHRD, 2016. Nores, Milagros et al. “Benefits of Early Childhood Interventions across the World: (Under) Investing in the Very Young” in Economics of Education Review 29 (2): 271–82, 2010. OECD. The High Cost of Low Educational Performance. Paris: OECD, 2010 USAID. JSP A Transformational Change: Evaluation of the Jordan School Construction and Rehabilitation Project. Amman: USAID, 2013. Unpublished. Woolner, P. et al. 2007. A Sound Foundation? What We Know about the Impact of Environments on Learning and the Implications for Building Schools for the Future. Oxford Review of Education 33(1): 47-70 World Bank. Second Education Reform for the Knowledge Economy Project: Loan Agreement (Loan No. 7702-JO). Washington, D.C.: World Bank, 2009. Unpublished. World Bank. Education Reform for the Knowledge Economy Project: Implementation Completion and Results Report (Report No. ICR00001208). Washington, D.C.: World Bank, 2009. 36 World Bank. Second Education Reform for the Knowledge Economy Project: Project Appraisal Document (Report No. 46824-JO). Washington, D.C.: World Bank, 2009. World Bank. Second Education Reform for the Knowledge Economy Project: Restructuring Paper (Report No. RES11824). Washington, D.C.: World Bank, 2013. Unpublished. World Bank. Second Education Reform for the Knowledge Economy Project: Restructuring Paper (Report No. RES18263). Washington, D.C.: World Bank, 2015. Unpublished. World Bank. Hashemite Kingdom of Jordan – Education Public Expenditure Review (P148394). Washington, D.C.: World Bank, 2016. Unpublished. World Bank. Assessment of Results and M&E of Operations, Education Portfolio: Second Education Reform for the Knowledge Economy (P105036). Washington, D.C.: World Bank, 2016. Unpublished. World Bank. Quality and Performance Review Learning Exercise: Education Global Practice. Washington, D.C.: World Bank, 2016. Unpublished. 37 Annex 7. PDO Indicator No. 1: Calculation of Actual Value Achieved NAFKE NAFKE NAFKE Original Revised Indicator 2008 2011 2014 indicator indicator (A) (B) (C) (C – A) (C – B) Grade 5 Mathematics 29 30 32 +3 +2 Grade 5 Science 50 44 46 -4 +2 Grade 5 Arabic 60 39 38 -22 -1 Grade 9 Mathematics 39 33 37 -2 +4 Grade 9 Science 46 36 41 -5 +5 Grade 9 Arabic 55 40 43 -12 +3 Grade 11 Mathematics 29 23 31 +2 +9 Grade 11 Science 41 28 31 -10 +3 Grade 11 Arabic 61 46 49 -12 +3 Average increase/decrease -7 +3 38