Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004512 IMPLEMENTATION COMPLETION AND RESULTS REPORT (LOAN 8446-MX) ON A LOAN IN THE AMOUNT OF US$350 MILLION TO THE UNITED MEXICAN STATES FOR A SCHOOL BASED MANAGEMENT PROJECT June 24, 2019 Education Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective January 29, 2019) Currency Unit = Mexican Peso (MXN) MXN 18.99 = US$1.00 FISCAL YEAR July 1 - June 30 Regional Vice President: Axel van Trotsenburg Country Director: Pablo Saavedra Senior Global Practice Director: Jaime Saavedra Chanduvi Practice Manager: Emanuela Di Gropello Task Team Leader(s): Monica Yanez Pagans ICR Main Contributor: Abril Alicia Ibarra Castaneda ABBREVIATIONS AND ACRONYMS AEL Local Education Authority (Autoridad Educativa Local) AGE School-based Management Support (Apoyo a la Gestión Escolar) CEAI Colegio de Estudios Avanzados de Iberoamérica CONAPASE Consejo Nacional de Participación Social en la Educación CONOCER Consejo Nacional de Normalización y Certificación de Competencias Laborales CPS Country Partnership Strategy DGAG Directorate of School Management (Dirección General Adjunta de Gestión) DGDGE General Directorate of School Management and Education (Dirección General de Desarrollo de la Gestión Educativa) DGE General Directorate of Evaluation (Dirección General de la Evaluación) DGEI General Directorate of Indigeous Education (Dirección General de Educación Indígena) ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares) FM Financial Management FY Fiscal Year GDP Gross Domestic Product GNI Gross National Income GoM Government of Mexico IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IE Impact Evaluation ISR Implementation Status and Results Report IERR Internal Economic Rate of Return INEE National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación) INEA National Institure for Adults Education (Instituto Nacional para la Educación de los Adultos) IPP Indigenous People Plan ITEA Tlaxcalteca Institute for Adults Education (Instituto Tlaxcalteca para la Educación de los Adultos) M&E Monitoring and Evaluation NAFIN Nacional Financiera, S.N.C., I.B.D. NDP National Development Plan (Plan Nacional de Desarrollo) OECD Organization for Economic Co-operation and Development OIC Internal Control Body (Órgano Interno de Control) PAD Project Appraisal Document PDO Project Development Objective PEC Quality Schools Program (Programa Escuelas de Calidad) PEC-FIDE Programa Escuelas de Calidad- Fortalecimiento e Inversión Directa a las Escuelas PED Dignified School Environment Program (Programa Escuelas Dignas) PEEARE Excelence School Program to Tackle Educational Gaps (Programa Escuelas de Excelencia para Abatir el Rezago Educativo) PETC Full Time Schools Program (Programa Escuelas de Tiempo Completo) PISA Program for International Student Assessments PLANEA National Plan for Learning Assessment (Plan Nacional para la Evaluación de los Aprendizajes) PRE Education Reform Program (Programa de la Reforma Educativa) PSE Education Sector Program (Programa Sectorial de Educación) RCTs Randomized Control Trials SBM School Based Management s.d. Standard Deviation SEB Subsecretariat of Basic Education (Subsecretaría de Educación Básica) SEMS Subsecretariat of Upper Secondary Education (Subsecretaría de Educación Media Superior) SEP Secretariat of Public Education (Secretaría de Educación Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) SIGED Education Management Information System (Sistema de Información y Gestión Educativa) SISAT Early Warning System (Sistema de Alerta Temprana) SPD Professional Teacher Service (Servicio Profesional Docente) WMS World Management Survey TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................10 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................11 C. EFFICIENCY ...........................................................................................................................19 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................19 E. OTHER OUTCOMES AND IMPACTS .........................................................................................19 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 20 A. KEY FACTORS DURING PREPARATION ...................................................................................20 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................21 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 23 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................23 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................25 C. BANK PERFORMANCE ...........................................................................................................26 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................27 V. LESSONS AND RECOMMENDATIONS ............................................................................. 28 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 29 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 47 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 49 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 50 ANNEX 5. BORROWER COMMENTS ...................................................................................... 55 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 57 ANNEX 7. THEORY OF CHANGE ............................................................................................ 59 ANNEX 8. CHANGES TO KEY INDICATORS ............................................................................. 61 The World Bank Mexico School Based Management Project (P147185) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P147185 Mexico School Based Management Project Country Financing Instrument Mexico Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency United Mexican States Secretaria de Educacion Publica Project Development Objective (PDO) Original PDO The objective of the Project is to improve schools' managerial capacity and parental participation to reduce dropout, repetition and failure rates among PEC Schools and PETC Schools. Revised PDO The objective of the Project is to improve schools' managerial capacity and parental participation to reduce dropout, repetitionand failure rates among Participating Schools. Page 1 of 62 The World Bank Mexico School Based Management Project (P147185) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 350,000,000 350,000,000 322,064,712 IBRD-84460 Total 350,000,000 350,000,000 322,064,712 Non-World Bank Financing 0 0 0 Borrower/Recipient 469,950,000 469,950,000 469,950,000 Total 469,950,000 469,950,000 469,950,000 Total Project Cost 819,950,000 819,950,000 792,014,712 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 24-Oct-2014 09-Jul-2015 10-Oct-2017 31-Dec-2018 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 26-Jul-2017 122.56 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Legal Covenants Change in Financial Management Other Change(s) 15-Aug-2018 270.48 Reallocation between Disbursement Categories 29-Nov-2018 303.77 Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial Page 2 of 62 The World Bank Mexico School Based Management Project (P147185) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 12-Dec-2014 Satisfactory Satisfactory 0 02 22-Jun-2015 Satisfactory Moderately Satisfactory 0 03 15-Dec-2015 Satisfactory Moderately Satisfactory 70.00 04 21-Jun-2016 Satisfactory Satisfactory 122.56 05 23-Dec-2016 Satisfactory Satisfactory 122.56 06 30-Jun-2017 Moderately Satisfactory Moderately Satisfactory 122.56 07 08-Dec-2017 Satisfactory Satisfactory 225.36 08 26-Jun-2018 Moderately Satisfactory Satisfactory 270.48 09 17-Dec-2018 Satisfactory Satisfactory 303.77 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Early Childhood Education 33 Public Administration - Education 1 Primary Education 33 Secondary Education 33 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Social Development and Protection 0 Social Inclusion 49 Participation and Civic Engagement 49 Page 3 of 62 The World Bank Mexico School Based Management Project (P147185) Human Development and Gender 0 Education 52 Access to Education 26 Education Financing 26 ADM STAFF Role At Approval At ICR Regional Vice President: Jorge Familiar Calderon Axel van Trotsenburg Country Director: Gerardo M. Corrochano Pablo Saavedra Senior Global Practice Director: Claudia Maria Costin Jaime Saavedra Chanduvi Practice Manager: Reema Nayar Emanuela Di Gropello Task Team Leader(s): Rafael E. De Hoyos Navarro Monica Yanez Pagans ICR Contributing Author: Abril Alicia Ibarra Castaneda Page 4 of 62 The World Bank Mexico School Based Management Project (P147185) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of Project Appraisal, Mexico's long-term economic growth was somewhat below expectations, limiting a potential rise in average living standards and more rapid progress on poverty reduction. In 2013, its Gross National Income (GNI) per capita was approximately 39 percent of the level observed in high income countries in the Organization for Economic Co-operation and Development (OECD), 1 the same proportion observed two decades earlier, signaling a lack of progress in economic convergence. Over the past three decades, annual Gross Domestic Product (GDP) growth averaged 2.4 percent and only 0.8 percent per capita. Growth decomposition exercises pointed to insufficient average productivity growth as the main cause of growth performance. To raise productivity and unleash growth, the Government launched in 2013 an ambitious structural reform agenda and enacted legislative changes in the areas of labor market regulation, education, telecommunication and competition policy, financial sector regulation, energy, and fiscal policy. There was a clear potential for the reforms to enhance potential output growth, which at Project Appraisal was estimated in the range of 2.5-3.0 percent. 2. Low learning outcomes comprised the biggest challenge faced by Mexico’s education system. Student learning, as measured by the Program for International Student Assessment 2012 (PISA) placed Mexico in the last position among OECD countries and in 53rd of the 65 participating countries. 2 In terms of education coverage, although Mexico was making significant progress in basic education, there were still students left out of school. At appraisal, net coverage was 87 percent for preschool (5 years-old), 96 percent for primary (6 to 12 years-old), and 87 percent for secondary (12 to 14 years-old). 3. The Education Reform was one of the 2013-18 administration’s priorities and aimed to accelerate growth and development by improving the quality of education while reducing access and achievement gaps between rich and poor. The 2013 Education Reform mainly consisted of: i) establishing a professional system for hiring, evaluating, training, and promoting teachers; ii) providing full autonomy to the National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación, INEE) to oversee all evaluation functions throughout the education system; (iii) establishing a federal census of education data and a national Education Management Information System; and (iv) fostering school autonomy and school-based management (SBM). 4. The Project supported the school autonomy component of the 2013 Education Reform. More autonomy at the school level sought to improve school infrastructure, provide relevant materials, and solve basic school operational problems, while promoting social participation so that students, teachers, and parents, under the school director’s leadership, could overcome their challenges. Changes to the General Education Law, derived from the Reform, stated that SBM programs should use evaluation results as feedback for continuous improvement and to develop annual school improvement plans (shared with the education authority and the school community), specifying their priorities and linking them to 1 Mexico is an upper middle-income country and a member of the OECD and the G20. 2 Mexico had slightly improved its Mathematics results in PISA over time (from 387 in 2000 to 413 in 2012), but reading results stagnated (from 422 in 2000 to 424 in 2012). Page 5 of 62 The World Bank Mexico School Based Management Project (P147185) verifiable targets. In addition, three of the seven strategies of the Programa Sectorial de Educación (PSE) 2013-18 focused on schools as the units of change and improvement. These included: (i) creating the conditions for schools to be at the center of the education system by allowing them to receive the support needed to achieve their objectives; (ii) increasing school management capacity to improve learning outcomes; and (iii) strengthening the relationship between the school and the broader community to promote social participation, transparency, and accountability. The school autonomy policy was mainly implemented through four programs: (i) the Schools of Excellence to End Educational Delay Program (Programa Escuelas de Excelencia para Abatir el Rezago Educativo, PEEARE); (ii) the Dignified Schools Environment Program (Programa Escuelas Dignas, PED); (iii) the Quality Schools Program (Programa Escuelas de Calidad, PEC); and (iv) the Full Time Schools Program (Programa Escuelas de Tiempo Completo, PETC). Under this framework, the World Bank loan was designed to support the school autonomy policy through PEC and PETC programs. 5. The financial and technical support to schools partially provided through the Bank-financed Project was the single most important means for schools to exert their autonomy, granted by the 2013 Education Reform. By supporting the school autonomy policy, the Project aimed to improve schools’ managerial capacity and parental participation to reduce dropout and improve the quality of education services in public basic education schools in Mexico. In the long run, the Project aimed to contribute to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by improving students’ future income through their staying in school longer and learning more. The Project was also an important component of services put forward in the World Bank Group’s Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 80800-MX), discussed by the Executive Directors on December 12, 2013, which is fully aligned with the goals of Mexico's National Development Plan (NDP) for 2013-18. Page 6 of 62 The World Bank Mexico School Based Management Project (P147185) Theory of Change (Results Chain) Project Development Objectives (PDOs) 6. The objective of the Project was to improve schools’ managerial capacity and parental participation to reduce dropout, repetition, and failure rates among Participating Schools. 3 Key Expected Outcomes and Outcome Indicators 7. The PDO was to be measured through the following indicators: a. Proportion of basic education schools in programs (PEC and PETC) to strengthen SBM with a director that has a “sufficient” level of managerial capacity. b. Proportion of basic education schools in PEC with parent associations participating in the design, monitoring, and adjustment of the school improvement plan. c. Dropout rate among basic education schools in programs to strengthen SBM (PEC and PETC). d. Gross failure rate among basic education schools in programs to strengthen SBM (PEC and PETC). e. Repetition rate among secondary school students in schools in programs to strengthen SBM (PEC and PETC). 3Participating Schools were clearly defined in the Project Restructuring, the Loan Agreement, and all other relevant Project documents as schools participating in the Programa de la Reforma Educativa (PRE) and in Programa de Escuelas de Tiempo Completo (PETC). Page 7 of 62 The World Bank Mexico School Based Management Project (P147185) Components 8. The Project included three components: 9. Component 1: Increasing School Autonomy and Parent Participation (Original allocation: US$342 million IBRD funding; Actual disbursed: US$321 million IBRD funding 4) Provision of direct financial support to PEC Schools to implement School Improvement Plans (Ruta de Mejora) agreed with parent associations. The school improvement plan is a strategic plan whereby the school’s technical council (comprised by the school director and teachers) identifies the school’s main challenges and goals and the means to achieve them. The direct support amounts are defined by a resource allocation formula using two criteria: exogenous conditions (i.e. enrollment, level of marginalization) and performance in quality indicators (i.e. student learning, graduation rate). Priority is given to schools serving marginalized and vulnerable populations and those with low achievement levels in the national standardized test and high dropout rates. The resources are used to support activities related to the school improvement plans, as opposed to recurrent costs of the school’s everyday functioning. 10. Component 2: Improving Schools’ Managerial Capacity (Original allocation: US$3.5 million IBRD funding; Actual disbursed: US$0 IBRD funding 5). Provision of technical assistance to PEC Schools and PETC Schools to strengthen school-based management through: a. Support for the development of a school dashboard for school supervisors and directors including performance indicators and best practices, among others. b. Support for: (a) the development and implementation of a capacity-building strategy for school directors and supervisors on: (i) the use of the school dashboard and its role in improving school management practices; (ii) the use of a classroom observation tool; (b) the improvement of schools’ managerial practices through the development and implementation of capacity-building activities for parents; and (c) the development of the information system needed to keep the school dashboards running, relevant, and up to date, including the preparation of guidelines describing an implementation and maintenance protocol for dashboards. 11. Component 3: Research and Innovation (Original allocation: US$3.6 million IBRD funding; Actual disbursed: US$0 IBRD funding). a. Provision of support for the development of a new instrument to measure Participating Schools’ managerial capacity, through the piloting, redesign (if necessary), implementation, and comparison of alternative questionnaires to measure managerial capacity. b. Provision of support to States to improve adherence to PEC’s Operational Rules through the carrying out of workshops and knowledge exchange activities. c. Provision of support for carrying out an assessment in at least two self-selected States which will estimate and quantify the effect of the intervention supported by the Project. 4 The disbursed portion of the loan allocated to Component 1 represents 39.9% of the original projection of the Component cost (US$804 million). The rest of the costs were financed by the GoM. 5 The GoM financed all costs for Component 2 (US$9.7 million) and Component 3 (US$5.4 million). Page 8 of 62 The World Bank Mexico School Based Management Project (P147185) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs and Outcome Targets 12. The PDO was formally revised in the July 26, 2017 restructuring to replace the terminology “PEC and PETC schools” with “Participating Schools.” This change had no substantive impact on the Project; therefore, there were no major implications on the original theory of change or the originally expected outcomes and targets. The revised PDO reads: “the objective of the Project is to improve schools’ managerial capacity and parental participation to reduce dropout, repetition and failure rates among Participating Schools.” The change was needed since the PEC program, one of the programs supported by the Project and stated in the PDO, was consolidated into a meta-program called the Education Reform Program (Programa de la Reforma Educativa, PRE). For consistency, reference to PEC and PETC was replaced throughout the Project by "Participating Schools." Participating Schools were clearly defined in the Loan Agreement and other relevant Project documents as schools that are part of PRE and/or PETC. Revised PDO Indicators 13. None of the PDO indicators suffered reductions in targets. While some indicators were replaced, their replacements were simply more accurate measures of the same outcomes. The PDO indicators were adjusted to reflect data availability and the change in the Project beneficiaries from "PEC and PETC Schools" to "Participating Schools” (mainly baseline conditions and progress among PRE beneficiaries). Of the 5 original PDO indicators, three were revised and two were replaced in the July 2017 restructuring. For a full explanation of the changes by indicator, please see Annex 8. The following are the revised key indicators: i. Proportion of a representative sample of Participating Schools with a director that scores 3 or above in the World Management Survey (WMS). Added to replace original PDO Indicator 1. ii. Proportion of a sample of Participating Schools that increase parental participation (score more than 1 in the School-based Management Support (Apoyo a la Gestión Escolar, AGE)). Added to replace original PDO Indicator 2. iii. Dropout rate among (primary and secondary) Participating Schools. iv. Primary and Secondary school gross failure rate among Participating Schools. v. Secondary school repetition rate among Participating schools. Revised Components 14. The three Project components were slightly revised in the July 2017 restructuring as follows: i. Component 1. Increasing School Autonomy and Parent Participation was adjusted to support the provision of Direct Support to foster school autonomy and implement school improvement plans in “Participating Schools” (PRE and PETC) instead of PEC schools only. At the time of restructuring, the Government of Mexico (GoM) requested that direct support to PETC schools also be included under Component 1. Although this implied more complex financial management of Project funds (funding two government programs instead of only one), it also helped PETC schools benefit from Component 1 and helped disburse Project funds more promptly. ii. Component 2. Improving Schools' Managerial Capacity was adjusted to support the provision of technical assistance to “Participating Schools” (instead of PEC and PETC schools) to strengthen SBM. The reference to the school dashboards was also removed because the Federal Government chose to have each state select the interventions to improve managerial skills of school directors and communities based on their specific context. Page 9 of 62 The World Bank Mexico School Based Management Project (P147185) iii. Component 3. Research and Innovation was adjusted to provide support to States to improve adherence to Operational Rules of "Participating Schools" (PRE and PETC) instead of PEC's Operational Rules. Other Changes 15. The Project's Financial Management (FM), including flow of funds, was modified to reflect PRE arrangements, which differ from those under PEC. In addition, FM included the flow of funds to support SBM in PETC schools. The restructuring also included a reallocation between disbursement categories to provide more financing for Direct Support to Participating Schools (Component 1). Finally, the Loan Agreement was amended to reflect the above changes and make eligible expenses for the PETC retroactive as of January 2016. Rationale for Changes and Their Implication on the Original Theory of Change 16. The July 2017 restructuring was needed to respond to the consolidation of national programs to strengthen SBM in Mexico. However, the description of the PDO, components and key activities mostly remained unchanged. The overall objective of the Project remained the same, since the Project continued supporting school autonomy and managerial practices of schools, but now under: (i) PRE's School Autonomy component by providing direct support to schools to implement school improvement plans; and (ii) PRE’s Managerial Practices component by developing and implementing capacity-building activities for school supervisors and directors. The PRE components supported by the Project follow the same approach as the former PEC program. Therefore, the theory of change remained unchanged. II. OUTCOME A. RELEVANCE OF PDOs 17. The PDO is aligned to the current World Bank Group’s CPS for FY14-FY19. It aligns with the CPS Pillar 2 “Increasing Social Prosperity” as updated through the Performance and Learning Review (Report No. 104752-MX) which has the expected objective of “improved access and quality in target education programs”. The CPS and the PDO are also consistent with Mexico’s National Development Plan (NDP) 2013-18, which has five main components: Peace, Inclusion, Quality of Education, Prosperity, and Global Responsibility. All the NDP components emphasize improving the quality of education and reducing access and achievement gaps between rich and poor to increase productivity and long-term growth. In addition, the Project fully supported the school autonomy aspects of the 2013 Education Reform. Finally, the Bank’s engagement with and support to Mexico, an upper-middle income country, is highly relevant since around half of its population is still poor. In addition, the evidence of best practices in key education policies in Mexico potentially has a valuable demonstration effect for other lower-income countries. Assessment of Relevance of PDOs and Rating Rating: High 18. The relevance of the PDO is rated High since the Project provided clear evidence of the alignment of the PDO to the expected outcomes of the current CPS and Mexico’s most current NDP (2013-18), as described above. Page 10 of 62 The World Bank Mexico School Based Management Project (P147185) B. ACHIEVEMENT OF PDOs (EFFICACY) 19. Even though the Project underwent a level 1 Restructuring in July 2017, this Implementation Completion and Results Report (ICR) does not conduct a split evaluation of the efficacy of the outcome indicators before and after the restructuring since the scope of activities and expected outcomes of the Project remained the same. The change in PDO merely reflects support to the same activities under different denominations in a recently consolidated government program. From the outset, the Project was designed to support the school autonomy policy as specific government programs evolve or shift. Given that PEC was consolidated into the PRE program, the Project was restructured to shift resources from PEC to PRE. In addition, the revised indicators reflected more detailed and accurate measures of the Project’s achievement without changing the Project’s scope or reducing its targets. Therefore, the achievement of each of the objectives is measured through the five PDO indicators revised at the July 2017 restructuring. 20. The Project’s expected outcomes were to improve managerial capacity and parental participation among Participating Schools to reduce dropout and improve quality of education, as measured by failure and repetition rates among Participating Schools. The ICR goes beyond the measurement of the PDO outcomes as stated in the Project Appraisal Document (PAD) by adding learning results as an expected outcome of this Project. Improved student learning (as measured by test scores) provides a broader view of the achievements of the Project in terms of its contribution to improving the quality of education. Although students’ learning results were not explicitly mentioned in the PDO, it was clear in both the PAD and the July 2017 restructuring paper, that the Project aimed to improve the quality of education, including learning results. For instance, the PAD mentions that “the two programs (initially supported by the Project) share the goal of contributing to students’ learning within a framework of more school autonomy, the use of evaluations to improve pedagogical strategies, better managerial capacities at the school level, and active social participation.” As explained in the objectives section (see paragraph 53), learning results were not included from the outset because: (i) the Bank was overly cautious of adding learning results as a key outcome indicator due, among others, to the short span of the Project to impact learning and to technical reasons (given the transition from the ENLACE national standardized test to the National Plan for Learning Assessment (Plan Nacional para la Evaluación de los Aprendizajes, PLANEA)) and the uncertainty on timing and availability of PLANEA results. Instead, failure and repetition rates were included as proxies for learning outcomes to measure the quality of education. In addition, it has become increasingly feasible and important to measure and include student learning outcomes as key outcome indicators to ensure children attain real learning in the classroom and are thus able to enter the job market as healthy, skilled, and productive adults (World Bank, Human Capital Project, 2018). Assessment of Achievement of Each Objective/Outcome Improve managerial capacity in Participating Schools. 21. This objective was measured through the key indicator “proportion of a representative sample of Participating Schools with a director that scores 3 or above in the World Management Survey (WMS).” 6 6The WMS is a well-known instrument developed by Bloom and others at the London School of Economics that has been widely used by the World Bank in several countries to measure managerial capacity in different areas, including education. The WMS has a scale from 1 to 5 to measure managerial capacity (1 being the lowest and 5 being the highest score). In the case of Mexico and other countries with low managerial capacity, there are two more intermediate scores, 1.5 and 2.5, to allow for a greater dispersion of the results. For more information, please visit: www.developingmanagement.org Page 11 of 62 The World Bank Mexico School Based Management Project (P147185) The share of Participating Schools with directors scoring 3 or more in the WMS increased from 21 percent in school year 2015-16 to 27 percent in 2017-18, exceeding the 25 percent target set at restructuring. School directors scoring 3 or more can use existing resources in an effective way to provide better education services. School directors’ managerial capacity was measured in a sample of 1,665 (PEC/PRE) schools. 22. Additional evidence from the results from 7 randomized control trials (RCTs) conducted in 7 of the 32 states in Mexico show that after the three years of implementation of the interventions supported by this Project, the school management index improved by 0.086 standard deviations (s.d.) per year 7 in Participating Schools where Project activities were closely monitored and implemented. 23. The Project (under Component 1) contributed to improving the managerial capacity of Participating Schools by providing direct support (based on national criteria) to implement the school improvement plans agreed with parents. The provision of direct financial support to schools: (i) worked as an incentive for school directors and local actors (including teachers and parents) to establish a common dialogue on how to address school needs to improve student learning (through the school improvement plans); (ii) allowed schools to exert their autonomy; and (iii) helped increase local transparency and accountability, since the school directors needed to report on the use of funds received to both the school community and education authorities. 24. The development and implementation of the capacity-building strategy for school directors and supervisors (Component 2), helped improve school directors’ managerial capacity (De Hoyos, Silveyra and Yanez 2017). All the supervisors and over 96 percent of school directors in Participating Schools received capacity-building activities in SBM. The capacity-building strategy mainly consisted of two sets of interventions. The first set designed and provided cascade training in school management to school supervisors who, in turn, trained school directors. This training strategy was conducted by the Dirección General de Desarrollo de la Gestión Educativa (DGDGE) in all 32 States. Examples of the cascade training in management and tools to improve learning provided include the following: a. Comprehensive school supervision course focused on how to help schools improve their student learning outcomes. The course, titled “Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos,” benefited 18,447 school supervisors. This included all school supervisors in basic education in the country and those that were on a clear track to become school supervisors. b. Training on the use of classroom observation tools to improve pedagogical practices and collection of performance indicators to assess basic reading, writing, and math skills. Overall, 7,694 supervisors were trained on a classroom observation tool and around 8,817 supervisors undertook the basic reading, writing, and math exploration tools training. Cascade training from supervisors to school directors benefited over 70,060 primary and secondary school directors (of 72,611). c. Eight workshops for trainer of trainers to a core group of 1,000 school supervisors, including key topics on best practices in SBM. d. Each school was also offered a paid time slot (one school day every month) to prepare, implement, and monitor its school improvement plan. This gave school directors and teachers a space to analyze school-level data and use it to improve students learning. 7The school management index represents schools’ managerial capacity as measured by the 5 WMS aspects: leadership, operations, monitoring, setting of objectives, and personnel. Page 12 of 62 The World Bank Mexico School Based Management Project (P147185) 25. The second set of interventions related to capacity-building under Component 2 consisted of training for school directors tailored to their specific management needs in the 7 States participating in the impact evaluation. This training aimed to help school directors improve their managerial capacity through a shared leadership approach, collaborative planning, and teamwork to improve learning. Examples of the training provided by the States include the following: a. Tlaxcala offered the Diplomado en Liderazgo para la Autonomía de Gestión Escolar to improve school leadership and certify school directors on the competency of Liderazgo para la Autonomía de Gestión en Centros de Trabajo Educativos, validated by the Consejo Nacional de Normalización y Certificación de Competencias Laborales (CONOCER). Training was conducted by the Colegio de Estudios Avanzados de Iberoamérica (CEAI) and included hands-on and theoretical approaches under a blended modality (4 face-to-face sessions/24 hours, 13 online sessions/136 hours, and 90 practice hours). The training focused on sharing experiences on how to solve specific school needs among a diverse group of facilitators including teachers, school leaders, and education experts. b. Puebla offered a Diploma program on school leadership imparted by WorldFund. Capacity-building activities under the Diploma program included 308 face-to-face training hours, 16 online hours, and personal coaching to school directors. Contents included the importance of building a school culture, understanding the role of teachers, parents, and students, and coaching to improve learning outcomes (including social and emotional skills). 26. The Secretariat of Public Education (Secretaría de Educación Pública, SEP) also developed a school dashboard information system called the Sistema de Alerta Temprana (SisAT). School directors and supervisors received training on the use of this dashboard. SisAT helped school supervisors and directors systematically collect relevant and up to date information on basic learning skills and performance indicators to monitor school improvement plans (i.e. students’ basic reading, writing and math skills). This was combined with the collection of frequent data on student attendance and student grades. Teaching practices (through the classroom observation tool), and the provision of feedback to teachers to improve their pedagogical practices was also a common practice at the school level, although this information was not always uploaded to the SisAT. The SisAT helped school directors access data to inform the monitoring of school improvement plans; these in turn helped them strengthen their managerial capacity. The SisAT is now working in 27 of 32 states in Mexico. 27. Finally, the research and innovation activities under Component 3 helped measure, monitor and thus improve schools’ managerial capacity (and assess the impact of the Project). Before the Project, there was no formal or robust measurement of school managerial capacity. The Project provided support to select and adjust an instrument to measure school managerial capacities in the Mexican context following international best practices. The adapted WMS provided, for the first time, a reliable measure of school managerial capacities in the country and their main dimensions (operations management, monitoring, target setting, people management, school leadership, and pedagogical practices). This measure helped fuel a discussion for schools to understand their specific challenges and some of the practices that effectively-managed schools follow. 8 8As previous research shows (i.e. De Hoyos, García, Patrinos, 2017), the sole provision of information to schools may have a positive effect on learning. Page 13 of 62 The World Bank Mexico School Based Management Project (P147185) Improve parental participation 28. This objective was measured through the key indicator “proportion of a sample of Participating Schools that increase parental participation.” Parental participation increased from 83.9 percent in the 2015-16 school year to 94.66 in 2017-18, exceeding the 90 percent target set at restructuring. Parental participation was measured with the School Management Autonomy questionnaire (Cuestionario de Autonomía de Gestión Escolar, AGE). The AGE is a national survey created to measure parental engagement and other aspects of SBM in 7 states. The baseline for this indicator was collected in a representative sample of 1,091 Participating Schools using the AGE questionnaire. 29. The Project (Component 1) contributed to increased parental involvement by providing direct support to Participating Schools to implement school improvement plans agreed with parents. Parents in schools that received support participated in the development of the school improvement plan and were involved in the monitoring process to ensure transparent use of the funds. 30. The Project (Component 2) also helped design capacity-building activities on SBM for parents to increase their participation. Although not all parents in Participating Schools were trained, 78.21 percent received some kind of SBM training. For instance, the DGDGE, in coordination with the Consejo Nacional de Participación Social en la Educación (CONAPASE), designed and delivered a capacity-building strategy for parents to increase their impact on students’ learning and development. The program, named Familias Educadoras, offered information and guidance for parents to effectively support their children at school. Familias Educadoras focused on: helping increase self-esteem and healthy emotional family relationships; guidance to set clear rules at home; specific techniques to support children with reading, writing and math at home; and parents’ role in school transparency and accountability. The results of an electronic beneficiary satisfaction survey of Familias Educadoras show that, overall, parents find this strategy useful and engaging. In terms of coverage, Familias Educadoras had only reached 3,000 schools by Project closing, but is expected to reach 18,386 basic education schools by 2018-19. 31. In addition, the 7 states participating in the impact evaluation designed tailored capacity-building activities to increase parental participation on school matters. For instance: 32. In Puebla, parents in Participating Schools received a 6-hour workshop (with two monthly sessions consisting of three hours each) called Familia Presente, delivered by school directors and SEP personnel. The workshop’s objective was to improve parental knowledge and practices that are effective to help strengthen student learning. Parents improved their communication skills and built confident relationships with their children, were informed of the importance of learning together with their children and understood the importance of having high expectations for their children’s performance. In addition, parents received information on the importance of participating in school activities and asking for support from teachers and other school actors when needed. 33. In Tlaxcala, intensive training was provided to a select group of 98 parents that were responsible for training and bringing in other parents to their children’s schools. The workshop, delivered by CEAI, lasted 20 hours (4 sessions of 5 hours each) and aimed to increase joint teacher-parent responsibility to improve students’ learning outcomes, as well as their involvement in the monitoring of the school improvement plans. Tlaxcala also developed a network called Un Domingo por Nuestros Hijos e Hijas, Page 14 of 62 The World Bank Mexico School Based Management Project (P147185) where parents met regularly over the weekends to brainstorm on how to improve their children’s learning and school autonomy from a social participation perspective. In addition, in 2015, the Instituto Tlaxcalteca para la Educación de los Adultos (ITEA) and Instituto Nacional de Educación de Adultos (INEA) helped design communication campaigns to help parents in Participating Schools meet their own learning needs, such as acquiring basic literacy and numeracy skills. 34. Finally, as mentioned in the previous PDO indicator, school supervisors and directors were also trained to increase parental involvement as part of the capacity-building strategy. Reduce Dropout Rates 35. This objective was measured through the key indicator “dropout rate among (primary and secondary) Participating Schools.” In primary, the dropout rate fell from 3.18 percent in 2014-15 school year to 2.56 in 2016-17 school year, exceeding the 2.91 target for 2017-18. In secondary schools, the dropout rate decreased considerably from 5.31 percent in school year 2014-15 to 2.12 percent in school year 2017-18, greatly exceeding the 4.80 percent target for the 2017-18 school year. 36. School grant schemes, such as the one supported under the Project, have shown positive impacts on lowering dropout rates (World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006). 9 This Project, in particular, contributed to reducing dropout rates by increasing school autonomy, parental participation, and schools’ managerial capacity. The Project helped school directors, teachers, students, and parents identify and address their local priorities. The school dashboard information system (SisAT) also allowed directors and even teachers to detect students at risk of dropping out and to focus on different strategies to keep them in the system. Schools may have also absorbed some direct costs of schooling, reducing the financial parental contributions and, thus, decreasing dropout rates among children whose parents were unable to cover such direct costs. Parents (with higher opportunity costs of sending their children to school) may have also decided to keep their children in school due to their increased involvement in the school’s issues. Parents also perceived that the relevance and quality of education increased with the activities supported under the Project. 10 For instance, parents perceived that there was more autonomy to make decisions at the school level and that school actors were better prepared to identify local needs and improve student learning outcomes. Reduce Gross Failure Rates 37. This objective was measured through the key indicator “primary and secondary school gross failure rate among Participating Schools.” For primary schools, the gross failure rate was calculated as the proportion of total students that failed the school year. For secondary schools, the gross failure rate was calculated as the proportion of total students that failed at least one subject in the school year. 9 The evidence cited comes from impact evaluations that isolated the impact of SBM interventions on the studied outcomes. However, since dropout rates are due to many other factors, it is hard to associate dropout to SBM interventions without an impact evaluation. 10 Evidence in Mexico suggests that school grants may contribute to improving learning results (Santibañez, Abreu and O'Donoghue, 2014; Cabrera and Campusano, 2016). Page 15 of 62 The World Bank Mexico School Based Management Project (P147185) Participating Primary schools’ gross failure rate slightly increased from 1.10 percent to 1.44 percent in the school year 2016-17, falling behind the 1.00 percent target. The increase in the gross failure rate for primary Participating Schools, however, should be taken cautiously. The changes observed are economically small and may be reflecting statistical noise, making them equivalent to no change 11. In addition, gross failure rates are driven by many other factors outside of the Project’s scope; for instance, changes in the national failure policy for primary school children may have negatively affected the gross failure rate in Participating Schools. Impact evaluations taking into account SBM interventions have found positive impacts on failure and repetition rates in Mexico (i.e. Skoufias and Shapiro, 2006). 38. The Participating Secondary schools’ gross failure rate slightly increased from 7.79 percent in 2013- 14 to 7.84 percent in 2017-18, falling behind the 3.26 percent target. The large difference between the current rate and the expected outcome is due to a mistake in the calculation of the baseline and target values during the Project restructuring (please see paragraph 63 and 68). The correct figures, however, show that secondary schools’ gross failure rate remained practically unchanged, with an increase of 0.06 percent throughout the span of the Project. 39. Although the Project’s activities may have a small positive impact on the overall gross failure rates, the activities under the Project are expected to mitigate in the short to medium term further increments in the gross failure rate among primary and secondary Participating Schools, making it likely to achieve the set targets in the future. As explained previously, the direct support to schools and capacity-building activities financed under the Project helped schools improve their managerial capacity and parental participation, which should increase schools’ responsiveness to local challenges, such as increments in gross failure rates (see theory of change). For instance, participating schools are now able to use data and design school improvement plans to support students that are more likely to fail the school year. Similarly, capacity-building interventions (such as training of school directors and supervisors in the classroom observation tools to provide feedback to teachers and monitor their school improvement plans), coupled with the collection and use of performance indicators through the SisAT, help identify low performers and find strategies to improve learning of students at a greater risk of failing the school year, which should contribute to reduced gross failure rates. Reduce Repetition Rates 40. This objective was measured through the key indicator “Secondary school repetition rate among Participating Schools.” Secondary school repetition rate among Participating Schools slightly increased from 0.63 percent (2014-15 school year) to 0.97 percent (2017-18), falling behind the 0.60 target. This result should be interpreted with caution given that the baseline values were already small and may fluctuate rapidly due to non-Project factors. The changes observed are economically small and may be reflecting statistical noise and are, therefore, equivalent to no change. In addition, repetition rates are driven by many other factors outside of the Project interventions. 41. Although the Project’s activities may have a small impact on helping decrease the overall repetition rates for participating secondary schools, Project activities are expected to contribute to achieving this 11 The last ISR for this Project suggested that the increase in primary gross failure rates could have been affected by the substantial reduction in dropout rates. However, there is not enough evidence to substantiate this argument. In addition, the increase is less than 0.4 percent and therefore the explanation in the main text seems more reasonable. Page 16 of 62 The World Bank Mexico School Based Management Project (P147185) objective in the medium term in the same way as they are expected to help reduce gross failure rates (see paragraph 39). 42. The indicators used to measure dropout, gross failure and repetition rates do not allow the establishment of a causal relationship between Project activities and the results obtained, mainly because these learning outcomes are affected by factors outside of the Project’s scope. Additional activities in both PRE and PETC may have also contributed to the results in dropout, gross failure and repetition rates. For instance, PRE had a large infrastructure component that helped schools with greater infrastructure needs to build classrooms, which could have contributed to helping parents think their children were better-off at school. Better infrastructure may have also contributed to improved learning. In the case of PETC, an impact evaluation (Silveyra, Yáñez & Bedoya, 2018) showed that the program had a positive impact on reducing dropouts and improving student learning outcomes. The impact evaluation cannot assess the mechanisms through which the different components of PETC, including school autonomy as described above, reduced dropouts and improved students’ learning outcomes. In addition to the extended school day and extra-curricular activities (i.e. music, English, etc.), PETC schools sometimes also include a meal program. The meal program, for example, could have worked as a magnet to retain students, and improve learning outcomes, especially among the poorest. Learning Results 43. This objective was measured through the key indicator “proportion of a representative sample of Participating Schools that improve their student’s learning,” measured by the National Standardized Test (Plan Nacional para las Evaluaciones de los Aprendizajes, PLANEA). Based on the overall results of impact evaluations conducted in 7 States, learning results were better in (T1) schools where Project interventions were closely monitored and implemented directly by the federal government (DGDGE) in coordination with the 7 States, as opposed to the (T2) schools that received similar interventions as part of the federal government’s national strategy to strengthen school autonomy (these schools may or may not have received the interventions, since the implementation was not overseen). In particular, students’ scores in language skills in T1 schools increased considerably and students’ learning improved on average (See Table 1). According to PISA equivalence scales, improving mathematics scores by 0.41 s.d. is equivalent to one additional year of education. This means that students in beneficiary schools accumulated 1.5 months of extra schooling per year. Page 17 of 62 The World Bank Mexico School Based Management Project (P147185) Table 1. Learning Results 44. The impact evaluations show that managerial capacity has a causal, positive but small impact on learning in Participating Schools in Mexico. Although the effects may not be transformational, they are cost-effective (see efficiency section) and given the impact evaluation results, are worth the investment. Furthermore, Project interventions, and in particular increased school managerial capacity, are expected to have lasting and larger effects in the future. As a meta-analysis 12 of SBM interventions shows, results in students learning are seen after 5 to 8 years of implementation (Gertler, Partinos and Rubio-Codina 2007). Therefore, the Project’s long-term effects on the targeted populations are expected to increase with time. 45. The results of the impact evaluations show with certainty that the Project contributed to improved learning results in the short term (3 years) by increasing schools’ managerial capacity (parental involvement may have also contributed to student learning but was not part of the impact evaluation). Overall, school autonomy, as measured by the activities supported by this Project, directly impacted school management (and parental participation) which, in turn, improved student learning outcomes. Justification of Overall Efficacy Rating Rating: Substantial 46. The overall efficacy rating of this Project is considered Substantial since the operation has achieved improved managerial capacity and parental participation, while decreasing dropout rates and improving learning in Participating Schools. Gross failure and repetition rates did not change, but it is difficult to assess if this was due to the effects of Project activities because there are many other factors that affect these variables. This ICR considers that the Project almost fully achieved its objectives since the results of the impact evaluation (De Hoyos et al forthcoming) show that it contributed to improved student learning results, which are a more robust measure of the quality of education than failure and repetition rates. 12 Examination of data from a number of independent studies of the same subject, in order to determine overall trends. Page 18 of 62 The World Bank Mexico School Based Management Project (P147185) C. EFFICIENCY Assessment of Efficiency and Rating 47. The Project’s benefits are Substantial. The economic analysis of the first two components of the Project shows positive results in both the cost-benefit analysis and the Internal Economic Rate of Return (IERR). After three years of implementation of Project interventions, the school management index improved by 0.086 s.d. per year, and student learning assessments in Math and Spanish improved, on average, by 0.054 s.d. per year. According to PISA equivalence scales, improving mathematics scores by 0.41 s.d. is equivalent to one additional year of education. This means that students in Participating Schools accumulated 0.132 more years of education per year of exposure to the program (0.054/0.41 = 0.132), which corresponds to 0.395 years over the duration of the program (0.132 × 3 = 0.395) or 1.5 months of extra schooling per year (0.132*12). Therefore, students who benefited from the Project will earn MX$784 more per year than those that did not through the effect of the Project on learning outcomes (50,771 × 0.117 × 0.132 = 784). This corresponds to an increase of 1.5 percent in per capita annual wages. (Please see Annex 4 for a complete explanation). 48. On average, the annual cost per student was of MX$291.2. Students who benefited from the Project earn MX$784 per year more than their peers. With an interest rate of 10 percent, the net present value of the income benefits of the average Project beneficiary is 6.7 times higher than its cost. Under these assumptions, the Project’s IRR is 21.5 percent. (Please see Annex 4 for a complete explanation and the methodology used). 49. DGDGE used existing systems and resources within SEP ensuring smooth project implementation, but disbursements faced a one-year delay given the consolidation of the SBM programs in Mexico, the fluctuations in the exchange rate, and the 2017 earthquake (please see section on Key Factors During Implementation). At closing, the Project had disbursed 92 percent of the funds; however, an extension to disburse the remaining funds until June 30, 2019 was granted. By the extended deadline, it is expected that about 99.4% of the Loan will be disbursed, leaving less than one percent undisbursed. D. JUSTIFICATION OF OVERALL OUTCOME RATING Rating: Satisfactory 50. The overall outcome rating of the Project is deemed Satisfactory. The relevance rating for the Project objectives is High, the efficacy rating is Substantial, and the efficiency rating is Substantial. There were only minor shortcomings in the operation’s achievement of its objectives. The combination of these ratings yields an overall Satisfactory rating. E. OTHER OUTCOMES AND IMPACTS Gender 51. Mexico has achieved gender parity in basic education. Female beneficiaries at the end of the Project accounted for 50.04 percent, while the set target was of 48.7 percent. Half of the student beneficiaries in Participating Schools were female, thus achieving gender parity. Page 19 of 62 The World Bank Mexico School Based Management Project (P147185) Institutional Strengthening 52. Since 2000, Mexico has provided direct support to schools to improve school autonomy and parental participation and the World Bank has supported SBM efforts since 2006. As the third World Bank Project supporting SBM in Mexico, this Project contributed to consolidating and strengthening a culture of school autonomy and SBM that are now embedded in the system. In particular, this Project helped reinforce the 2013 Education Reform and supported the institutionalization of the school autonomy policy. The policy has endured despite changes in the programs through which the provision of direct support to schools is implemented. The new administration has also continued to support the federal programs to improve school autonomy despite reducing its budget. Poverty Reduction and Shared Prosperity 53. The Project contributed to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity. The Project helped strengthen schools’ managerial capacity and parental participation which, in turn, contributed to reduced dropout rates in Participating Schools. The Project also improved learning outcomes in Participating Schools. All Participating Schools are public schools, many of which are situated in poor areas (on average, 78.33 percent and 51.28 percent of PRE and PETC schools are in highly and very highly marginalized areas). Because of the Project, students in these schools will stay in school longer and will learn more. As more Project beneficiaries enter the labor market (and the additional years of schooling and learning achievement of the workforce increases), the economic benefits of the Project are also expected to increase (see Annex 4). More human capital will translate to higher personal or household incomes, less poverty and, if the increases in human capital take place among disadvantaged households (which is the case in this Project), it will translate to lower income inequality in the long run. Other Unintended Outcomes and Impacts 54. The school autonomy and SBM policies in basic education have permeated into the upper secondary education system. The Secretaría de Educación Media Superior (SEMS) launched in recent years the Fondo para fortalecer la autonomía de gestión en planteles de Educación Media Superior to provide direct financial and technical support to schools following the school autonomy approach that has been undertaken for several years in basic education. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 55. Objectives. The objectives laid out in the PDO were clear, measurable, and based on the available evidence on the impact of SBM initiatives. Based on the literature, the PDO aimed to improve schools’ managerial capacity and parental participation to decrease dropout, failure and repetition rates (World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006). Learning results were not included in the PDO from the outset because the Bank was cautious of adding learning results as key outcome indicators due to, among others, a short Project lifespan that would not allow enough time to impact learning. In addition, there was no baseline for test scores for Participating Schools. While the Project was being designed, the National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares, ENLACE) was under revision and there was no certainty on how learning results were going to be Page 20 of 62 The World Bank Mexico School Based Management Project (P147185) measured in the future, 13 so gross failure and repetition rates were chosen as proxies to measure learning. However, dropout, failure, and repetition rates (as well as learning results) are education outcomes that may be affected by factors outside of the Project activities. The only way of measuring the real impact of the Project on such outcomes is through a rigorous impact evaluation, which was also conducted under the Project. 56. Design. The Project design was simple and sought a smooth implementation of the Project. The operational logic behind Component 1 was straightforward: the funds should reach and be administered directly by schools. Component 2 was in tune with the available international evidence and designed to ensure that schools were able to use the resources from Component 1 to improve education service delivery. Component 3 was crucial to estimate and quantify the effect of the Project interventions by including the design and implementation of a rigorous impact evaluation. 57. A drawback of the Project’s design was to allocate funds to Components 2 and 3. The two previous Projects’ experiences warned about the challenges associated with the procurement of technical assistance services, mainly related to the client’s preference to follow local procedures (see procurement section for a complete explanation). All the activities under Components 2 and 3 were undertaken using Government budget instead of the Loan as originally planned in the PAD. It would have been desirable to include Components 2 and 3 as financed with counterpart funding only from the outset. 58. Risk and Mitigation Measures. Few risks that materialized may have needed more attention at appraisal. Nevertheless, none of the materialized risks mentioned below jeopardized the achievement of the PDO: (i) The consolidation of PEC into PRE was not considered a major risk at Project design, but it materialized in 2016 requiring a Level 1 Restructuring (the risk of PEC being discontinued was rated as moderate); (ii) Macroeconomic risks were rated as moderate, but oil prices went down in 2015 and 2016, severely affecting the education budget for SBM programs and thus the number of Project beneficiaries. In addition, the exchange rate increased considerably from the beginning of the Project to closing. A clearer mitigation strategy for these risks would have been desirable. B. KEY FACTORS DURING IMPLEMENTATION Coordination and engagement 59. The three Government agencies involved in the implementation of the Project: Secretaría de Hacienda y Crédito Público (SHCP), National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D, NAFIN), and SEP-DGDGE worked as a team to ensure that the Project was implemented as planned to achieve its development objectives. Disbursements slowed in 2016 due to the consolidation of programs to strengthen SBM in Mexico and the need to restructure the Project. This restructuring took almost a year to complete due to uncertainty from the Government on how and when the Project should finance the new program and if the Project should also finance direct support to PETC schools (given the increases in the exchange rate). Another challenge during Project implementation was that there were 13 In the June 2017 restructuring, the Bank team did not add learning outcomes as part of the PDO even though the new national standardized test (Plan Nacional para las Evaluaciones de los Aprendizajes, PLANEA) was already established. As previously stated, the Bank was cautious in using learning results as measured by test scores as a key outcome indicator. Measurement of learning outcomes, however, was maintained in the Project’s impact evaluation. Page 21 of 62 The World Bank Mexico School Based Management Project (P147185) cases where the direct support to schools arrived late in the school year, preventing schools from implementing their school improvement plan as planned. SEP addressed this by speeding up the process for transferring funds to schools, for example by keeping a trust fund to manage the direct support to schools’ funds and transferring resources directly from the federal government (bypassing the States). 60. Finally, the States’ uneven capacity to implement the national school director and supervisor training strategy was mitigated by SEP by having the DGDE carry out the coordination and provision of training for all school supervisors in the country. Nevertheless, the capacity-building for school directors was achieved at different levels and paces by states and districts given their own implementation capacity and limitations. For instance, in Tabasco and Durango, there is frequent rotation of personnel, affecting the implementation of Project activities. Therefore, it is important to provide each state with a suite of services to implement school autonomy and accountability initiatives, according to their particular needs and shortcomings. Factors subject to World Bank control Adequacy of Supervision 61. The Bank conducted regular supervision missions at least twice a year that included meetings with SHCP, NAFIN, and SEP, as well as field visits to Participating Schools, all of which helped identify implementation challenges in a timely manner. For instance, through the July 2017 Level 1 Restructuring, the Project was adjusted to respond to the consolidation of the SBM government programs in Mexico. 62. The Bank also sustained close collaboration with SEP in the design and implementation of the impact evaluation. For instance, the Bank provided: (i) experts’ time to ensure the robust design of the impact evaluation and the accuracy of the data collection and analysis; (ii) help to select an adequate instrument to measure schools’ managerial capacity; (iii) financial support for the piloting of the instrument, training of enumerators, and data analysis; (iv) continuous technical assistance tailored to the federal government and the 7 states participating in the impact evaluation, including the design and coordination of numerous conferences and workshops to share best practices and international experiences to inform the selection and implementation of adequate interventions to improve schools’ managerial capacity; and (iv) guidance to select/develop VALORA, an instrument to measure the degree to which the capacity-building strategy was implemented in the 7 states. Overall, the Bank accompanied the Government throughout the impact evaluation process, monitored its timeline closely, and helped ensure that any technical and administrative challenges were overcome. Furthermore, in coordination with SEP, the Bank created a webpage as a repository for the evaluation, including information on the: (i) design and preparation; (ii) lines of intervention; (iii) measurement instruments; (iv) implementation; (iv) results; and (v) good practices. The website is available under the domain: http://escuelaalcentro.com/. 63. The Bank also provided guidance and support related to M&E. For instance, after a mistake in the baseline data and target for secondary schools’ gross failure rates was reported by SEP, the Bank hired a consultant to review all data related to the Project indicators. SEP collaborated fully with the consultant, providing all relevant databases and information. After a detailed review, no further errors were detected. Page 22 of 62 The World Bank Mexico School Based Management Project (P147185) Factors outside the control of the Government or the World Bank Macroeconomic Environment 64. As mentioned in the risks section, the fall in the world price of oil in 2015 and 2016 produced fiscal austerity in Mexico that resulted in, among others, the reduction of the education budget from 2016 onwards. The fiscal austerity affected the budget of SBM programs, including PRE and PETC and, although at the beginning of the Project a significant increase in students participating in SBM programs in Mexico was expected, the number of beneficiaries actually decreased. The exchange rate also increased considerably from the beginning to the end of the Project. At Project preparation, the Peso to Dollar exchange rate was around 13:1, while it was almost 20:1 by Project closing. In the July 2017 restructuring, the Government of Mexico decided to add PETC schools under Component 1 to ensure that all Project funds could be disbursed despite the changes in the exchange rate. Natural disasters 65. The earthquake that affected Mexico in September 2017 affected approximately 15,000 schools (5,000 had major damages), requiring a significant amount of resources to be redirected towards the reconstruction of the affected schools. The reduction in the amount of available resources from the Government significantly affected the projected number of beneficiaries of both the PRE and the PETC. To redirect existing limited resources, however, the Government adopted strict targeting criteria towards more vulnerable schools, including indigenous, rural, and those located in highly marginalized and marginalized communities. This focus had a significant impact on improving equity in allocating resources to schools. Although the targeted number of beneficiaries was not met, better-targeted schools received more resources. 66. The September 2017 earthquake also damaged several SEP buildings, including the data analysis buildings. This natural disaster further slowed M&E reporting since SEP needed to recover lost data and reallocate personnel responsible for analyzing and reporting indicator data. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 67. The operation’s theory of change was clear and based on available evidence on similar SBM initiatives (see Annex 7). Each of the key Project indicators were matched to monitor the achievement of each of the Project objectives. The M&E design mainly relied on official data from the Encuesta 911 (the main survey used by the Government to update official education indicators). The M&E design also contributed to incorporating a concrete measure of schools’ managerial capacity (one of the PDO objectives) into SEP’s M&E agenda. The Project provided support to choose an instrument through the piloting, redesign, implementation, and comparison of alternative questionnaires to measure schools’ managerial capacity. The M&E plan helped SEP systematically measure school directors’ managerial capacity for the first time in Mexico. 68. The results framework had some flaws, most of which were corrected in the 2017 restructuring (see Annex 8). Nevertheless, the following remained through the Project: (i) the target for the dropout Page 23 of 62 The World Bank Mexico School Based Management Project (P147185) rate for Participating Schools was underestimated; (ii) the baseline and targets for gross failure and repetition rates were already small from the outset and did not represent significant changes (for instance, the difference between the baseline and the target for participating secondary schools’ repetition rates was only of a 0.03 percent reduction); (iii) the participating secondary schools’ gross failure rate baseline and target were inaccurate in the 2017 restructuring; 14 (iv) the number of project beneficiaries was not updated; and (v) learning results were not added as PDO indicators. M&E Implementation 69. The methodology to calculate each of the Project indicators was clearly described in the PAD and in the July 2017 restructuring paper. The methodology for most indicators followed conventional and local measurements of education indicators such as dropout, failure, and repetition rates. In the case of school managerial capacity, the methodology to collect and analyze data to calculate this indicator followed the WMS guidelines, which has been used in more than 15 different countries. The methodology to measure parental participation was adjusted to the AGE national survey guidelines in the July 2017 restructuring to make data collection for this indicator less expensive and more reliable. There were delays and a few inaccuracies in the reporting of information by SEP to the Bank, as well as high rotation of SEP personnel responsible for analyzing and updating data for Project indicators. Despite this, SEP was diligent in systematizing the information collected (i.e. keeping all relevant files together). M&E Utilization 70. For the first time, the SEP generated information related to the schools’ managerial capacity by using the WMS. The WMS results have been used to inform the school autonomy policy design at the federal, state, and school levels. Each of the 7 States that undertook the WMS received a diagnostic with specific feedback pointing to management areas for improvement, which helped them adjust and tailor capacity-building strategies for schools principals and supervisors. A similar diagnostic has been received by each of the schools participating in the WMS so that each school director can identify and address its managerial shortcomings. In addition, SEP has used the WMS results to shape the national school autonomy policy. Similarly, the information on the AGE questionnaire has been useful to adjust the strategy to increase parental participation and engagement at federal, state, and local levels. Justification of Overall Rating of Quality of M&E Rating: Substantial 71. The rating for the overall quality of M&E is considered Substantial given the moderate shortcomings in the M&E system’s design and implementation. For instance, there were delays in the collection and analysis of the data to update PDO indicators. Nevertheless, the M&E system was generally sufficient to assess the achievement of the objectives and test the links in the results chain. 14 The baseline at restructuring was calculated by SEP at 3.49 percent for PRE and PETC schools for the 2014-15 school year but in March 2018, SEP reported that the correct baseline figure for the 2014-15 school year was 7.79 percent instead. Since the PAD baseline was reported at 9.64 percent for PEC and PETC schools, it seems feasible that the 3.49 percent reported at restructuring was incorrect. Page 24 of 62 The World Bank Mexico School Based Management Project (P147185) B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 72. This was a category C Project with a low Social Risk Rating. The Project triggered OP/BP4.10 Indigenous Peoples. Therefore, an Indigenous Peoples Plan (IPP) was approved and published on the Bank and SEP webpages in August 2014. The IPP was updated at the mid-term review (2016), in consultation with the Dirección General de Eduacación Indígena (DGEI), teachers and indigenous leaders, and published in early 2017. The overall implementation of the IPP was satisfactory, especially among school communities that speak Indigenous languages (in Mexico, there are 68 indigenous languages and 364 variants). 73. The main results of the IPP were: i. Increased awareness of SEP staff at the central level, as well as supervisors, directors and teachers, that Indigenous Peoples may be bilingual and require differentiated strategies to ensure their participation in SEP Programs. ii. Promotion of the coordination of DGDGE with other departments within SEP around the topic of Indigenous schools and the pertinence of processes and instruments used for M&E and Supervision (DGDGE coordinated with the DGEI, the General Directorate of Curriculum (DGDC), and others). iii. Improved communication of guidelines, norms, plans, and other important messages by addressing communications in the language of the Indigenous communities to reach parents and students in their native language – i.e. (a) in the SisAT Program, the instructions given to students for reading and writing exercises were translated into two main Indigenous languages (Tutunacú and Náhuatl) thus enabling students to understand instructions well before performing the exercise in Spanish; (b) instruments to monitor the Indigenous PRE and PETC schools were improved. Monitoring of Indigenous schools increased from 8 percent in 2016 to 16 percent in 2017; (c) translation of administrative information (banners/posters) to local languages to improve governance and fiduciary transparency is underway. iv. Yearly published PRE guidelines now include a clause to alert education authorities (Autoridades Educativas Locales, AEL) that information on Programs, processes, and plans must be communicated to the community in the local language. v. Indigenous schools were incorporated into the pilot program of a school-based strategy. 104 of 997 schools in this program were Indigenous. 74. Financial Management (FM) and audit. Project Aide Memoires indicated a consistent and satisfactory FM performance, except for the FM supervisions that took place from November 2016 through October 2017, where performance was rated Moderately Satisfactory (MS). FM performance was downgraded during this period mainly due to a lack of disbursements while the Project was restructured to disburse funds to schools in the newly created PRE program. 75. The audit performance was uniformly strong. Annual, independent audits of Project accounts were delivered on time and produced in general clean, unqualified opinions for all years of implementation, except for the 2017 audit report where the opinion was qualified due to observations identified by the auditors (Órgano Interno de Control, OIC) from SEP, which were promptly addressed by the Project. The final Implementation Status and Results Report (ISR) rated FM performance as Satisfactory. Page 25 of 62 The World Bank Mexico School Based Management Project (P147185) 76. FM supervision was conducted in accordance with arrangements stipulated in the legal agreement. The FM risk rating remained Substantial throughout Project implementation due to the Project’s size, scope and complexity. The Bank was able to dedicate greater time and attention to FM oversight as implementation progressed, which translated into visible results. 77. Procurement. As per the Project Procurement visits to participating states, school-level procurement was conducted as agreed in the Operational Manual. SEP could not use the funds allocated to Categories 2 and 3 mainly due to budgetary and regulatory reasons related to the internal budgetary calendar and budgetary constraints that were aggravated by the state of emergency caused by the September 2017 earthquakes. 78. Project design in Mexico should consider the challenges associated with the procurement of technical assistance services with Bank financing. SEP, as other federal entities, is required to follow local budgetary procedures that are difficult to reconcile with national procurement procedures, making procurement of capacity-building activities burdensome for SEP. SEP could thus not finance training or impact evaluation expenses with Project funds as they are not considered “investments” and by Law are not subject to be financed with foreign debt. C. BANK PERFORMANCE Quality at Entry 79. The Bank provided technical assistance to the Government to prepare a Project with strategic relevance to strengthen schools’ managerial capacity and parental participation, in line with the school autonomy component of the 2013 Education Reform. The Bank’s contribution for quality at entry included sharing international best practices and available evidence on SBM initiatives to inform Project preparation. For instance, the Bank conducted research on available instruments to measure schools’ managerial capacity to help the Government select an adequate instrument for the Mexican context and measure schools’ managerial capacity for the first time. The Bank also provided information on relevant interventions, such as the use of classroom observation tools and students’ performance data to improve teaching practices. Project design considered poverty and social development aspects by targeting public marginalized schools and preparing a plan to address indigenous populations’ specific needs. In terms of FM, the Project used the supported programs’ flow of funds. The safeguards and fiduciary teams were continuously involved in relevant aspects of the Project design. On the other hand, the Bank missed the opportunity to assign counterpart funding instead of loan funds to capacity-building and research activities under the Project, even though lessons learned from the two previous Projects warned of the challenge of using loan funds for these activities. While the Project’s M&E framework had moderate shortcomings, including the underestimation of school dropouts and overestimation of gross failure and repetition rates, it was sufficient to measure the Project impacts. Quality of Supervision 80. The Bank sustained a continuous dialogue with the client, which facilitated the close monitoring of Project implementation. For instance, the Bank began a Level 1 Restructuring in 2016 as soon as the PEC program was consolidated into the PRE program, and disbursements were put on hold until the restructuring was completed. If the Bank had not started the restructuring as soon as the changes in the SBM programs occurred, disbursements would have lagged even further. Page 26 of 62 The World Bank Mexico School Based Management Project (P147185) 81. The Bank also helped overcome technical, administrative, and procurement challenges to undertake the impact evaluation of the Project. Under the impact evaluation, the Bank helped ensure that the interventions to improve schools’ managerial capacity were undertaken as designed at Project appraisal. This engagement was crucial to the successful completion of the impact evaluation, which was a major achievement of this Project since the two previous projects were unable to carry out or finalize the planned impact evaluations. 82. The safeguards and fiduciary teams continuously supervised and developed follow-up reports on the implementation of the IPP and followed up on detailed aspects of the direct support to schools’ disbursements, audits, etc. This was coupled with building staff capacity in the Directorate of School Management (Dirección General de Desarrollo de la Gestión Educativa) on both fiduciary and technical aspects. On the other hand, there were moderate shortcomings. For instance, the above-mentioned restructuring missed the opportunity to update the number of beneficiaries and further adjust the Project’s indicators (i.e. removing gross failure and repetition rates and adding learning results). The Bank, however, continues to work closely with the Government to ensure a smooth transition after Project closing. Justification of Overall Rating of Bank Performance Rating: Satisfactory 83. As described above, there were only minor shortcomings in Quality at Entry and Quality of Supervision; therefore, the rating for overall Bank Performance is deemed Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 84. Mexico has consolidated its school autonomy policy in basic education through the provision of direct support to schools and capacity building to local stakeholders since 2000. Schools have improved their managerial capacity and increased parental participation, thus reducing dropout rates and improving students’ learning results. It is also likely that gross failure and repetition rates will decrease in the future due to Project activities. The main risk to the development outcomes of the Project is that a new administration (which took office in December 2018) would stop financing direct support to schools. The focus on rigorous assessments in the Project’s design, however, helps ensure that interventions can be sustained over the long term and survive political transitions, bolstered by a solid evidence base. For instance, in September 2018, the Bank helped SEP prepare a short note for the incoming administration on the benefits of SBM programs, including information on the impact of the Project, to inform decision- making regarding future funding of these activities. The new administration continues to allocate funds to PRE and PETC. While the PRE and PETC budgets for 2019-20 school year were reduced, they are still considerable. It is uncertain if this support will continue in the future, but should the Government stop financing PRE and PETC or other new programs with similar activities, school directors are expected to continue applying what they have learned on the effective use of available resources and on the managerial skills they acquired, even without direct support from SEP. The risk to the development outcome is considered moderate since, overall, the development outcomes of the Project are expected to be maintained. Page 27 of 62 The World Bank Mexico School Based Management Project (P147185) V. LESSONS AND RECOMMENDATIONS 85. The adoption of national school autonomy policies largely depends on the provision of direct resources and, more importantly, capacity-building activities provided to school supervisors, directors and parents to effectively use local resources to improve education service delivery. The 2013 Reform gave all schools the legal autonomy to improve school infrastructure, purchase educational materials, and solve basic operational problems, while promoting social participation so students, teachers and parents, under the school director’s leadership, can overcome the challenges each school faces. Schools that could exert their autonomy were those that received direct support and capacity-building through SBM programs such as the ones supported under this Project. 86. Direct support to schools works best when accompanied with targeted training to improve schools’ managerial capacity and parental involvement. Schools’ managerial capacity in Mexico is correlated to learning results and can be improved through a sound capacity-building strategy for school directors and supervisors (de Hoyos, et al forthcoming). Therefore, Projects supporting SBM initiatives should place time and effort on developing local stakeholders’ capacity. 87. Including indicators that are driven by many factors outside of a project’s scope to measure the impact of Project activities is counterproductive. In this particular case, including failure and repetition rates as key outcome indicators proved that results can increase or decrease merely due to statistical noise or other reasons not related to the Project, making it difficult to attribute changes to Project activities. Better indicators to measure the impact and attribution of projects should be selected in the future to avoid assessing Projects based mainly on outside factors. 88. Whenever possible, projects should include learning outcomes as part of the PDO and key outcome indicators. Bank projects should be more ambitious in measuring learning outcomes, as their ultimate goal is to tackle the learning crisis effectively, as stated in the 2018 World Development Report. 89. The Bank plays a key role in providing technical expertise and support to ensure that impact evaluations of project activities are implemented and finalized, contributing to building and sharing evidence on relevant education topics such as SBM. Having a strong analytical team with ample expertise in the design and implementation of impact evaluations helped the Government overcome different challenges in its undertaking. For instance, the Bank established a clear timeline and closely monitored impact evaluation activities, helped identify an instrument to measure impact, provided training and guidance to the firms hired to collect the data, oversaw the data analysis, and wrote several notes/reports on the evaluation, among others. In collaboration with SEP, the Bank also developed a webpage to document all relevant aspects of the evaluation to share the experience and its results within the country and abroad. 90. Using M&E data to help stakeholders understand their shortcomings and act upon them is a low hanging fruit and helps improve schools’ managerial capacity. As part of the Project, school directors participating in the impact evaluation received a personalized diagnostic of their managerial capacity. This information helped them understand what areas needed improvement and act more easily upon them. . Page 28 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve schools' managerial capacity and parental participation to reduce dropout, repetition, and failure rates among Participating Schools Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of a Percentage 21.00 25.00 27.00 representative sample of Participating Schools with a 01-Aug-2014 31-Dec-2018 31-Dec-2018 director that scores 3 or above in the World Management Survey Comments (achievements against targets): Target exceeded 108%. The share of Participating Schools with directors scoring 3 or more in the WMS increased from 21 percent in school year 2015-16 to 27 percent in 2017-18, exceeding the 25 percent target set at restructuring. The World Management Survey (WMS), used to calculate this indicator, is a well-known instrument that has been widely used in several countries to measure managerial capacity in different areas, including education. The WMS has a scale from 1 to 5 to measure managerial capacity (1 being the lowest and 5 being the highest score). School directors scoring 3 or more in the WMS can use existing resources in an effective way to provide better education services. Additional evidence shows that after the three years of implementation of the interventions supported by this Project, the school management index (schools’ managerial capacity as measured by the 5 WMS aspects: leadership, operations, monitoring, setting of objectives, and personnel) improved by 0.086 standard deviations (s.d.) per year in Participating Schools where Project activities were closely monitored and implemented. Page 29 of 62 The World Bank Mexico School Based Management Project (P147185) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of a sample of Percentage 83.90 90.00 94.66 Participating Schools that increase parental 01-Aug-2014 31-Dec-2018 31-Dec-2018 participation (score more than 1 in the AGE) Comments (achievements against targets): Target exceeded (105% achieved). Parental participation increased from 83.9 percent in the 2015-16 school year to 94.66 in 2017-18, exceeding the 90 percent target set at restructuring. Parental participation was measured with the School Management Autonomy questionnaire (Cuestionario de Autonomía de Gestión Escolar, AGE). The AGE is a national survey created to measure parental engagement and other aspects of SBM in 7 out of the 32 states in Mexico. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Dropout rate among Percentage 3.18 3.48 2.91 2.56 Participating Schools (primary) 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Dropout rate among Percentage 5.31 5.04 4.80 2.12 Participating Schools (secondary) 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target exceeded (112% and 156% achieved for primary and secondary respectively). In primary, the dropout rate fell from 3.18 percent in the 2014-15 school year to 2.56 in the 2016-17 school year, exceeding the 2.91 target for 2017-18. In secondary schools, the dropout rate decreased considerably from 5.31 percent in school year 2014-15 to 2.12 percent in school year 2016-17, greatly exceeding the 4.80 percent target for the 2017-18 school year. (Actual results are presented for school year 2016-17 against targets for school year 2017-18 because actual results for school year 2017-18 are not yet available.) Page 30 of 62 The World Bank Mexico School Based Management Project (P147185) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Gross failure rate among Percentage 1.10 0.09 1.00 1.44 Participating Schools (primary) 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Gross failure rate among Percentage 3.49 8.50 3.26 7.84 Participating Schools (secondary) 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target not met. (-44% and -140% achieved for primary and secondary respectively). Participating Primary schools’ gross failure rate slightly increased from 1.10 percent to 1.44 percent in the school year 2016-17, falling behind the 1.00 percent target. The increase in the gross failure rate for primary Participating Schools, however, should be taken cautiously. The changes observed are economically small and may be reflecting statistical noise, making them equivalent to no change. In addition, gross failure rates are driven by many other factors outside of the Project’s scope; for instance, changes in the national failure policy for primary school children may have negatively affected the gross failure rate in Participating Schools. The Participating Secondary schools’ gross failure rate slightly increased from 7.79 percent in 2013-14 to 7.84 percent in 2016-17, falling well behind the 3.26 percent target. The large difference between the current rate and the expected outcome is due to a mistake in the calculation of the baseline and target values during the Project restructuring (please see paragraphs 63 and 68). The correct figures, however, show that secondary schools’ gross failure rate remained practically unchanged, with an increase of 0.06 percent throughout the span of the Project. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Repetition rate among Percentage 0.63 0.67 0.60 0.97 Participating Schools (secondary) 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Page 31 of 62 The World Bank Mexico School Based Management Project (P147185) Comments (achievements against targets): Target not met (target -62% achieved). Secondary school repetition rate among Participating Schools slightly increased from 0.63 percent (2014-15 school year) to 0.97 percent (2017-18), falling behind the 0.60 target. This result should be interpreted with caution given that the baseline values were already small and may fluctuate rapidly due to non-Project factors. The changes observed are economically small and may be reflecting statistical noise and are, therefore, equivalent to no change. In addition, repetition rates are driven by many other factors outside of the Project interventions. A.2 Intermediate Results Indicators Component: Increasing School Autonomy and Parent Participation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion General national criteria for Yes/No N Y Y the application of the formula to allocate direct 01-Aug-2014 31-Dec-2018 31-Dec-2018 transfers to schools adopted by all States Comments (achievements against targets): Target met. There were national criteria for the application of the formula to allocate direct transfers to schools throughout the duration of the Project, ensuring that resources were allocated transparently and that the incentives of different stakeholders were aligned to improve the quality of education. Although the formula ideally should have kept the two criteria (i.e. exogenous conditions and performance in quality indicators) throughout the Project, changes in the SBM Programs required that the formula’s criteria remain mainly on equity criteria such as the locality’s (or community’s) level of marginalization and the total number of students taking into account the available resources. Component: Improving Schools' Managerial Capacity Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 32 of 62 The World Bank Mexico School Based Management Project (P147185) Target Completion Design of capacity building Yes/No N Y Y activities on SBM for school supervisors 01-Aug-2014 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target met. Capacity-building activities for school supervisors were designed as originally planned. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of supervisors of Percentage 0.00 80.00 127.70 Participating Schools that receive capacity building 01-Aug-2014 31-Dec-2018 31-Dec-2018 activities in SBM Comments (achievements against targets): Target exceeded. All school supervisors and those on a clear path to become supervisors in participating schools received capacity-building activities in SBM. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Design of capacity building Yes/No N Y Y activities on SBM for school directors 01-Aug-2014 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target met. Capacity-building activities for school directors were designed as originally planned. Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 33 of 62 The World Bank Mexico School Based Management Project (P147185) Target Completion Design of capacity building Yes/No N Y Y activities on SBM for parents 01-Aug-2014 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target met. Capacity building activities for parents were designed as originally planned. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of Participating Percentage 0.00 90.00 78.21 Schools that provide capacity building activities to parents 01-Aug-2014 31-Dec-2018 31-Dec-2018 on SBM Comments (achievements against targets): Target not met. 78.21 percent of parents at participating schools received capacity-building activities on SBM, not meeting the 90 percent target. Although SEP, the States, and schools made the effort to train most parents, some were difficult to engage given other commitments. However, parental participation in participating schools did increase as measured by the AGE questionnaires (See comment on PDO indicator 2). Component: Research and Innovation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion New instrument to measure Yes/No N Y Y schools' managerial capacities designed and 01-Aug-2014 31-Dec-2018 31-Dec-2018 implemented Page 34 of 62 The World Bank Mexico School Based Management Project (P147185) Comments (achievements against targets): Target met. In order to measure schools’ managerial capacities, the WMS was selected and adapted to the Mexican context. The instrument was implemented in 7 out of the 32 states in Mexico. This instrument allows for national and international benchmarks on schools’ managerial capacity and has been widely used in other countries. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Baseline data collected, Yes/No N Y Y analyzed, and presented to education authorities (for the 01-Aug-2014 31-Dec-2018 31-Dec-2018 assessment of interventions on Participating Schools) Comments (achievements against targets): Target met. Baseline data for the impact evaluation was collected during the first semester of the school year 2015-16 and data presented in August 2016. Data was presented to national and state education authorities and each school director participating in the WMS received an individual diagnostic of their school’s managerial capacity. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion First follow-up data Yes/No N Y Y collected, analyzed, and presented to the education 01-Aug-2014 31-Dec-2018 31-Dec-2018 authorities for policy adaptation/re-design (for the assessment of interventions on Participating Schools) Page 35 of 62 The World Bank Mexico School Based Management Project (P147185) Comments (achievements against targets): Target met. First follow-up data collection for the impact evaluation began during the first semester of the 2016-17 school year. Data was presented in October 2017 to national and state education authorities and each school director that participated in the WMS received an individual diagnostic of their school’s managerial capacity after one year of implementation. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Second follow-up data Yes/No N Y Y collected, analyzed, and presented to the education 01-Aug-2014 31-Dec-2018 31-Dec-2018 authorities for policy adaptation/redesign (for the assessment of interventions on Participating Schools) Comments (achievements against targets): Target met. Second follow-up data for the impact evaluation was collected during school year 2017- 18 and data was presented in November 2018 to national and state education authorities. Each school director that participated in the WMS will receive an individual diagnostic of their school’s managerial capacity after two years of implementation. Unlinked Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 6900000.00 8900000.00 4986417.00 01-Aug-2014 31-Dec-2018 31-Dec-2018 Female beneficiaries Percentage 48.63 48.70 49.17 Page 36 of 62 The World Bank Mexico School Based Management Project (P147185) 01-Aug-2014 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target not met. There were 4,986,417.00 direct project beneficiaries (students in participating schools) by Project closing, while the target was 8,900,000 students. The education budget was severely affected by the fiscal austerity resulting from the fall in oil prices in 2015 and 2016, which significantly reduce the budget for SBM programs and thus the number of Project beneficiaries. The exchange rate also increased considerably from the beginning to the end of the Project. Therefore, although at the beginning of the Project a significant increase in students participating in SBM programs in Mexico was expected, the number of beneficiaries actually decreased. In addition, the September 2017 earthquake affected approximately 15,000 schools, requiring that a significant amount of resources be redirected to support the reconstruction of affected schools. The reduction in the amount of available resources from the Government significantly affected the projected number of beneficiaries of both the PRE and the PETC. However, in order to redirect existing limited resources, the Government adopted strict targeting criteria towards more vulnerable schools, including indigenous, rural, and those located in highly marginalized and marginalized communities. This focus had a significant impact on improving equity in allocating resources to schools. Although the targeted number of beneficiaries was not met, better-targeted schools received more resources. Female beneficiaries target met. Female beneficiaries at Project closing accounted for 50.04 percent, while the target was of 48.7 percent. Half of the student beneficiaries in participating schools were female, thus achieving gender parity. Page 37 of 62 The World Bank Mexico School Based Management Project (P147185) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve Schools’ Managerial Capacity 1. Proportion of a representative sample of Participating Schools with a director that scores 3 or Outcome Indicators above in the World Management Survey 1. Design of capacity-building activities on SBM for school supervisors. The DGDGE coordinated the design of four training courses for school supervisors. 2. Percentage of supervisors of PEC/PRE and PETC schools that receive SBM capacity-building activities. All basic education school supervisors were trained in at least one of the SBM capacity-building activities. 3. Design of capacity-building activities on SBM for school directors. The course on a classroom observation tool and the course on basic reading, writing, and math exploration tools were adpated to be delivered to school directors by school supervisors in a cascade scheme. In addition, the 7 states participating in the impact evaluation also designed tailored training for school directors in their states. 4. Percentage of school directors of PEC/PRE and PETC schools that receive capacity-building activities in SBM. Cascade training from supervisors to school directors benefited most Intermediate Results Indicators school directors in Participating Schools, exceeding the target of 80% school directors trained. 5. The WMS instrument was selected and adpated to the Mexican context to measure schools' managerial capacities (output). 6. The WMS was implemented in 7 of 32 states in Mexico (output). 7. Baseline data collected, analyzed, and presented to education authorities (for the assessment of the Project inteventions) 8. First follow-up data collected, analyzed, and presented to the education authorities for policy adaptation/re-design (for the assessment of the Project interventions) 9. Second follow-up data collected, analyzed, and presented to the education authorities for policy adaptation/redesign (for the assessment of the Project interventions) 10. School directors participating in the WMS received an individual diagnostic of their school’s managerial capacity (output). Page 38 of 62 The World Bank Mexico School Based Management Project (P147185) 11. The 7 states applying the WMS received an aggregate diagnostic at the state level for their schools’ managerial capacities (output). 1. Component 2. Courses designed for school supervisors: (i) 1 course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos; (ii) 1 course on a classroom observation tool; (iii) 1 course on basic reading, writing, and math exploration tools; and, (iv) 8 workshops for trainer of trainers, including key topics on school management best practices. 2. Component 2. Number of school supervisors trained (i) All school supervisors and those that were on a clear track to become school supervisors (18,447) received the course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos. (ii) 7,694 supervisors (74 percent of the total number of supervisors in the country) were trained on a classroom observation tool (iii) 8,817 supervisors undertook the basic reading, writing, and math exploration tools Key Outputs by Component training and 1,000 received the training of trainers workshops. 3. Component 2. Design capacity-building activities for school directors. (i) 1 course on a classroom observation tool; (ii) 1 course on basic reading, writing, and math exploration tools; and, (iii) 7 capacity-building strategies tailored for the 7 states participating in the impact evaluation 4. Component 2. Number of school directors trained (i) Over 76,000 school directors were trained in the two first courses abovementioned. (ii) Over 800 school directors were directly trained in the 7 states in the third course abovementioned. 5. Component 2. The WMS was selected, piloted and adapted to measure school managerial capacity. 6. Component 3. The impact evaluation baseline, first and second follow-up were analyzed Page 39 of 62 The World Bank Mexico School Based Management Project (P147185) and presented to education authorities including SEB, DGDE, Local Education authorities in the 7 participating states. 7. Component 3. In addition, 7 states and all school supervisors and directors participating in the impact evaluation received a tailored diagnostic with WMS results. Objective/Outcome 2: Increase Parental Participation Outcome Indicators 1. Proportion of a sample of Participating Schools that increase parental participation 1. Design of capacity-building activities on SBM for parents, coordinated by the DGDGE. 2. 78.21 percent of parents at participating schools received capacity-building activities on Intermediate Results Indicators SBM. 1. Component 2. Courses designed for parents: (i) Familias Educadoras (ii) 7 capacity-building strategies for parents in the 7 States participating in the impact Key Outputs by Component evalution. 2. Component 2. Number of parents trained. (i) Parents in 18,386 schools by the end of the school year 2018-19. Objective/Outcome 3: Reduce Dropout Rate Among Participating Schools (PRE-PETC) Outcome Indicators 1. Dropout rate among Participating Schools (PRE-PETC) 1. Increased school autonomy through direct support to schools. 2. Increased parental participation through direct support and capacity-building Intermediate Results Indicators 3. Increased school managerial capacity through direct support and capacity-building 4. Increased education quality (impact evaluation) 1. Component 1. General national criteria for the application of the formula to allocate direct transfers to schools adopted by all States. Key Outputs by Component 2. Component 1. Number of schools that received direct support 3. Component 1. Number of beneficiaries (students) Page 40 of 62 The World Bank Mexico School Based Management Project (P147185) 8. Component 2. Courses designed for school supervisors: (i) 1 course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos; (ii) 1 course on a classroom observation tool; (iii) 1 course on basic reading, writing, and math exploration tools; and, (iv) 8 workshops for trainer of trainers, including key topics on school management best practices. 9. Component 2. Number of school supervisors trained (i) All school supervisors and those that were on a clear track to become school supervisors (18,447) received the course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos. (ii) 7,694 supervisors (74 percent of the total number of supervisors in the country) were trained in a classroom observation tool (iii) 8,817 supervisors undertook the basic reading, writing, and math exploration tools training and 1,000 received the training of trainers workshops 10. Component 2. Design capacity-building activities for school directors. (i) 1 course on a classroom observation tool; (ii) 1 course on basic reading, writing, and math exploration tools; and, (iii) 7 capacity-building strategies tailored for the 7 states participating in the impact evaluation 11. Component 2. Number of school directors trained (i) Over 76,000 school directors were trained in the two first courses abovementioned. (ii) Over 800 school directors were directly trained in the 7 states in the third course abovementioned. 12. Component 2. The WMS was selected, piloted and adapted to measure school managerial capacity. 13. Component 3. The impact evaluation baseline, first and second follow-up were analyzed and presented to education authorities including SEB, DGDE, Local Education authorities in the 7 participating states. Page 41 of 62 The World Bank Mexico School Based Management Project (P147185) Component 3. In addition, 7 states and all school supervisors and directors participating in the impact evaluation received a tailored diagnostic with WMS results. Objective/Outcome 4: Reduce Gross Failure Rate Among Participating Schools (PRE-PETC) Outcome Indicators 1. Gross failure rate among Participating Schools (PRE-PETC). 1. Increased school autonomy through direct support to schools. 2. Increased parental participation through direct support and capacity-building Intermediate Results Indicators 3. Increased school managerial capacity through direct support and capacity-building 4. Increased education quality (impact evaluation) 1. Component 1. General national criteria for the application of the formula to allocate direct transfers to schools adopted by all Stated. 2. Component 1. Number of schools that received direct support 3. Component 1. Number of beneficiaries (students) 14. Component 2. Courses designed for school supervisors: (i) 1 course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos; (ii) 1 course on a classroom observation tool; (iii) 1 course on basic reading, writing, and math exploration tools; and, Key Outputs by Component (iv) 8 workshops for trainer of trainers, including key topics on school management (linked to the achievement of the best practices. Objective/Outcome 2) 15. Component 2. Number of school supervisors trained (i) All school supervisors and those that were on a clear track to become school supervisors (18,447) received the course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos. (ii) 7,694 supervisors (74 percent of the total number of supervisors in the country) were trained in a classroom observation tool (iii) 8,817 supervisors undertook the basic reading, writing, and math exploration tools training; and 1,000 received the training of trainers workshops 16. Component 2. Design capacity-building activities for school directors. (i) 1 course on a classroom observation tool; Page 42 of 62 The World Bank Mexico School Based Management Project (P147185) (ii) 1 course on basic reading, writing, and math exploration tools; and, (iii) 7 capacity-building strategies tailored for the 7 states participating in the impact evaluation 17. Component 2. Number of school directors trained (i) Over 76,000 school directors were trained in the two first courses abovementioned. (ii) Over 800 school directors were directly trained in the 7 states in the third course abovementioned. 18. Component 2. The WMS was selected, piloted and adapted to measure school managerial capacity. 19. Component 3. The impact evaluation baseline, first and second follow-up were analyzed and presented to education authorities including SEB, DGDE, Local Education authorities in the 7 participating states. Component 3. In addition, 7 states and all school supervisors and directors participating in the impact evaluation received a tailored diagnostic with WMS results. Objective/Outcome 5. Reduced repetition rate among Participating Schools (PRE-PETC) Outcome Indicators 1. Repetition rate among Participating Schools (PRE-PETC) 1. Increased school autonomy through direct support to schools. 2. Increased parental participation through direct support and capacity-building Intermediate Results Indicators 3. Increased school managerial capacity through direct support and capacity-building 4. Increased education quality (impact evaluation) 1. Component 1. General national criteria for the application of the formula to allocate direct transfers to schools adopted by all States. 2. Component 1. Number of schools that received direct support Key Outputs by Component 3. Component 1. Number of beneficiaries (students) 20. Component 2. Courses designed for school supervisors: (i) 1 course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos; Page 43 of 62 The World Bank Mexico School Based Management Project (P147185) (ii) 1 course on a classroom observation tool; (iii) 1 course on basic reading, writing, and math exploration tools; and, (iv) 8 workshops for trainer of trainers, including key topics on school management best practices. 21. Component 2. Number of school supervisors trained (i) All school supervisors and those that were on a clear track to become school supervisors (18,447) received the course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos. (ii) 7,694 supervisors (74 percent of the total number of supervisors in the country) were trained in a classroom observation tool (iii) 8,817 supervisors undertook the basic reading, writing, and math exploration tools training; and 1,000 received the training of trainers workshops 22. Component 2. Design capacity-building activities for school directors. (i) 1 course on a classroom observation tool; (ii) 1 course on basic reading, writing, and math exploration tools; and, (iii) 7 capacity-building strategies tailored for the 7 states participating in the impact evaluation 23. Component 2. Number of school directors trained (i) Over 76,000 school directors were trained in the two first courses abovementioned. (ii) Over 800 school directors were directly trained in the 7 states in the third course abovementioned. 24. Component 2. The WMS was selected, piloted and adapted to measure school managerial capacity. 25. Component 3. The impact evaluation baseline, first and second follow-up were analyzed and presented to education authorities including SEB, DGDE, Local Education authorities in the 7 participating states. Component 3. In addition, 7 states and all school supervisors and directors participating in the impact evaluation received a tailored diagnostic with WMS results. Page 44 of 62 The World Bank Mexico School Based Management Project (P147185) Objective/Outcome 6. Improved Student learning results 1. Students’ scores in language skills as measured by PLANEA Outcome Indicators 2. Average students’ scores as measured by PLANEA 1. Increased school autonomy through direct support to schools. 2. Increased parental participation through direct support and capacity building Intermediate Results Indicators 3. Increased school managerial capacity through direct support and capacity building 4. Increased education quality (impact evaluation) 1. Component 1. General national criteria for the application of the formula to allocate direct transfers to schools adopted by all States. 2. Component 1. Number of schools that received direct support 3. Component 1. Number of beneficiaries (students) 4. Component 2. Courses designed for school supervisors: (i) 1 course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos; (ii) 1 course on a classroom observation tool; (iii) 1 course on basic reading, writing, and math exploration tools; and, (iv) 8 workshops for trainer of trainers, including key topics on school management best practices. Key Outputs by Component 5. Component 2. Number of school supervisors trained (i) All school supervisors and those that were on a clear track to become school supervisors (18,447) received the course Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos. (ii) 7,694 supervisors (74 percent of the total number of supervisors in the country) were trained in a classroom observation tool (iii) 8,817 supervisors undertook the basic reading, writing, and math exploration tools training; and 1,000 received the training of trainers workshops 6. Component 2. Design capacity-building activities for school directors. (i) 1 course on a classroom observation tool; (ii) 1 course on basic reading, writing, and math exploration tools; and, (iii) 7 capacity-building strategies tailored for the 7 states participating in the impact Page 45 of 62 The World Bank Mexico School Based Management Project (P147185) evaluation 7. Component 2. Number of school directors trained (i) Over 76,000 school directors were trained in the two first courses abovementioned. (ii) Over 800 school directors were directly trained in the 7 states in the third course abovementioned. 8. Component 2. The WMS was selected, piloted and adapted to measure school managerial capacity. 9. Component 3. The impact evaluation baseline, first and second follow-up were analyzed and presented to education authorities including SEB, DGDE, Local Education authorities in the 7 participating states. Component 3. In addition, 7 states and all school supervisors and directors participating in the impact evaluation received a tailored diagnostic with WMS results. Page 46 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Raja Bentaouet Kattan Task Team Leader(s) Rafael de Hoyos Task Team Leader(s) Ciro Avitabile Team Member Wendy Cunningham Team Member Janet Entwistle Team Member Juan Carlos Serrano Financial Management Specialist Supervision/ICR Monica Yanez Pagans Task Team Leader(s) Gabriel Penaloza Procurement Specialist(s) Daniel Chalupowicz Financial Management Specialist Rafael E. De Hoyos Navarro Team Member Abril Alicia Ibarra Castaneda Team Member Antonella Novali Team Member Robert H. Montgomery Environmental Specialist Gabriela Grinsteins Counsel Marcela Lucia Silveyra de la Garza Team Member Arelia Jacive Lopez Castaneda Social Specialist Jasmine Anne Pineda Team Member Page 47 of 62 The World Bank Mexico School Based Management Project (P147185) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 22.440 110,923.12 FY15 34.637 154,043.27 Total 57.08 264,966.39 Supervision/ICR FY15 2.268 108,086.38 FY16 23.165 140,322.92 FY17 52.755 290,973.28 FY18 36.122 203,835.41 FY19 43.874 239,015.13 Total 158.18 982,233.12 Page 48 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (%) Increasing School Autonomy 342.00 321.19 91.77 and Parent Participation Improving Schools' 3.50 0.00 0 Managerial Capacity Research and Innovation 3.625 0.00 0 Front End Fee 0.875 0.875 100 Total 350.00 322.06 92.02 Page 49 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 4. EFFICIENCY ANALYSIS Background and Context 1. The Project included the design and implementation of seven Randomized Control Trials (RCTs) that were implemented in 7 out of the 32 States in Mexico benefiting from this Project. The use of an RCT to assess the impact of an intervention on a set of outcomes of interest is considered the gold standard within the impact evaluation literature to quantify causal impacts. In this Project, these 7 RCTs were designed and implemented to measure in a rigorous manner how the activities implemented as part of the Project impacted both schools’ managerial capacities and student learning outcomes. Moreover, the international evidence in this area is very thin, so the results of these RCTs are also creating important public goods and adding substantially to our global knowledge on the potential of strengthening school autonomy and management to improve education outcomes. 2. The implementation of the 7 RCTs designed as part of this Project was complex, taking a full three years (school years 2015-16, 2016-17, 2017-18). It is worth highlighting that this process was fully owned by the Government, with its implementation fully funded with local resources. 3. All 32 States in Mexico that benefited from the activities supported by this Project were invited to participate in the RCTs. The Government selected only 7 States based on the availability of resources, which is fully documented in the webpage created as a repository for these impact evaluations: http://escuelaalcentro.com/. The 7 States that participated in the impact evaluation were Puebla, Guanajuato, Durango, Estado de Mexico, Tlaxcala, Morelos, and Tabasco. 4. The design and implementation of these 7 RCTs should be put in the context of the long-term expansion of the Government’s national strategy to increase school autonomy and strengthen school management capacities, which has been promoted by different administrations of the Government in different formats since 2001. 15 5. The efficiency analysis presented in this section is divided in two parts: (a) Project’s efficiency; and (b) Sensitivity analysis of the results. The Project’s efficiency estimates the economic benefits and costs of the Project and the internal economic rate of return (IERR). The economic benefits of the Project are estimated based on the results from these 7 RCTs on student learning outcomes impacted through the school autonomy and management channels. The costs of the Project are estimated based on administrative data provided by the Government. Project’s Efficiency: Economic Benefits of Participating in the Project Benefits 6. The first component of the Project included the provision of direct support to schools to implement their improvement plans (Rutas de Mejora), which is directly related to the concept of school 15The guidelines for school-based management are based on lessons learned from more than 13 years of the Programa Escuelas de Calidad (PEC) that has operated in Mexico since 2001 and has been supported by the Bank since 2006. The policy of strengthening school-based management was implemented through several national programs, among others, PEC and the Programa Escuelas de Tiempo Completo (PETC). Page 50 of 62 The World Bank Mexico School Based Management Project (P147185) autonomy. The second component comprised the design and implementation of tools to support schools to improve their managerial capacity. The third component included the design of a set of impact evaluations to quantify the causal impact of the implementation of the activities supported by the Project. The channels through which school autonomy and management impact education outcomes cannot be assessed or disentangled from the 7 RCTs designed and implemented as part of this Project. Therefore, the economic benefits of the activities supported under components 1 and 2 are quantified jointly to conduct this Project efficiency analysis. 7. This efficiency analysis quantifies the economic benefits of the activities supported by the Project using the theory of change presented in Figure 4.1. This assumes that school autonomy, as measured by the activities supported by this Project, directly impacts school management and student learning outcomes in the short term (3 years), which will then translate into higher labor income of the students in the long term (40+ years). Figure 4.1. Theory of Change Students' future School autonomy School management Student learning income after joining the labor market 8. To measure the short-term effects of the Project on school management and student learning outcomes, the results from the 7 RCTs designed and implemented as part of this Project are used. 16 Although student learning outcomes are only high-level outcomes of this Project, the design and implementation of the RCTs allows us to quantify these long-term impacts. To measure the long-term effect of the Project on lifetime earnings, we use international equivalences of learning outcomes and years of education, and estimations of the returns to years of education to express the benefits in monetary terms. 9. The results from the 7 RCTs show that after the three years of implementation of the interventions supported by this Project, the school management index improved by 0.086 s.d. per year, and student learning assessments in Math and Spanish improved on average by 0.054 s.d. per year. 10. International assessment equivalences between learning outcomes and school attainment are used to translate the student learning improvements to years of education. According to PISA equivalence scales, improving mathematics scores by 0.41 s.d. is equivalent to one additional year of education. This means that students in Project beneficiary schools accumulated 0.132 more years of education per year of exposure to the Project (0.054/0.41 = 0.132), which corresponds to 0.395 years over the duration of the three years of the Project (0.132 × 3 = 0.395) or 1.5 months of extra schooling per year (0.132*12). 16By short-term effects of the Project we are referring to the effects of the activities supported by the Project during its 3 years of implementation. Long-term effects refer to the impacts on beneficiaries once they are adults are reach the labor market. Page 51 of 62 The World Bank Mexico School Based Management Project (P147185) 11. After this, the Mincer Equation is used, which relates years of schooling with wages as follows: � ̂1 + ′ ln( ) = � + + ̂ , where Yi is the logarithm of hourly wage of individual i; Si is the years of schooling of i; X is a vector of controls; ̂ , and � , � are the estimated parameters; and ̂ is the error term. 12. The monetary effects of the Project can be captured by the increase in the years of schooling times the estimated returns to education, as described in the following equation: ̂ ∗ ∆ , = ∗ where are the monetary benefits of the Project at time t, ∆ captures the impact in years of schooling, ̂ corresponds to the returns to education, and stands for the expected labor income in year t for a Project beneficiary student. 13. The Mincer equation has been estimated in several studies for Mexico. Most of the studies use data from the Household Income Expenditure Survey (ENIGH, for its acronym in Spanish) to estimate the returns to education. 17 According to Bracho and Zamudio (1994), an extra year of schooling increases the hourly wage on average by 11.7 percent.18 Barceinas (1999) estimates this figure to be 12.8 percent. 19 We will use 11.7 percent for the main calculations, but different scenarios are used to check for the robustness of the results (see Sensitivity Analysis section). The national mean yearly labor income per capita for 2016 according to ENIGH is MX$50,771, which is used as the reference income. 20 The reference income is used as the constant wage through the professional life assuming there is no inflation or economic growth. 14. Results show that students who benefited from the Project will earn MX$784 more per year than those that did not benefit through the effect of the Project on learning outcomes. 21 This corresponds to an increase of 1.5 percent in per capita annual wages. Costs 15. The national programs through which the Government supported the strengthening of school autonomy and management were mainly PEC and PETC. PEC stopped operating in school year 2016-17. Starting in school year 2016-17, PEC was fully replaced by the Programa de la Reforma Educativa (PRE), which included several components beyond school autonomy and management (e.g. school infrastructure). 16. For the Project’s efficiency analysis, we use the average annual cost of PEC per student for school year 2015-16, which is MX$291.2. We do not use the costs of PETC and PRE because they include 17 The ENIGH is a nationally-representative household survey in Mexico. 18 Bracho, Teresa and Zamudio, Andrés, (1994), Los rendimientos económicos de la escolaridad en México, 1989, Economía Mexicana NUEVA ÉPOCA, III, issue 2, p. 345-377. 19 Barceinas, F. (1999) Función de ingresos y rendimiento de la educación en México. Estudios Económicos. Universidad Autónoma Metropolitana. México. 20 The national mean yearly income per household for 2016 is MX$186,083. 21 50,771 × 0.117 × 0.132 = 784 Page 52 of 62 The World Bank Mexico School Based Management Project (P147185) components beyond school autonomy and management (the activities financed under this Project) and are not directly comparable with those reported for PEC that can be fully attributed to the activities supported under this Project. Cost-Benefit Analysis 17. The model assumes that students who benefited from the Project, who are on average 10 years old when the investment is made, enter the workforce at age 24 and retire at age 65 (after 41 years of work) and earn MX$784 per year more than their peers. With an interest rate of 10 percent, the net present value of the income benefits of the average Project beneficiary is 6.7 times higher than its cost. Under these assumptions, the Project yields an IRR of 21.5 percent. 22 Table 4.1. Summary of Economic Analysis Baseline scenario parameters Average age when the investment was made 10 Age when entering workforce (years) 24 Retirement age (years) 65 Time in the labor market (years) 41 Annual interest rate 10% Additional years of education per year of Project beneficiaries 0.132 Return to additional year of education 0.117 Annual per capita labor income 2016 MX$ (ENIGH) 50,771 Increase on labor income per year due to the Project (MX$) 784 Student beneficiaries PEC student beneficiaries 2015-2016 6,178,516 Costs per student per year (MX$) PEC annual cost per student (including administrative costs) 291.2 18. When interpreting the results, it is important to consider some caveats of the analysis, which are mostly driven by the lack of data. First, the analysis is a lower bound of benefits given that it is based only on the individual student benefits of education through wages. However, there are other pecuniary and non-pecuniary individual and social benefits to education that are not included in the estimations because of lack of information and simplicity of analysis (that is, noncognitive skills, health and nutrition benefits, reduction in crime rates, productivity). In this same line, the Project is expected to have positive indirect impacts over several wellbeing outcomes such as school management, as found in other countries (Hanushek et al 2013 and Bloom et al 2015). 23 Second, schools that were used as counterfactuals to assess 22 The PAD estimated an IERR of only 1.95 percent, which was calculated with no information available to guess the impact and benefits of the program. 23 Hanushek, E.A., Link, S. and Woessmann, L. (2013). ‘Does school autonomy make sense everywhere? Panel estimates from PISA’, Journal of Development Economics, Vol. 104 (September), pp. 212–32. Bloom, N., R. Lemos, R. Sadun and J. Van Reenen (2015). "Does Management Matter in schools?," Economic Journal, Royal Economic Society, Vol. 0 (584), pages 647-674, May. Page 53 of 62 The World Bank Mexico School Based Management Project (P147185) the impact of the interventions supported by the Project might have benefited indirectly by at least some of the activities supported by the Project, as the school autonomy and management was being phased-in by the Government during the time of the study as part of an ambitious education reform to improve education outcomes. To this end, the impacts of the activities supported by this program can be considered lower bound estimates of the actual benefits of the Project. Sensitivity Analysis 19. As mentioned earlier, the analysis is based on several parameters and assumptions that can be relaxed or modified to check the robustness of results. Estimations of results for alternative interest rates (5 percent, 7 percent) result in positive benefit-cost ratios ranging from 6.2 to 13.6. Using alternative lower returns to education set up at 0.101 (Montenegro and Patrinos 2014) yields consistent results. Table 4.2. Benefit-Cost Ratios by Scenarios Benefit-Cost Ratio Scenarios 5% 7% 10% Baseline 23.9 14 6.7 Lower returns to education 20.5 12 5.6 Page 54 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 5. BORROWER COMMENTS The Borrower’s comments on this report presented in this Annex have been translated into English and redacted by the Bank. Any mistakes are the sole responsibility of the Bank. 1. As the Project Implementing Agency, the Secretariat of Public Education (SEP), through its General Directorate for the Development of Educational Management (DGDGE) in the Sub secretariat of Basic Education (SEB), considers that the Implementation Completion and Results Report (ICR) collects relevant and reliable information on the design, development, implementation, and results of the Project. In particular, the ICR captures the alignment and contribution of the Project to the policy to strengthen school-based management and autonomy introduced in 2013 by the Government of Mexico, which reflects the relevance of the Project's goal for basic education in Mexico. 2. Likewise, the ICR summarizes and compiles the information that the DGDGE has been providing throughout the life of the Project through semi-annual financial reports, semi-annual monitoring reports, and technical supervision missions, among others, on the progress of each of the Project’s components. This shows that both the Bank and the SEP adhered to the objectives and the strategy for their achievement throughout Project implementation. In this sense, the ICR provides an external and objective view of the Project's achievements, which constitutes a solid and independent basis to recognize its achievements and identify the challenges that arose not only for management of the Project, but also for the educational policy on school management and autonomy. 3. The ratings assessed by the World Bank team on effectiveness, efficiency, monitoring and evaluation are considered adequate, given that the Project showed positive results in improving the management skills of school directors which, in turn, resulted in favorable results in student learning. This is undoubtedly the most outstanding result of the Project, and its measurement was possible through the collaboration with the World Bank, who contributed its experience and knowledge for the design of the impact evaluation of the PRE and the PETC. 4. The Project also contributed to greater family involvement in the Ruta de Mejora and in the development of their children’s basic skills, which could in part explain the improvements in dropout rates in participating schools. 5. The ICR is also clear in presenting the Project’s limitations in decreasing the failure and repetition rates in participating schools. However, as explained in the report, these data could only be noise and not statistical differences considering that evaluations carried out on projects to strengthen school autonomy have shown a reduction in these indicators. 6. The monitoring of some of the indicators presented challenges for the DGDGE team because of the rotation of staff and the changes to the Formato 911, through which the SEP collects the necessary information to calculate the main educational indicators. However, both the DGDGE and the World Bank assigned personnel with technical knowledge and knowledge of the Mexican educational system to carefully analyze the data and overcome these difficulties. 7. The Bank provided technical assistance to the DGDGE in the design of the Project and in the Page 55 of 62 The World Bank Mexico School Based Management Project (P147185) search for mechanisms to measure its results in a rigorous and objective manner. Thus, the Bank collaborated with the SEP to select the most appropriate instrument to measure schools’ managerial capacity and designed an experimental evaluation to measure, for the first time, schools’ managerial capacity and their contribution to the improvement of student learning. In this process, the Bank offered constant feedback to the DGDGE and the seven entities participating in the impact evaluation, participating actively in the communication and dissemination of the impact evaluation results to the local educational authorities and to the participating schools. The Bank also provided feedback to the DGDGE on the interventions developed to strengthen schools’ managerial skills. During implementation support missions, the Bank had the opportunity to exchange views with school directors, school supervisors, and parents on the pertinence of the interventions to strengthen schools’ managerial capacity in the context of the conditions and culture of Mexican schools. This dialogue allowed the identification of key elements already in place in Mexico for good school management and the challenges to generate greater improvements in this area. 8. An important contribution from the World Bank was its sharing evidence of good practices from other contexts, which will be important to continue considering that educational changes are complex and that governments have limited time to promote them. It is therefore necessary to have clear references to effective policies. 9. Finally, in terms of financial management and procurement, the SEP considers that the proposed rating adequately responds to what happened during Project implementation. The technical support of Financial Management and Procurement Specialists was satisfactory, with the specialists showing sensitivity and awareness of the circumstances and challenges of the Project at all times, an understanding of all Project dimensions, and providing support whenever required. This assistance helped install technical capacity in the local unit. 10. A number of areas of improvement have, however, been identified: first, taking into account how disbursements behaved, all were made under Category 1 (Direct support to schools), with several reallocations between categories throughout the life of the Project (from Categories 2 and 3 to Category 1). It is therefore suggested to consider the possibility of making disbursements only against Category 1 in similar operations in the future. Second, related to the previous point, Category 2 (incremental costs) ceased to be operational as it could not include travel allowances and costs, which is why no disbursements were made against it. Third, for Category 3 (consulting services, among others) to be maintained, greater knowledge of international financing organizations’ procurement policies should be promoted within the government. The implementing agency found an internal lack of knowledge and rejection of these policies when it tried to advance procurement processes. On the other hand and related to procurement, the replacement of the obsolete SEPA system with the STEP system was considered a success. 11. We would like to finish this evaluation with a special thanks to the Managers of the Project, Rafael de Hoyos and Mónica Yañez, as well as to all their team, because they have been our best allies in the successful performance of the Project, always contributing solutions, pragmatism, technical rigor and, above all, a passion to contribute to the improvement of education in Mexico. Page 56 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 6. SUPPORTING DOCUMENTS Bloom, N., R. Lemos, R. Sadun and J. Van Reenen (2015). "Does Management Matter in schools?," Economic Journal, Royal Economic Society, Vol. 0 (584), pages 647-674, May. Cabrera-Hernández y Peréz-Campuzano (2016) INFORME FINAL. Estudio Exploratorio Sobre el Impacto del Programa Escuelas de Calidad 2016. CIDE. De Hoyos, R., V. Garcia, H.A. Partrinos (June 2017). Economics of Education Review Volume 58, Pages 123-140 De Hoyos,R., M. Silveyra and M. Yañez (2017) “La Escuela al Centro. Impacto del Programa a un Año de Implementación” nota del Banco Mundial no publicada. Gertler, Partinos and Rubio-Codina (2007) Impact Evaluation for School-Based Management Reform. The World Bank. Poverty Reduction and Economic Management. Thematic group on Poverty Analysis Monitoring and Impact Evaluation. Gertler, Patrinos and Rubio-Codina (2012) Empowering Parents to Improve Education. Evidence from Rural Mexico. Journal of Development Economics. 99 (1): 68-79. Hanushek, Eric A. & Link, Susanne & Woessmann, Ludger, (2013) “Does school autonomy make sense everywhere? Panel estimates from PISA,” Journal of Development Economics, Elsevier, vol. 104(C), pages 212-232. Kremer, Michael, Conner Brannen y Rachel Glennerster (2013) “The Challenge of Education and Learning in the Developing World”, Science, Vol. 340, Issue 6130, pp. 297-300 Krishnaratne, S., White, H. y Carpenter, E., (2013). “Quality education for all children? What works in education in developing countries”, Working Paper 20. New Delhi: International Initiative for Impact Evaluation (3ie) McEwan, Patrick J. (2015) “Improving Learning in Primary Schools of Developing Countries: A Meta- Analysis of Randomized Experiments”, Review of Educational Research, September 2015, Vol. 85, No. 3, pp. 353 –394 Murnane, Willet y Cardenas. 2006. Did Participation of Schools in Programa Escuelas de Calidad (PEC) Influence Student Outcomes? Harvard University, School of Education. Nicholas Bloom & Renata Lemos & Raffaella Sadun & John Van Reenen, (2015) “Does Management Matter in schools?,” Economic Journal, Royal Economic Society, vol. 0(584), pages 647-674, 05. Santibañez, Abreu y O'Donoghue (2014) School based management effects: Resources or governance change? Evidence from Mexico. Economics of Education Review, 2014, vol. 39, issue C, pages 97-109 Page 57 of 62 The World Bank Mexico School Based Management Project (P147185) Skoufias, E. y J. Shapiro. 2006. “The Pitfalls of Evaluating a School Grants Program Using Non-experimental Data.” World Bank Policy Research Working Paper No. 4036, Washington, D.C. UNICEF (2016) Niñas y niños fuera de la Escuela. Mexico. UNICEF Report SEMS, Encuesta Nacional de Deserción en la Educación Media Superior, ENADEMS 2011/121. World Bank (2018) World Development Report -Learning to realize Education’s Promise 2018. World Bank (July 2017) Restructuring Paper. School Based Management Project. Mexico. P 147185. World Bank (April 2014) Project Appraisal Document. School Based Management Project. Mexico. P147185. Page 58 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 7. THEORY OF CHANGE 1. School-based management (SBM) refers to a combination of the following elements: (i) autonomy to make decisions at the school level; (ii) self-evaluation for continuous improvement; (iii) collaborative planning and social participation to achieve concrete goals; and (iv) local transparency and accountability. All these elements, under certain circumstances (i.e. minimal institutional capacity at school and state level) have been identified by the literature to help improve the quality of education (Hanushek, 2013). SBM programs have also proven more effective when applied for a longer period of time (4 to 5 years or more) (Gertler et al., 2007). 2. The premise behind SBM initiatives is that school actors, including school directors, teachers, students, and parents, best know what goes on in their schools and are better positioned to identify and address their needs. Therefore, the theory of change is that providing financial support to schools, coupled with adequate capacity-building for local actors, promotes SBM. These two components (resources + training) enable schools to exert their autonomy and strengthen the dialogue and communication within the school community (school directors, parents, teachers) to make decisions related to their own priorities and to plan and implement school improvement plans accordingly. For instance, Santibañez, Abreu and O'Donoghue, 2014; Cabrera and Campusano, 2016 also found that PEC-FIDE, a small SBM pilot program in Mexico, observed an increase in test scores as a result of giving large rural schools more financial resources to meet pressing equipment, material, and infrastructure needs. 3. In particular, the successful implementation of SBM schemes largely depends on schools having the self-management capacity of local stakeholders; their ability to conduct a candid self-assessment; and the development and implementation of school improvement plans. This Project emphasized the importance of providing capacity-building activities to improve the schools’ ability to use the local resources (also considered in the Project) to provide better education services in their specific context. International evidence suggests a strong positive correlation between schools’ managerial capacity and learning results (Bloom et al., 2014; 2015.) An impact evaluation (IE) conducted under this Project found similar results for Mexico (De Hoyos, forthcoming). In the broader education literature, training and support to local stakeholders in the management of school grants has also been linked to improved SBM and learning outcomes (Pradhan et al., 2011; Heyward et al., 2011; Gertler et al., 2012; Blimpo and Evans, 2011; Khattri, Ling et al., 2010; Duflo, 2014). 4. Another crucial aspect of SBM success is parental involvement. Increasing parental participation in school matters has been linked to better attendance, behavior and test scores. There is a significant positive correlation between school, family, and community involvement and student success (Henderson and Mapp 2002). Parent engagement has also shown positive results in improving the school environment itself (Gertler, Patrinos and Rubio-Codina 2012). Furthermore, parental engagement in school improves the performance of all students at the school, not just their own children. The more comprehensive and well-planned the partnership between school and home, the higher the student achievement (Henderson and Berla 1995). The relationships between the school and the broader community also promote social participation, transparency in resource usages, and accountability (Gertler et al 2007). Page 59 of 62 The World Bank Mexico School Based Management Project (P147185) 5. SBM schemes, including direct financial and technical support to schools, such as the one supported under this Project, have shown positive impacts on lowering dropout, failure and repetition rates (World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006). SBM initiatives can also help reduce parental financial contributions to schools, thus, increasing enrollment rates and help avoiding fund leakages by reaching schools directly (World Bank 2014, UNESCO/IIEP). Page 60 of 62 The World Bank Mexico School Based Management Project (P147185) ANNEX 8. CHANGES TO KEY INDICATORS The following table describes the indicators before and after restructuring and provides a detailed rationale for these changes. Revised Indicator (at July 2017 Original Indicator Rationale for changes Restructuring) Proportion of basic education Proportion of a representative Added to replace original PDO schools in programs (PEC and sample of Participating Schools Indicator 1, which lacked a clear PETC) to strengthen SBM with a with a director that scores 3 or benchmark since “sufficient” was director that has a “sufficient” above in the World not clearly determined given that level of managerial capacity. Management Survey. the design of the data collection Baseline: 10.50 percent Baseline: 21 percent, 2015-16 instruments to be used was Target: 40 percent Target: 25 percent, 2017-2018 pending. In addition, the baseline data became available until December 2015 and included a representative sample of all schools in the country and not only Project beneficiaries. To correct these inaccuracies, this indicator was dropped and replaced in the July 2017 restructuring. Proportion of basic education Proportion of a sample of Added to replace original PDO schools in PEC with parents’ Participating Schools that Indicator 2 replaced because the associations participating in the increase parental participation methodologies used to measure design, monitoring, and (score more than 1 in the AGE). parental participation were adjustment of the school Baseline: 83.9 percent in 2015- costly and comparability of the improvement plan. 16 data collected in different years Baseline: 72.70 percent, 2011- Target: 90 percent in 2017- was also questionable. 12 2018 Target: 78 percent, 2017-18 Dropout rate among basic Dropout rate among (primary The wording for indicator 3 was education schools in programs and secondary) Participating adjusted to reflect the revised to strengthen SBM (PEC and Schools. PDO. "Basic education schools in PETC). Primary Primary programs to strengthen SBM" Baseline: 3.83 percent, 2012-13Baseline: 3.18 percent, 2014- was replaced with "Participating Target: 3.48 percent, 2017-18 15 Schools." In addition, data for Secondary Target: 2.91 percent, 2017-18 this indicator was added for PRE Baseline: 5.44 percent, 2012-13Secondary and PETC schools as of the 2014- Target: 5.04 percent, 2017- Baseline: 5.31 percent, 2014- 15 school year to reflect the 2018 15 change in Project beneficiaries. Target 4.80 percent, 2017-18 Gross failure rate among basic Primary and Secondary school Same as above Page 61 of 62 The World Bank Mexico School Based Management Project (P147185) education schools in programs gross failure rate among to strengthen SBM (PEC and Participating Schools. PETC). Primary Primary Baseline: 1.10 percent, 2014- Baseline: 1.30 percent, 2012-13 Target: 0.09 percent, 2017-18 15 Secondary Target: 1.00 percent, 2017-18 Baseline: 9.64 percent, 2012-13 Secondary Target: 8.50 in 2017-18 Baseline: 3.49 percent, 2014- 15 Target: 3.26 percent, 2017-18. Repetition rate among Secondary school repetition Same as above secondary school students in rate among Participating schools in programs to Schools. Baseline: 0.63 percent, strengthen SBM (PEC and PETC) 2014-15 Baseline: 0.87 percent, 2012- Target: 0.60 percent, 13 2017-18 Target: 0.67 percent, 2017-18 Page 62 of 62