Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Report Number: ICRR0021756 1. Project Data Project ID Project Name P147185 MX School Based Management Project Country Practice Area(Lead) Mexico Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-84460 31-Dec-2018 348,220,127.39 Bank Approval Date Closing Date (Actual) 24-Oct-2014 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 350,000,000.00 0.00 Revised Commitment 350,000,000.00 0.00 Actual 322,064,712.39 0.00 Prepared by Reviewed by ICR Review Coordinator Group Susan Ann Caceres Christopher David Nelson Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) According to the Lending Agreement (p. 6) and the appraisal document (p.vi) the development objectives were “to improve schools’ managerial capacity and parental participation to reduce repetition, drop-out, and failure rates among Programa Escuelas de Calidad (PEC) schools and Programa Escuelas de Tiempo Completo (PETC) schools”. The project was restructured on July 26, 2017 to clarify that the project supported participating schools that consolidated the PEC and PETC programs under the Education Reform Program. The revised objectives were “to improve schools’ managerial capacity and parental participation to reduce drop-out, repetition, and failure rates among Participating Schools”. During the restructuring key associated outcome targets were revised downwards to correct for data changes. No split rating is applied, as the restructuring resulted in no substantive change either to the objective or to targets/indicators. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Oct-2014 c. Will a split evaluation be undertaken? No d. Components There were three components: Increasing School Autonomy and Parent Participation (at appraisal, US$342.000 million; actual, US$321.2 million) supported resources to priority schools to develop school improvement plans. Improving Schools’ Managerial Capacity (at appraisal, US$3.500 million; actual, US$0) financed a school dashboard for supervisors and directors; it also supported capacity development for school managers; and parents. Research and Innovation (at appraisal, US$3.625 million; actual US$0) provided resources to assess changes in managerial capacity in at least two self-selected States to quantify the effect of the project supported interventions and improve adherence to Operational Rules among schools. In the July 2017 restructuring some changes were made to the components at the request of the Government. Direct support was meant to strengthen school-based management for “Participating Schools” (instead of PEC and PETC schools, to align with new acronymn). The reference to the school dashboards was removed. It also clarified that each state selected Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) interventions to improve managerial skills of school directors and communities, based on the specific context. The revisions motivated changes in the indicators. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs: Project costs were US$322 million, 8 percent less than the appraisal estimate of US$350 million with all of these resources being utilized for the first component. However, the ICR (p. 18) noted that all resources (99.4%) were expected to be expended by June 30, 2019- the extended deadline granted by the Bank. Government resources ultimately covered all costs for components two and three for research undertaken by the project, due to procurement regulations for financing technical assistance activities. The team reported that the resources devoted by the government for components two and three exceeded (US$ 3.6 million) the planned amount (US$ 3.5 million). Financing: IBRD financed the loan. Borrower Contribution: According to the appraisal document (p. 56), total project costs were expected to be US$785 million, where IBRD financed US$300 million. Thus, the Borrower was expected to contribute US$485 million; however, contributions were slightly lower (US$470 million). Dates: The operation was approved by the Board on October 24, 2014 and became effective on July 9, 2015. The period complied with the condition that effectiveness occur within 18 months of approval. In July 2017 there was a level two restructuring, which made changes to the results framework, objective, component cost, legal covenants and arrangements for financial management. The operation closed on schedule on December 31, 2018. However, the ICR (p. 18) reported an extension was granted until June 30, 2019 to disburse the remaining funds. 3. Relevance of Objectives Rationale The Government of Mexico’s National Development Plan promotes more equitable human capital accumulation to reduce poverty. The Education Reform (2013-18) focuses on improving the low levels of learning to increase productivity, as the education system has already attained high levels of net enrollment in pre-primary for 5 year old (87 percent), primary (96 percent) and lower secondary education (87 percent), but issues remain with the quality of teaching and learning. The Education Reform Program promotes school autonomy and school-based management (among other aspects that include: a professional teacher management system and collection of school data) to improve quality. The appraisal document (p. 3) notes the importance of the integration of these elements, but lacks detail on the pathway autonomy plays in improving education quality, which was the development challenge focused in this operation. Thus, areas emphasized in the Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Reform Program were substantially aligned with the objectives of the operation, as the Reform noted the important role for schools, parents, school managers, and suggested evaluation data that could be used by schools when they were developing their annual school plans. The World Bank Group’s Country Partnership Strategy (FY14-FY19) (at entry and closure) was aligned with the objectives of the Government and the objectives of the operation. The Second Pillar of the Partnership Strategy (Increasing Social Prosperity) focuses on improved access and quality in education to improve inequities in educational outcomes and improve productivity between the rich and the poor. Thus, the objectives of improving schools’ managerial capacity, improving parental participation, and reducing repetition, drop-out, and failure rates were substantially aligned with the Government's focus, as well as the Bank's priorities throughout the life of the operation. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve managerial capacity in Participating Schools Rationale The theory of change focused on improving the operational aspects of the school- particularly school-based management and the capacity of school leaders and parents to improve the education system and its quality. Improvements were expected from the training of instructional leaders and the provision of direct resources to schools to implement improvement plans. The key indicator to evaluate this objective was “the proportion of a representative sample of Participating Schools with a director that scores 3 or above in the World Management Survey (WMS)”. The evidence-based theory of change was logical, but relied on several assumptions to be simultaneously present (beyond the supported activities). Decentralized level capacity is needed to support schools. One aspect of the capacity is the efficient transfer of resources to schools, as these resources are critical inputs for schools to implement improvement plans. School leaders may also need additional technical assistance at the decentralized level (beyond the training sessions) to identify the most effective strategies to improve education services. Since teachers were not directly supported, an assumption in the theory of change was that school directors would ensure teachers implement activities to foster student learning and school directors monitor the pedagogical practices of teachers. An additional assumption in the theory of change was the presence of adequate Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) instructional leadership skills and adequate performance among the school directors and supervisors. The linking of these key issues is not immediately clear in the relationship between improved educational outcomes and managerial improvements. Outputs: Direct support was provided to schools to implement the school improvement plans each year (Table 1). National criteria were developed to allocate resources directly to the schools. All States adopted the criteria (i.e. Marginalization of locality and number of students) (ICR p. 32). The ICR reported that the financial resources were an incentive for school directors to work with parents and teachers in how to improve student learning. The directors had to report on the use of the funds, thus providing a mechanism to increase transparency and accountability. Table 1 Number of grants by program and year year PEC PETC 2015-2016 42,614 na 2016-2017 22,296 25,032 2017-2018 18,693 25,134 2018-2019 pending pending All school supervisors and directors in participating schools took part in the capacity-building program. The number of school supervisors trained ranged from 74 percent of supervisors in the country (7,694) to all supervisors and those on a clear path to become supervisors (18,447)[CDN1] . Over 76,000 school directors were trained by supervisors through the cascade training program. This cascade training program was implemented in 32 states. Topics of the school director and supervisor training varied across states focusing on themes such as conducting classroom observations and strategies to improve student learning, suggesting that the training appropriately focused on instructional leadership skills- not just managerial responsibilities. But the ICR gives no information on whether the variable approach was beneficial, nor whether some states performed better than others on their managerial improvements. There was an additional capacity building program, but the content of the program varied in the seven States participating in the impact evaluation. This additional training was provided to 800 school directors in the seven States. The training was targeted to the specific needs of the school and its director, but was related to building the school culture, coaching strategies, and strategies to improve student learning. There was reference in the ICR to the use of measurement systems in the schools, but no reference to what difference these tools made to school performance and only cursory reference to how these tools were used in discussions around changing how the schools function. Outcomes: Capacity was increased for school directors and supervisors. This indicator was measured in participating schools with the WMS instrument, which measures managerial capacity. The scale in the WMS is 1 (lowest score) to 5 (highest score). Directors Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) scoring three or more in the WMS increased from 21 percent in 2015/2016 to 27 percent in 2017/2018, which exceeded the revised target (25 percent). The ICR noted those scoring three (or higher) use existing resources in an effective way to provide better educational services. The ICR also reported that the WMS served as a tool to provide feedback to schools to understand their challenges. The sample of school directors’ taking the WMS was based on a 1,665 school national sample, not just project supported schools. The ICR also noted that the instrument was applied in several other countries, thus permitting benchmarking. The team shared data that showed other countries had lower scores than Mexico, but nearly all OECD countries scored slightly higher. Country scores ranged from 1.7 (lowest) to 2.9 (highest) with Mexico falling somewhere in the middle (2.3). The ICR provides no disaggregated data to show whether different approaches had different impacts in the states where bespoke approaches were applied. This makes it difficult to judge whether project support determined the shift or whether other factors were at play. The school management index improved by 0.086 standard deviations per year in schools participating in an impact evaluation. The ICR qualified that this was after 3 years of support and where activities were closely monitored. The impact evaluation found that managerial capacity had a small impact on learning in these schools (ICR p. 17), though the ICR makes little reference to potential qualifying factors here with regards to other issues driving a change in the way school supervisors were behaving. It should also be noted that the participating directors in these seven States received additional training. Rating Substantial OBJECTIVE 2 Objective Improve parental participation Rationale The theory of change focused on how parents were involved in their children’s schools- particularly school-based management and the capacity of school leaders and parents to function as a means to improve the education system and its quality. The project claimed to strengthen the culture of school autonomy and school-based management through direct financial support to schools. School-based management depends upon parental participation to ensure transparency and accountability, and in this project the expectation was that parents would hold the system accountable and be more involved in their children’s education. The key indicator to evaluate this objective was the “proportion of a sample of Participating Schools that increase parental participation”, but there is no explicit linking of the degree of parental engagement in school plans. This is inferred in the theory. Parental participation was measured in seven States with the School Management Autonomy questionnaire (Cuestionario de Autonomía de Gestión Escolar (AGE)). Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Outputs: Parents in 78.21 percent of participating schools or 18,386 schools by the end of the project received school-based management training, which did not meet the target (90%). Parents had a role in the development of the school improvement plans and monitoring the use of school grant resources[CDN1] , but the ICR does not describe the role. The training discussed how parents can better support their children’s learning, which is an important topic to focus on. A survey found parents satisfied with the training, but the ICR does not provide detail on the differentiation of approaches in the different regions, nor does it detail if and which approaches work best in changing and influencing the behavior of parents. As part of capacity building training for school directors and supervisors, they learned how to involve parents in the learning of their children[CDN2] , which is important to also include in managerial training, but the ICR does not provide detail on whether this has been applied and where it has worked well. The theory is inferred rather than illustrated by evidence. Outcomes: Parental participation increased from 83.9 percent in the 2015-16 school year to 94.66 in 2017-18[CDN1] , which exceeded the revised target (90 percent). The participation was measured by the School Management Autonomy questionnaire (Cuestionario de Autonomía de Gestión Escolar (AGE). The AGE survey was conducted in seven States among a sample of 1,091 Participating Schools. The AGE was developed by the Secretary of Public Education. The differentiation in approaches among the impacted evaluation regions should have been better illustrated in the ICR to ensure confidence that the activities were driving the change in behavior among parents. No detail is provided on which were the more effective tactics in changing participation rates. Rating Substantial OBJECTIVE 3 Objective Reduce Dropout Rates Rationale The theory of change of the project was that school grants and capacity building of school directors and parents would improve school autonomy and parental participation, which would have an impact on what parents, teachers, and school managers would do to impact students dropping out or remaining in school. The ICR asserts (without evidence) that schools may have reduced the financial cost parents have to contribute from the school grants (i.e. where parents were previously asked to contribute to school supplies or school maintenance), which could have helped to reduce barriers to school participation. For the project resources intended to support enhancements to school infrastructure and delivery of a meal program, an impact evaluation was Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) undertaken of the PETC program and found an association with reducing student drop-out and student learning (ICR p. 17). The key indicator to evaluate this objective was“ the dropout rate among (primary and secondary) Participating Schools.” Outputs: A school dashboard information system (SisAT) was developed. It allowed directors and teachers to identify at risk students to detect which students may need interventions to support their issues. Half of the students in Participating Schools were female. Outcomes The dropout rate decreased in primary schools from 3.18 percent in 2014-15 school year to 2.56 in 2016-17 school year, which met the revised target (2.91 percent). Results from the 2017 and 2018 school year were not available at the time of the writing of the ICR. The dropout rate decreased in secondary schools from 5.31 percent in school year 2014-15 to 2.12 percent in school year 2017- 18, meeting the revised target (4.80 percent) and the original target (5.04 percent). Results from the 2017 and 2018 school year were not available at the time of the writing of the ICR. The ICR reports that caution should be used in interpreting this indicator, as it is affected by aspects outside the control of the project. However, it could also be that interventions likely to impact this indicator were not directly supported by the project. It could also be that the indicators were not an appropriate measure of changes in the objectives, as there is not sufficient detail on interim changes that may be required to sufficiently link the SBM changes to the retention rates in schools. Rating Substantial OBJECTIVE 4 Objective Reduce Gross Failure Rates Rationale The theory of change for this aspect of the project was that school grants and capacity building of school directors and parents would improve school autonomy and parental participation, which over time would have an impact on reducing student failure (and in the long-term, learning). The project sought to strengthen the culture of school autonomy and school-based management through direct financial support to schools, as previous evaluations found a relationship between school grant schemes and a Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) reduction in the failure rate. The key indicator to evaluate this objective was“ the failure rate among (primary and secondary) Participating Schools.” The theory of change did not include information on direct interventions related to teachers, but indirectly referred to the actions of principals as a contributing factor. It would take the direct activities of principals to use data to diagnose what teachers may be doing that could contribute to failure rates and work with teachers to develop interventions. Outputs: Same as reported in increased managerial capacity and parental participation. Outcomes: The gross failure rate for primary schools was 1.1 percent in 2014 and increased to 1.44 percent n 2018, not meeting the target (1 percent). Thus, the gross failure rate slightly increased over the course of the project. The gross failure rate for secondary schools was 7.79 percent in 2013/201 and increased to 7.84 percent in 2018, not meeting the revised target (3.26 percent). The ICR noted that there was a mistake in the calculation of the baseline and target values (p. 31). The ICR reported that this data should be taken cautiously (p. 16) as the rate is related to other factors outside the control of the project such as national policy, but this applies equally to the positive shift in indicators for both drop-out rates and parental participation. Another potential explanation may be that the project supported activities were not adequate to have an impact upon the indicator. Another way of viewing decreased failure is increased learning. While the project did not collect specific indicators to measure changes in learning, an impact evaluation in seven States with approximately 44,000 students assessed the impact of project supported activities such as school grants and capacity building upon learning. It showed better results in reading and on average (than math), and where implementation monitoring occurred, compared with other federal schools under the program. It is not known whether the comparison schools received the same interventions, as implementation was not monitored. In addition, the ICR did not disaggregate data across the country to understand learning outcomes for project districts compared with other districts. The ICR translated this increase to signify 1.5 months of extra schooling per year based on PISA equivalents. Rating Modest OBJECTIVE 5 Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Objective Reduce repetition rate Rationale The theory of change of the project was that school grants and capacity building of school directors and parents would improve school autonomy and parental participation, which would have an impact on what parents and school managers would do to impact the repetition rate in school. This project strengthened the culture of school autonomy and school-based management through direct financial support to schools, as previous evaluations found a relationship between school grant schemes and reduction in the repetition rate. The key indicator to evaluate this objective was “the repetition rate among (secondary) Participating Schools.” Output: Same as presented in increased managerial capacity and parental participation. Outcome: The secondary school repetition rate among Participating Schools slightly increased from 0.63 percent (2014-15 school year) to 0.97 percent (2017-18), not meeting the target (0.60). The ICR indicated that this result should be interpreted cautiously as many factors impact repetition (p. 16), but this applies equally across the range of indicators illustrating project success. Another potential explanation may be that project supported interventions were not adequate to impact this indicator. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project improved managerial capacity and parental participation, although there may be issues with attribution as other factors may be encouraging the participation of parents. There have been modest changes in failure and repetition rates; however, a forthcoming impact evaluation suggests improved learning, which is a better measure of education quality. However, the ICR did not disaggregate data across the country to understand whether learning outcomes for project districts were better than others. The overall efficacy rating is substantial, which is indicative of sufficient evidence illustrating a strong contribution to the operation’s achievement of its objectives. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Overall Efficacy Rating Substantial 5. Efficiency Economic analyses were prepared during preparation and at closure. At preparation the loan was expected to result in a return of US$23.3 million. Project interventions were also expected to increase schooling by an additional 0.032 years for project beneficiaries (p. 53). An economic analysis prepared at closure calculated an internal rate of return of 21.5 percent. The ICR asserts project interventions provided value for money. The average annual cost per student of the project was of MX$291.2. The ICR estimated that students who benefited from the project would earn MX$784 more per year due to its positive impact on learning, which corresponded with a one and half percent increase in per person annual wages. The benefits were 6.7 times higher than cost. These estimates were based on the evidence that improvements in school management and autonomy (through direct resources to schools and training) will improve student learning, which results in greater student earnings in the labor market. The estimate is based on students entering the workforce at age 24 and remaining until 65 years. It also assumed students remaining in school and receiving quality education. These aspects were critical for the estimated benefits to materialize. The analysis; however, does not provide data on comparable cost efficiency. There were delays related to the program consolidation and impact from earthquake. However, these factors did not negatively impact implementation. Other aspects of administrative efficiency were not described in the ICR. Given the reduction in available budgetary resources from the fall in oil prices, the number of beneficiaries could not be met. The Government adopted a targeting criteria (i.e. marginalized communities), resulting in benefit to more vulnerable schools. While the project disbursed 92 percent of project funding, the ICR reported that nearly all money was expected to be disbursed by the extension of June 30, 2019. Considering all of these aspects, project efficiency is rated substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  21.50  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is assessed across the entire project and is rated Substantial. Both efficacy and efficiency are also rated Substantial overall. Thus, the outcome is rated Satisfactory, indicative of minor shortcomings in the operation’s achievement of its objectives. a. Outcome Rating Satisfactory 7. Risk to Development Outcome There are several factors that help mitigate the risk to development outcome. The implementation of the capacity building of school directors, school supervisors, and parents continues. Areas supported in the project such as school autonomy and management are part of the Government’s Reform Program, giving a formal endorsement of these aspects. There are a number of continued risks to the development outcome. The risk of natural disasters such as earthquakes and hurricanes remain, which could reduce expected resources to address damaged infrastructure, as occurred during the operation, instead of efforts to sustain school-based management. On the other hand, if there is contingency funding within line ministries, this risk may be minimal. There is also a risk that political support may not continue for direct support to schools. However, a new government was installed in December 2018 and has continued to support the Reform program at a decreased level. Considering all of these factors, the risk is rated moderate. 8. Assessment of Bank Performance a.Quality-at-Entry Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) Design was simple and based on prior lessons and international evidence of school-based management. Results from impact evaluations and a meta-analysis were used during preparation. The Bank team worked with the Implementing Agency, the Secretariat of Public Education (SEP), through its General Directorate for the Development of Educational Management (DGDGE) in the Sub secretariat of Basic Education to prepare the operation. There were minor weaknesses. It was foreseeable that counterpart funding would be needed to finance components 2 and 3, given past experience. Bank resources could have been better used for other purposes. While risks were appropriately identified (i.e. decrease in oil prices resulted in decreased education budget that impacted the number of project beneficiaries), these risks impacted implementation, suggesting mitigation measures were insufficient. There were shortcomings in the indicators selected (see monitoring and evaluation). Design may have missed inclusion of activities to directly reduce repetition, drop-out, and failure with its reliance on schools identifying and implementing adequate measures in their plans. The design would also have benefitted from a clearer linking of the PDO indicators to interim measures of school transformation. There is little evidence illustrating what aspects of schools needed to change to ensure the benefits of SBM were likely to lead to improved educational outcomes. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank team provided active and proactive supervision. They coordinated with the government agencies responsible for overseeing the project Secretaría de Hacienda y Crédito Público (SHCP), National Financial Development Bank, and SEP- DGDGE. Regular supervision missions were conducted, including visits to participating schools. The Government’s completion report noted the positive contribution of the Bank team. It particularly appreciated the team’s pragmatism and technical knowledge (ICR p. 56). The Bank team supported the design and implementation of the impact evaluation. This support ensured valid instruments were used to measure parental participation and managerial capacity. The Bank actively supervised safeguard and financial management. With this support, the ICR (p. 26) noted visible results. When challenges arose, the Bank team addressed the issues. The Bank team supported the Government’s program consolidation in the project restructuring. The restructuring took time, since the government needed to ensure the best use of project resources in the context of fluctuations in exchange rate (ICR p.21). It ultimately decided to support PETC schools. When resources were slow to get to some schools, the Bank team worked with the implementing agency to bypass the States and provide resources directly to schools. When baseline data issues emerged, the Bank team hired a data consultant to review data and ensure its accuracy. There were shortcomings. The ICR suggested the restructuring could have been more successful if it had made further adjustments in the Results Framework such as removing failure rates and including learning (as these data were available) and updating the number of project beneficiaries. Likewise, the Bank team should have better dealt with the issue of Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) attribution once it was clear that government funding would cover the full budget allocation for components 2 and 3 of the project. This financing issue muddied the linking of the project design and Bank involvement, even though in principle, the team supported all aspects of the design. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Before the Project, there was no formal or robust measurement of school managerial capacity. The Project provided support to select and adapt an instrument to measure school director capacity. The adapted WMS measured: operation management, monitoring, target setting, human resource management, school leadership, and pedagogical practices. Indicators used in the Results Framework included: rates of dropout, gross failure and repetition (as proxies for education quality), as well as measures of parental participation and managerial capacity. The ICR appropriately states that the rates of drop-out, failure, and repetition were not adequate to detect changes in managerial capacity and parental participation and were not effective measures of the project supported activities. Indicators did not provide immediate outcomes of the implementation of school-based management or principal training- except for output measures. Measures of student learning were not included in the Results Framework from the outset because the short period of the project and uncertainty related to baseline measurement, as the national assessment was being revised during preparation. Indicators relied on official Government data from Encuesta 911.The project also included a rigorous impact evaluation. The impact evaluation examined the effect of project interventions upon various outcomes. The impact evaluation was not designed to detect the causal pathways of the program and this limited its utility in truly understanding the difference SBM approaches make to student outcomes. The government selected the States of Puebla, Guanajuato, Durango, Estado de Mexico, Tlaxcala, Morelos, and Tabasco, after invitation to all States (ICR p.50). The ICR pointed out shortcomings in the data in the Results Framework. First, the target for the dropout rate was underestimated. Second, the targets for gross failure and repetition rates were modest (i.e. repetition rates was only of a 0.03 percent reduction). Third, there were inaccuracies in the failure rate baseline. Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) b. M&E Implementation Most of the shortcomings in the results framework and data inaccuracies were corrected during the 2017 restructuring. The methodology to measure parental participation was changed during the restructuring to make data collection less expensive and more reliable. However, the number of project beneficiaries was not changed during the restructuring, despite knowing the project would not reach the target, given the decline in government resources from the fall in oil prices (i.e. reached 4,986,417 students compared with the target of 8,900,000 students). The ICR argued that the project should have also included learning results, given that this outcome was part of the impact evaluation and by the time of restructuring there was clarity on the revised learning assessment. An impact evaluation was implemented in seven states. The evaluation supplemented the indicators that were tracked to provide outcomes of project supported interventions. There were challenges with the implementation of monitoring and evaluation that were outside the control of the project. Data were lost during an earthquake, which slowed reporting. The implementing agency reallocated staff to address this issue. There were inaccuracies in government’s school data, which were the result of staff rotation and changes in the school data (ICR p. 55). Data were reviewed by a consultant to ensure accuracy. There were delays and inaccuracies in the reporting of information by the Implementing Agency and there was no effort to address the limitations of the design in terms of developing useful proxies for intermediate outcomes. c. M&E Utilization Data from the WMS and AGE surveys were used by the Implementing Agency to help inform school autonomy policies. Data from these instruments were discussed with school directors. Moreover, data collected from the impact evaluation were shared with national and State education officials, but the ICR does not provide evidence that the field assessment teams were regularly interacting with school administrators or encouraging iteration in approaches. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified a Category C for environmental assessment purposes, as it did not include construction or Infrastructure rehabilitation. Indigenous Peoples Policy (OP 4.10) was triggered. An Indigenous Peoples Plan (IPP) was approved and published. An updated plan was disclosed in 2017 after consultation with Indigenous leaders and Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) teachers. The ICR reported increased awareness that indigenous students may be bilingual and need differentiation by teachers and schools. The ICR also noted improvements in communication strategies to outreach with indigenous parents. b. Fiduciary Compliance Financial management was rated Satisfactory, except during a short period when disbursements slowed between November 2016 and October 2017. During the restructuring, changes were made to financial management arrangements to fund project resources more rapidly to schools. Compliance with financial management and reporting was adequate. Reports were timely and unqualified, except for the 2017 audit report, which was appropriately corrected. Procurement was limited in the operation. As the government used its resources to pay for the technical assistance activities associated with component two and three, since government regulations require compliance with local procurement procedures. c. Unintended impacts (Positive or Negative) None reported. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) 12. Lessons The ICR (p. 28-29) contained a narrative that IEG synthesized into the following lessons: School-based management requires both direct resources to schools and capacity-building activities to school supervisors, directors and parents. School director capacity is correlated with student learning (ICR p. 28). Instructional leaders and parents need training in how to effectively use financial resources (and local resources) and improve education services to improve student learning. The training was not just about managerial capacity, but instructional leadership skills. The ICR also noted that delayed resources to schools negatively impacted the activities in the school improvement plans, as these resources helped schools overcome challenges. Restructuring should ensure internal consistency and project teams should consider including measurement of student learning. The objective and indicators in the results framework must align with each other. While the project restructuring corrected some weaknesses in the Results Framework, it missed the opportunity to add an indicator to measure learning (an aspect covered by the impact evaluation) and reduce the number of project beneficiaries from reduced resources. It also missed the opportunity to more effectively capture intermediate changes that could be expected from the project interventions. The restructuring continued to use indicators that measured drop-out, failure and repetition rates, which may not have been good measures of the project activities. The ICR proposed more ambitious measures to include student learning outcomes, as advanced by the Bank’s 2018 World Development Report. Additional benefits are attained when projects use analytical knowledge and data. In this operation, the Bank supported the government design and implementation of the impact evaluation and helped identify instruments to measure complex aspects such as managerial capacity or parental participation. Data from the impact evaluation and survey instruments were utilized not just for project monitoring and evaluation, but other purposes. However, more nuanced and revealing use of this resource can be illustrative of where disaggregated information can be applied to changes and approaches in one region over another. Results from the surveys were shared with school directors who participated in the impact evaluation and received a personalized diagnostic to understand shortcomings in managerial capacity. Data from the WMS and AGE surveys were used by SEP to help inform school autonomy policies and the Bank’s policy dialogue with the government. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX School Based Management Project (P147185) A field assessment could provide evidence about the roll-out of school-based management across the country and provide additional evidence from the impact evaluation and data not provided in the ICR such as disaggregation of achievements amongst different states. 14. Comments on Quality of ICR The quality of the ICR is substantial. The ICR is candid and well written. It provides a concise description of the project, its activities, and the factors affecting implementation, though at times more disaggregated information on variation in the performance of the project could have been useful. It contains analysis of shortcomings in design, preparation, implementation, and Bank performance. The annex provided a detailed explanation of the efficiency analysis, but there is no real benchmarking in the efficiency analysis. The completion report reported all indictors and evidence available from the impact evaluation, but it does not provide much on counterfactual performance, nor on how the project worked better than the reforms taking place elsewhere in the country. Aspects not specified in the ICR were clarified with the team. The Borrower’s completion report was useful and supplemented the information contained in the ICRR. a. Quality of ICR Rating Substantial Page 18 of 18