Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Report Number: ICRR0021779 1. Project Data Project ID Project Name P118904 Learning, Equity and Accountability Prog Country Practice Area(Lead) Philippines Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-83440 31-Aug-2018 295,200,000.00 Bank Approval Date Closing Date (Actual) 18-Mar-2014 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 295,200,000.00 0.00 Actual 295,200,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Susan Ann Caceres Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) According to the Lending Agreement (p. 6) and the appraisal document (p. vi), the development objective was “to improve the quality of grades 1 to 3 reading and math skills of children in target regions and target schools, with a special focus on those belonging to targeted disadvantaged groups." Target regions were defined in the appraisal document (p. 3) as “Regions 5, 8, 9, CAR, and Caraga." Target schools were defined in the appraisal document (p. 3) as “a subset of elementary and/or high schools in the target regions implementing the relevant project components." The appraisal document (p. 3) defined targeted disadvantaged groups as “indigenous peoples; persons with disabilities; children living in remote or difficult-to-access locations; and out-of-school children and youth." The project was restructured in November 2017 to make changes in the results framework, implementation schedule, and definition of disadvantaged groups. It redefined disadvantaged group as “children living in remote or difficult-to-access areas and learners who attend schools with high concentration of indigenous people (66 percent or higher)” (ICR, p. 11). Children with disabilities were no longer part of the definition of disadvantaged groups, as the Department of Education believed that early grade reading assessment (EGRA) and early grade math assessment (EGMA) were not appropriate instruments to assess them. In addition, the Department pointed out that there were limited programs under the project to support children with disabilities and limited ability of teachers to identify them given inadequate definition (ICR 27). As a result, this group was removed from indicators and monitoring. No split rating is applied in this Review, as the restructuring did not change the objective or make substantive changes to targets/indicators. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components There were three components. Each component comprised disbursement linked indicators (DLIs) that included a mix of activities, outputs, and outcomes (see Section 4 for the specific DLIs). Improvement of Teaching and Learning in Grades 1 to 3 (at appraisal, US$180.25 million; actual, US$175.45 million) was to support EGRA and EGMA, which are validated assessments intended to be diagnostic tools. It also was to support training for teachers in reading and math skills, as well as training related to interpretation of EGRA and EGMA results to address skill gaps. At least six Mother Tongue versions of EGRA and EGMA were to be developed and administered. Principals were Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) to be coached to be able to support teachers. The Department of Education was to establish guidelines on assessment and refine the Teacher Induction Program to include content for early literacy and numeracy education and assessment. Strengthening of Accountability and Incentives of Department of Education Employees (at appraisal, US$82.5 million; actual, US$ 82.5 million) was to support the development of a Performance Monitoring System (PMS) to provide real-time information on each indicator. It also was to provide resources to redesign a School Report Card (SRC) and strengthen the Department’s financial management capacity. Improvement of Program Design for Targeting Disadvantaged Groups (at appraisal, US$36.5 million; actual, US$36.5 million) was to support an information system to produce data about existing programs serving disadvantaged groups to identify gaps. An evaluation of four programs serving disadvantaged groups was to be completed. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Actual project costs were $295.20 million, nearly all the expected amount (US$ 300 million). The amount undisbursed related to one DLI that was unmet (i.e number of teachers trained). Costs for each component were consistent with estimated amounts. Financing: IBRD financed the loan. Borrower Contribution: The Borrower did not contribute, nor was it expected. Dates: The operation was approved by the Board on March 18, 2014 and became effective on June 24, 2014. Level two restructurings occurred in September 2015 and May 2016 that made changes to definition and/or targets of the DLIs and disbursement arrangements. In November 2017 there was another level two restructuring that made changes to the results framework, implementation schedule, and definition of disadvantaged groups (as outlined in Section 2a). The operation closed four months later than expected, on December 31, 2018. The extension was done to finalize independent verification and a program review of disadvantaged students, as well develop an implementation plan for the information system strategy for disadvantaged students. 3. Relevance of Objectives Rationale Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Inclusive growth remains a challenge with high income inequality in the country (ICR, p. 5). For this reason, one of the main pillars of the Government’s Development Plan promotes equalizing access to development opportunities for all Filipinos. Within the education sector, an important area to address is learning, since many primary level students are non- readers. Scores on the National Achievement Test (NAT) for third graders decreased from 61.74 and 61.94 in 2009-2010 to 54.42 and 57.23 in 2011-2012. Low levels of learning are more pronounced based on ethnicity, disability, geography, and among indigenous people. One particular issue was the preparation of teachers for the subject areas of reading and math. The National Development Plan focuses on teacher competencies, continuous professional development of teachers, and use of data to make evidence-based policy decisions to support student learning. The aims of the Basic Education Sector Reform Agenda were consistent with the objectives of the operation. The Reform focuses on increasing mastery of the curriculum by students and promoting mother tongue multi-lingual education. The Department of Education developed programs targeting specific disadvantaged groups, including programs serving children with disabilities. The objectives of the World Bank 2010-2013 Country Assistance Strategy (active at preparation) were to improve public service delivery and governance with specific focus on better access and quality of basic education services. The Country Partnership Strategy (2015-2019) (current at closure) focuses upon empowerment of the poor and vulnerable and maintains a focus on quality education services. The objectives advanced in the operation were also aligned with the Bank’s Education Sector Strategy that emphasizes Learning for All and the messages from the 2018 Development Report. Thus, the objectives were highly relevant to country context and Bank strategies throughout the life of the operation, as early learning is a critical outcome of quality education services. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve grades 1 to 3 reading and math skills in target regions and schools with special focus on disadvantaged groups Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Rationale Improvements in education quality were needed to increase student learning. The theory of change focused on early grade literacy and numeracy skills, since these foundational skills were lacking in many students. The project addressed numeracy and literacy skills through a comprehensive set of interventions such as: training teachers, providing instructional guidelines and reading materials, giving follow-up support through coaching and monitoring for teachers, and assessing the learning of students. The project implemented EGRAs and EGMAs as diagnostic tools. Importantly, the project connected the instructional and assessment elements via teacher training. The theory of change was logical and based on literature connecting important aspects of student learning. It recognized that teachers and their training alone cannot improve student learning; an enabling environment of the education system is also needed. Thus, the project simultaneously addressed and supported aspects such as instructional leadership via principal and head teacher training; tools and incentives to monitor student learning via performance-based bonuses; a results-based performance management system, and SRCs. Since financial management was an issue in the education system, the project also supported this aspect. Finally, studies and analysis of disaggregated data were planned as a feedback loop regarding programming for disadvantaged students. An underlying assumption was that these programs were adequate, or that deficiencies would be corrected with newly collected knowledge and data. The project’s theory of change also supported the Department of Education to develop assessment tools in mother tongue languages so that data were appropriate to many of the groups of disadvantaged students. It also supported linguistic maps to better understand needs. However, the results chains between improved student learning and the results-based performance management system, SRCs, and financial management were not well developed. Part of the theory of change centered on incentivizing performance through DLIs. However, some of DLIs (specifically DLIs 6- 9) lacked measurement specificity. Specific DLIs included: · DLI 1 [US$23.25m]: Versions of EGRA and EGMA tools developed for each selected mother tongue in each targeted region. · DLI 2 [US$90m]: Teachers from identified schools trained to improve their capability to effectively teach grades 1-3 reading and math. · DLI 3 [US$30m]: School principals/head teachers from identified schools trained to strengthen field-level support systems for improved early grade reading and math teaching. · DLI 4 [US$12m]: Development of an enabling policy environment for the effective implementation of grades 1-3 reading and math programs in the target regions. · DLI 5 [US$25m]: Linguistic map for each targeted region developed for improved targeting and implementation of the grades 1-3 reading and math programs. · DLI 6 [US$30m]: Capacity of Department of Education to implement its performance incentive scheme improved. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) · DLI 7 [US$30m]: Development of an enabling policy environment for the use of SRCs in the target regions. · DLI 8 [US$22.5m]: Financial management capacity of Department of Education central, regional, and division offices improved. · DLI 9 [US$12.5m]: Improvement of data management for educational programs serving the target disadvantaged groups in the target regions. · DLI 10 [US$24m]: Comprehensive review completed on selected Department of Education educational programs serving the target disadvantaged groups in the target regions. Outputs: By project closing, project activities had reached 14,397 elementary and secondary schools with 150,584 teachers and 4,377,208 learners in the five target regions. The Department of Education introduced a blended approach to teacher training that combined the original cascade training with school-level activities through School Learning Action Cells (SLACs). The new plan allowed smaller groups of teachers from each school to participate in division-level training and then return to their schools and deliver training to other teachers through the SLACs. This was envisioned to improve teacher quality. The project trained 34,074 teachers in grades 1-3, which nearly met the target (35,994 teachers) for this DLI. The 10-day training focused on recognized methods to improve early grade reading and math. The modality of training was a cascade training for principals and teachers who attended a divisional or regional training and then transmitted to colleagues at their school. During the DLI verification process, teachers in 400 randomly selected schools were interviewed and reported the training was relevant and useful. Teachers reported “they were introduced to different strategies in teaching beginning reading that they had not encountered in their pre-service training." They reported using the knowledge to develop materials for teaching literacy and numeracy and believed they were better in assessing reading difficulties and developing appropriate reading interventions (ICR, p. 19). The ICR also noted that the SLACs permitted follow-up training, which research shows is needed for training to be effective. 9,865 principals or head teachers in project-supported schools also received training in early literacy and math to support efforts by teachers, meeting the target (9,865). The target for this DLI was changed during the restructuring, as the Enhanced Basic Education Information System revealed fewer principals in the five regions than originally estimated. Including principals in the training is important for training programs, as instructional leaders need to support teachers in their application of training. School leaders were positive about the usefulness of the training (ICR, p. 19). SRCs were developed for all project schools. The ICR noted that schools were required to use the EGRA and EGMA data to share with communities/parents. The Department of Education issued a memorandum mandating that all schools present SRCs to their communities. Nearly 30 percent of all schools were trained in the use of the SRCs. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Mother tongue tools for EGRA and EGMA were developed. Prior to the project, there were no appropriate language tools, and during the project 50 tools were developed, meeting the target (50), representing 10 tools for each of the project regions. This was a DLI. The ICR noted that the tools were validated and developed for five mother tongue languages: Bikol, Chavacano, Ilokano, Sinugbuanong Binisaya, and Waray (p. 18). Several policies were formulated by the Department of Education to strengthen early literacy and numeracy and teacher induction policy. A policy framework and reforms were designed to address poor reading comprehension of learners in early grades. The framework specified that data from the EGRA and EGMA should be used to identify teacher training needs and formulate plans to improve teaching and learning outcomes. The ICR reported improved data and information management of the programs serving disadvantaged groups. The ICR also noted a delay in “developing the information system based on a strategy” (p. 41). The Department of Education developed a new financial management operations manual, and 126,000 elementary and secondary financial management staff were trained on the manual. 921 regional and division office accountants, budget officers, and bookkeepers were trained. The ICR (p. 23) reported that trainees noted that “the content was helpful, and the system was easy to use." 13,612 staff were trained on the performance management system (based on DLI sample verification among 400 schools). Training in performance management occurred across the five target regions, their 47 division offices, and 14,021 schools. The ICR (page 26) stated that the DLI related to capacity to implement the performance incentive scheme was considered to be fully achieved because the sample found more staff trained than the target, and the mechanism was an efficient way to award performance bonuses. The system set out specific criteria, mechanisms, and processes for performance target setting, monitoring, evaluation, and development planning for schools and offices. The Performance Management Committee was established as an appeals committee. This system included a defined feedback mechanism to effectively and efficiently ensure that appeals were well documented and properly addressed. The ICR noted that most issues related to the Performance Based Bonuses (p. 30). Linguistic maps were developed to improve targeting of programs. The ICR (page 20) noted that “teachers had become more responsive to the linguistic needs of children." The maps were used by school administrators in their allocation of resources, including teachers and materials, and by the Department of Education to assign new teachers to schools. Four programs serving disadvantaged students were reviewed. The ICR did not discuss what was found in the review, nor how the information was used to strengthen the programs going forward. Outcomes: Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) EGRA and EGMA were administered on a sample of project-supported schools. The sample comprised first grade in school year 2015-2016, second grade in 2016-2017, and third grade in 2017-2018. Different schools and students were sampled each year, based on a random selection of regions (i.e. level of accessibility and remoteness) (ICR, p. 15). The project team clarified that this purposeful sample was designed in this manner to address the DLIs and capacity of the Department of Education. For this reason, a sample design that would have provided longitudinal student data was rejected. Students scoring zero in comprehension (oral reading comprehension) in EGRA using the mother tongue version decreased from 81 percent in 2015 to 66 percent in 2019, which exceeded the target (69 percent, or 15 percent decrease). Students with a score of at least 60 percent in EGMA (number decimation) in the mother tongue version increased from 36 percent in 2015 to 54 percent in 2019, which exceeded the target (51 percent). Students from disadvantaged groups with scores of zero in oral reading comprehension in EGRA mother tongue version decreased from 86 percent in 2015 to 31 percent in 2019, which exceeded the target (77 percent or 9 percent decrease). Students living in remote or difficult-to-access areas with scores of zero in comprehension in EGRA mother tongue version decreased from 92 percent in 2015 to 61 percent in 2019, which exceeded the target (83 percent or 8 percent decrease). Students in schools with high concentration of indigenous people with scores of zero in EGRA using the mother tongue version decreased from 81 percent in 2015 to 55 percent in 2019, which exceeded the target (72 percent or 9 percent decrease). Students from disadvantaged backgrounds with a score of at least 60 percent in EGMA in the mother tongue version increased from 23 percent in 2015 to 62 percent in 2019, which exceeded the target (32 percent or 9 percent increase). Students living in remote areas with a score of at least 60 percent in EGMA increased from 13 percent in 2015 to 85 percent in 2019, which exceeded the target (22 percent or 9 percent increase). Students in schools with high concentration of indigenous people with a score of at least 60 percent in EGMA increased from 33 percent in 2015 to 55 percent in 2019, which exceeded the target (42 percent or 9 percent increase). Students from the designated disadvantaged groups showed similar gains as their non-disadvantaged peers. The ICR (p. 22) also reported that “girls scored higher than boys on all nine EGRA sub-tasks, and on the EGMA sub-tasks girls consistently scored higher than boys in all tasks except number discrimination (boys scored 0.33 percentage points higher) and word problems (boys scored 0.04 percentage points higher)." The project team also reported that between results from Round 3 and Round 1, there was a 46.75 percentage point decrease of pupils with zero scores in the EGRA Oral Reading Comprehension subtest of the EGRA from the baseline (from 78.86% to 32.11%). There was a 54.46% decrease of pupils with zero scores in EGRA Oral Reading Comprehension among disadvantaged pupils (from 82.88% to 28.42%). There was a 54.29 percentage point increase of pupils with at a score of at least 60% in the EGMA Number Discrimination subtest from the baseline (from 35.92% to 90.21%). There was a 63.41 percentage point increase from the baseline of pupils with at least a score of 60% in the EGMA Number Discrimination subtest among disadvantaged pupils (from 28.84% to 88.25%). Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Across the target regions, average performance increased between 2015 to 2017 for all subtasks -- particularly letter name knowledge (20.8% in 2015 to 52.6% 2017), oral reading fluency (words per minute from 6.5 in 2015 to 44.0 in 2017), and dictation (from 17.2% correct in 2015 to 52.8% correct in 2017). The percentage of students who could provide no correct scores on listening comprehension decreased from 16% in 2015 to 12% in 2017, and in oral reading comprehension from 81% in 2015 to 32% in 2017. Scores on these two indicators were worse for remote schools, but the gap narrowed between 2015 and 2017 (ICR, p. 50, no exact data provided). Increasing EGRA and EGMA scores were found by the United States Agency for International Development (USAID) in testing in La Union (Region 1), Ilocos Norte (Region 1), Illocos Sur (Region 1), Cebu (Region VII), and Bohol (Region VII). Since these regions do not overlap with the Bank-supported project, but supported similar interventions as this project, the ICR suggested that similar effects were likely attained by this project. The effect sizes attained in the USAID projects ranged from 0.15 (oral fluency) to 0.47 (reading comprehension) (ICR, p. 18). Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Gains in reading and math scores were found for students in the target regions, schools and disadvantaged groups. The increase was similar among disadvantaged and non-disadvantaged students. However, targets were only modestly ambitious, and there was no control group. The targets for nine out of the 10 DLIs were met, with the unmet one related to the number of teachers trained. Efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis was not conducted during preparation. Instead, the established links between early foundational skills and later earnings were presented in the appraisal document and completion report, which provided a solid justification for the investment. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) A cost-benefit analysis estimated the private returns to schools comparing the benefits of project-supported interventions with project costs. First, the effect sizes from international evidence were presented; effects on learning outcomes ranged from 0.15 to 0.47 standard deviation increase. Then, this effect was assumed to have an impact upon earnings (2.55 to 10.34 percent) (ICR. p. 50). The ICR acknowledged that the estimate was not based on actual benefits from the project and did not disaggregate by gender or disadvantaged group (p. 53). Finally, the private returns were estimated to be 9.8 percent based on local data and were expected to result in increased earnings ranging from 2.6 percent (low estimate) to 19.6 percent (high estimate). The net present value ranged from US$592 million to US$4,548 million. The analysis did not take into account investments in SRCs or financial management. The ICR provided a number of other metrics suggesting the investment provided value for money. Efficiency was attained in relation to project unit costs for training. Project costs were consistent with government norms, ranging from per person US$180 to US$486 (ICR, p. 55). Project training cost per teacher was on average US$338.40 per participant for the 10-day national training and US$142.40 for the school-based modality. In addition, the government rolled the training to all 17 regions beyond the targeted regions of the operation. The Department of Education's financial requirements became less cumbersome to schools, thus freeing school principals’ time for teacher support. For its improved financial management, the Department of Education received a Good Practice Award from the government, for "resolving recurrent issues in project implementation” (ICR p. 23). The project fully disbursed US$295.20 million (98.4 percent), as only US$4.80 million was cancelled for not fully meeting one DLI. The ICR suggested that the Department of Education was transparent in its mechanism to provide performance-based bonuses. The ICR also indicated some moderate implementation inefficiencies. Issues included turnover in personnel (ICR, p. 25) and lagging disbursements (p. 26). These implementation inefficiencies did not produce significant impacts or delays. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  0  Not Applicable 0 ICR Estimate  0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Relevance of objectives is assessed across the entire project and is rated High. Efficacy is rated Substantial overall. Efficiency is also rated Substantial across the entire project. Thus, the outcome is rated Satisfactory, indicative of only minor shortcomings in the operation’s preparation and implementation. a. Outcome Rating Satisfactory 7. Risk to Development Outcome There are several factors that help mitigate the risk to the development outcome. Activities supported under the project have been institutionalized by the Department of Education, including diagnostic tools, policies and guidelines for early literacy and numeracy, and financial management manuals. One factor that has assisted in the institutionalization of the interventions is that they were anchored in the Basic Education Sector Reform Program. The early literacy training program and school-based modality used in this project are being used nationally. SRCs have been rolled out nationally. The financial management operating manual continues to be used throughout the education system for both project and non-project schools. The government has increased public investment in education in the past eight years, which suggests that supported interventions will be sustained. Moreover, the current project supported by the World Bank continues these financing interventions. 8. Assessment of Bank Performance a.Quality-at-Entry Preparation had a number of strengths. Design built upon evidence and lessons, for example, the use of the independent verification agent and solid logic for project interventions. The results framework and indicators were well developed. The DLI process incorporated appropriate rigor, and reimbursement was based on evidence that the minimum eligible expenditure program was incurred by the government within 15 months of submission. Risks such as low capacity, governance, and implementation structure were appropriately identified during entry; however, the ICR noted that in addition to the measures that were developed, technical assistance would have been beneficial (p. 24). The ICR pointed out some shortcomings at entry. The Bank team did not adequately anticipate context-specific challenges to operationalizing EGRA and EGMA or analyze the data (see Section 9). These challenges had to be addressed during restructuring. Some issues with the DLIs were not anticipated; specifically, a lengthy process for the Department of Education and Ministry of Finance to submit withdrawal applications, and the specific time periods for achievement of Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) DLIs, created issues. Given that most of the disbursements were contingent upon verification (62 percent), these factors created implementation delays. Finally, without a comparison group, the data did not demonstrate attribution of observed outcomes to Bank-supported interventions. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision proactively addressed implementation issues. The Mid-term Review led to modifications to the definition of disadvantaged groups to ensure appropriate measurement of indicators. For example, the Department of Education believed EGRA and EGMA were not appropriate assessments for children with disabilities. Eligible expenditure programs were also simplified during implementation, and the restructurings helped reduce lags in reimbursements. The definition and timing of the DLIs were changed to resolve challenges. The Bank team provided comments to improve verification reports so that they were responsive to the required disbursement rules and verification processes. The team also ensured that the data was disagreated by disadvantaged groups by triangulating other data from the government's conditional cash transfer program and school-level data. The Bank team continuously monitored development impact. Task team leaders (TTLs) used supervision ratings to signal needed changes. For example, ratings were lowered until data issues with monitoring and evaluation were addressed to permit appropriate disaggregation by disadvantaged groups. There were minor shortcomings. Over the course of the operation, there were four TTLs. The transitions between TTLs were not always smooth, as there was lack of continuity of opinions among TTLs in relation to project deliverables and DLIs. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) The DLIs and indicators in the results framework were a mix of input, output, and outcome indicators that reflected the project's logic and achievement of the objective. EGRA and EGMA were a core part of monitoring and evaluation, as well as overall project interventions. However, targets were set low for improvements in EGRA and EGMA. Some DLIs lacked specificity in measurement (i.e. Capacity of DepEd to implement its performance incentive scheme improved; Development of an enabling policy environment for the use of SRCs in the target regions; Financial management capacity of DepEd central, regional, division offices improved; Improvement of data management for educational programs serving the target disadvantaged groups in the target regions). The institutional arrangements for data collection, reporting, analysis were the responsibility of the Department of Education and the independent verification agent. b. M&E Implementation Data collection surrounding EGRA and EGMA encountered challenges. There was a delay in collecting baseline EGRA and EGMA data, as it took longer to develop the mother-tongue versions for EGRA and EGMA. Data collection involved a tablet to capture the student’s responses in these two tools. The first round only collected data related to one disadvantaged group (children living in remote areas). Thus, results disaggregated by disadvantaged groups were initially not available, but in subsequent years became available with collection of additional student information. The Department of Education believed EGRA and EGMA were not appropriate to assess children with disabilities. The Department of Education developed a separate assessment to measure Grade 3 student performance, believing it could be adapted for students with mild disabilities. These data were not reported in the ICR, as this group was removed from indicators and monitoring. Monitoring and evaluation also comprised other sources of data. There were independent assessments of programs and policy implementation. For example, aspects highlighted in one assessment were comprehensive language mapping with widest possible use of languages and inclusion of elders in the selection committee for recruiting mother-tongue teachers (ICR, p. 28). The DLI verification process constituted another source of data by providing quantitative verification and qualitative assessment of SRCs and training received by principals and teachers in a sample of schools. c. M&E Utilization Monitoring and evaluation data were used by the Department of Education. Data from EGRA and EGMA helped refine mother-tongue policies. Early Language, Literacy and Numeracy Assessments incorporated domains from both the EGRA and EGMA tools to provide literacy and numeracy data. Moreover, the verification informed the Department of Education’s own verification process and helped it understand how to better identify issues and provide assistance in the field. Project monitoring and evaluation data have also been used by schools. EGRA and EGMA data were used by teachers to help develop interventions for Indigenous students. Principals used language maps in their hiring of teachers. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) M&E Quality Rating Substantial 10. Other Issues a. Safeguards During preparation, two safeguard policies were triggered: Environmental Assessment (OP 4.01) and Indigenous Peoples (OP 4.10). The project was rated Category C for Environmental Assessment, as the project did not finance civil works. An environmental assessment was not required. The Indigenous Peoples safeguard policy was relevant, as the project benefited this population group. The government updated the existing Indigenous Peoples Planning Framework. This policy was monitored during implementation. Given that the project's results framework specified disaggregated data related to this group, this was an additional level of monitoring conducted by the project. The Department of Education built a database related to mother-tongue materials and teacher training. A language map was developed during the operation. Safeguard ratings were consistently satisfactory. b. Fiduciary Compliance The project used existing government accounting systems. Risks such as limited capacity, absence of an integrated financial management manual, and unclear policies and processes were identified during preparation. For this reason, a DLI was established to support improved capacity. During the project, the Department of Education developed a manual for financial management aligned with appropriate accounting policies. The manual was posted on the website of the Department of Education. The Department of Education also trained staff in all schools in proper financial management strategies. For its improved financial management, the Department of Education received a Good Practice Award from the government, for "resolving recurrent issues in project implementation” (ICR, p. 23). Ratings for financial management improved over the course of the operation from moderately unsatisfactory to moderately satisfactory. The Bank team worked with the government to improve disbursements by modifying DLI periods and simplifying eligible expenditure programs, ensuring that the government could seek reimbursement for achievements at the time they were realized. No significant issues were noted in relation to eligible expenditures. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Procurement was rated moderately satisfactory or satisfactory during the life of the operation. Procurement followed government procedures that were acceptable to the Bank and based on eligible expenditures. There was a delay in procurement of the contracting of the DLI verification agent. c. Unintended impacts (Positive or Negative) None reported. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons IEG has synthesized the lessons presented in the ICR to the following: Using "within budget" financing to support improvements in the education system can also support improvements in country systems. This operation employed disbursement-linked indicators (DLIs) to finance a mix of activities, outputs, and outcomes. The DLIs supported appropriate policies for improving early reading and numeracy. These policies have been adopted and institutionalized. Appropriate design of DLIs is an important issue for TTLs. In the case of this project, a clear understanding of how country systems operated was lacking during preparation, resulting in limited flexibility for the government, implementation delays, and the need to address during restructuring. The use of a verification process is an important complement of DLIs. Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Follow-up support is critical component to teacher training. One-off training is ineffective. For this reason, the Department of Education introduced a blended approach to teacher training that combined the original cascade training with school-level activities through School Learning Action Cells (SLACs). The new plan allowed smaller groups of teachers from each school to participate in division-level training and then return to their schools and deliver training to other teachers through the SLACs. This was envisioned as a method of improving teacher quality; however, teachers still reported a need for mentoring or follow-up support, which the literature shows is critical. Including student learning in a project development objective is feasible. The project used internationally recognized tools to measure reading and math improvement, since tracking improvements is critical. The project demonstrated that it is possible to improve learning outcomes, and thus, feasible to have an ambitious PDO (i.e. improved student learning). However, an evaluation with a control group is needed to attribute observed outcomes to project interventions. In addition, monitoring and evaluation must also clearly define and measure target groups. 13. Assessment Recommended? No 14. Comments on Quality of ICR The quality of the ICR was substantial. The ICR was well written. It provided a concise description of the project, its activities, and the factors affecting implementation. It contained analysis of shortcomings in design, preparation, implementation, and Bank performance. The annex provided a detailed explanation of the efficiency analysis and additional data. The ICR reported on all indicators and supplemented its analysis with additional data and evidence. It did not, however, provide much information about the contribution of capacity building efforts to decentralized-level staff or discuss the use of SRCs or the performance-based bonus mechanism. Unclear aspects or those not specified in the ICR were clarified with the project team. a. Quality of ICR Rating Substantial Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Learning, Equity and Accountability Prog (P118904) Page 17 of 17