Document of The World Bank Report No: ICR00003000 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-41550 IDA-50800 IDA-H2080) ON A CREDIT IN THE AMOUNT OF SDR 3.6 MILLION (US$ 5.0 MILLION EQUIVALENT) AND GRANT IN THE AMOUNT OF SDR 3.6 MILLION (US$ 5.0 MILLION EQUIVALENT) AND ADDITIONAL CREDIT IN THE AMOUNT OF SDR 0.7 MILLION (US$1 MILLION EQUIVALENT) TO THE REPUBLIC OF MOLDOVA FOR THE QUALITY EDUCATION IN RURAL AREAS OF MOLDOVA PROJECT May 26, 2014 Human Development Sector Unit Belarus, Moldova and Ukraine Country Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective April 15, 2014) Currency Unit = Moldovan Leu (MDL) MDL 1.00 = US$ 0.07 US$ 1.00 = MDL 13.42 January 1 – December 31 ABBREVIATIONS AND ACRONYMS AF Additional Financing AFQERM Additional Financing to the Quality Education in Rural Areas of Moldova CAS Country Assistance Strategy CEE Central and Eastern Europe CIS Commonwealth of Independent States CPS Country Partnership Strategy DAMEP Data Analysis, Monitoring and Evaluation Policies Unit DO Development Objective ECA Europe and Central Asia EGPRS Economic Growth and Poverty Reduction Strategy EMIS Education Management Information System EU European Union GDP Gross Domestic Product GoM Government of Moldova ICB International Competitive Bidding IDA International Development Association IEA International Association for the Evaluation of Educational Achievement IMF International Monetary Fund IP Implementation Progress ISR Implementation Status and Results Report MDL Moldovan Leu MDG Millennium Development Goal M&E Monitoring and Evaluation MoEd Ministry of Education MoF Ministry of Finance MTEF Medium-Term Expenditure Framework MTR Mid-Term Review NBS National Bureau of Statistics NGRAC National Grant Review and Approval Committee NPV Net Present Value OECD Organization for Economic Co-operation and Development PAD Project Appraisal Document PDO Project Development Objective PHRD Japan Policy and Human Resources Development Fund PISA Program for International Student Assessment PPP Purchasing Power Parity PR Public Relations i PSF Per-student Based Financing QAA Quality Assurance Agency QEA Quality at Entry QERM Quality Education in Rural Areas of Moldova QSA Quality of Supervision RF Results Framework RGRAC Regional Grant Review and Approval Committee SIL Specific Investment Loan TIMSS Trends in International Mathematics and Science Study TTL Task Team Leader Vice President: Laura Tuck Country Director: Qimiao Fan Sector Manager: Alberto Rodriguez Project Team Leader: Andrea Guedes ICR Team Leader: Anna Olefir ii CONTENTS A. Basic Information...................................................................................................... iv B. Key Dates .................................................................................................................. iv C. Ratings Summary ...................................................................................................... iv D. Sector and Theme Codes ........................................................................................... v E. Bank Staff ................................................................................................................... v F. Results Framework Analysis ..................................................................................... vi G. Ratings of Project Performance in ISRs ................................................................... ix H. Restructuring (if any) ................................................................................................. x I. Disbursement Profile .................................................................................................. xi 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 6 3. Assessment of Outcomes .......................................................................................... 11 4. Assessment of Risk to Development Outcome......................................................... 18 5. Assessment of Bank and Borrower Performance ..................................................... 19 6. Lessons Learned ....................................................................................................... 20 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 21 Annex 1. Project Costs and Financing .......................................................................... 22 Annex 2. Outputs by Component ................................................................................. 23 Annex 3. Economic and Financial Analysis ................................................................. 32 Annex 5. Beneficiary Survey Results ........................................................................... 43 Annex 6. Stakeholder Workshop Report and Results................................................... 46 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 47 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 58 Annex 9. List of Supporting Documents ...................................................................... 59 Map: IBRD 33448 FIGURES Figure 1: Benefits of the Quality Grants ........................................................................... 26 Figure 2: Teachers’ and Students’ Perceptions Regarding the Frequency of Use of Equipment and Teaching Materials Obtained under Equity Grants ................................. 27 Figure 3: Savings from the Structural Education Reform Program .................................. 36 Figure 4: Student-teacher Ratio and Average School Size in Moldova, 1991-2013 ........ 36 Figure 5: Trends in primary and secondary education spending in 2009-2012 ................ 37 Figure 6: Employed Persons Aged 15-24 with Upper Secondary Education in Moldova: Occupation versus Field of Study, 2007-2012 .................................................................. 39 Figure 7: Share of Unemployed Persons Aged 15-24 with Upper Secondary Education in Moldova, 2007-2011 ......................................................................................................... 39 TABLES Table 1: Public Expenditure on Education in Moldova, 2007-2013 ................................ 34 Table 2: Correlations between Proportional Budget Change and Variables Related to Cost per Student or Equity ........................................................................................................ 38 iii A. Basic Information Quality Education in Country: Moldova Project Name: the Rural Areas of Moldova IDA-41550, IDA- Project ID: P090340 L/C/TF Number(s): 50800, IDA-H2080 ICR Date: 05/26/2014 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Borrower: MOLDOVA Original Total USD 10.00M Disbursed Amount: USD 11.87M Commitment: Revised Amount: USD 10.82M Environmental Category: C Implementing Agencies: Ministry of Education of the Republic of Moldova Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/26/2005 Effectiveness: 08/07/2006 08/07/2006 05/27/2010; 03/20/2012; Appraisal: 10/06/2005 Restructuring(s): 05/17/2012; 12/13/2013 Approval: 03/14/2006 Mid-term Review: 05/30/2008 11/18/2009 Closing: 05/31/2010 11/30/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Low Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Satisfactory Unsatisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Moderately Overall Borrower Performance: Performance: Moderately Satisfactory Unsatisfactory iv C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Unsatisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 35 35 Primary education 17 17 Secondary education 28 28 Sub-national government administration 20 20 Theme Code (as % of total Bank financing) Administrative and civil service reform 14 14 Decentralization 14 14 Education for all 29 29 Education for the knowledge economy 14 14 Rural services and infrastructure 29 29 E. Bank Staff Positions At ICR At Approval Vice President: Laura Tuck Shigeo Katsu Country Director: Qimiao Fan Paul G. Bermingham Sector Manager: Alberto Rodriguez Maureen Anne McLaughlin Project Team Leader: Andrea C. Guedes Ana Maria Parchuc De Jeria Figueroa ICR Team Leader: Anna Olefir ICR Primary Author: Anna Olefir v F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The main objective of the Project is to increase the quality of education in rural areas in Moldova. Revised Project Development Objectives (as approved by original approving authority) The objective of the Project is to assist the Recipient in improving the quality of education in rural areas in Moldova. (a) PDO Indicator(s) Original Target Actual Value Values (from Formally Revised Achieved at Indicator Baseline Value approval Target Values Completion or documents) Target Years Indicator 1: Increase in average learning achievement of students in rural schools Increasing the overall average and subjects’ examinations averages Overall weighted avg. Overall weighted from the baseline of 7.13 avg. 6.25, 2010/2011 that is: Romanian 6.80 Increase by (8% Romanian 6.08 Value Overall weighted avg. Russian 6.48 cumulative) from the Russian 6.89 (quantitative or 7.01 Math 6.87 2005/2006 baseline Mathematics 5.42 qualitative) Romanian 6.76 Geography 7.10 Geography 6.26 Russian 7.15 Languages 7.39 Languages 6.29 Mathematics 7.03 Physics 7.29 Physics 5.77 Geography 7.00 Languages 7.06 Physics 6.99 Date achieved 10/09/2006 05/31/2010 03/20/2012 10/31/2013 Not achieved. Due to an overhaul in the administration of the Baccalaureate exams in Comments 2010, the baseline was changed from 2005/2006 to 2010/2011 (under the Additional (incl. % Credit approved by the Board on March 20, 2012). However, in subsequent years, the achievement) Ministry of Education introduced further changes in the exam’s application methodology, making exams not comparable across years. The difference in achievement between secondary school students in rural and Indicator 2: urban schools, on average, is decreased. Overall weighted avg. 6.26 (rural)/ Overall weighted avg. At least 5% 6.24 (urban) 7.26 (rural)/ reduction over Romanian 6.08/6.08 Value 7.68 (urban) baseline in year 3 Mathematics (quantitative or Romanian 7.12/7.55 and subsequently at 5.55/5.37 qualitative) Math 7.02/7.58 least 5% reduction Geography Geography 7.36/7.82 over year 3 6.36/6.21 Physics 7.15/7.50 (10% cumulative) Physics 5.96/5.70 Date achieved 10/09/2006 05/31/2010 10/31/2013 While this indicator has been achieved, the Bank’s supervision and ICR teams have Comments serious misgivings about the data. In particular, the most recent data showed rural (incl. % students outperforming urban ones, which is highly unusual and raises red flags on the achievement) reliability of the results. vi (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1. Number of subject/grade level curriculum programs completed and disseminated Value (quantitative or 0 18 17 17 qualitative) Date achieved 03/14/2006 05/31/2010 04/29/2008 02/18/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 2. Number of new curricular guides for existing textbooks Value (quantitative or 0 18 17 17 qualitative) Date achieved 03/14/2006 05/31/2010 04/29/2008 02/18/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 3. Number of new curricular guides for new textbooks Value (quantitative or 0 18 17 17 qualitative) Date achieved 03/14/2006 05/31/2010 04/29/2008 02/18/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 4. Teachers trained in the use of the new curriculum in the 10 targeted rayons Value All 10th - 12th grade (quantitative or 0 100% 9,000 teachers teachers trained in qualitative) new curriculum Date achieved 03/14/2006 05/31/2010 04/29/2008 05/12/2011 Comments End of Project target achieved. Target was 9,000 but 8,284 teachers is the total (incl. % number of teachers in the 10 targeted rayons that required training and received it. achievement) Indicator 5. Percent of students in rural schools who have new textbooks for grade 12 mathematics, biology, chemistry and physics Value (quantitative or 0 100% 100% 100% qualitative) Date achieved 03/14/2006 05/31/2010 04/29/2008 11/02/2011 Comments (incl. % End of Project target achieved. achievement) Indicator 6. International assessment administered and disseminated Public Moldova participated Value No participation in PISA disseminations of in PISA 2009 Plus (quantitative or and TIMSS international results (Newspapers and disseminated qualitative) assessments report results) results publically Date achieved 03/14/2006 05/31/2010 05/10/2013 Comments End of Project target achieved. In 2013, Moldova also registered for PISA 2015. vii (incl. % achievement) Indicator 7. Number of schools that were equipped through equity grants USD 0.5 million Value disbursement for (quantitative or 0 1,176 schools 1,190 schools Rural School Equity qualitative) and Quality grants Date achieved 03/14/2006 05/31/2010 04/29/2008 02/18/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 8. Number of lyceums that were equipped through quality grants USD 0.5 million Value disbursement for (quantitative or 0 300 schools 304 schools Rural School Equity qualitative) and Quality grants Date achieved 03/14/2006 05/31/2010 04/29/2008 02/18/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 9. Utilization and user-satisfaction assessment report on equipment 'Satisfactory' Value according to surveys, (quantitative or None Satisfactory focus groups, qualitative) classroom observations Date achieved 03/14/2006 05/31/2010 10/25/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 10. Quality of Rural School Quality Grant proposals 'Good' according to Value criteria established by (quantitative or N/A (No proposals) Very good Good Rayon Evaluation and qualitative) Approval Committees Date achieved 03/14/2006 05/31/2010 04/29/2008 10/25/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 11. Implementing formula-based funding Formula-based Value Formula piloted 100% funding is (quantitative or No formula in two rural (nationwide) implemented qualitative) rayons nationwide Date achieved 10/09/2006 05/31/2010 04/29/2008 05/10/2013 Comments (incl. % End of Project target exceeded. achievement) Indicator 12. Percent of schools reorganized in the two pilot rayons Value (quantitative or 0 30% 100% qualitative) Date achieved 03/14/2006 05/31/2010 10/25/2010 viii Comments (incl. % End of Project target achieved. achievement) Indicator 13. Number of staff trained in education policy making (central and local level) 90: Value 90: 20 (central (quantitative or 0 100% of total staff 20 (central level) and level) and 70 qualitative) 70 (local level) (local level) Date achieved 03/14/2006 05/31/2010 04/29/2008 10/25/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 14. Number of staff trained in IT (central and local level) 90: Value 90: 20 (central (quantitative or 0 100% 20 (central level) and level) and 70 qualitative) 70 (local level) (local level) Date achieved 03/14/2006 05/31/2010 04/29/2008 10/25/2010 Comments (incl. % End of Project target achieved. achievement) Indicator 15. Annual education report prepared and disseminated by January 30 next year Value Reports prepared and (quantitative or 0 1 disseminated annually qualitative) Date achieved 03/14/2006 05/31/2010 03/27/2008 Comments (incl. % End of Project target achieved. achievement) G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 05/16/2006 Satisfactory Satisfactory 0.00 2 11/09/2006 Satisfactory Satisfactory 0.50 3 02/26/2007 Satisfactory Satisfactory 0.50 4 04/02/2007 Satisfactory Satisfactory 0.50 5 05/30/2007 Satisfactory Satisfactory 1.00 6 05/13/2008 Satisfactory Satisfactory 3.65 7 11/21/2008 Satisfactory Moderately Satisfactory 4.40 8 06/19/2009 Moderately Satisfactory Moderately Satisfactory 5.83 9 07/29/2009 Moderately Satisfactory Moderately Satisfactory 5.83 10 02/19/2010 Moderately Satisfactory Moderately Satisfactory 5.93 11 04/02/2010 Moderately Satisfactory Satisfactory 5.93 12 12/08/2010 Satisfactory Satisfactory 7.24 13 05/28/2011 Satisfactory Satisfactory 8.11 14 11/23/2011 Satisfactory Satisfactory 8.40 15 06/20/2012 Satisfactory Satisfactory 11.12 16 12/30/2012 Moderately Unsatisfactory Satisfactory 11.41 17 06/18/2013 Moderately Unsatisfactory Satisfactory 11.71 18 11/23/2013 Moderately Unsatisfactory Satisfactory 11.84 ix H. Restructuring (if any) The Project was restructured four times, as described in the table below. Restructuring Board ISR Ratings at Amount Reason for Restructuring & Key Date(s) Approved Restructuring Disbursed at Changes Made PDO DO IP Restructuring Change in USD millions 05/27/2010 N MS S 5.93 Reallocation of the proceeds and extension of closing date. 03/20/2012 Y S S 8.40 Approval of Additional Financing in the amount of US$1 million equivalent; revision of PDO to align with that of the original Financing Agreement; revision of the Results Framework; and extension of closing date. 05/17/2012 N S S 11.12 Extension of closing date until effectiveness of the Additional Financing. 12/13/2013 N MS S 11.84 Partial cancellation of unused allocations from the Additional Financing (of a small amount of USD180,000). x I. Disbursement Profile Note: Disbursement profile automatically adds two “Education For All” Fast Track Initiative (EFA-FTI) grants of US$8.8 million that Moldova benefitted from in 2006-2010. These grants were not part of the QERM Project (P090340). xi 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Of all the transition economies in the region, Moldova had the hardest fall in the 1990s. A tentative revival in 1997 was squashed by the financial crisis in Russia, with devastating effects for Moldova’s export, the government budget, and debt. In addition to rising poverty resulting from the economic downturn in the 1990s, the country also suffered from territorial secession and armed conflict. Between 2000 and 2004, the economy began to recover: real GDP increased by more than 30 percent and the poverty rate declined by more than a half. However, the driving force behind this growth was increased consumption, fueled by rising real wages and workers’ remittances (some 25 percent of Moldova’s active population was working abroad). At 27 percent of GDP, remittances were reducing the incidence of poverty. Nevertheless, at the time of Project design, Moldova had the highest poverty rate in Eastern Europe and the Caucasus. The proportion of the population below the national poverty line was estimated at 49 percent in 2002, and the rural poor predominated, accounting for 68 percent of all poor people. 2. Education issues and GoM’s strategy. Moldova had expressed strong commitment to education reform and improving educational quality, access, and efficiency. This commitment was clearly formulated in the 2004 Economic Growth and Poverty Reduction Strategy (EGPRS), which also recognized and adopted the Millennium Development Goals (MDGs). The EGPRS encompassed four objectives for education: (i) to improve the quality of education; (ii) to increase access to education, particularly for children from vulnerable families; (iii) to raise the efficiency of management within the education sector; and (iv) to improve the integration of children with special educational needs and vulnerable children. The EGPRS was integrated with the Medium-Term Expenditure Framework (MTEF) for 2005-2007. 3. The plans were based on progress already made in modernizing education, much of it associated with the earlier Bank-financed General Education Project (P008558) implemented in 1997-2004. However, significant challenges in education still remained, such as:  Quality: Learning outcomes suggested that there were still problems with quality at the secondary level, but the results of the 2003 international assessment Trends in International Mathematics and Science Study (TIMSS) for grade 4 suggested that there was hope for improvement arising from the reforms in curriculum, textbooks, teacher training, and assessment that were underway: the relatively high proportion of 4th grade students demonstrating problem-solving capacity was particularly encouraging. However, the TIMSS report also showed that the shortage of resources for teaching materials and equipment was still acute. The low level of teachers’ salaries had an adverse effect on the quality of learning outcomes at all levels, making it difficult to attract good young teachers into the profession. Inequities in students’ outcomes reflected parental background as well as school quality, and 1 children from more educated and prosperous families and those in urban areas got better results than those from poorer and rural households.  Access: Moldova lagged behind many of its neighbors in terms of access to education. For primary and gymnasium levels (grades 1 to 9), the gross enrollment rate was 95 percent in 2003 – lower than rates in the European Union (EU) member states and candidate countries of Central and Eastern Europe (CEE), but higher than in other western Commonwealth of Independent States (CIS) countries. At the upper secondary level (grades 10 to 12), the overall gross enrollment rate of 42 percent was less than half that achieved by CEE EU members and well below that in other comparator countries. Disparities in enrollment rates also reflected the higher rate of dropouts of children from poorer and rural families. Children whose parents were working abroad also tended to absent themselves from school and drop out prematurely.  Efficiency: There was a downward trend in the number of young people of school age. Between 1990 and 2003, the number of 7- to 15-year-olds decreased by 20 percent. Between 2003 and 2010, the number of children in this age group was expected to continue falling (by 32 percent for the basic education age group) and a 22 percent fall was expected in the number of 16- to 18-year-olds. The fall in the number of school-age children was reflected in a decrease in the average student- teacher ratios over recent years (14.7 in primary and secondary schools), with worrying variations between schools. Likewise, average class size had also been falling. In sum, there were many small schools with small classes and low student- teacher ratios, particularly in rural areas. The large number of non-teaching relative to teaching staff and high heating costs, due in part to poor insulation of large school buildings, were some of the other issues adding to the concerns about low efficiency and high costs.  Public finance of education was in crisis, partly due to the methodology used to allocate budgets to schools. 4. Rationale for Bank involvement. The Bank’s support to the sector for the QERM Project was in line with the strategic and human development priorities outlined in Moldova’s Country Assistance Strategy (CAS) FY05-08, which focused on supporting interventions at the local and community levels in rural areas and small cities, which inter alia were aimed at improving the quality of and access to social services, including education services. The QERM Project supported the GoM’s education program, articulated in the 2004 EGPRS. The Project was also nested in the first strategy and action plan for the education sector (2006-2008) of the Ministry of Education (MoEd).1 1 The strategy and action plan was essentially a consolidation of the proposals made in various documents, particularly the EGPRS and the “Education for All National Action Plan,” but also the Moldova Village Program, and the Neighborhood Action Plan for European Integration; it also linked the overall plan for the sector to the costing of the MTEF. 2 The subsequent Additional Financing (AF) supported activities consistent with the Project Development Objective (PDO) and the Country Partnership Strategy (CPS) FY09-13. 1.2 Original Project Development Objectives (PDO) and Key Indicators 5. The original PDO was to increase the quality of education in rural areas of Moldova. The Project included two key outcome indicators: (i) an increase in the average learning achievement of students in rural schools; and (ii) a reduction in the gap in average achievement between rural and urban schools. 1.3 Revised PDO and Key Indicators 6. The PDO underwent an editorial change at the time of approval of the AF to ensure its consistency with the Project’s legal documents. The revised PDO was as follows: “The objective of the Project is to assist the Recipient in improving the quality of education in the rural areas in Moldova.” This change was strictly editorial and of no bearing to the Project’s objectives, thus the ICR uses the revised PDO to assess the project. 7. The key indicators were revised twice during the Project: first in April 2008, when baseline data became available; and again at the time of the AF. In 2008, the outcome indicators and targets were modified as follows: (i) Improving learning in schools from rural areas (Revised target: increasing the overall average and subject’s examinations averages by 5 percent instead of the original 8 percent from the 2005/2006 baseline); (ii) Difference in average results of secondary school students from rural and urban area is decreasing (Revised target: Reduction of the average discrepancy in the exams for students from rural and urban areas at least 2 percent a year (6 percent cumulative)). At the time of AF on March 20, 2012, the change to the baseline of the first outcome indicator from 2005/2006 to 2010/2011 was formally made due to overhaul of the Baccalaureate exam system in Moldova (Revised target: Increasing the overall average and subject’s examinations averages from the baseline of 2010/2011). 1.4 Main Beneficiaries 8. The Project focused on rural areas and its direct beneficiaries encompass the upper secondary students in rural areas of Moldova, as well as rural teachers and school directors as well as central and local education and finance authorities. At the same time, the Project had positive repercussions for beneficiaries in both rural and urban areas. 1.5 Original Components 9. The original Project consisted of four components: (i) Improving Teaching and Learning in Rural Schools; (ii) Increasing Access and Equity in Rural Schools; (iii) Increasing Efficiency in the Use of Resources; and (iv) Strengthening Education Planning and Monitoring. 3 10. Component 1: Improving Teaching and Learning in Rural Schools (US$2.86 million). The principal objective of this component was to improve the quality and relevance of teaching and learning in rural schools, in particular for the upper grades of the secondary education level (grades 10-12). This was meant to be achieved by strengthening the MoEd’s three core pedagogical systems: curriculum (upgrading and modernizing the curriculum and providing new educational materials and textbooks), teaching (professional development for teachers and staff in the structure and content of the new curriculum), and evaluation (objective, professional, and efficient evaluation, monitoring, and student assessment). This component encompassed three subcomponents: (i) Modernizing the Curriculum for Grades 10-12; (ii) Teacher Professional Development for Grades 10-12; and (iii) Establishment of National Centre for Examinations and Quality Assurance under the MoEd. 11. Component 2. Increasing Access and Equity in Rural Schools (US$5.15 million). The main objective of this component was to provide poor rural schools and communities with additional resources to improve the quality of and access to adequate education services. Another objective was to strengthen the capacity of schools to manage priorities, resources, and educational outcomes. These objectives were to be accomplished by providing additional resources to eligible schools, which in turn were required to manage them (including the needs identification and prioritization of a resource allocation strategy) in a transparent and efficient manner and to be accountable for results. This component comprised two subcomponents: (i) Rural Schools Equity Grants; and (iii) Rural Schools Quality Grants. 12. Component 3. Increasing Efficiency in the Use of Resources (US$1.96 million). The main objective of this component was to improve the efficiency of the system for allocating public money to schools and the use of resources within schools, in the context of the projected demographic trends (school-age population, class sizes, and student-staff ratios were falling and were expected to continue to decline). The component sought to build on the existing formula-based funding approach, with emphasis on using the signals that it generates to trigger policy action. This component was subdivided into three subcomponents: (i) Education Budgeting Improvement; (ii) Formula Funding and School Autonomy Pilots; and (iii) Grants for the Optimization of School Networks in Selected Rayons. 13. Component 4. Strengthening Education Planning and Monitoring (US$0.40 million). The main objective of this component was to build capacity at the MoEd to better manage the education system by strengthening its capacity to promote and monitor programs. The component’s objective was meant to be met through: (i) development of human resources of the education sector in policy and monitoring at the ministerial level; and (ii) improvement in education management and administration at the rayon level (i.e., rayon education administration departments). This component encompassed two subcomponents: (i) Policy and Monitoring Development; and (ii) Rayon Education Administration Improvement. 4 1.6 Revised Components 14. The original components were not modified during Project implementation. The AF supported additional technical assistance to facilitate reform efforts, as listed below:  Component 1 “Improving Teaching and Learning”: development of standards for receiving schools; 2 continued support for Moldova’s capacity to participate in international assessments; and a strengthened ability to monitor the quality of education through support to the Examination Agency.  Component 3 “Increasing Efficiency in the Use of Resources”: capacity building and training to education personnel at all levels for implementation of formula-based financing and school optimization; and capacity building for strengthening the school mapping database.  Component 4 “Strengthening Education Planning and Monitoring”: continued support to public relations (PR) activities to support the implementation of the school reform program. 1.7 Other Significant Changes 15. Revised Results Framework. The Results Framework (RF) underwent revisions during implementation: (i) In 2008, the RF was revised to reflect recently collected baseline data; (ii) intermediate indicators were revised during the October 2009 MTR; and (iii) at the time of the AF approval. In 2008 targets for outcome and output indicators were modified based on the baseline data obtained. Also, two output indicators were dropped: (i) improvement of test scores as measured by annual examinations at grades 4, 9, and 12, as they were not independently conducted assessments; and (ii) expenditure on quality-related line items as a percent of total school expenditure, due to the lack of data at the time. Two additional indicators related to new curriculum programs were also added. 16. The key intermediate outcome indicators were again revised during the MTR conducted in October 2009. In retrospect, while these modifications were appropriate, one of the intermediate indicators dropped during the MTR -- the empirical evaluation of teaching and learning on samples of 8th and 10th grade students -- could have been a good alternative for measuring Project outcomes. Finally, the MTR introduced third-party evaluations with a view to more completely evaluating outcomes (which were subsequently implemented, as shown in Annex 2). A final revision to the RF occurred with the approval of the AF in March 2012. Specifically, the baseline for the indicator “Improved learning in rural schools, as measured by average achievement scores in Baccalaureate examinations” was changed to account for modifications in the exam’s 2 Schools that receive students from closed or reorganized schools. 5 methodology; and (ii) there was an increase in the end of Project target for the intermediate indicator “Implementing formula based funding in two pilot rayons.” 17. Extension of the closing date. The Project was originally designed to be implemented over a four-year period but ended up having cumulative extensions adding to 42 months: (i) in May 2010, the Bank agreed with a 24-month extension until May 30, 2012, to make up for initial implementation delays; (ii) in March 2012, as part of the AF, the Board approved an 18-month extension, with a new closing date of November 30, 2013, to implement the additional activities; and (iii) in May 2012, the Project was extended by five months to allow time for the AF to become effective. 18. Additional Financing. The GoM requested the AF in the amount of US$1 million equivalent for technical assistance to scale up the activities initiated under the Project and to support implementation of its Structural Reform Program in the education sector, as well as to help avoid a financing gap between the Project and a follow-up operation. The GoM also asked for an 18-month extension of the closing date of the Project to accommodate the AF. The AF was approved by the Board on March 20, 2012. 19. Partial cancellation of the Additional Credit. Following the request from the Borrower dated November 28, 2013, the Bank partially cancelled US$180,000 equivalent from the Additional Credit (IDA-50800). 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design, and Quality at Entry 20. Soundness of background analysis. The Project was designed taking into consideration findings and recommendations obtained from analytical work conducted as part of Project preparation, earlier Bank and OECD work reviewing the Moldova education system, a series of studies financed by a Japan Policy and Human Resources Development Fund (PHRD) grant, and lessons learned from the implementation of the Moldova General Education Project. A social assessment was conducted as part of Project preparation, capturing the opinions and recommendations of teachers, school principals, and representatives of rayon education departments. There were also consultations with Moldovan decision makers, civil society, and donor partners to assess their commitment to improving the governance of service delivery through increased support of bottom-up approaches. 21. Risk assessment. The risks were appropriately identified and rated and adequate mitigation measures were put in place. 22. Results framework. The RF for the Project was weak on several fronts: (i) it had disconnect and attribution problems; (ii) it did not allow depiction of key important development outcomes achieved by the Project (e.g., improved efficiency of the education sector was not captured by the PDO-level indicators or the PDO itself); (iii) it 6 focused on narrowing down the performance gap between urban and rural areas although the Project contributed in significant ways to improving the quality of teaching and learning in Moldova as a whole, with positive repercussions well beyond rural areas; and (iv) it relied on only two outcome indicators, both derived from the Baccalaureate exam results (which has attribution, measurement and timing problems) as further discussed in section 3.2. 2.2 Implementation 23. Overall, the Project was implemented in a satisfactory way, with the interventions related to improving efficiency having a much broader scope and impact than was originally anticipated. 24. Major factors affecting implementation. Key factors that affected Project implementation encompass: i. Factors outside the control of government:  A significant period of political uncertainty. Project preparation and implementation cycles transcended four elected Parliaments, with a significant period of political uncertainty in 2009-2010 that affected the work of the government and the pace of reforms in the sector. There were also five Ministers of Education and five Executive Directors during Project implementation, which caused delays in implementation of some Project components, notably prior to the MTR. ii. Factors generally subject to government control:  Mounting fiscal pressures, particularly during the financial crisis of 2009. Government spending on education in 2009 reached 9.4 percent of GDP, way above the regional average of around 5 percent, putting the efficiency reforms in the education sector among the most urgent priorities for the country (and thus entailing a larger scope of work under Component 3, which supported “Increasing Efficiency in the Use of Resources”). Also, under these circumstances, the MTEF for the MoEd required a decreasing budget, and as a result prioritization of the education programs was challenging but the Government tackled it. iii. Factors generally subject to implementing agency control:  A very wide and politically sensitive agenda for the MoEd to tackle. Enhancing the quality of education and strengthening the Baccalaureate exam system was conducted in a time of tight fiscal pressures and politically sensitive school network consolidation reforms.  SDR to USD exchange rate fluctuations resulted in a higher actual amount of credit proceeds than originally planned in USD. This enabled procurement of school buses for transport of students in rural areas (to hub schools from closed or reorganized schools), an activity not originally envisaged under the Project. 25. Albeit with some delays, the government implemented politically and socially sensitive education reforms, including application of a per-student financing mechanism nationwide, substantial rationalization of the school network, and an overhaul of the Baccalaureate exam system. The GoM’s program and its objectives were both visionary and very much in line with the sector’s needs. In this sense, the Project’s support was 7 critical in helping maintain the momentum for the reform program and in continuing to provide the GoM with strategically placed technical assistance. 26. Mid-Term Review. An MTR was conducted in October 2009, around eight months before the Project’s original closing date. The DO and IP ratings at the time were Moderately Satisfactory. As mentioned above, as a result of political turmoil, there were delays in the implementation of some Project components prior to the MTR. After that, however, implementation remained steady for the last four years of the Project. 2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 27. Design. The Project’s M&E design was weak. The RF had disconnect and attribution problems. It did not fully convey the development hypothesis implicit in the Project or the cause and effect linkages between the intermediate results and the PDO. Thus, many Project achievements were not captured, including better efficiency of the education sector and creation of an improved learning environment in both urban and rural areas. Indeed, a significant part of the Project’s interventions, including Component 3 “Increasing Efficiency in the Use of Resources” (which accounted for 43 percent of actual Project costs), was not reflected in the outcome indicators. While most of the Project-supported interventions had national coverage, the PDO focused only on rural areas. 28. At the time of Project design and later during the preparation of the AF, the Bank's team opted to use Baccalaureate exam results as a proxy for the achievement of the PDO. In hindsight, this decision proved to be wrong on many fronts. Student assessment results are not a reliable indicator to assess this type of project, as there are generally attribution, measurement, and timing problems associated with this indicator. In terms of attribution, the ICR team is of the opinion that the PDO should have been defined in more constrained terms, referring to beneficiaries’ schools only and comparing the envisaged changes with a control group of comparable non-beneficiary schools. As to the measurement issue, Baccalaureate exams are not designed to measure quality improvements, but rather provide a gateway to university entrance. Further, in the case of Moldova, this decision proved to be even less adequate as, on one hand, the MoEd changed the exam's application methodology during Project implementation and, on the other hand, introduced a new curriculum into the classroom without reflecting this change in the Baccalaureate exam. The timing issues associated with this indicator include, for example, the fact that the exam results for the school year 2010/11 (conducted in May-June 2011) were made available in a national report only around October-November 2011. 29. Implementation. In line with M&E design, Results Framework and the Arrangements for Results Monitoring of the Project presented in Annex 3 of the QERM PAD were used by the MoEd to monitor the Project implementation. The heads of the MoEd technical departments and rayon staff routinely reported on components/subcomponents progress, outputs and implementation shortcomings, and Bank supervision visits used these tables to review the progress with the project 8 implementation. At the same time, information on the key outcome indicators was not accurately and timely reported to the Bank by the Quality Assurance Agency (QAA) of the MoEd preventing the Bank team from revising the outcome indicators for the project during four restructurings. The preliminary data from Bacca 2011/2012 made available at the time of ISR 16 resulted in the downgrade of the DO rating from Satisfactory to Moderately Unsatisfactory as the Bank team found many inaccuracies in the data (this was the first set of data following the new baseline of Bacca 2010/2011 established at the time of AF). In January 2013 the MoEd finally informed the Bank that it had changed the exams application methodology and resubmitted the exam results. Even then, the MoEd subsequently submitted revised results to the Bank as there have been still issues with data accuracy. 30. Utilization. The M&E framework served as a good basis for monitoring progress on activities and output indicators. At the time of approval of the Additional Credit, the Bank team was informed of the changes introduced by the MoEd to the Baccalaureate exam in 2010/2011 and revised the baseline for the PDO indicator accordingly. However, in 2011/2012 and 2012/2013, the MoEd continued “fine-tuning” the exam and application methodology without informing the Bank in a timely manner. For example, the MoEd submitted explanations of the changes to the Baccalaureate exam application methodology for the 2011/2012 school year only in January 2013 (less than 11 months before Project closing). A similar situation occurred in the subsequent year (2012/2013), and clarifications were provided to the Bank in October 2013, one month before Project closing. This did not provide sufficient time to revise the outcome indicators or to design and implement an alternative and sound ex-post evaluation. 2.4 Safeguard and Fiduciary Compliance 31. Safeguards. The QERM Project did not trigger any safeguards. 32. Procurement. Procurement ratings oscillated between Satisfactory and Moderately Satisfactory. Lower ratings were given in the first years of implementation since the MoEd lacked procurement capacity and a general understanding of procurement procedures. Procurement post-reviews found that procurement processes were generally reliable, timely, and transparent, with some corrective actions requested by the Bank related to the evaluation of candidates. Procurement aspects were generally addressed in a timely and qualitative manner. 33. One of the major procurement activities under the Project was procurement of 35 school buses to provide a means of transport to students in rural areas of Moldova. The purchase of buses was completed in January 2012. The procurement followed International Competitive Bidding (ICB) procedures. Shortly after the buses started operating, the Minister of Education received several complaints from rayons and schools about bus suppliers’ alleged non-compliance with their contractual obligations. The MoEd also raised issues related to the high prices of the buses. In response to the concerns raised, the Bank conducted an audit on buses in November 2012. The audit led by transportation and procurement specialists, included a review of the technical and 9 procurement aspects of the process. A transport specialist visited two rayons, reviewed the technical conditions and maintenance of the buses, and verified the degree of compliance of the bus supplier with the conditions of the supply contracts. The team also reviewed general operating conditions and itineraries of the buses in the two rayons. The audit concluded that all buses inspected were in good condition and technically sound, and met all the requirements of the technical specifications of the contracts. The Bank also found that the supplier met the conditions of the contracts in terms of warranty obligations. Technical problems had been resolved with reasonable delays. Further, the audit concluded that the poor quality of roads was causing problems in the buses. 34. The procurement review concluded that the ICB procedure was carried out according to Bank guidelines and the contract was awarded to the lowest evaluated compliant bidder in accordance with the requirements and evaluation criteria of the Bidding Documents. The issue of overpricing was also dismissed as, in addition to the price of the buses, the contract price included the required additional services such as transportation, insurance, training for drivers, and three years’ warranty, as provided by specific contract clauses. 35. Financial Management. Generally, financial management arrangements were satisfactory throughout the Project implementation. Yet using country systems for financial management created some challenges at the beginning. All problems were eventually overcome, resulting in efficient disbursement of IDA credit funds. Quarterly interim financial management reports were submitted on time and according to a format agreed with the Bank; such reports were always found acceptable. Annual audit reports were submitted in a timely manner and with an unqualified audit opinion. In addition to the audit of Project financial statements, Project auditors were required to review the existence and condition of assets purchased out of Project funds, especially the assets procured for rural schools. Initially, there were some deficiencies noted in storage, maintenance and accounting of the assets, which were eliminated later on. 2.5 Post-completion Operation/Next Phase 36. Improving the quality, relevance, and efficiency of the education system continues to be one of Moldova’s main priorities, as strong and robust economic growth and development will necessarily have to rely on the country’s human capital. To support the government’s priorities, a follow-up operation financed by IDA, the Moldova Education Reform Program, was approved by the Board in early 2013. It will continue supporting key activities initiated or implemented under the QERM Project, including: the development of a student-level Education Management Information System (EMIS) for primary and secondary education; (ii) continued participation in the Program for International Student Assessment (PISA) to monitor quality improvements; (iii) hub school upgrades based on some of the quality assurance standards developed under the QERM operation; and (iv) implementation of efficiency measures such as aligning class sizes with commonly accepted international standards, adjusting the size of the existing school network and reallocating students to “receiving” schools, and continued implementation of per-student financing nationwide. 10 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design, and Implementation Rating: Substantial 37. The Project was highly relevant at the time it was designed and remains equally relevant today, pertinent to the country’s development priority to provide schooling aligned with labor market needs and those of the broader economy, as stated in the Moldova 2020 National Development Strategy. Quality-enhancing policies substantiated by efficient financing in the education sector were vital to the advancement of the sector and in accordance with national priorities. Likewise, Project objectives were consistent with the CAS FY05-08, and the Project’s focus on improving the quality of primary and secondary education in rural areas was also fully aligned with the Bank’s Education Strategy and its three pillars of investing early, investing smartly, and investing for all. At the time of closing, Project relevance remained high in terms of country priorities and its activities with respect to strengthening the quality of education and enhancing the efficiency and management of Moldova’s education system. The Project also remained consistent with the Country Partnership Strategy FY09-13. Therefore, the relevance of objectives of the Project is rated as High. 38. Relevance of project design and implementation is rated Modest as: (i) component 1 of the project was not ‘rural focused’ despite the PDO: it tackled curriculum reform, teacher development, and textbook provision for grades 10-12 (textbooks for grade 12 only); (ii) equity and quality grants under component 2 activities were too small and incongruously targeted to provide poor rural schools with additional resources to improve quality of and access to adequate education services; and (iii) lack of proper coordination with the QAA in the course of project implementation had a detrimental impact on the Project’s final rating. 39. The overall Relevance of Objectives, Design, and Implementation is rated Substantial, reflecting High rating for relevance of objectives and Modest rating for relevance of design and implementation. 3.2 Achievement of Project Development Objectives Efficacy Rating: Modest 40. The project was transformational on a number of fronts as shown below. 41. Component 1. Improving Teaching and Learning in Rural Schools. New standards for secondary education were developed and approved and the curriculum was upgraded in line with these standards. Based on existing data and the beneficiaries’ assessment, there is strong awareness of the new and improved curriculum in place for upper secondary education, albeit further reform is needed to make it more focused on the development of core skills and competencies like problem solving or communication skills. The Project made a significant contribution by publishing 17 curriculum guides to assist teachers to improve teaching and learning in classrooms using the revised 11 curriculum. Anecdotal evidence gathered during the field visits (as reported in the MTR aide memoire) suggests that these guides are highly valued by teachers. Also, the new textbooks for grade 12 in mathematics, physics and astronomy, chemistry and biology were published and widely distributed (to all rural and urban lyceums in the country – 28,500 textbooks for each subject, 21,000 in Romanian language and 7,500 in Russian). All teachers for grades 10-12 in 10 selected rayons received training. In April 2007, over 8,000 teachers were trained on the curriculum introduced in 2006. In addition, in November 2008, the MoEd (outside of Project finds) through the Institute of Education Sciences trained almost 400 trainers (one per rural lyceum), who then trained over 16,000 teachers and administrators on the general teaching methodology. According to the results of the MTR, the MoEd evaluated both training sessions and found them to be successful. In summary, it is notable that in terms of the number of teachers trained, the target was exceeded. 42. The Quality Assurance Agency was established and institutionalized within the MoEd, and the Project contributed to its capacity building through various training programs for its staff. The members of the QAA also benefitted from a number of international study trips. The QAA led Moldova’s participation in the TIMSS 2007 3 and the PISA 2009 Plus international assessment of student learning, which was successfully implemented and the results have been widely disseminated and discussed).4 Results of PISA 2009 Plus became available only in 2012 and thus the MoEd decided not to participate in PISA 2012 assessment which would have given robust data on education quality developments and student learning outcomes (including by urban/rural divide) in the country. The QAA also worked on modernization and improvement of the Baccalaureate exam system, including strengthening its integrity. As a result, it is now considered as a credible instrument and is driving students to study harder (see Annex 5 for beneficiaries’ assessment results). Under the AF, the Project continued to support the MoEd to participate in international assessments (e.g., preparations to participate in the PISA 2015 assessment, to which Moldova subscribed) and to strengthen the MoEd’s ability to monitor the quality of education through support to the QAA (for design and analysis of the tests, and maintenance of the ADPS database of Moldovan students’ Baccalaureate exam results). 43. Also under the AF, quality assurance standards for schools were developed and will serve as the basis for school upgrades under the follow-up Moldova Education Reform Project. Overall, by Project completion, upper secondary education students and teachers had benefited from: a revised curriculum in all 17 subject areas; training on the new curriculum and new curriculum guidelines; and textbooks distributed to all 12th grade students in math, physics, biology, and chemistry (mostly in rural areas). At the same time, the Ministry's QAA continued to improve the Baccalaureate exam, and Moldova joined and completed the PISA 2009 Plus assessment, publicly disclosing its results. 3 Although the results were not acceptable to the International Association for the Evaluation of Educational Achievement – IEA. 4 Even though Moldova’s PISA results were quite poor compared to OECD and regional comparators. 12 Likewise, 4th and 9th grade national assessments are now carried out annually, and there are ongoing efforts to improve the quality and reliability of those tests. By financing the above inputs, teaching and learning conditions have improved in schools, mostly in rural schools but also in urban ones. At the same time, it is not possible for the team to determine whether these improved conditions translated into better learning. Although the reliability of the Baccalaureate exam results today is believed to be better than in 2006, it is not possible to derive any trend from this, much less attribute it to the Project. 44. Component 2. Increasing Access and Equity in Rural Schools. At appraisal, about 1,100 schools were expected to benefit; early ISRs set that target at 1,176. The Project surpassed targets for the number of schools that received equity and quality grants. At Project completion, 1,190 equity grants and 304 quality grants had been awarded to rural schools 5 . Rural schools, more poorly equipped compared to urban ones, were provided through equity grants with additional resources (learning materials such as maps, geometry triangles, dictionaries and other books, sports equipment, and other supplies). The MoEd developed criteria for selection of schools and reviewed school proposals jointly with rayon authorities. As a result of this process, only qualified schools with the capacity to use these grants received them. The quality grants were also awarded on a competitive basis and focused on improving conditions in chemistry, biology, and physics school labs by providing equipment and supplies (like visual aids, equipment and materials for experiments and so on, as well as a computer with a projector). An evaluation of this component conducted in 2010 found that the majority of principals, teachers, and students felt that the grants had improved conditions in their schools (see Annex 2 for details). 45. Also, the results of the social assessment of beneficiaries’ perceptions of the education changes, conducted as part of the ICR preparation, showed that both students and teachers in rural areas consider that, as a result of the endowment of educational institutions with equipment and training of teaching staff, the level of mastering of school subjects by students has increased and students’ skills and competences improved (including logical, analytical, and critical thinking). Equipment procured through equity grants included school lab equipment, projectors, computers, etc. These materials are used widely and are seen to contribute to student learning. The effect looked more prominent in smaller schools in rural areas (important as the most vulnerable groups at risk of poverty in Moldova remain those with low education levels and those in rural areas). Lessons became more interactive and encouraged participative learning. Most teachers underlined that the curriculum guides (developed and distributed in the framework of the Project) provided new opportunities to teaching staff in enhancing their pedagogical competences and served as an important support in designing and conducting lessons and evaluating learning results (see Annex 5 for beneficiaries’ assessment results). 5 4 schools of national subordination out of 304 under quality grants were urban. 13 46. In addition, during the ICR mission in October 2013, the team had the opportunity to visit schools that had benefited from the equity and quality grants, and confirmed that in most cases these materials had been the only ones received since Moldova’s independence in 1991. The team’s perception is that the equipment was much more relevant and of greater potential impact in smaller schools, which typically do not have any other resources (e.g., parent-teacher associations) to draw upon. At the same time, according to teachers, there is still a lack of school lab equipment, understandable given the one-time quality/equity grant interventions under the Project. Teachers also reported issues with the quality and durability of biology and chemistry lab equipment. 47. Component 3. Increasing Efficiency in the Use of Resources. This component surpassed its original objective as it not only supported the establishment of a more efficient budget allocation system, but also set the stage for broader education reforms, including a national school rationalization program that has already impacted the efficiency of resource allocation and created fiscal space for quality improvements (see outputs by component in Annex 2 and the economic and financial analysis section presented in Annex 3). Per-student based financing of schools ended up being implemented nationwide after it was successfully piloted in two rayons. School rationalization has already led to an adjustment in the school network, including halting the inefficient pulverization of resources across schools that are too small to function efficiently and with acceptable quality. Overall, the average class size in the country grew from 19.0 in the 2010 school year to 20.2 in 2013. In the same period, the student-teacher ratio for grades 1-12 showed a positive trend, increasing from 10.5:1 to 11.0:1. 48. A competent team of highly experienced local consultants of the MoEd (that had been working on the formula-based funding) worked closely with the Ministry of Finance (MoF), the Treasury, rayon authorities, and school principals and accountants on the development of school budgets – initially for pilot rayons and eventually for all primary and secondary schools in Moldova. The team conducted multiple seminars involving central and local education and finance authorities to train them on the new financing methodology and school budget preparation. Also, a national feasibility study of the optimization of the school network was prepared based on the school mapping exercise, and served as the basis for development of the Structural Reform Program in the education sector approved by the GoM in 2011. In addition, numerous discussions, conferences, and other events were conducted with the participation of the government, Parliament, rayon authorities, communities, school management, teachers, parents, students, and mass media all over the country to raise support for the much needed but politically sensitive school network optimization reform. 49. Under per-student based financing mechanism the schools were also granted the right to allocate resources in line with their needs and priorities. An assessment of school directors’ job satisfaction and sense of empowerment as a result of these changes showed noticeable positive differences for schools in those rayons that switched to per-student financing (pilots at the time) including in terms of improved transparency of budget allocations, school autonomy, incentives for efficiency and education quality (details are available in the Annex 2). Also, to support the school optimization reform, in particular 14 transportation of students from optimized to merged schools in rural areas, the Project procured 35 school buses that were delivered in 2010-2012 (10 to the first two pilot rayons and the rest to those subsequently joining the pilot). 50. Component 4. Strengthening Education Planning and Monitoring. At appraisal, the critical institutional shortcoming of the MoEd was emphasized. The MoEd had seen a significant number of staff leave and had little capacity to carry out its strategic mandate. The staff of the MoEd Department for Information Technologies was complemented with consultants to accelerate the institutional capacity building and development of the Education Management Information System (EMIS), intended to better monitor the development and performance of the education sector. The MoEd had several restructurings during the course of the Project, which led to creation of the Analysis, Monitoring, and Policy Evaluation Department. Its specialists, together with other ministry specialists responsible for policy development and heads of directorates (overall 20 participants), received training on: (i) educational policies, development, and implementation; and (ii) techniques for developing education policy documents. In addition, 70 rayon-level staff received training in administration and educational management. Also, a PR strategy was developed and conducted to support the education reform program. Continued support to PR activities was also provided under the AF to the Project. 51. The project also supported training activities for rayon administrators at the local level. Rayons are a critical element in monitoring the education system’s performance. Training was provided in the areas of education administration, education management, and data and information management in support of the national program. Overall, 94 rayon-level staff received training in ICT in education, meeting the target for this activity. Activities planned under this component were completed and training targets for both components were met. Together with the AF, the Project supported: (i) design of an EMIS; (ii) training of central and rayon-level staff on formula-based funding, policy making, and ICT-related matters; (iii) creation and strengthening of the Ministry's QAA to prepare, conduct, and analyze student assessments; (iv) Moldova's participation in the PISA 2009 Plus international assessment; and (v) dissemination of education information to stakeholders, such as the results of the PISA 2009 Plus and Baccalaureate exam results. On a less positive note, while the MoEd’s institutional capacity is probably better today than in 2006, it remains relatively weak and concentrated among a small group. That is partially due to the difficulty in attracting qualified professionals into public service in Moldova, a situation not unique to the education sector. The EMIS was developed and data entry was conducted in November-December 2013 (the work is continued under the follow-up Moldova Education Reform Project). At the same time, its sustainability will depend on the political will of the Ministry's leadership to support it, use it for policy making, and allocate proper resources for its regular update and maintenance. 52. Notwithstanding the positive findings, neither the Board-approved outcome indicators nor additional information and evidence collected during the preparation of this ICR are conclusive to support a rating above Modest. 15 3.3 Efficiency Rating: Substantial 53. The Project was efficient on a number of fronts. The economic analysis confirms that the Project’s investments of US$12.09 million in 2006-2013 were well justified and the interventions brought benefits and savings far exceeding the Project’s initial costs. The most substantial benefits of the Project are the gains in sector efficiency that surpassed the Project's original objectives. Per-student based financing of schools ended up being implemented nationwide after it was successfully piloted in selected rayons and municipalities. School rationalization has already led to an adjustment of the school network. From the 2009/10 to the 2013/14 school year, 131 schools were closed, representing 9 percent of the total. Maintaining these schools in 2013 alone would have required government spending of US$26 million, or an additional 5 percent of total education sector spending. In addition, through consolidation of the school network and an increase in average class and school sizes, existing capacities are used more efficiently, freeing up resources for quality-enhancing inputs. Also, as a result of per-capita financing, schools are now able to reallocate resources according to their needs, allowing schools to spend resources according to their priorities. 54. At the same time, the Project costs of US$12.09 million are relatively small – equivalent to less than 1 percent of public expenditure on primary and secondary education over the eight-year period 2006-2013. The economic analysis at appraisal took the framework of a traditional cost-benefit analysis, though no estimates of the net present value (NPV) were made at that time. Given the high uncertainty associated with predicting the labor market outcomes of individuals and linking those outcomes to investments made today, the cost-benefit analysis at the ICR stage focused on the benefits of the Project associated with the fiscal savings from the efficiency measures targeted under Component 3. The NPV of the savings was calculated based on a discount rate of 10 percent and covered the period of 10 years from Project completion (2013-2033). Under these assumptions, the savings from the school consolidation alone amount to about US$154.6 million in NPV terms, equivalent to 29.6 percent of Moldova’s education budget in 2013. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory 55. Project relevance was substantial at its design and remains so today. Likewise, Project interventions resulted in current and future gains in efficiency. Conceding the shortcomings in the outcome indicators, alternative indicators are assessed on whether the Project achieved its objective of assisting Moldova in improving the quality of education in rural areas. That task presented two challenges. The first was to determine whether there was any improvement in the quality of education in rural areas. The second was establishing whether those improvements could be attributed to the Project. A key difficulty in this process is that the overall quality of education data for Moldova is poor, particularly data that would support measuring the Project's impact. Traditional education indicators such as enrollment, repetition, and graduation and dropout rates, while 16 important, speak little to the quality of education and are also problematic in Moldova. Further, the country does not have a track record of continuous participation in international assessments, which could show some trend in performance. Thus, Project- financed inputs/activities that have contributed to improvements in education quality are identified. A social assessment of beneficiaries’ perceptions was conducted to gauge the impact of the reforms introduced under the Project, and visited several rural schools to “review” their conditions and to talk to teachers and directors about the use of school grants. The key conclusions of the survey and of those visits are that teaching conditions improved in the schools surveyed, particularly in terms of the availability of teaching and learning materials. Despite the additional information and evidence collected, it was not possible to substantiate whether the Project contributed to improving education quality in rural areas. As such, the overall outcome of this Project is rated as Moderately Unsatisfactory. 3.5 Overarching Themes, Other Outcomes, and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 56. The project focused on most vulnerable groups at risk of poverty in Moldova (those with low education levels and those in rural areas). Gender gaps in upper secondary education are found to be negligible. (b) Institutional Change/Strengthening 57. One major institutional accomplishment of the Project was the establishment and equipment of the Quality Assurance Agency (QAA), including: the development and installation of an automated data processing system (ADPS) and training of relevant staff in managing the Baccalaureate exam database; capacity building and training of QAA personnel to conduct national and participate in international assessments; and strengthening of the QAA’s ability to monitor the quality of education in Moldova. As shown in the results of the beneficiaries’ assessment, strengthening the integrity of the Baccalaureate exam system laid the groundwork for a more meritocratic way of determining university admission and driving students to learn harder. (c) Other Unintended Outcomes and Impacts 58. The grant writing training received by schools carried over to other areas, which allowed some schools to independently raise funds for school investments from other sources. Also, as reported by several school principals, with the per-student based funding, schools now recognize the need to improve the quality of the education they offer to attract more students to their institutions. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 59. As part of the ICR preparation, a social assessment was conducted in early 2014 to document beneficiaries’ perceptions of the overall project impact on the quality of education in rural areas. Survey respondents perceived an improvement in the quality of 17 rural schools in recent years and attributed to the project the following interventions: (i) provided laboratory equipment and visual aids that helped students better understand abstract scientific concepts; (ii) supported the development and training of teachers on a new competence-based curriculum (which is more practical than the old curriculum, and helped students improve their communication, team work, and learning skills); (iii) enhanced school principals’ autonomy, allowing them to seek efficiency savings and reallocate funding, including towards quality-enhancing investments; and (iv) helped improve the quality and reliability of national student assessments that, together with the measures on strengthening the integrity of the Baccalaureate exam system, laid the groundwork for a more merit-based way of determining university admission and increased students’ academic efforts. The general perceptions of trends in education quality are positive over the time that the educators have been in their jobs (between 4 and 20 years). 4. Assessment of Risk to Development Outcome Rating: Low 60. The key project achievements have been institutionalized including Quality Assurance Agency, curriculum for 10-12 grades, per-student based financing, autonomy of schools in managing financial and largely human resources. It will be difficult to revert these arrangements. For example, for the second year now the primary and secondary education system operates nationwide under the new financing mechanism. The GoM is unlikely to give up on this methodology and it will be difficult to take back the wider decision making powers granted to schools. Of course, the elections pose threat to optimization reform (school closures and reorganization and staff reduction) but nevertheless, overall rating for Risk to development outcomes stands Low. 61. Also, a follow-up project financed by IDA –Education Reform Project (P127388)— will continue providing support to the MoEd to ensure that school optimization is accompanied by unwavering efforts to improve the quality of schools and of education in general, so that stakeholders can benefit from it and support the changes. In particular, it will work to ensure that: (i) the MoEd has the institutional tools it needs to continue steering the education sector and measuring its achievements; (ii) a large number of schools meet adequate operational standards; and (iii) teachers and principals receive training and compensation commensurate with the roles expected of them. In addition, Global Partnership for Social Accountability grant promotes the use of social accountability tools in the education sector. Making stakeholders (students, parents, teachers and communities in general) informed about their schools’ performance will enable them to hold schools and teachers accountable for quality of education services provided and use of resources (in particular, in the context of the ongoing education reform and wider autonomy of schools in managing resources granted under the new per- student based financing mechanism). 18 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 62. During Project preparation, the Bank: made sound technical analysis using findings and recommendations from analytical work conducted as part of Project preparation and lessons learned from the implementation of the completed Moldova General Education Project; involved multiple stakeholders, capturing their opinions and recommendations and reflecting these in project design; and correctly identified risks and mitigation measures for the operation. At appraisal, the critical institutional shortcomings of the MoEd were emphasized, including its low capacity to carry out its strategic mandate. While this was acknowledged, the allocation to the MoEd team’s capacity building to better manage the education system (supported by Component 4) was envisaged to be only 4 percent of total Project costs. Also, weaknesses in the RF during Project preparation (as described in sections 2.1 and 2.3), regrettably resulted in a Moderately Unsatisfactory rating for the overall Project outcomes. (b) Quality of Supervision Rating: Moderately Satisfactory 63. Implementation was under the responsibility of five task team leaders (TTLs). There was appropriate continuity in the reporting despite the transition in leadership. Bank supervision took place on a regular basis, providing appropriate advice and observations to counterparts. A local education staff regularly followed up on implementation. Eighteen ISRs were prepared during the life of the Project, about one every six months, except during the transfer period from one TTL to another, when the ISR update took 12 months. There were regular implementation support visits to Moldova, including fiduciary supervisions. By and large, implementation support visits included the appropriate skills mix to support and advise the government on areas targeted by the Project. The findings of each supervision visit were conveyed to Bank management clearly and in a candid fashion, including implementation shortcomings and potential risks (for example, the situation with the Baccalaureate exam). Ratings were in line with the situation described in the supervision reports. When the Bank team was informed of the changes introduced by the MoEd to the Baccalaureate exam it revised the baseline for the PDO indicator accordingly. However, the MoEd continued to make changes to the Baccalaureate exam without timely informing the Bank. Importantly, all the flaws with the PDO-level indicators were revealed very late in the process (11 months before project closing) not providing sufficient time to revise the outcome indicators. However, given the failure to revise them, the ICR rates the Bank’s supervision performance as Moderately Satisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 64. The overall Bank performance is rated Moderately Unsatisfactory, accounting for the final rating of the development objective. 19 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 65. The Project preparation and implementation cycles transcended four elected Parliaments, with a significant period of political uncertainty in 2009-2010. During this period, Parliamentary elections were conducted in April 2009, July 2009, and November 2010, affecting the work of the government and the pace of reforms in the sector. Nevertheless, although with some delays, the government implemented politically and socially sensitive education reforms, including implementation of a per-student financing mechanism nationwide and substantial rationalization of the school network. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 66. The MoEd completed all activities envisaged under the Project, albeit with initial delays, and took important but politically challenging decisions in the process, including leadership over school consolidation, piloting and subsequent implementation of a per- student-based financing mechanism, and overhauling and strengthening the integrity of the Baccalaureate exam system. There were five Ministers of Education and five Executive Directors during Project implementation. The political uncertainties caused delays in the implementation of some Project components prior to the MTR, but since then the implementation remained steady for the past four years. Fiduciary performance was also mostly satisfactory. The advice given and the actions agreed during the Bank missions and reflected in the aide memoires were implemented. 67. The key shortcoming of the MoEd during implementation was its failure to inform the Bank in a timely manner of the changes to the Baccalaureate exam, as well as the results. Only in January 2013 did the MoEd submit the Baccalaureate exam results, including: (i) corrected baseline data for the 2010/11 school year; and (ii) achieved scores for the 2011/12 school year, with explanations on the changes to the methodology. Even then, the MoEd subsequently submitted revised results to the Bank as the provided set had many mistakes. Unfortunately, this shortcoming had significant implications for the Project’s final rating. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 68. The overall Borrower performance is rated as Moderately Satisfactory, reflecting the Satisfactory rating of the government’s performance and the Moderately Satisfactory rating of the implementing agency’s performance. 6. Lessons Learned 69. Project design must be based on a well-defined RF, allowing capturing key results influenced by the project. The RF should convey the cause and effect linkages between the intermediate results and the objective (avoiding a disconnect between the two), and 20 should clearly depict the development hypothesis implicit in the project. For example, while at appraisal it was acknowledged that the Project had several features contributing in significant ways to improving the quality of teaching and learning in Moldova as a whole, with positive repercussions well beyond rural areas, but this was not reflected in the M&E framework. Instead, the RF focused on narrowing the performance gap between urban and rural areas. In terms of quality improvements in rural areas, the PDO and outcome indicators should have been defined in more constrained terms, referring to beneficiary schools only and comparing the envisaged changes with a control group of comparable non-benefitted schools. 70. Both top-down and bottom-up approaches are needed for the efficiency reforms to be successful. The MoEd’s ownership was important to guide the policy agenda, while increased autonomy granted to local stakeholders empowered them through greater decision-making authority and the more flexible and predictable financing defined by a transparent formula (in particular, decisions about managing staff and school budget, including reallocation of funds across budget lines). The assessment of school directors’ sense of empowerment showed a number of noticeable positive differences between pilot rayons (which switched to a per-student financing mechanism and received greater autonomy in managing their budgets) and non-pilot rayons (see Annex 2). The assessment showed that school autonomy provides a number of advantages as part of a comprehensive strategy for school and education system improvements, including the incentive to use funds more efficiently. In addition, school directors also believed that school autonomy can have a positive impact on education quality. 71. Adequate resourcing for capacity-building activities is fundamental. The PAD acknowledged the MoEd’s low capacity to carry out its strategic mandate due to the low capacity of its staff. In addition, the risks section emphasized that the ambitious plan of school network rationalization could impact the MoEd's capacity to implement the Project. At the same time, an insufficiently low percentage of Project funds (only 4 percent) was allocated for building capacity of the Ministry of Education to better manage the education system and to promote and monitor policies and programs. This was partially addressed at the time of approval of the AF, which supported technical assistance to the MoEd for implementation of the initiated reforms. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 72. Upon reviewing the draft ICR, the Borrower confirmed that they have no further comments. (b) Cofinanciers 73. Not applicable. (c) Other partners and stakeholders 74. Not applicable. 21 Annex 1. Project Costs and Financing QERM including AFQERM Project Costs by Component (in USD millions equivalent) Appraisal Estimate Actual/Latest Percentage of Components (USD mln.) Estimate (USD mln.) Appraisal Component. Improving Teaching and Learning in Rural Schools 2.78 3.56 128% Component 2. Increasing Access and Equity in Rural Schools 5.11 2.56 50% Component 3. Increasing Efficiency in the Use of Resources 2.27 5.17 228% Component 4. Strengthening Education Planning and Monitoring including operating expenditures 0.52 0.8 154% Total Baseline Cost 10.68 12.09 113% Physical Contingencies 0.2 - Price Contingencies 0.34 - Total Project Costs 11.19 12.09 108% Note: Costs including Borrower/Recipient contribution of US$0.19 mln. Totals may not add up due to rounding. Actual amounts are higher than originally planned due to SRD to USD currency exchange rate fluctuations. (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (USD mln.) (USD mln.) Borrower 0.19 0.19 100% IDA-41550 (QERM) Credit 5.00 5.56 111% IDA-H2080 (QERM) Grant 5.00 5.55 111% Total QERM 10.00 11.11 111% IDA-50800 (AF QERM) Credit 1.00 0.79 79% Total Project Costs 11.19 12.09 108% Note: Actual amounts are higher than originally planned due to SRD to USD currency exchange rate fluctuations. 22 Annex 2. Outputs by Component 75. Component 1. Improving Teaching and Learning in Rural Schools. The principal objective of this component was to improve the quality and relevance of teaching and learning in rural schools, in particular for the upper grades of secondary education (grades 10-12). This was meant to be achieved by strengthening the MoEd’s three core pedagogical systems: curriculum (upgrading and modernizing the curriculum and providing new educational materials and textbooks), teaching (professional development for teachers and staff in the structure and content of the new curriculum), and evaluation (objective, professional, and efficient evaluation, monitoring, and student assessment). This component encompassed three subcomponents: (i) Modernizing the Curriculum for Grades 10-12; (ii) Teacher Professional Development for Grades 10-12; and (iii) Establishment of National Centre for Examinations and Quality Assurance 6 under the MoEd. 76. Subcomponent 1.1. Modernizing the Curriculum for Grades 10-12. New standards for secondary education were developed and approved and the curriculum was upgraded in line with these standards. Based on existing data and the beneficiaries’ assessment, it is known that there is an improved curriculum in place for upper secondary education, albeit further reform is needed to make it less heavy and more focused on the development of core skills and competencies like problem solving or communication skills. The Project made a significant contribution by publishing 17 curriculum guides to assist teachers to improve teaching and learning in classrooms using the revised curriculum. Anecdotal evidence gathered during the field visits (as reported in the MTR aide memoire) suggests that these guides are highly valued by teachers. Also, the new textbooks for grade 12 in mathematics, physics and astronomy, chemistry and biology were published and widely distributed (to all rural and urban lyceums in the country – 28,500 textbooks for each subject, 21,000 in Romanian language and 7,500 in Russian). 77. Subcomponent 1.2. Teacher Professional Development for Grades 10-12. All teachers for grades 10-12 in 10 selected rayons received training. In April 2007, over 8,000 teachers were trained on the curriculum introduced in 2006. In addition, in November 2008, the MoEd (outside of Project finds) through the Institute of Education Sciences trained almost 400 trainers (one per rural lyceum), who then trained over 16,000 teachers and administrators on the general teaching methodology. According to the results of the MTR, the MoEd evaluated both training sessions and found them to be successful. In summary, it is notable that in terms of the number of teachers trained, this subcomponent’s target was exceeded. 78. Subcomponent 1.3. Establishment of National Centre for Examinations and Quality Assurance under the MoEd. The Centre was established and institutionalized within the MoEd, and the Project contributed to its capacity building through various trainings of its staff. The members of the National Centre also benefitted from a number 6 Currently the Quality Assurance Agency (QAA). 23 of international study trips. The Centre led Moldova’s participation in the TIMSS 20077 and the PISA 2009 Plus international assessment of student learning, which was successfully implemented and the results have been widely disseminated and discussed).8 The Centre also worked on modernization and improvement of the Baccalaureate exam system, including strengthening its integrity. Under the AF, the Project continued to support the MoEd to participate in international assessments (e.g., preparations to participate in the PISA 2015 assessment, to which Moldova subscribed) and to strengthen the MoEd’s ability to monitor the quality of education through support to the QAA (for design and analysis of the tests, and maintenance of the ADPS database of Moldovan students’ Baccalaureate exam results). 79. Also under the AF, quality assurance standards for schools were developed and will serve as the basis for school upgrades under the follow-up Moldova Education Reform Project. Overall, by Project completion, upper secondary education students and teachers had benefited from: a revised curriculum in all 17 subject areas; training on the new curriculum and new curriculum guidelines; and textbooks distributed to all 12th grade students in math, physics, biology, and chemistry (mostly in rural areas). At the same time, the Ministry's QAA continued to improve the Baccalaureate exam, and Moldova joined and completed the PISA 2009 Plus assessment, publicly disclosing its results. Likewise, 4th and 9th grade national assessments are now carried out annually, and there are ongoing efforts to improve the quality and reliability of those tests. 80. The ICR team assessment is that by financing the above inputs, teaching and learning conditions have improved in schools, mostly in rural schools but also in urban ones. At the same time, it is not possible for the team to determine whether these improved conditions translated into better learning. Although the reliability of the Baccalaureate exam results today is believed to be better than in 2006, it is not possible to derive any trend from this, much less attribute it to the Project. At the same time, since the Bank's team found no evidence of whether these improvements translated into better teaching and learning in rural classrooms compared to urban ones, the achievements under this component are rated Moderately Unsatisfactory. 81. Component 2. Increasing Access and Equity in Rural Schools. The main objective of this component was to provide poor rural schools and communities with additional resources to improve the quality of and access to adequate education services. Another objective was to strengthen schools’ capacity to manage priorities, resources, and educational outcomes. These objectives were meant to be accomplished by providing additional resources to eligible schools, which in turn were required to manage them (including the needs identification and prioritization of resource allocation strategy) in a transparent and efficient manner and to be accountable for results. This component 7 Although the results were not acceptable to the International Association for the Evaluation of Educational Achievement – IEA. 8 Even though Moldova’s PISA results were quite poor compared to OECD and regional comparators. 24 comprised two subcomponents: (i) Rural Schools Equity Grants; and (ii) Rural Schools Quality Grants. 82. At appraisal, about 1,100 schools were expected to benefit; early ISRs set that target at 1,176. The Project surpassed targets for the number of schools that received equity and quality grants. At Project completion, 1,190 equity grants and 304 quality grants had been awarded (i.e., schools receiving quality grants were also eligible for equity grants). The equity grants were offered to gymnasiums, middle schools, and lyceums, and the quality grants to lyceums. Grants were used for the purchase of educational inputs such as science laboratories and teaching materials. The MoEd developed criteria for selection of schools and reviewed school proposals jointly with rayon authorities. As a result of this process, only qualified schools with the capacity to use these grants received them. 83. Subcomponent 2.1. Rural Schools Equity Grants. All schools satisfying the eligibility criteria received a small grant of US$3,000-US3,500 per school based on the size of their student population. Considering the relatively small size of the equity grants, at appraisal schools were anticipated to and did select inexpensive learning materials such as maps, geometry triangles, dictionaries and other books, sports equipment, and other supplies. Rural schools which are more poorly equipped compared to urban ones were thus equipped through equity grants with these additional resources 9 . The Project provided 1,190 equity grants, surpassing its target of 1176 grants. 84. Subcomponent 2.2. Rural Schools Quality Grants. The quality grants were also awarded on a competitive basis and focused on improving conditions in chemistry, biology, and physics school labs by providing equipment and supplies (like visual aids, equipment and materials for experiments and so on, as well as a computer with a projector). The Project at appraisal set a target of 300 rural schools for quality grants. At Project completion, 304 schools had received quality grants. Also, throughout Project implementation, the schools received capacity-building training and assistance for quality grant proposal preparation. It was found during the ICR field visits that the training schools received in grant writing was useful and later allowed school managers to independently raise funds for school investments from other sources. 85. An evaluation of this component was conducted in 2010.10The utilization and user satisfaction assessment of the equipment purchased with the grants was considered 9 Some rural schools have not received any supplies for around two decades. 10 Report on Sociological Study under the Project "Quality Education in Rural Areas of Moldova", Centre for Sociological Research and Marketing 'CBS AXA', Chisinau 2010. At the first stage of the study, interviewing covered: 168 school directors; 545 teachers from gymnasiums; 139 teachers of biology, chemistry, and physics from secondary schools; 1,624 students studying in grades 8 and 9; and 471 students studying in grades 11 and 12. Sampling procedure for the equity and quality grant beneficiaries was as follows: probabilistic, stratified, representative of institutions benefiting from the Project; stratification criteria: the residence, administrative-territorial units, type of institution, beneficiaries from both components or only the equity component; and random selection method. The sample size for equity grants was 161 institutions (52 lyceums, 36 general middle schools, and 73 gymnasiums) and for quality 25 satisfactory. It found that the majority of principals, teachers, and students felt that the grants had improved conditions in their schools. The quality grants, which provided equipment and materials for school labs, made an especially positive impact on the perceptions of those interviewed (Figure 1). Figure 1: Benefits of the Quality Grants Improvement of the material support of the disciplines 31.6% Increase of the level of teaching-learning 25.7% Increase of the interest of the students 12.5% Made the work of the teachers easier 10.3% Possibility to conduct practical lessons 5.9% Renewing of the equipment 0.7% Don’t know 13.2% 0% 5% 10% 15% 20% 25% 30% 35% 86. In general, equipping institutions with didactic materials and equipment was assessed positively by teachers: around 76.6 percent of them stated they were satisfied with the materials received. More teachers (83.8 percent) noted that the received equipment and materials helped them when teaching new topics and 86.7 percent of teachers believed that the equipment and new materials contributed to students’ better understanding of the topics. However, 29.6 percent of teachers underlined the fact that they were dissatisfied with the materials received, because some were not priority-driven for their discipline. Chemistry teachers were the most dissatisfied (35.7 percent). Over 56 percent of teachers agreed that the received didactic materials and equipment contributed to changing the teaching methods of their discipline. Also, according to 80 percent of teachers, the possibilities for realizing practical lessons increased (an opinion most popular among physics teachers). 87. According to students, teachers who most often used11 the equipment and didactic materials during lessons taught geography (71.7 percent), physical education (71.4 percent), mathematics (68.8 percent), physics (68.5 percent), and history (65.9 percent). Teachers’ responses were somewhat different with regard to the frequency of use: three- quarters (75 percent) of computer science teachers reported using the materials often or very often, followed by teachers of chemistry (73.6 percent), history (73.1 percent), and Romanian language and literature (73 percent). Teachers’ and students’ perceptions grants, 50 institutions (42 beneficiaries from both components, 8 beneficiaries from the quality component). At the second stage, the study covered surveys among teachers (147); surveys among students (509); seven focus groups with students; five focus groups discussions with parents; four in-depth interviews with directors; and 12 in-depth interviews with teachers. 11 Response options were "very often" and "often." 26 regarding the frequency of the use of equipment and teaching materials obtained under equity grants are presented in Figure 2. Figure 2: Teachers’ and Students’ Perceptions Regarding the Frequency of Use of Equipment and Teaching Materials Obtained under Equity Grants Very often 21.8% 9.6% Often 36.9% 57.7% Seldom 25.9% 23.5% Students Very seldom 8.3% 5.0% Teachers Never 7.1% 1.0% Don’t know 3.3% 0% 10% 20% 30% 40% 50% 60% 70% 88. Overall, the evaluation found that information about the QERM was known by 61.4 percent of teachers. Over 70 percent of teachers evaluated the selection process for equipment and didactic materials as transparent and objective. The quality of the equipment was assessed an average of 7.8 on a scale from 1 to 10 (where 1 means “very poor quality” and 10 means “very good quality”). However, around 20 percent of the teachers faced problems in using the new equipment. About one-third of the teachers considered the main shortcoming of the grants to be the limited financial resources provided to the institutions compared with the real needs of the school. 89. During the ICR mission in October 2013, the team had the opportunity to visit schools that had benefited from the equity and quality grants, and confirmed that in most cases these materials had been the only ones received since Moldova’s independence in 1991. The team’s perception is that the equipment was much more relevant and of greater potential impact in smaller schools, which typically do not have any other resources (e.g., parent-teacher associations) to draw upon. At the same time, according to teachers, there is still a lack of school lab equipment, understandable given the one-time quality/equity grant interventions under the Project. Teachers also reported issues with the quality and durability of biology and chemistry lab equipment (see Annex 5 for beneficiaries’ assessment results). As such, the final assessment of the component's performance is Moderately Satisfactory. 90. Component 3. Increasing Efficiency in the Use of Resources. The main objective of this component was to improve the efficiency of the system for allocating public money to schools and of the use of resources within schools, in the context of the projected demographic trends (school-age population, class sizes, and student-staff ratios were falling and were expected to continue to decline). The component sought to build on the existing formula-based funding approach, with emphasis on using the signals that it generates to trigger policy action. This component was further subdivided into three subcomponents: (i) Education Budgeting Improvement; (ii) Formula Funding and School 27 Autonomy Pilots; and (iii) Grants for the Optimization of School Networks in Selected Rayons. 91. Subcomponent 3.1. Education Budgeting Improvement. A competent team of highly experienced local consultants of the MoEd (that had been working on the formula- based funding) also worked closely with the Ministry of Finance (MoF), the Treasury, rayon authorities, and school principals and accountants on the development of school budgets – first for two pilot rayons and later expanded to all primary and secondary schools in Moldova. The team conducted multiple seminars involving central and local education and finance authorities to train them on the new financing methodology and school budget preparation. The subcomponent also included preparation of the detailed, analysis-based Medium Term Expenditure Framework (MTEF) projections for the whole education sector. While the MoEd jointly with the MoF developed the MTEF for various periods (e.g., MTEF for 2009-2011), the decision at the MTR was to drop the activity of development of the methodology for MTEF in education. The MTR mission recommended that the MoEd restart this work with its own funds and personnel once the country’s financial situation improves (after the financial crisis of 2009) and when there is a need for MTEF budgeting. This is now the case and the MTEF for education is implemented by the MoEd together with the MoF and other stakeholders. 92. Subcomponent 3.2. Formula Funding and School Autonomy Pilots. By May 2009, all required analytical work on simulation of the formula-based funding had been conducted. Following some delays due to political uncertainties, the GoM in 2009 approved a resolution for piloting per-student based financing and a budget ceiling for primary and secondary education. By Project completion, a new school funding formula had been adopted and implemented, initially in two pilot rayons (Rîşcani and Căuşeni) in 2010. Nine additional rayons and two municipalities were added in 2012. The system was rolled out nationwide in January 2013. The logic behind per-student formula-based financing is simple: expanding the autonomy of school directors and allowing them to manage their own budgets empowers local actors to make better informed decisions related to their schools. Giving a predictable budget defined by a transparent formula to a school’s management and allowing it to allocate resources in the manner it judges best produces a more efficient outcome. At the same time, local actors should be held accountable for the quality of service provision. 93. Indeed, formula-based financing offers a transparent mechanism for budget allocation and creates positive and negative incentives for rayons to rationalize school networks. Schools that were too small to remain open were forced to close and their students transferred to neighboring schools. The schools were also granted the right to allocate resources in line with their priorities. As a result, in spite of the tight fiscal environment, primary and secondary schools in both pilot rayons found the fiscal space to allocate resources for capital investments, fixed assets, and capital repairs, and for the first time since 2006, they could afford capital spending above the country average. 28 94. An assessment of school directors’ job satisfaction and sense of empowerment as a result of these changes showed noticeable differences between the pilot and non-pilot rayons.12 The assessment drew the following findings in each area of interest:  Transparency of budget allocations: a larger share of school directors from pilot rayons knew their school budget in detail (95 percent compared to 83 percent in non-pilot rayons) and thought that resources in their rayons were allocated transparently (62 percent versus 38 percent);  School autonomy: the majority of school directors in pilot rayons wished to have full autonomy, while in non-pilot rayons, administrators wanted subordination to rayon or municipal authorities;  Sense of empowerment: the majority of school directors in pilot rayons thought that they (or their school boards) had considerable responsibility for deciding on budget allocations within the school and allocating funds for teachers’ professional development, while in non-pilot rayons this responsibility was viewed as being held primarily in the mayor’s office;  Incentives for efficiency: a larger share of directors in pilot rayons compared to non-pilot rayons thought that their current system of funding provided incentives to create bigger classes and to reduce the number of non-teaching staff;  Job satisfaction: looking ahead, school directors in pilot rayons were slightly less optimistic about serving in their job than their non-pilot counterparts;13 and  Education quality: the vast majority of school directors in pilot and non-pilot rayons believed that school autonomy can have a positive impact on education quality. 95. Since 2012, the GoM has expanded the pilots to an additional nine rayons and two municipalities, and rolled out per-student financing nationwide in January 2013. It merits stating that during the process of closing schools, the MoEd protected certain schools, whenever alternative ones were not feasible or adequate for their students, to ensure that the rationalization process did not result in student dropouts. Overall, the average class size in the country grew from 19.0 in the 2010 school year to 20.2 in 2013. In the same period, the student-teacher ratio for grades 1-12 showed a positive trend, increasing from 10.5:1 to 11.0:1. 96. Subcomponent 3.3. Grants for the Optimization of School Networks in Selected Rayons. The piloting of per-student formula financing and optimization of the school network initially took place in two rayons – Rîşcani and Căuşeni (rayons from different regions of the country, with different demographic trends and school network composition). In both rayons, optimization grant investment plans were developed and 12 The assessment covered 241 participating school principals (72 from the pilot rayons of Rîşcani and Căuşeni and 169 from the non-pilot rayons of Donduseni, Anenii Noi, Drochia, and Cimislia). Social Impact Assessment of School Optimization Reforms in Moldova , Magenta Consulting, March 2011. 13 This finding may be the result of the transition period and the need for more information materials and training for school directors joining the new financing scheme —therefore, the follow-up Moldova Education Reform Project supports an updated modern system for training school directors. 29 sent to the MoEd. To support the school optimization reform, in particular transportation of students from optimized to merged schools in rural areas, the Project procured 35 school buses that were delivered in 2010-2012 (10 to the first two pilot rayons and the rest to those subsequently joining the pilot). Also, a national feasibility study of the optimization of the school network was prepared based on the school mapping exercise, and served as the basis for development of the Structural Reform Program in the education sector approved by the GoM in 2011. In addition, numerous discussions, conferences, and other events were conducted with the participation of the government, Parliament, rayon authorities, communities, school management, teachers, parents, students, and mass media all over the country to raise support for the much needed but politically sensitive school network optimization reform. 97. This component surpassed its original objective as it not only supported the establishment of a more efficient budget allocation system in the education sector, but also set the stage for broader education reforms, including a national school rationalization program that has already impacted the efficiency of resource allocation and created fiscal space for quality improvements (see subcomponent 3.2. above and the economic and financial analysis section presented in Annex 3). An unexpected impact of these changes, as reported by several school principals, is that schools now recognize the need to improve the quality of the education they offer to attract more students and ensure their financial sustainability, as well as to increase the resource envelope available to them. For that reason, the component is rated as Highly Satisfactory. 98. Component 4. Strengthening Education Planning and Monitoring. The main objective of this component was to build capacity at the MoEd to better manage the education system by strengthening its capacity to promote and monitor programs. This objective was meant to be met through: (i) development of the education sector’s human resources in policy and monitoring at the ministerial level, and (ii) improvement in the education management and administration in rayon education administration departments. This component encompassed two subcomponents: (i) Policy and Monitoring Development; and (ii) Rayon Education Administration Improvement. 99. Subcomponent 4.1. Policy and Monitoring Development. At appraisal, the critical institutional shortcoming of the MoEd was emphasized. The MoEd had seen a significant number of staff leave and had little capacity to carry out its strategic mandate. The staff of the MoEd Department for Information Technologies was complemented with consultants to accelerate the institutional capacity building and development of the Education Management Information System (EMIS), intended to better monitor the development and performance of the education sector. The MoEd had several restructurings during the course of the Project, which led to creation of the Analysis, Monitoring, and Policy Evaluation Department. Its specialists, together with other ministry specialists responsible for policy development and heads of directorates (overall 20 participants), received training on: (i) educational policies, development, and implementation; and (ii) techniques for developing education policy documents. In addition, 70 rayon-level staff received training in administration and educational management. Also, a PR strategy was developed and conducted to support the education 30 reform program. Continued support to PR activities was also provided under the AF to the Project. 100. Subcomponent 4.2. Rayon Education Administration Improvement. This subcomponent supported training activities for rayon administrators at the local level. Rayons are a critical element in monitoring the education system’s performance. Training was provided in the areas of education administration, education management, and data and information management in support of the national program. Overall, 94 rayon-level staff received training in ICT in education, meeting the subcomponent target for this activity. 101. Activities planned under this component were completed and training targets for both components were met. Together with the AF, the Project supported: (i) design of an EMIS; (ii) training of central and rayon-level staff on formula-based funding, policy making, and ICT-related matters; (iii) creation and strengthening of the Ministry's QAA to prepare, conduct, and analyze student assessments; (iv) Moldova's participation in the PISA 2009 Plus international assessment; and (v) dissemination of education information to stakeholders, such as the results of the PISA 2009 Plus and Baccalaureate exam results. On a less positive note, while the MoEd’s institutional capacity is probably better today than in 2006, it remains relatively weak and concentrated among a small group. That is partially due to the difficulty in attracting qualified professionals into public service in Moldova, a situation not unique to the education sector. The EMIS was developed and data entry was conducted in November-December 2013 (the work is continued under the follow-up Moldova Education Reform Project). Still, the EMIS’s sustainability will depend on the political will of the Ministry's leadership to support it, use it for policy making, and allocate proper resources for its regular update and maintenance. In light of these considerations, the ICR team feels that the rating for this component is Moderately Satisfactory. At the same time, the team recognizes that those objectives were too ambitious given the scope of interventions supported by the Project, including the actual financing allocated to them, which represented 6 percent of the actual overall Project cost. 31 Annex 3. Economic and Financial Analysis 102. This annex presents the economic and financial analysis of the QERM Project in two parts. The first is a discussion of the macroeconomic and fiscal trends, including the situation in Moldova’s education sector with a particular focus on the primary and secondary education subsector. The second summarizes the economic analysis carried out to identify the economic gains resulting from implementation of the Project. Macroeconomic and Sector Context 103. The Republic of Moldova, a resource-poor and landlocked country, is undergoing a difficult transition from a planned to a market economy. Although a deep economic collapse marked its first decade of independence,14 economic growth resumed in 2000 and continued at an average annual rate of 5.9 percent until the 2008 global financial crisis. The economy recovered from the 2008-09 global economic crisis with average annual GDP growth exceeding 5 percent over 2010-2013. As a result, Moldova experienced the highest cumulative GDP growth among all regional partners’ relative to the pre-crisis year of 2007. At the same time, as a small open economy in which agriculture has a significant role, Moldova’s growth performance has been strong but volatile, reflecting vulnerability to climatic and global economic conditions. 104. Moldova’s recent economic performance reduced poverty and promoted shared prosperity. The national poverty and extreme poverty rates fell from 30.2 percent and 4.5 percent, respectively, in 2006 to 16.6 percent and 0.6 percent in 2012, making Moldova one of the world’s top performers in terms of poverty reduction. Similarly, consumption growth among the bottom 40 percent of the population outpaced average consumption growth: estimates for 2006-11 suggest an annualized overall growth in consumption of 2.9 percent over the period, as compared to 5.8 percent for the bottom 40 percent. These developments were driven by economic growth and the associated growth in earnings, as well as by an increase in private transfers such as remittances. Despite a sharp decline in poverty, Moldova remains one of the poorest countries in Europe. Based on the Europe and Central Asia (ECA) standardized poverty lines of US$5/day and US$2.50/day at Purchasing Power Parity (PPP), 55 percent of the population was poor and 10 percent was extremely poor in 2011. The most vulnerable groups at risk of poverty in Moldova remain those with low education levels, households with three or more children, those in rural areas, families relying on self-employment, the elderly, and Roma. Also, labor force participation in Moldova is extremely low, particularly among women, rural residents, youth (15-24 years old), older workers (55 years old and more), and those with low educational attainment. 105. Indeed, education represents an important factor that influences the level of poverty. There is also substantial evidence that human capital – defined as the accumulated stock 14 During 1991-1999, GDP declined by almost 60 percent (with the exception of 1997, when GDP grew 1.6 percent). 32 of education, knowledge, and skills – is important for economic development and growth. 15 Not surprisingly, improving the quality, relevance, and efficiency of the education system is one of the main priorities of the GoM. However, the demographic and fiscal realities of the country have not made it easy for the government to fulfill its mandate in education. Over the years, Moldova’s education sector has witnessed uneven education quality and lack of efficiency. 106. Moldova’s performance on OECD’s Programme for International Student Assessment (PISA) 2009 Plus (supported under the Project) is a substantive illustration of the symptoms of quality challenges faced by the system. Its results showed that the performance of the country’s 15-year-olds in reading, math, and science is among the lowest in the region. The performance gap with OECD peers was equivalent to a little more than two years of schooling. Overall, around 60 percent of Moldova's 15-year-olds were shown to lack the basic levels of proficiency in reading and math literacy needed to participate effectively and productively in society. Widespread cheating and corrupt practices demotivated students to study. 107. In addition, as a result of demographic decline, Moldova’s student population has shrunk significantly (by more than 50 percent since 1991), but the school network has not adjusted accordingly. The traditional system of education financing in Moldova has provided few incentives to improve sector efficiency by reducing resource utilization in response to a shrinking school-age population. The old financing system tied resources to teachers not students, thus making teachers a “free resource” from the point of view of local authorities. This arrangement has left Moldova with some of the smallest classes and lowest student-teacher ratios in the world, mainly in the country’s rural areas. In 2009, the GoM spent 9.4 percent of GDP on education, way above the regional average of about 5 percent, indicating the urgent need for efficiency reforms in the sector. 108. Moldova’s education sector was characterized by an oversized network of increasingly small schools and a high number of poorly paid and aging teachers (with relevant consequences for quality and the social prestige of the profession). The 2011 feasibility study conducted under the Project examined the network of 970 rural primary schools, gymnasiums, and lyceums and identified up to 283 low-enrollment institutions (29 percent of the rural school network) that could be optimized in the short term without impairing access to education for students in rural areas. Overall, it concluded that up to half of Moldova’s rural schools may need to be reorganized or shut down in the next three to five years to counteract the sharp population declines that have taken place over the last 20 years.16 An average school in Moldova operated at 54 percent of the capacity for which it was designed, leading to wasteful expenditures in the form of heating bills and public utilities. As a result, a high proportion of the education budget went toward 15 See: Sianesi and Van Reenen (2003), Eichengreen et al. (2013) and EBRD Transition Report (2013). Available at: www.tr.ebrd.com 16 Policy options to support the school network optimization process in the Republic of Moldova Feasibility Study, IPP (2010). 33 the financing of personnel expenditures and maintenance costs of school buildings (including expenditures on heating and utilities), crowding out much needed quality- enhancing investment in capital and educational materials. 109. Under Project support, in 2009 the government developed a new formula-based funding mechanism and school optimization plan and implemented them in two pilot rayons beginning in 2010. In 2011, based on the strong analytical work conducted under the Project (in particular, completion of a feasibility study on school optimization) and under World Bank Public Expenditure Review technical assistance, the GoM embarked on a set of ambitious initiatives reflected in the Action Plan for Education Structural Reform Implementation prepared by the MoEd and endorsed by the International Monetary Fund (IMF). 17 Activities under the QERM Project both informed the development of the government’s Education Reform Program and supported its implementation. 110. Since January 2012, the government has expanded formula-based financing to nine additional rural rayons (Criuleni, Cantemir, Călăraşi, Nisporeni, Taraclia, Ştefan Vodă, Glodeni, Leova, and Sîngerei) and two urban municipalities (Bălţi and Chişinău).18 With the piloting of per-student financing under the Project, the GoM took critical steps to improve the efficiency of resource allocation in primary and secondary education. This commitment was further strengthened by Parliament’s passage on April 26, 2012 of the much needed and groundbreaking package of amendments to the existing laws on education, labor, and local public finance. These legal changes created an enabling environment that allowed implementation of efficiency reform measures, such as the introduction of per-student financing nationwide and the change in the responsibility of a school from the mayor to the rayon council, thus enabling rationalization of school networks. Subsequently, in January 2013, the government rolled out the new per-student financing mechanism nationwide.19 Table 1: Public Expenditure on Education in Moldova, 2007-2013 2011 2012 2013 2009 (structural (13 (PSF 2007 2008 (2 pilots) 2010 reform) pilots) nationwide) Expenditures (million leu) 4,249 5,178 5,666 6,575 6,869 7,397 7,075* Percent of budget expenditure 19.0 19.8 20.7 22.4 21.4 20.9 18.6 Percent of nominal GDP 8.0 8.2 9.4 9.2 8.3 8.4 7.1 Source: MoF and BOOST database. Note: * Excluding special funds of higher education institutions. 17 Government Resolution No. 484 of July 5, 2011. 18 Government Resolution No. 535 of July 14, 2011. 19 Government Resolution No. 728 of October 10, 2012. 34 111. With these reforms in the sector, the GoM was able to bring education spending onto a fiscally sustainable footing. The education budget was approved and executed largely in line with MTEF projections for the period As a result, total education spending decreased as a percent of GDP from 9.4 percent in 2009 to 7.1 percent in 2013. Economic analysis 112. The economic analysis both at appraisal and at the time of ICR preparation took the framework of a traditional cost-benefit analysis (though no estimates of the NPV were made at the time of appraisal). 113. The actual Project costs of US$12.09 million are relatively small – equivalent to less than 1 percent of public expenditure on primary and secondary education over the eight- year period of 2006-2013. At the same time, the QERM Project has a number of benefits and economic gains. The most substantial benefits of this Project are: (i) the gain in efficiency in the primary and secondary education subsector (also, due to consolidation of school networks and increases in class and school sizes, existing capacities are used more efficiently, freeing up resources for quality-enhancing inputs); and (ii) creation of an improved learning environment in upper secondary schools. 114. Primary and secondary education is the largest subsector in Moldova’s education budget, representing around half of the education spending. From 2005 to 2010, public spending in this subsector grew 2.4 times (28 percent annually on average). The oversized and overstaffed network would have continued to require increasing resources for its maintenance: if the trends of 2005-2010 persisted, spending on primary and secondary education would have reached MDL 4.59 billion by 2013 ( 35 115. Figure 3). Over 2010 to 2013 – that is, since the launch of the per-student financing pilots followed by school consolidation and efficiency reforms in the sector – an additional MDL 1.45 billion (US$108 million) would have been required, 38.9 percent of the primary and secondary education budget or 20.6 percent of the total education budget in 2013. Figure 3: Savings from the Structural Education Reform Program Source: Calculations based on MoF and BOOST data. 116. From the 2009/10 to the 2013/14 school year, 131 schools were closed, or 9 percent of the total. Maintaining these schools in 2013 alone would have required an additional MDL 357 million from the government (an additional 5 percent of the total education spending). The MoEd also conducted a reorganization of school networks and created hub schools. During 2005-2013, 206 general education institutions were reorganized. This process consisted of transforming lyceums into gymnasiums (lower secondary education institutions) and gymnasiums into primary schools. As a result, average class size increased from 19.0 in 2010 to 20.2 in 2013 (in comparison, in 2002 and 2003 there were 22 students per class) and the student-teacher ratio increased from 10.5 in 2011 (the lowest since 1991) to 11.0 in 2013 (Figure 4). Figure 4: Student-teacher Ratio and Average School Size in Moldova, 1991-2013 36 Source: NBS data. 117. Optimization is not a goal in itself. The main purpose of the reforms was to improve the quality of education through consolidation of schools and better utilization of resources (human and physical), and was a long-overdue response to the effects of the demographic transition. Indeed, through the consolidated school network and increases in class and school sizes, existing facilities were used more efficiently, freeing up resources for quality-enhancing inputs such as increased investments into teaching aids and methodological literature, books and periodicals, training, informational technologies, and personal computers ( 118. The nationwide application of per-student financing of education in 2013 facilitated the restructuring of the school network with two main impacts: efficiency and equity. Under the more flexible per-student financing arrangement, school principals are able to reallocate funds as they see fit within the budget envelope determined by the financing formula. The size of the budget gain was found to be positively correlated with a number of school characteristics, including average class size, the student-teacher ratio, the ratio of students to non-teaching staff, building capacity utilization, average number of students per grade, and total number of students. This indicates that the formula is more favorable to lower-cost schools than was the old input-based financing system. The relationship between proportional budget change and the prime drivers of per-student costs—class size and the ratios of students to teachers and students to non-teaching staff—is consistent with this conclusion. All three variables generally rise in line with proportional budget change. The index of community deprivation—which measures the relative well-being of the communities in which the schools are located—has a very small and statistically significant inverse correlation with proportional budget change, indicating that schools in more deprived areas tend to experience a positive budget change (see Table 2). 119. Figure 5). This can already be seen in the period since the initiation of the first pilots and subsequent roll-out of the optimization and efficiency reform in the sector, though there is still room for improvement relative to the pre-crisis period. 120. The nationwide application of per-student financing of education in 2013 facilitated the restructuring of the school network with two main impacts: efficiency and equity. Under the more flexible per-student financing arrangement, school principals are able to reallocate funds as they see fit within the budget envelope determined by the financing formula. The size of the budget gain was found to be positively correlated with a number of school characteristics, including average class size, the student-teacher ratio, the ratio of students to non-teaching staff, building capacity utilization, average number of students per grade, and total number of students. This indicates that the formula is more favorable to lower-cost schools than was the old input-based financing system. The relationship between proportional budget change and the prime drivers of per-student costs—class size and the ratios of students to teachers and students to non-teaching staff—is consistent with this conclusion. All three variables generally rise in line with proportional budget change. The index of community deprivation—which measures the relative well-being of the communities in which the schools are located—has a very 37 small and statistically significant inverse correlation with proportional budget change, indicating that schools in more deprived areas tend to experience a positive budget change (see Table 2). Figure 5: Trends in primary and secondary education spending in 2009-2012 180% 160% 2009-2012 140% average 120% 100% 80% 60% 40% 20% 0% Source: BOOST public expenditure data (on the basis of MoF Treasury data). Note: Spending of primary schools, gymnasiums, lyceums, and general secondary schools only (selected categories). Table 2: Correlations between Proportional Budget Change and Variables Related to Cost per Student or Equity Variable Correlation coefficient Average class size 0.34 Student teacher ratio 0.50 Student to non-teaching staff ratio 0.20 Building capacity utilization 0.08 Total expenditure per student -0.52 Total number of students 0.13 Number of students per grade 0.14 Total expenditure -0.12 Community deprivation index -0.10 Note: All correlation coefficients are significant at the p<0.01 level. 121. The formula promotes efficiency, benefitting larger schools over smaller schools, and incentivizes rayons and cities to reorganize small, costly, and inefficient schools. It also favors rural schools over urban ones, which in the case of Moldova suggests that the formula promotes efficiency by enabling urban schools to reduce costs by decreasing the number of small classes without impairing access. Overall, the uniform per-student amount gives favorable efficiency incentives in Moldova by encouraging rayons and cities to reorganize their school networks to ensure that schools can finance their costs within their respective budget allocations. 38 122. The results of the beneficiaries’ analysis also showed that Project activities contributed to improving the quality of education in rural areas as follows. Laboratory equipment and visual aids provided under the Project helped students to better understand abstract scientific concepts. The development of a new competence-based curriculum and training of teachers to apply it helped students improve their communication, team work, and learning skills. Enhanced autonomy of schools allowed school leaders to make efficiency savings and reallocate funding towards school needs and quality-enhancing investments. Improved quality and reliability of national student assessments together with the measures on strengthening the integrity of the Baccalaureate exam system laid the groundwork for a more merit-based way of determining university admission and increased students’ efforts in the classroom. 123. Upper secondary education (the focus of the QERM Project in terms of quality improvements) plays an important role in the progression from compulsory education either to further education (in particular, higher education) or to the world of work. Given the lack of national data regarding the progression of upper secondary graduates to higher education institutions by urban-rural divide (plus the existing quota system for publicly- financed seats), the labor market outcomes of graduates were reviewed, with the caveat that these largely depend upon labor market conditions (of both the supply and demand side). A substantial share of Moldova’s employed population (19 percent) has an upper secondary education degree: 16 percent in urban areas and 21 percent in rural areas. According to National Bureau of Statistics (NBS) data, after a drop in 2010, graduates with an upper secondary education degree increasingly report being able to find and take on jobs above their education level (Figure 6). The share of unemployed 15- to 24-year- olds with an upper secondary education in rural areas is declining (Figure 7), although the gap in the disposable labor-related income of graduates from rural and urban areas is widening. Figure 6: Employed Persons Aged 15-24 with Upper Secondary Education in Moldova: Occupation versus Field of Study, 2007-2012 100% job corresponds to education 80% level/study 60% below education level 40% 20% above education level 0% 2007 2008 2009 2010 2011 2012 2007 2008 2009 2010 2011 2012 2007 2008 2009 2010 2011 2012 equivalent, but unrelated Employed Urban Rural Source: National Bureau of Statistics, Labor Force Survey data. Figure 7: Share of Unemployed Persons Aged 15-24 with Upper Secondary Education in Moldova, 2007-2011 39 35% 30% 25% 20% Upper secondary (average) 15% Urban 10% Rural 5% 0% 2007 2008 2009 2010 2011 15-24 years Source: National Bureau of Statistics, Labor Force Survey data. 124. Given the high uncertainty associated with predicting the labor market outcomes of individuals and linking those outcomes to investments made today, the cost-benefit analysis focused on the benefits of the Project associated with the fiscal savings from the efficiency measures targeted under Component 3. The NPV of the savings is calculated based on a discount rate of 10 percent and covers the period of 10 years from Project completion (2013-2033). Under these assumptions, the savings from the school consolidation alone amounts to about US$154.6 million in NPV terms, equivalent to 29.6 percent of Moldova’s education budget in 2013. 40 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Responsibility/ Name Title Unit Specialty Lending Ana Maria Parchuc de Jeria Project Team Leader, Sr. Human ECSHD Figueroa Development Specialist Hideki Mori Sr. Social Protection Specialist ECSHD Robin S. Horn Lead Education Specialist ECSHD Ala Pinzari Operations Officer ECSH2 Antonio Lim Operations Officer ECSHD Nadejda A. Mochinova Sr. Program Assistant ECSHD Tamara Ursu Program Assistant ECCMD Nicholay Chistyakov Sr. Finance Officer LOAGl Daria Goldstein Sr. Counsel LEGEC Bogdan Constantinescu Sr. Financial Management Specialist ECSPS Anna Wielogorska Sr. Procurement Specialist ECSPS Martin Godfrey Consultant ECSHD Supervision/ICR Ana Maria Parchuc de Jeria Project Team Leader, Sr. Human ECSHD Figueroa Development Specialist Scherezad Joya Monami Project Team Leader, Sr. Education SASED Latif Specialist Project Team Leader, Sr. Operations Andrea Guedes ECSH2 Officer Project Team Leader, Senior Nina Arnhold MNSHE Education Specialist Project Team Leader, Lead Education Ana Maria Sandi ECSH2 Specialist Anna Olefir HD Operations Officer ECSH2 Lucas Gortazar Junior Professional Associate ECSH2 Igor Kheyfets Economist ECSH2 Lars M. Sondergaard Country Sector Coordinator EASHE Irina Babich Financial Management Specialist ECSO3 Raluca Marina Banioti Program Assistant ECCRO Andrei Busuioc Sr. Financial Management Specialist ECCAT Oxana Druta Financial Management Analyst ECSO3 Anna Goodman Program Assistant ECSHD Robin S. Horn Adviser PA9SS Antonio C. Lim Operations Officer ECSEG Mariana Doina Moarcas Sr. Operations Officer ECSH2 Nadejda A. Mochinova Resource Management Officer AFTRM Ala Pinzari Operations Officer ECSH2 Ana Maria Sandi Consultant ECSH2 Irina Shmeliova Procurement Specialist ECSO2 41 Patricia K. Tibbetts Consultant SASED Tamara Ursu Program Assistant ECCMD Kirill Vasiliev Education Specialist ECSH2 Anna L Wielogorska Sr. Procurement Specialist EASR1 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) USD Thousands Stage of Project Cycle No. of Staff Weeks (including travel and consultant costs) Lending FY05 22.18 144.22 FY06 40.01 200.23 FY07 43.42 270.09 FY08 38.46 232.10 Total: 144.07 846.64 Supervision/ICR FY09 34.65 107.22 FY10 46.53 124.82 FY11 52.35 170.09 FY12 25.56 116.50 FY13 14.61 59.88 FY14 10.96 60.90 Total: 184.66 639.41 42 Annex 5. Beneficiary Survey Results 125. As part of the ICR preparation, a study was conducted to document beneficiaries’ perceptions of the overall impact of the QERM Project on the quality of education in rural areas. The study also took into account the social assessment conducted in 2005 in preparation of the Project to gauge beneficiaries’ perceptions of Project interventions, and replicated its design to have a baseline for comparison. The 2005 assessment encompassed 21 in-depth interviews among high-level rayon officials, mayors, school directors, teachers, and parents. In addition, five focus groups were conducted among teachers, students, and parents to discuss the major problems faced by schools and the proposed solutions. 126. The most pressing problems identified in 2005 were: (i) the lack of modern quality teaching and didactic aids as well as appropriate school premises; and (ii) the shortage of sufficient and satisfactory teaching staff. Most respondents agreed that these issues were compounded by the high level of curricula complexity, low quality of textbooks, poor school libraries, and lack of computers and internet access. Increased autonomy of schools was seen as beneficial in the opinion of some respondents; the low number of students will make it inevitable in the future to establish school networks, which could help improve school facilities/maintenance and solve the problem of teacher shortages in particular subjects. In addition, taking into consideration the needs, views, and recommendations raised in other studies,20 specific activities were built into the Project to address and mitigate concerns expressed by stakeholders. For example, regarding more participatory local government, the Project included the introduction of formula-based funding together with decentralization and higher financial autonomy of schools. 127. The 2014 study covered the same three rayons as those in the initial social assessment (Făleşti in the North, Teleneşti in the Center, and Leova in the South). In each of the three rayons, rural high schools (lyceums) were targeted, bringing the total number of schools to 21. Overall, 30 in-depth interviews with key informants (education department officials, school directors, and principals) and 15 focus groups (with parents, students, and teachers) were conducted. To gauge beneficiaries’ perceptions of the impact of Project interventions, it covered the relevant areas of curriculum, teaching and learning materials, school financing and autonomy, school grants, and learning assessments. Schools principals and teachers were the ones found to be most familiar with the Project. They identified it with the laboratory equipment received through the quality and equity grants, participation in the competence-based curriculum training for teachers and provision of the associated curriculum guides, and the financial autonomy training for principals and accountants. 20 Changing the Way Our Services are Governed? Bottom-up Governance and Service Delivery in Moldova: Opportunities and Challenges for Effective Mechanisms of Participation (2005). 43 128. The results showed that Project activities contributed to improving the quality of education in rural areas, as the Project: (i) provided laboratory equipment and visual aids that helped students to better understand abstract scientific concepts; (ii) supported the development of and trained teachers on a new competence-based curriculum (which is more practical than the old curriculum, and helped students improve their communication, team work, and learning skills); (iii) enhanced the autonomy of school principals, allowing them to seek efficiency savings and reallocate funding including towards quality-enhancing investments; and (iv) helped improve the quality and reliability of national student assessments that, together with the measures on strengthening the integrity of the Baccalaureate exam system, laid the groundwork for a more merit-based way of determining university admission and increased students’ academic efforts. 129. The new competence-based curriculum is considered by beneficiaries to be more practical, applied, and clear than the old one, emphasizing interactive participation from students. This has resulted in more student interest in the subject matter and greater focus on peer learning. Students now have better communication, teamwork, and learning skills, as well as “life skills.” Several principals cited the curriculum reform as one of the factors contributing to the improvement in learning over time. The new curriculum was significantly simplified. However, some principals and teachers think that it is still too heavy. A greater focus on interdisciplinary approaches is needed (e.g., between physics and math), as well as better alignment of curriculum and teaching materials and textbooks. Also, while the curriculum is competence-based, evaluation still largely focuses on reproduction of material learned rather than on competencies. Nevertheless, most teachers confirmed that the curriculum guides provided for all subjects in high school under the Project were very useful and timely and supported introduction of the new curriculum into the teaching process. Most teachers were satisfied with the content of the guides and considered them useful in preparing and conducting lessons and in evaluation. 130. School principals had positive perceptions towards per-student-based financing and wider autonomy of educational institutions, stating that this provides them with the opportunity to solve the most pressing problems of their institutions. Principals generally preferred the new financial autonomy regime, despite having doubts about the new responsibilities prior to its rollout. None preferred to go back to having these decisions made at the municipal level, and all stated that all per-student allocations reach the educational institutions. School managers also reported receiving sufficient support (of high quality and usefulness) and training (abundant in information) in implementing the new measures of school financial autonomy. They have succeeded in undertaking some improvements in terms of equipping their institutions. Savings have been channeled to installing new energy efficient windows and purchasing new teaching materials or computers. Procurement processes have become more efficient because school directors know better what they need as opposed to municipal accountants, who were the de facto decision makers under the previous system. The new financing mechanism is seen to work well for schools with high capacity utilization. 44 131. Beneficiaries reported that prior to the quality and equity grants provided under the Project, schools rarely received equipment (some had not received new supplies for around two decades). Equipment procured through grants included projectors, computers, etc. These materials are used widely and were seen to contribute to student learning. In general, school principals and teachers confirmed that the schools’ lab equipment was useful. The labs served as visual aids for students, very important for helping them learn interactively, visualize abstract concepts, and better prepare to study science-related disciplines. At the same time, teachers reported issues with the quality and durability of biology and chemistry lab equipment. Some teachers mentioned the lack of freedom to select certain items of lab equipment within the total amount of the grant. Rather, they received sets that sometimes contained items the school would prefer to replace with other necessary equipment. There is still a lack of school lab equipment, understandable given the one-time quality/equity grant interventions under the Project. At the same time, the training that schools received in grant writing under the Project was reported as useful, allowing school managers to independently raise funds from other sources for school investments later on. 132. According to beneficiaries, thanks to the strengthened integrity of the Baccalaureate exam system, both students and teachers started to treat studies more seriously, particularly from grade 10, though some students opted for vocational education for fear of failing the exam. Now students get a score based on what they know, motivating them to study harder; this is expected to have a positive impact. These views are generally shared by most school principals and teachers. In the long run, the reform of the Baccalaureate exam system is likely to improve the quality and meritocracy of education by boosting student effort and forcing students to study to receive higher results. Parents and students in schools confirmed that they had been informed about the new exam rules, as well as the exam results for their school. 133. Lastly, in terms of awareness of Moldova’s participation in international assessments, it was mostly representatives of rayon education departments who were familiar with the PISA 2009 Plus and its results, as they were discussed at a meeting of rayon education heads in Chisinau, as well as at rayon meetings. Some math teachers were also aware of the PISA as it was discussed during the methodological meetings for teachers in terms of the results and related issues of practical application of students’ knowledge. These educators’ general perceptions of trends in education quality were positive over the time that they had been in their jobs (ranging from 4 to 20 years). Nevertheless, despite the fact that the equipment of rural schools has considerably improved over the last five years, there is no conclusive evidence that the gap in the quality of education in rural and urban areas has narrowed, since urban schools are generally better off due to the greater capacity of authorities and parents in urban areas to support them. 45 Annex 6. Stakeholder Workshop Report and Results Not applicable 46 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 134. Project achievements are reviewed in light of what was planned, what was modified during implementation (with justification for the causes of changes), the actions undertaken, and the results obtained. Component 1. Improving Teaching and Learning in Rural Schools 135. The main objective of this component was to improve the quality and relevance of teaching and learning in rural schools, especially in lyceums (grades 10-12), by strengthening the three basic (MoEd) pedagogical systems: curriculum, teaching, and assessment. The component was implemented by the MoEd Department for Preschool and General Education and by the QAA. The component consisted of three subcomponents. Subcomponent 1.1. Modernizing Curriculum for Grades 10-12 136. The basic objective was to complete the modernization of the national curriculum by finalizing and revising its design, implementing a new curriculum for grades 10-12, and supporting the new curriculum with new textbooks and teaching materials. Implemented activities focused on the main approved directions and the results are as follows: (i) A Reference Framework for the National Curriculum was developed, edited, and distributed in all lyceums in the country; (ii) Guidelines for the implementation of the modernized curriculum for 17 school subjects were developed; (iii) Training on the guidelines was provided to 8,284 teachers from lyceums; (iv) 1,190 schools from rural areas were endowed with equipment for some disciplines and their curricular areas strengthened under the equity grants program. Subcomponent 1.2. Teacher Professional Development for Grades 10-12 137. The objective of this subcomponent was to deliver training to teachers from lyceums in rural areas to support implementation of curriculum for grades 10-12 (secondary level). The achievements of this subcomponent are as follows: (i) Standards were developed for teachers’ continuous education training for general secondary education; (ii) Base courses (10 guides for training school teachers) were developed. 3,000 copies of each of the 10 guidelines were published in Romanian and 500 copies in Russian, for a total of 35,000 copies; (iii) Methodological training of teachers was conducted in the 10 selected rayons; (iv) The training provider’s performance was assessed; (v) Ten training centers at the rayon level (Rîşcani, Donduşeni, Floreşti, Rezina, Teleneşti, Nisporeni, Anenii Noi, Leova, Cimişlia, and Gagauz Yeri) were equipped with training equipment (notebooks, printers, and projectors). The procurement and distribution of equipment was completed in September 2009. 47 138. To achieve these results, the MoEd approved the regulation of assessment and accreditation of institutions (and continuous education training programs) by Order No. 203 on March 29, 2008. Meanwhile, due to changes in the MoEd structure, the assessment function was transferred to the QAA. 139. Methodological competence development of teachers from rural lyceums was conducted with the effort of the MoEd, the Training and Science Institute, rayon education departments, lyceums’ administrations, and Project consultants. At the central level, the Training and Science Institute trained 394 local trainers in November 2008. At the local level, trainings were organized with support from rayon education departments: 451 seminars were held on methodological competences development for 16,909 teachers in lyceums, of which 11,875 or 70.2 percent were from rural areas (369 schools), while 5,034 teachers, or 29.8 percent, were from urban areas (82 lyceums). Subcomponent 1.3. Establishment of National Centre for Examinations and Quality Assurance 140. The objective of this subcomponent was to establish a National Centre for Examination and Quality Assurance under the MoEd (currently the QAA), to effectively assess students, examinations and testing, and accreditation and to strengthen national capacities for measurement in education. Another objective was to strengthen the security of exams, especially the Baccalaureate exam, which is highly important in Moldova’s education system. To achieve these objectives, the following activities were undertaken: (i) Establishment of the Quality Assurance Agency The QAA was created based on Government Decision No. 1469, dated December 2006. The goal of the QAA is: “assuring quality education, carrying state’s policy on external evaluation of the secondary general educational system, and improving national capacity in educational measurement.” (ii) Development and installation of specialized software for the QAA Development and installation of the ADPS (automated data processing system) software was done between February and October 2008. In December 2008, the QAA organized training courses for local trainers from rayon education departments and for 120 staff responsible for managing the database of the 2009 Baccalaureate exam for general secondary education institutions, higher education institutions, and republican status lyceums. Additional training for 440 users was organized in January 2009. (iii) Provision of equipment for the Evaluation and Assessment Agency (now QAA) The multiplication equipment, advanced computers, scanners, projector, servers, and furniture for the QAA were delivered by June 2008. (iv) Organization of study visit for Moldovan specialist in evaluation A study visit was organized from November 10–17, 2007 to the National Examination Centre in Lithuania. The quality of the management process of pre- 48 university education assessment, the Republic of Lithuania’s experience in evaluation, and other aspects of the evaluation process and review were studied.21 (v) Training of teaching staff on in-classroom formative evaluation methods Training on this topic was carried out in 2008. Training of trainers was held in 10 republican vocational centres (Rîşcani, Donduseni, Floreşti, Rezina, Telenesti, Nisporeni, Anenii Noi, Leova, Cimişlia, and Gagauz Yery). The training was organized for 173 local trainers, who conducted the training for lyceum teachers. These trainings were organized in each rayon with support from the rayon education departments. (vi) Implementation of baseline survey in 2007 A baseline evaluation of the basic capabilities of 8th and 10th grade students was made by the State Institute of Continuing Education (SICE). The main study objective was to prepare a baseline of education results to enable comparison of Moldova with other countries. The study included:  Collection of data to monitor the capability of students in mathematics and native language based on the curriculum for grades 8 and 10;  An assessment of the quality of pre-university education in rural areas;  A comparison of students’ results depending on their residential environment;  A comparison of students’ current test results with results obtained in other national and international evaluations;  Identification of variables that affect students' abilities, such as training, language, time spent working on computers, Internet use in preparing lessons, parents' level of education, and level of study students intend to obtain. (vii) Support for completion of tests at the end of primary cycle and exams at the end of the gymnasium cycle in 2007, 2008, and 2009, and for completion of the annual Baccalaureate exam. The most important activity was implementation by the QAA of the final assessments for grades 4 and 9, the Baccalaureate exam, and the graduation exam at the end of 11th grade of general school. For these purposes, the Project supported the QAA in: 21 Topics of study and research were: the development of statistical basis, its correlation between areas and steps of school, and participation of the Republic of Lithuania in international assessments (PISA and TIMSS). Within these objectives, participants heard and discussed communications presented by employees of: the National Examination Center of Lithuania, the Ministry of Education and Science, the Center for Development of Education, the Municipality of Vilnius, the Center of Information Technologies, two gymnasiums, and National Agency for the Evaluation of Schools. Studying the practice of the Republic of Lithuania in the field of assessment of academic results by the National Examination Center had a positive impact on QAA’s activities on: organization and evolution of the exam administration procedure in Moldova; development of exam programs; development of the items bank; organization of the pre-test; content establishment; correlation of the structure of the test exam to the level of curricular areas; development of a test focused on competences and administration of the test during examination; and development of the statistical basis and its correlation to areas and steps of the school. 49  Organizing trainings for the heads of rayon education divisions and the heads and deputy heads of training for republican methodological centers;  Publishing programs for all subjects on which examinations are held for the school steps and a newsletter (in the state language and in Russian) that contained exclusively regulations on the organization and conduct of examinations and other acts aimed at the evaluation process;  Launching the QAA’s webpage (http://www.aee.edu.md);  Providing online access to the centralized database to any student who takes exam, allowing them to review their test results. (viii) Participation in international tests One of the objectives of the QAA was to encourage Moldova’s participation in international tests and assessments of school performance. With the financial support of the Project, the QAA under MoEd ensured Moldova’s participation in the TIMSS 2007 and the PISA 2009+ assessments. By participating in the PISA 2009+ test, Moldova met the following PISA objectives: (i) evaluation of the skills of 15-year-old students who participated in the test; (ii) identification of the skills students need to integrate into society; (iii) and analysis of the knowledge application students have gained in school in reading/understanding of text, mathematics, and science as well as skills in solving concrete life problems. The Project covered the costs related to preparation of training materials, organization of a pre-test, basic PISA 2009 + testing requirements, etc. Additional Financing for Quality Education Activities 141. Under the AF for the Quality Education Financing Agreement, Component 1 was charged with undertaking the following additional activities: (i) Carrying out studies to support the development of standards for receiving schools The Quality Assurance Standards developed for schools with Project support were approved by the MoEd Order No. 970, dated November 10, 2013. (ii) Strengthening the ability to monitor the quality of education The Project: supported the QAA in the development of various academic skills assessment tools; developed a brochure containing assessment tool samples; and delivered a 4-day training for group coordinators and members of groups that developed tests for the final assessments in primary and secondary education. Component 2. Increasing Access and Equity in Rural Schools 142. The objective of this component was to grant schools and poor rural communities additional resources to improve the quality of and access to educational services. Also, eligible schools would be helped to learn to manage the additional resources in a transparent and efficient manner and to be accountable for results. Responsibility for implementation of Component 2 was given to the MoEd’s budget department, and was implemented through two subcomponents. 50 Subcomponent 2.1. Rural Schools Equity Grants 143. The purpose of this subcomponent was to provide educational institutions in rural areas with basic didactic materials and equipment for subjects of study and curricular areas by providing a grant estimated at about US$2,000 to each eligible school. To achieve these objectives, the following activities were undertaken: (i) Purchase of teaching equipment All rural schools in the country that fit eligibility criteria set by the MoEd received equity grants. The list of eligible institutions was developed within the school mapping project database and carried out during the pre-Project phase. Eligibility criteria developed by the MoEd were placed in catalogs of equipment and educational materials published during the pre-Project phase and distributed to all educational institutions of the country (two catalogs per institution). Schools selected equipment and didactic materials from these catalogs based on the amount allocated and according to their current needs. Equity grants comprised the two curricular areas of arts and technology and language and communication, and disciplines such as physical education, biology, chemistry, physics, mathematics, geography, IT, and history. Equipment and didactic materials were purchased in October – December, 2006. (ii) Distribution of teaching equipment to schools in rural areas All types of educational institutions in rural areas benefited from equity grants, except for primary schools and other types of schools with less than 80 students. Equipment and didactic materials purchased within that tender were distributed to 1,190 eligible institutions in rural areas (gymnasiums, secondary schools, and lyceums) from 2007 until the beginning of 2008. Equipment and materials included: illustrative teaching materials, some tools, molds and equipment for arts and technology; a sports inventory for physical education; tools and tables for mathematics; teaching materials for Romanian, Russian, and foreign languages; maps and globes for geography; tables for informatics; maps and tables for history; and some equipment for physics, biology, and chemistry. Subcomponent 2.2. Rural Schools Quality Grants 144. The objective of this subcomponent was to provide quality grants on a competitive basis to eligible rural lyceums to supply basic didactic equipment of laboratories for physics, chemistry, and biology as well as to supply them with information on techniques, thus improving the quality of studies in these disciplines. Also, the subcomponent was intended to encourage lyceums to manage their resources effectively, to prioritize needs, and to be accountable for school performance results. The following activities were taken to provide schools with equipment: (i) Criteria development and evaluation of grant proposals Quality grants were awarded via a competition to which all eligible lyceums from rural areas registered at the time of the announcement could apply. To participate in the competition, rural lyceums had to: prepare and submit a project proposal for school unit development; demonstrate the need for resources to achieve their goals; 51 and prove their availability to contribute 10 percent of the size of the grant, provided through services or in cash. All stages of granting the quality grants were made in accordance with the Grants Operations Manual (GOM) component of the QERM’s Project Operational Manual approved by the World Bank. (ii) Assistance to schools in developing grant proposals To assist rural lyceums in preparing and presenting grant proposals, the first phase of training seminars for school directors and members of Regional Grant Review and Approval Committee (RGRAC), created under all rayon education departments, took place between October and December 2006. Of the 343 rural lyceums registered by September 1, 2006, 336 participants from lyceums (98 percent) and 144 members of RGRAC sites attended the first stage of training. By the end of 2006, all lyceums in rural areas had worked on development of projects for participation in the competition, and 251 lyceums presented projects for the competition. Under decision of the National Grant Review and Approval Committee (NGRAC), MoEd Order No. 134 of March 16, 2007, quality grants were awarded to 131 lyceums. Beginning in summer 2007, the program of training for school directors, members of the focus groups, and RGRAC members continued: 220 people attended seminars with focus groups, and 228 people attended the second stage of training for school directors. By the end of 2007, 199 projects had been drafted and presented for the second phase of the contest. After evaluation, based on NGRAC decision, by MoEd Order No. 102 of February 26, 2008, quality grants were awarded to 173 lyceums. (iii) Purchase and delivery of teaching equipment Extensive laboratory equipment for physics, chemistry, biology and IT technology was purchased for the 131 schools selected in the first phase of the contest and distributed to them by the end of the 2007/08 school year. International tender for the procurement of equipment for the 173 lyceums selected in the second phase of the competition took place on May 23, 2008, and the equipment was distributed by the end of 2008. (iv) Impact assessment of the grants programs A study of the grants’ impacts was initiated to: determine the degree of equipment use in the educational process; establish gaps in procurement and distribution of equipment; measure the immediate effects of the use of grants in the educational process; evaluate the impact of the training program for lyceum directors in the preparation and implementation of the projects; evaluate the capacity of the school management in project implementation; and determine the visible and measurable impacts of the Project on target groups. The first phase of this survey took place from February to April 2009. A report on the survey’s findings on initial achievements was submitted to the MoEd on May 12, 2009. The survey’s conclusions were that:  The process for selection of equipment and teaching materials was generally regarded as transparent and objective. 52  The quality of equipment received was assessed as satisfactory for both types of grants.  Project benefits identified by teachers were: improvements in the technical materials base; facilitated teaching of the material and its assimilation by students; increased opportunities for implementation of practical work; and greater opportunities for students to perform practical work and to work in small groups.  The shortcomings were mainly as follows: a few materials were not satisfactory and not all materials requested were received (notebooks, educational software); some were received without being needed (selection not of particular items but of the whole module); there was a long process for materials distribution; there was unfair granting of certain subjects; there was incomplete training of teachers in use of equipment; and there was lack of sufficient information about the Project. Component 3. Increasing Efficiency in the Use of Resources 145. The basic objective of this component was to increase the efficiency of the public financial allocation to schools and to encourage more efficient use of resources in schools, in the context of projected demographic changes (school-age population, class sizes, and student/staff ratios), including the use of formula-based funding in all schools. Responsibility for implementation of the component, which consists of three subcomponents, was delegated to the MoEd budget department. Subcomponent 3.1. Education Budgeting Improvement 146. The objective of this subcomponent was to improve and increase the capacity and role of the MoEd in preparation of annual budgets and its capacity in costing activities based on MTEF projections for the education sector. 147. A local consultant worked with MoEd staff and used sector data to prepare the methodology that formed the basis for two new regulations developed in consultation with the MoF: (i) regulation of the relationship between the MoEd and other central authorities with subordinate educational institutions; and (ii) regulation of the relationship between the MoEd and rayon education departments. 148. For the purpose of training specialists involved in MoEd budget costing, data collection and preparation MTEF forecasts, an organizational plan to conduct seminars in the area was developed, but methodology of preparation of MTEF was not developed. MTEF development work is currently managed by the MoF. Subcomponent 3.2. Formula Funding and School Autonomy Pilots 149. The objectives of this subcomponent were to:  Pilot a system of self-financing and governance of schools in selected rayons to increase efficiency in resource utilization by the end of 2010;  Approve a formula-based financing mechanism for primary and general education schools by the end of 2010;  Implement the formula across the country by the end of Project life; 53  Collect data on dropout rates of students affected by the school rationalization; and  Provide training to personnel responsible for formula-based financing and school optimization. (i) Formula piloting To develop a new formula-based funding mechanism that would meet the necessary requirements and standards, an international consultant with extensive experience in the field was hired competitively to provide assistance to the MoEd. Together with the MoEd budget department and MoF staff, the consultant prepared three variants of formula-based funding. In July 2008, the variant of the formula considered most acceptable for Moldova (variant 1B) was approved. To estimate formula parameters, data from 1,202 schools (81 percent of the total), covering 357,000 students (76 percent), were used. The percentage of schools reorganized in two pilot rayons based on the new formula- based funding mechanism was as follows:  2007/08 - 40 percent of all schools  2008/09 - 80 percent of all schools  2009/10 - 100 percent of schools. (ii) Formula approval and implementation across the country The final formula now in use was approved by Government Decision No. 728, dated October 2, 2012. As of January 1, 2013, all school budgets from primary and general education system in the country have been approved based on the provisions of the new formula. (iii) Data on dropout rates The dropout indicator was needed to better monitor the school network optimization reform’s effects. This indicator was considered and introduced into the database during implementation of the school censorship and improvement of the school mapping system (and creation of the new student-level Education Management Information System). This helps tracking the reform process (monitor dropouts and support mitigation measures), especially given the uncertainty of the previously available data. (iv) Training on formula application and school optimization A series of trainings were organized during the formula’s pilot and after its approval. In June 2009, a team of consultants prepared and organized two informative seminars on the new formula-based funding and school autonomy in the pilot rayons. Seminars in both rayons were attended by over 200 people, including heads of rayon councils, heads of rayon education departments, heads of finance departments, school managers, and accountants. The seminars were deemed successful, registering the interest of local representatives in application of the new formula as well as in the expansion of school autonomy. Between December 2012 and February 2013, another round of training on the formula’s application was delivered by Project budget experts. Other trainings were organized at the request of local authorities through the end of the Project. 54 Subcomponent 3.3. Grant for the Optimization of School Networks in Selected Rayons 150. This subcomponent had two main activities:  Conduct of a feasibility study; and  Implementation of a grant program for optimization in pilot rayons and evaluation and dissemination of the results. 151. During the first quarter of 2009, the Project team conducted a study on the "Current situation of general secondary education in pilot rayons Căuşeni and Rîşcani," referring to the school network, provision of space for educational institutions, provision of human resources, and school spending. The analysis identified several odd situations that required clarification. For a deeper analysis of the situation in schools, an "Assessment of Educational Institutions" was developed, enabling the collection of additional information as necessary to improve analysis of information in both pilot rayons’ schools and improve the formula used for funding. Key findings include: a) The analysis of pilot rayons’ schools’ building capacity and number of students found that about 41.8 percent of the existing capacities were not used, creating undue costs, which in turn reflected on the current state of schools: in Căuşeni, the average rate of capacity utilization of the premises was 62.5 percent; in Rîşcani, the average rate of capacity utilization of the buildings was 52.5 percent, 10 percent less compared to that in Căuşeni. b) The analysis of schools’ human resources found that the student/teacher ratio was 11.8 on average in the Căuşeni rayon, and the ratio of students/other staff was 10.9. The student/teacher ratio in Rîşcani was 10.8 on average, or 8.8 percent less compared to Căuşeni, while the student/other staff ratio was 10.6. c) The total expenditure planned for 2009 for educational institutions in Căuşeni constituted 60.34 million leu, or 4880.35 lei per student on average, while in Rîşcani, total expenditure planned was 37.34 million lei, or 4785.98 lei per student on average. Expenditure for payment of goods and services constituted an average of 23 percent of the total in Căuşeni and 24.7 percent in Rîşcani. The largest shares of these costs were for: meals for students (about 6.3 percent and 6.2 percent of total spending in Căuşeni and Rîşcani, respectively); gas (4.9 percent and 4.8 percent, respectively); fuel (3.1 percent and 4.8 percent, respectively); electricity (2.3 percent and 2.1 percent, respectively); and current costs for repairs (1.4 percent and 1.7 percent, respectively). 152. To encourage and support pilot rayons and to create improved conditions for the schools under constituency, an expenditure of US$1 million was planned to benefit schools undergoing restructuring, if they matched the criteria to be developed. The funds were used to procure 10 buses. Additionally, 11 hub schools benefited from one computer class each. In nine rayons, 304 blackboards were delivered. 55 Component 4. Strengthening Education Planning and Monitoring 153. The main objective of this component was to develop the MoEd’s capacity to better manage the education system by strengthening its capacity to promote and monitor policies and programs. Implementation of this objective was made possible through focused activities targeted at: (i) Development of human resources in the education sector in policy and monitoring at the ministerial level; and (ii) Improvement in education management and administration at the rayon level. 154. The Project financed the initial mobilization of investments through on-the-job training in policy formulation and monitoring. The component consists of two subcomponents. Subcomponent 4.1. Policies and Monitoring Development 155. The former Computerization and Didactic Provision of Education Directorate was responsible for implementation of this subcomponent. According to the original concept, a Strategic Policy and Program Group (SPPG) had to be created to consolidate and make more cohesive the various educational-related initiatives. 156. When the MoEd structure changed at the end of 2006, “The data analysis, monitoring and evaluation policies unit” (DAMEP) was created. Subsequently, a new “Concept and Action Plan” for the entire subcomponent was developed and approved in early 2007. These changes required a new concept for the subcomponent and a review of the Project Operational Manual (developed and approved by the MoEd Order in April 2007). Out of Project funding, both directorates responsible for the component were equipped with IT equipment, and DAMEP was also equipped with office furniture. 157. Training of MoEd staff in educational policy development was done by Centre "Education 2000 +", Romania, in consortium with the Institute for Public Policy from Moldova, which organized a training in the first quarter of 2008. Training focused on two modules: I - Educational Policies Development and Implementation; and II - Techniques for Developing Educational Policy Documents. Twenty persons attended these training seminars, and each participant received course support, books, and other materials related to policy and education management. Subcomponent 4.2. Rayon Education Administration Improvement 158. This subcomponent supported training activities for representatives of rayon administrations to improve management capacity at the local level. Training focused on: (i) administration of education; (ii) educational management; and (iii) management of data and information required to support the national program. i) Training activities The purpose of these activities was to ensure effective training of rayon staff. Rayon education divisions were equipped with the necessary materials to conduct training 56 seminars and other related activities. This equipment was purchased together with the endowment of the MoEd Directorates described under Component 4.1. In 2008 and 2009, a series of trainings were conducted on two modules: a) Implementation of ICT in education; and b) Data management. Seminars were attended by 94 people in the first round of trainings and 70 people in second round. Participants were technical specialists responsible for IT, heads of methodological centers, and specialists (e.g., economists from rayon education departments). ii) The public information campaign The results of the public information campaign were most visible in the last year of Project implementation. A series of articles about the school network optimization were published in central and local newspapers and two roundtables were organized on the school optimization reform. A documentary was produced that covered the positive results of the school network reorganization and hub school creation. It reported on improvements in the educational process and the changes made to optimize education spending. 57 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable 58 Annex 9. List of Supporting Documents 1. Bonilla-Chacin M., and R. Nayar. 2004. “Education and Poverty in Moldova.” Technical Background Paper for the report Recession, Recovery and Poverty in Moldova. World Bank, Washington, DC. 2. International Development Association. 2004. Country Assistance Strategy. IDA, Washington DC. 3. FTI Secretariat. 2004. Education for All Fast-Track Initiative: Framework Document. World Bank, Washington DC. 4. IMF. 2004. “Republic of Moldova: 2003 Article IV Consultation – Staff Report; and Public Information Notice on the Executive Board Discussion.” IMF Country Report №04/39. IMF, Washington, DC. 5. Institute of Public Policy. 2003. Reforming Education System. EPIGRAF, Chisinau. 6. International Development Association and International Monetary Fund. 2004. Joint Staff Assessment of the Poverty Reduction Paper. IDA and IMF, Washington, DC. 7. OECD. 2002. Thematic Review of National Policies for Education – Moldova. CCNM/DEELSA/ED, OECD, Paris. 8. Institute of Public Policy. 2010. Policy options to support the school network optimization process in the Republic of Moldova Feasibility Study. 9. Republic of Moldova. 2003. Economic Growth and Poverty Reduction Strategy (2004-2006). Chisinau, Moldova. 10. Republic of Moldova. 2004a. Medium Term Expenditure Framework (2005-2007). Chisinau, MTEF Team, Ministry of Finance. 11. Republic of Moldova. 2004b. Education for All National Action Plan (2004-2007) Chisinau, EFA Team. 12. Tibi C., S. Berryman, and M. Peleah. 2002. Moldova’s Education Sector: a Financing Strategy to Leverage System-Wide Improvement. World Bank, Washington, DC. 13. World Bank. 2004. Recession, Recovery and Poverty in Moldova. Report №28024- MD. World Bank, Washington, DC. 14. World Bank. 2004b. HD Strategy Note – Moldova. World Bank, Washington, DC. 59 15. World Bank. 2005. Moldova Education Policy Note: Analysis in Support of Improvements in Quality, Equity and Efficiency in the Education Sector. Report №32882-MD. World Bank, Washington, DC. 16. World Bank. 2009. Country Partnership Strategy for Moldova 2009-2012. World Bank, Washington, DC. 17. World Bank. 2011. Moldova's Path to Quality in General Education: The Plan for System Optimization. World Bank PER Policy Note. World Bank, Washington, DC. 60 IBRD 33448R 27°E 28°E 29°E 30°E Dnes tr To Vinnytsya UKRA INE To Chernivtsi Moghiliov- To Vinnytsya Ocnita Podolski Briceni MOLDOVA Donduseni B To Chernivtsi Edinet Soroca e Drochia s 48°N Camenca 48°N Rîscani s Floresti Nist ru a Costesti Soldanesti ˘ r Glodeni Balti Rîbnita Rezina Balatina a Pr Sîngerei ut To Voznesens'k r Falesti ˘ Telenesti 0 10 20 30 40 Kilometers a Chiperceni b Orhei 0 10 20 30 Miles RO MAN I A Sculeni Dubasari ˘ TRANSNISTRIA i To Pascani Mt. Balanesti Calarasi ˘˘ Criuleni (430 m) a Ungheni Straseni ˘ Grigoriopol Nisporeni ˘ Stauceni To Zhmerynka 47°N ˘ CHISINAU 47°N ˘ Lapusna Ialoveni Anenii Noi Tiraspol Hîncesti Bender Leuseni (Tighina) Slobozia ˘ Cainari Causeni ˘ To Odesa Plain Cimislia c ˘N Stefan-Voda ist To Birlad ea ru g Leova Bu This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank Comrat Basarabeasca Group, any judgment on the legal status of any territory, or any To Birlad endorsement or acceptance of such boundaries. Cantemir 27°E ˘ ˘ GAGAUZIA Ceadîr- To Artsyz UKRA INE Lunga MO LDO VA 46°N 46°N Prut SELECTED CITIES AND TOWNS Cahul Taraclia AUTONOMOUS TERRITORIAL UNIT CAPITALS ˘ ˘ GAGAUZIA RAIONS OR MUNICIPALITIES CAPITALS* ˘ Vulcanesti NATIONAL CAPITAL RIVERS MAIN ROADS RAILROADS To Imayil B l ack AUTONOMOUS TERRITORIAL UNIT BOUNDARIES To Bucharest Sea and Constanta RAIONS OR MUNICIPALITIES BOUNDARIES INTERNATIONAL BOUNDARIES *Names of the raions or municipalities are identical to their capitals. 28°E 29°E 30°E MAY 2007