Document of The World Bank Report No: ICR2786 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3420) ON A GRANT IN THE AMOUNT OF SDR 6.4 MILLION (US$ 10 MILLION EQUIVALENT) TO THE UNITED NATIONS INTERIM ADMINISTRATION MISSION IN KOSOVO FOR THE BENEFIT OF KOSOVO, SERBIA FOR A INSTITUTIONAL DEVELOPMENT FOR EDUCATION PROJECT June 9, 2014 Human Development Sector Unit South East Europe Country Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2013) Currency Unit = EUR EUR 1.00 = USD 1.37665 USD 1.00 = SDR 0.64935 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CPS Country Partnership Strategy EMIS Education Management Information System EPIP Education Participation and Improvement Project ERR Economic Rate of Return FRR Financial Rate of Return GoK Government of Kosovo ICR Implementation and Completion Results Report IDEP Institutional Development for Education Project M&E Monitoring and Evaluation MED Municipal Education Department MEST Ministry of Education, Science and Technology of Kosovo MFE Ministry of Finance and Economy of Kosovo MTEF Medium-Term Expenditure Framework MTR Mir-Term Review NATO North Atlantic Treaty Organization NGO Non-Governmental Organization NPV Net Present Value OECD Organization for Economic Co-operation and Development PAD Project Appraisal Document PCT Project Coordination Team PDO Project Development Objectives PISA Programme for International Student Assessment PISG Provisional Institutions of Self-Government POM Project Operations Manual PSC Project Steering Committee SCTL State Council for Teacher Licensing SDG School Development Grant SDR Special Drawing Rights UN United Nations UNICEF United Nations Children’s Fund UNMIK United Nations Interim Administration Mission in Kosovo Vice President: Laura Tuck Country Director: Ellen A. Goldstein Sector Manager: Andrea Guedes Project Team Leader: Flora Kelmendi ICR Team Leader: Janssen Teixeira ICR Primary Author: Janssen Teixeira KOSOVO INSTITUTIONAL DEVELOPMENT FOR EDUCATION PROJECT CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 7 3. Assessment of Outcomes .......................................................................................... 11 4. Assessment of Risk to Development Outcome......................................................... 17 5. Assessment of Bank and Borrower Performance ..................................................... 17 6. Lessons Learned ....................................................................................................... 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 20 Annex 1. Project Costs and Financing .......................................................................... 21 Annex 2. Outputs by Component ................................................................................. 22 Annex 3. Economic and Financial Analysis ................................................................. 26 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 30 Annex 5. Beneficiary Survey Results ........................................................................... 32 Annex 6. Stakeholder Workshop Report and Results................................................... 33 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 34 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 43 Annex 9. List of Supporting Documents ...................................................................... 44 Map. ……………………………………………………………………….………….45 A. Basic Information Institutional Country: Kosovo Project Name: Development for Education Project Project ID: P102174 L/C/TF Number(s): IDA-H3420 ICR Date: 06/10/2014 ICR Type: Core ICR UN INTERIM Lending Instrument: SIL Borrower: ADMINISTRATION KOSOVO - UNMIK Original Total XDR 6.40M Disbursed Amount: XDR 6.21M Commitment: Revised Amount: XDR 6.21M Environmental Category: B Implementing Agencies: Ministry of Education, Science and Technology Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 04/30/2007 Effectiveness: 12/14/2007 12/14/2007 03/04/2009 08/16/2010 Appraisal: 10/29/2007 Restructuring(s): 11/18/2011 06/11/2013 Approval: 12/13/2007 Mid-term Review: 12/15/2010 04/11/2011 Closing: 06/30/2012 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Moderately Satisfactory Unsatisfactory Quality of Supervision: Moderately Satisfactory Implementing Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 21 21 Primary education 39 39 Secondary education 30 30 Sub-national government administration 7 7 Tertiary education 3 3 Theme Code (as % of total Bank financing) Conflict prevention and post-conflict reconstruction 17 17 Decentralization 17 17 Education for all 33 33 Education for the knowledge economy 33 33 E. Bank Staff Positions At ICR At Approval Vice President: Laura Tuck Shigeo Katsu Country Director: Ellen A. Goldstein Orsalia Kalantzopoulos Sector Manager: Andrea C. Guedes Mamta Murthi Project Team Leader: Flora Kelmendi Keiko Miwa ICR Team Leader: Janssen Edelweiss Nunes Teixeira ICR Primary Author: Janssen Edelweiss Nunes Teixeira F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The long-term overall goal of the proposed project is to support the government in the implementation of the Strategy for the Development of Pre-University Education in Kosovo and the Strategy for the Development of Higher Education. The specific objective of the proposed project is to strengthen systems, institutions and management capacities needed for education quality improvements. Revised Project Development Objectives (as approved by original approving authority) The PDO remained unchanged throughout the project lifetime. (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years MEST establishes baseline of assessment and examination instruments to Indicator 1 : measure the student learning outcomes at grade 5 by 2010. The instruments for Grade 5 assessment Value No learning outcome G5: are developed and quantitative or Analysis prepared assessment disseminated in place, and the Qualitative) analysis of data was prepared. Date achieved 11/13/2007 12/31/2011 12/31/2013 12/23/2013 Comments (incl. % Target achieved. This indicator was revised in the 3rd restructuring. achievement) MEST/Ministry of Finance and Economy (MFE) monitors, analyzes and Indicator 2 : publishes capital and recurrent expenditures (by level of education, by municipality, per student, by spending composition) each year Yearly expenditures Value Report reports are quantitative or Raw data, but no analysis No change disseminated published by the Qualitative) MFE Date achieved 11/13/2007 12/31/2011 12/31/2013 12/23/2013 Comments (incl. % Target achieved. achievement) 60 percent of municipalities are able to report, using EMIS data, on the status of Indicator 3 : dropout and retention disaggregated by gender and community by 2012 Value quantitative or 0% 80% 60% 80% Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2013 12/23/2013 Comments Target achieved. The target of this indicator was revised from 80 percent to 60 (incl. % percent in the 3rd restructuring, but the final value was exactly 80 percent (as achievement) originally planned). 60 percent of schools receiving School Grants produce school improvement Indicator 4 : plans and monitoring reports by the Project Closing Date. Value quantitative or 0% 60% 60% Qualitative) Date achieved 11/13/2007 12/31/2011 12/23/2013 Comments (incl. % Target achieved. achievement) Indicator 5 : 70 percent of teachers are licensed as regular teachers by 2011 Value quantitative or 0% 70% 75% Qualitative) Date achieved 11/13/2007 12/31/2011 12/23/2013 Comments (incl. % Target achieved. achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years A primary and secondary education law which clarifies roles and responsibilities Indicator 1 : for delivery of education among various levels of education system is adopted by the Parliament by June 2011. Value The Law was (quantitative Existing law Implemented Implemented adopted in or Qualitative) 2011. Date achieved 11/13/2007 12/31/2011 06/30/2012 12/23/2013 Comments Target achieved. The Law was adopted in 2011. By-laws required to guide (incl. % implementation developed, adopted and being used by MEST and other relevant achievement) stakeholders. Timely preparation/ revision of per capita funding formula that will direct Indicator 2 : resources to areas of need, and its implementation. Value The funding Implementatio (quantitative Formula from 2003 Review formula was revised n or Qualitative) and is being used Date achieved 11/13/2007 12/31/2011 06/30/2012 12/23/2013 Comments Target achieved. The reviewed per-capita funding formula was used for the 2014 (incl. % budget allocation. This is the fourth year that formula has been in use. The achievement) formula is reviewed each year. The number of municipalities which allocate recurrent budget to schools Indicator 3 : according to funding formula. Value 0 15 Nationwide 31 (quantitative (37) or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target not achieved. Since 2013, the total number of municipalities increased to (incl. % 38. The seven municipalities that are not using the formula are those with a achievement) majority of Serbian population. The number of municipalities which prepare and implement municipal education Indicator 4 : development plans Value (quantitative 0 15 No change 3 or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments This indicator was dropped in the 3rd restructuring process, as an evaluation of (incl. % the implementation of the municipal grants showed limited impact of this achievement) activity. Number of schools implementing SDGs which are able to establish quality Indicator 5 : improvements indicators at school level and monitor them Value (quantitative 0 240 181 0 or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target not achieved. The monitoring of quality improvement indicators had not (incl. % been done. achievement) Indicator 6 : % of teachers who have completed the minimum in-service training requirements Value (quantitative 0% 20% 10% 22.2% or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments (incl. % Target achieved achievement) The Council for Teacher Licensing is established, and teacher licensing, Indicator 7 : professional development and performance evaluation systems are in place by project closing date The Council for Teacher Licensing Value was established, but (quantitative Not existing Functional No change only the licensing or Qualitative) aspects are functional Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target not achieved. This indicator was revised in the 3rd restructuring, by (incl. % dropping the reference to curriculum from the previous title. The licensing achievement) aspects are functional, while the other two aspects are not fully functional. The availability and quality of annual EMIS report, consisting of statistics and Indicator 8 : analysis of education trends Value Report available, but no Report used for Report Report (quantitative analysis decision making disseminated disseminated or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target achieved. The EMIS software has been finalized and is in use. On the (incl. % basis of EMIS data, the MEST produced annual report (with UNICEF) with achievement) statistics and analysis of trends by 2013 Indicator 9 : System for Learning assessment at the primary level Value (quantitative No Yes No change Yes or Qualitative) Date achieved 12/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments (incl. % Target achieved. This (core) indicator was introduced in the 3rd restructuring achievement) Number of additional qualified primary teachers resulting from project Indicator 10 : interventions Value (quantitative 0 17,666 No change 15,529 or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Target not achieved. This (core) indicator was introduced in the 3rd Comments restructuring. To get a regular teaching license teachers have to have a university (incl. % degree, pedagogical training, and have completed an inception period as beginner achievement) teachers The availability of school mapping, and costed investment plan and maintenance Indicator 11 : guidelines for school facilities by December 2011 The school mapping is completed, and the investment plan and Value Implementatio maintenance (quantitative Not existing Updating n guidelines are or Qualitative) finalized, approved and under implementation by the MEST Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments (incl. % Target achieved achievement) Indicator 12 : Development of new school construction design standards Manual with school Value Standards construction design (quantitative Not existing Not applicable formally standards adopted or Qualitative) adopted by the MEST in July 2011 Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target achieved. In the 3rd restructuring process, this indicator replaced the one (incl. % on “the % of new school construction which use the new design standards” achievement) Indicator 13 : The law on higher education is amended and adopted Value (quantitative Existing law Not applicable Implemented Law adopted or Qualitative) Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments Target achieved. This indicator was added in the 3rd restructuring. The Law on (incl. % Higher Education has been revised and adopted, and by-laws developed and achievement) implemented The study on the size and shape of higher education in Kosovo is used by MEST Indicator 14 : to develop a concrete strategy which proposes a cost-effective expansion of the system The recommendations of the study were incorporated in the Higher Education Value Implementation Report part of the sector (quantitative Study not available steps begin disseminated strategy. But the or Qualitative) strategy falls short of including a proposal for a cost- effective expansion of the system Date achieved 11/13/2007 12/31/2011 12/31/2012 12/23/2013 Comments (incl. % Target not achieved achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 01/10/2008 Satisfactory Satisfactory 0.00 2 04/16/2008 Satisfactory Satisfactory 0.00 3 10/10/2008 Satisfactory Satisfactory 0.00 4 12/15/2008 Moderately Satisfactory Moderately Satisfactory 0.00 Moderately Moderately 5 06/17/2009 0.18 Unsatisfactory Unsatisfactory Moderately Moderately 6 09/08/2009 0.18 Unsatisfactory Unsatisfactory 7 10/15/2009 Moderately Satisfactory Moderately Satisfactory 0.31 8 11/17/2009 Moderately Satisfactory Moderately Satisfactory 0.31 9 03/18/2010 Moderately Satisfactory Moderately Satisfactory 0.52 Moderately 10 04/27/2010 Moderately Satisfactory 0.52 Unsatisfactory Moderately 11 09/23/2010 Moderately Satisfactory 1.19 Unsatisfactory Moderately 12 10/31/2010 Moderately Satisfactory 1.80 Unsatisfactory Moderately 13 05/30/2011 Moderately Satisfactory 2.37 Unsatisfactory Moderately 14 12/25/2011 Moderately Satisfactory 3.75 Unsatisfactory Moderately 15 06/26/2012 Moderately Satisfactory 4.74 Unsatisfactory 16 07/24/2012 Moderately Satisfactory Moderately Satisfactory 4.81 17 11/17/2012 Moderately Satisfactory Satisfactory 6.46 18 06/18/2013 Moderately Satisfactory Moderately Satisfactory 7.74 19 12/23/2013 Satisfactory Moderately Satisfactory 9.08 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Changes of arrangements for 03/04/2009 MS MS 0.00 the implementation of municipal grants. Revision of composition of the Project Steering Committee and 08/16/2010 MS MU 1.19 the responsible for the role of Secretary of this committee. The closing date was extended by one year, from June 30, 2012 to June 30, 2013, to allow for the construction of the model basic education school to be 11/18/2011 MU MS 3.62 completed. Also, the disbursement categories, funding allocations and percentages of expenditures to be financed were revised. The closing date was extended by six months, from June 30, 2013 to December 31, 2013, to allow for the conclusion of the 06/11/2013 MS S 7.74 model basic education school and its furnishing and equipping, since the first extension was not sufficient for ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions the completion of these activities. I. Disbursement Profile 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. The recent history of Kosovo’s education system is highly unusual and important for understanding the challenges faced by the Government of Kosovo (GoK). After the break out of Yugoslavia in the early 1990s, a series of measures against the Kosovo Albanian population took place, including a ban on teaching in the Albanian language and the introduction of Serbian as the only official language for formal education. Due to ethnic segregation in schools, changes to the school curriculum developed in Belgrade, and the dismissal of Kosovo Albanian teaching staff, the Albanian population decided to establish an unofficial education system at all levels, known as the “parallel system.” Financing came from remittances from abroad, community money, and in kind contributions. Symbolic stipends were paid to teachers. During the period of intensified conflict between 1997 and 1999, education provision was severely curtailed in war-affected areas. During the North Atlantic Treaty Organization (NATO) campaign in 1999, education services were largely suspended. 2. Following the end of the war in June 1999, the international community, namely the United Nations Interim Administration Mission in Kosovo (UNMIK) was left to rebuild the public sector, including the education system. At this point, the parallel system was transferred into the official system, and the foundations of the system were set by UNMIK, which was entrusted with the mandate to govern Kosovo. The decisions of the UNMIK included creating an inventory of all teaching and non-teaching staff, setting the basic parameters of teaching hours, and setting teachers’ salaries. In 2001, elections were held in Kosovo, which resulted in the creation of the Parliament, and the establishment of new Provisional Institutions of Self-Government (PISG) of Kosovo. The PISG had a mandate, with oversight from the international community, to establish the public administration and run public services. In its effort to rebuild education, the newly established provisional government in Kosovo faced a number of challenges. After an initial post- conflict acceleration in 2000, economic growth had weakened in line with declining donor resources. Unemployment was pervasive and particularly problematic among young people. About 45 percent of the population in Kosovo was poor, with another 18 percent vulnerable to poverty. Building human capital and attracting foreign investments were key priorities for future economic growth and poverty reduction. 3. By the time of the preparation of the Institutional Development for Education Project (IDEP), Kosovo was entering a new phase in its history as its final political status was being negotiated. In February 2008, following several United Nations’ facilitated attempts to negotiate a final status, Kosovo’s Government declared independence and agreed to adhere to the principles of the Ahtisaari Status Settlement. 1 This marked a change in the sense that Kosovo institutions were 1 In early February 2007, the Special Envoy of the Secretary-General of the United Nations (UN), Martti Ahtisaari, presented a draft Comprehensive Proposal for the Kosovo Status Settlement intended to lead the way to independence. The proposal envisaged Kosovo’s future as a multi-ethnic society, governing itself democratically and with full respect for the rule of law, the highest level of internationally recognized human rights and fundamental freedoms, and which promotes the peaceful and prosperous existence of all its inhabitants. While this proposal was accepted by the local government in Kosovo and backed by the international community, it was strongly opposed by Serbia and Russia. Various UN-facilitated attempts to negotiate an agreement left Kosovo's status unresolved. Supported by the international community, Kosovo's government declared independence from Serbia on February 17, 2008. 1 fully responsible for most domains, with some exceptions. Following independence, the Ministry of Education, Science and Technology of Kosovo (MEST) was solely responsible for education reform and policy. The municipalities became the units of local self-government, as proposed by the Ahtisaari package, with full and exclusive power in various areas, as follows: provision of education services (pre-primary, primary, and secondary education); registration and licensing of educational institutions; and recruitment, payment of salaries, and training of teachers and educational administrators. Within this context, the MEST had to quickly adapt to the shift in system governance. The IDEP support was geared towards strengthening and mainstreaming these efforts. 4. The government had identified education as one of the three key sectors for Kosovo’s growth prospects in its strategy and in the Medium-Term Expenditure Framework (MTEF). Additionally, the government recognized the importance of education in preventing future conflicts and facilitating social reconstruction. Despite the lack of quality information on educational outcomes, it was assumed that Kosovo’s performance was low compared to other countries in Southeast Europe, considering the lack of a teacher licensing mechanism, the practice of multiple shifts in schools, and the limited availability and quality of teaching and learning materials. The sector was characterized by low enrollment in secondary and higher education (88 percent and 18 percent in 2005 respectively), despite the young and growing population. The high population growth rate had increased the demand for school places and had led many schools to operate in multiple shifts. The capacity of the government to manage the education system was low, as well as its ability to coordinate efforts and institutionalize improvements promoted after the conflict. To face these challenges, the MEST developed two educational strategies as follows: the Strategy for the Development of Pre-University Education in Kosovo 2007-2017, and the Strategy for Development of Higher Education in Kosovo 2005-2015. Education was one of the few sectors in Kosovo where the Government’s ambitions were articulated into detailed and credible sectoral development plans. 1.2 Original Project Development Objectives (PDO) and Key Indicators 5. The PDO as stated in the Financing Agreement is as follows: the objective of the Project is to strengthen systems, institutions and management capacities needed for education quality improvements. The original key outcome indicators are: • MEST establishes baseline of the student learning outcomes at Grades 5 and 9 by 2010. • MEST/Ministry of Finance and Economy (MFE) monitors, analyzes and publishes capital and recurrent expenditures (by level of education, by municipality, per student, by spending composition) each year. • 80 percent of municipalities supported under the project are able to report on the status of dropout and retention disaggregated by gender and community by 2011. • 60 percent of schools supported under the project produce annual school improvement plans and monitoring reports by 2011. • 70 percent of teachers are licensed as regular teachers by 2011. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. Although the PDO remained unchanged throughout the project lifetime, the key outcome indicators were revised in 2011 (see matrix below). The IDEP was restructured after the Mid- term Review (MTR) of April 2011 inter alia to improve the clarity, measurability and relevance of project indicators, and to reflect modifications to some component activities. The definition of 2 the indicator on students’ assessment changed, as the assessment of grade 9 was already being carried out by the MEST when the project was launched. Additionally, the target of the indicator on reports on retention and drop-outs was revised from 80 percent to 60 percent, but it turned out that this change was not necessary, as the final value of this indicator was exactly 80 percent. Original Outcome Indicators Revised Outcome Indicators MEST establishes baseline of the student learning MEST establishes baseline of assessment and outcomes at Grades 5 and 9 by 2010 examination instruments to measure the student learning outcomes at grade 5 by 2010 MEST/Ministry of Finance and Economy (MFE) MEST/MFE monitors and publishes recurrent monitors, analyzes and publishes capital and expenditure each year (*) recurrent expenditures (by level of education, by municipality, per student, by spending composition) each year 80 percent of municipalities supported under the 60 percent of municipalities are able to report using project are able to report on the status of dropout Education Management Information System (EMIS) and retention disaggregated by gender and data on the status of dropout and retention community by 2011 disaggregated by gender and community by 2012 60 percent of schools supported under the project 60 percent of schools receiving school development produce annual school improvement plans and grants produce school improvement plans and monitoring reports by 2011 monitoring reports by 2011 (**) 70 percent of teachers are licensed as regular % of teachers licensed as regular teacher teachers by 2011 (*) The 3rd Amendment to the Financing Agreement establishes the following slightly different title for this indicator: MEST/MFE monitors, analysis and publishes capital and recurrent expenditures (by level of education, by municipality, per student, by spending composition) each year. (**) The 3rd Amendment to the Financing Agreement establishes the following slightly different title for this indicator: Sixty percent (60%) of schools receiving School Grants produce school improvement plans and monitoring reports by the Project Closing Date. 1.4 Main Beneficiaries 7. No specific groups of target beneficiaries were clearly identified in the PAD, although it does mention that schools with a high proportion of students from poor families and minorities would be targeted to benefit from the school development grants. Since the IDEP supported the implementation of key elements of the MEST’s Pre-University and Higher Education Strategies and had nationwide coverage, all students, teachers and managers of these sub-sectors were expected to benefit from the Project. 1.5 Original Components 8. The IDEP contained four components, each representing a key element of the Government’s two education strategies: (i) Strengthening the organization and financing of the education system, with particular attention to the legal framework and funding formula. This component focused on decentralizing the education system delivery including: • developing an appropriate legal framework and management structures and processes for primary, secondary, and tertiary education which clarify the roles and responsibilities of each level and the accountability mechanisms of the education system; • strengthening the leadership, decision making and resource management at the municipality and school levels through training and municipal development grants; 3 • piloting of formula-based funding from the municipality to schools and enhancement of school financial autonomy; and • providing school development grants which would promote activities to improve the quality of teaching-learning conditions and enhance inclusive education at school level. (ii) Building institutions and management capacities to promote quality improvements in primary and secondary education. This component would build on activities that had been carried out in Kosovo since 2000, which aimed at improving the quality of education including: • implementing teachers' professional development, including in-service training, and licensing system as developed by MEST in 2004 and 2007; • implementing the assessment of students' learning outcomes, and establishing the baselines to monitor learning outcomes in the long-term, and the review of the pilot matura (high school leaving exam) and its full implementation; • strengthening the institutional capacities and management processes of the key department in MEST concerning the quality of education, including teacher training, curriculum development, and assessment; and • providing training and an enabling operational environment for the use of the EMIS for analysis of key policy issues and decision making by MEST and municipalities. (iii) Creating conditions to introduce efficient and appropriate designs and reduce multiple shifts in Kosovo's schools to address the current and future constraints in learning spaces. This component focused on addressing the constraints in learning space, including: • school mapping, as well as preparing the school facility investment and maintenance plans; • developing standard designs for classrooms and school facilities, which are fiscally sustainable, appropriate for new teaching methods, and energy efficient; and • constructing two model schools (one primary and one secondary) based on the new design standards and school mapping. (iv) Strengthening management capacity in higher education, at system and institutional levels to support the implementation of Higher Education Strategy 2005-2015. This component was designed to support the implementation of the Strategy for Development of Higher Education 2005-2015, including: • supporting MEST to develop amendments to the Law on Higher Education and to draft a new law on financing higher education; • building the capacity of MEST departments and staff responsible for the functions defined in the amended Law on Higher Education; and • conducting feasibility studies of: (a) a cost-effective expansion strategy for the Kosovo higher education system (size and configuration, institutional diversification); and (b) the financial sustainability of the Kosovo higher education system (financing needs, cost- sharing, scholarships and student loans, budget allocation mechanisms). 9. Causal linkages between project components and outcome indicators. All four components as originally planned financed activities that would strengthen systems, institutions and management capacities needed for education quality improvements. Components 1, 2, and 3 focused on pre-university education while Component 4 focused on higher education. However, the five outcome indicators relate only to Components 1 and 2, as shown in the table below. No outcome indicators were included to measure the contributions of Components 3 and 4 to the PDO. 4 Components Original Outcome Indicators Component 1 - Strengthening MEST/MFE monitors, analyzes and publishes capital and recurrent the organization and financing expenditures (by level of education, by municipality, per student, by of the education system, with spending composition) each year particular attention to the legal 60 percent of schools supported under the project produce annual school framework and funding improvement plans and monitoring reports by 2011 formula Component 2 - Building MEST establishes baseline of the student learning outcomes at Grades 5 institutions and management and 9 by 2010 capacity to promote quality 80 percent of municipalities supported under the project are able to improvements in primary and report on the status of dropout and retention disaggregated by gender secondary education and community by 2011 70 percent of teachers are licensed as regular teachers by 2011 1.6 Revised Components 10. The basic structure of the four components did not change throughout the operational cycle. However, as a result of the MTR of April 2011, two activities originally planned were modified as described in the table below. Because these modifications did not alter the PDO, they were approved as part of a Level Two restructuring. The other components and activities remained unchanged. Components and Activities Modifications Component 2 – Building institutions and management Curriculum development was dropped from this capacities to promote quality improvements in primary activity, given the availability of financial and secondary education. resources outside the project. Activity - Strengthening the institutional capacities and management processes of the key department in MEST concerning the quality of education, including teacher training, curriculum development and assessment. Component 3 - Creating conditions to introduce efficient The MEST decided to construct only one urban and appropriate designs and reduce multiple shifts in basic education school (grades 0 to 9), rather than Kosovo's schools. the rural primary school (grades 1-5) and the Activity – Constructing two model schools based on the urban secondary school (grades 10-12) that were new design standards and school mapping. originally planned. As the cost of the basic school was higher than the total amount originally allocated to the civil works, it was agreed that the construction of the secondary school would be financed by the government (see Section 1.7 below). 1.7 Other significant changes 11. The Project was approved on December 13, 2007 became effective on December 14, 2007. It was originally expected to close on June 30, 2012. The Project underwent four restructurings throughout the six years it was active, all of which were Level Two processes. These restructurings reflected changes in the project schedule, implementation arrangements and financing, as stated in the next paragraphs. 5 12. 1st Restructuring (Corrective). On March 4, 2009, the first amendment to the financing agreement 2 was approved to reflect the arrangements for the implementation of municipal grants, which were anticipated in the PAD but not mentioned in the original financing agreement. The appendix to the financing agreement on definitions was updated accordingly. Additionally, the disbursement table was updated to include the provisions for municipal grants under the first disbursement category, without changing any funding allocations of this table. 13. 2nd Restructuring (Corrective). On August 16, 2010, the second amendment to the financing agreement 3 was approved to reflect a revised composition of the Project Steering Committee (PSC) and the responsibility for the role of Secretary of this committee. The schedule of meetings of the PSC and the date of adoption of the Project Operations Manual (POM) by this committee were also adjusted. 14. 3rd Restructuring (Corrective). On November 18, 2011, the closing date was extended by one year, from June 30, 2012 to June 30, 2013, to allow for the construction of the model basic education school to be completed. Also, the disbursement categories, funding allocations and percentages of expenditures to be financed were revised, as indicated below. These changes were agreed to accommodate the revision of project activities as well as to reflect the government’s commitment to co-finance the construction of the model basic school (grades 0-9). Category Amount of the Grant Percentage of Allocated (in SDR) Expenditures to be Financed Current Revised Current Revised Current Revised 1. Goods, works, consultants’ No change 2,550,000 2,310,000 100% 100% services, training, school grants, municipal grants and incremental operating costs under Part A 4 2. Goods, works, consultants’ No change 1,470,000 1,382,000 100% 100% services, training and incremental operating costs under Part B 3. Goods, works, consultants’ 3(a). Goods, 1,400,000 908,000 100% 100% services, training and incremental consultants’ operating costs under Part C services, training and incremental operating costs under Part C 3(b). Works 1,350,000 100% 75% under Part C 4. Goods, works, consultants’ No change 340,000 156,000 100% 100% services, training and incremental operating costs under Part D 5. Unallocated No change 640,000 294,000 Total Amount 6,400,000 6,400,000 15. 4th Restructuring (Corrective). On June 11, 2013, the closing date was extended by six months, from June 30, 2013 to December 31, 2013, to allow for the finalization of the conclusion 2 Signed by the UNMIK on April 23, 2009. 3 Signed by the Government of Kosovo on September 2, 2010. 4 In fact, the description of this disbursement category changed in the 1st restructuring (dated March 4, 2009). 6 of the model basic education school and its furnishing and equipping, since the first extension was not sufficient for the completion of these activities. Also, the funding allocations were revised to reflect implementation progress and shortcomings, as follows: Category Amount of the Grant Percentage of Allocated (in SDR) Expenditures Current Revised to be Financed 1. Goods, works, consultants’ services, training, school 2,310,000 2,218,670 100% grants, municipal grants and incremental operating costs under Part A 2. Goods, works, consultants’ services, training and 1,382,000 1,640,930 100% incremental operating costs under Part B 3(a). Goods, consultants’ services, training and 908,000 1,021,770 100% incremental operating costs under Part C 3(b). Works under Part C 1,350,000 1,350,000 75% 4. Goods, works, consultants’ services, training and 156,000 137,010 100% incremental operating costs under Part D 5. Unallocated 294,000 31,620 Total Amount 6,400,000 6,400,000 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Strengths 16. Continuity of support. There was a strong rationale for Bank involvement in Kosovo’s education sector. Despite the uncertain nature of the future status of Kosovo 5, which posed high risks to any Bank-supported operation, the engagement of the Bank with the UNMIK was critical. The Bank was one of the few development partners that had sustained its support in education, since the conflict of 1999. At preparation, the Bank had supported the first Education and Health Project (2000-2003) and the Education Participation and Improvement Project (EPIP) (2003- 2006), which respectively aimed to improve efficiency and equity in public resource allocation for education, and to improve educational attainment and enhance the access of vulnerable groups to education. 17. Strategic alignment with country and Bank priorities. The PDO was strategically relevant for Kosovo’s context at preparation, in which institutions and capacities needed to be built right after the conflict (see Section 1.1). The Project’s objectives were aligned with the Interim Strategy Note for Kosovo 2007-2008, which had identified education as the main area of Bank’s support at that time. The IDEP also aimed to support the government in the implementation of the Strategy for the Development of Pre-University Education in Kosovo 2007-2017, and the Strategy for Development of Higher Education in Kosovo 2005-2015 18. Incorporation of lessons learned. The IDEP incorporated lessons learned from the EPIP’s implementation, specifically concerning community participation and school board involvement in interventions carried out at the local level in a post-conflict environment. 5 Kosovo was declared a new state in 2007 and was in the early stages of building its institutions at project preparation. 7 Weaknesses 19. Complexity and ambitiousness of design. The project design was overly complex and ambitious considering the capacity at the central, regional and local levels observed in 2007. The project components included activities like teachers’ licensing and performance assessment, the implementation of which required politically sensitive initiatives and strong institutional capacity. However, it is important to highlight that task teams usually face difficulties while preparing projects in conflict-affected contexts like the one in Kososo, in which it is hard to identify the full set of factors that could potentially affect the implementation, and key stakeholder would likely to survive transitions. In addition to this, the enabling conditions for the implementation of education financing reforms were not fully assessed at preparation, which added challenges to the subsequent implementation of the Project. In fact, when the IDEP closed, about 50 percent of municipalities were still hesitating to delegate autonomy to schools for undertaking procurement activities due to legal constraints, which were not identified at preparation. The PAD mentions neither this constraint, nor any related mitigating measure. 20. Preparation of civil works subcomponent. The project design failed to properly assess the issues surrounding civil works at preparation, including site selection and accurate cost estimates for both schools. The contract for the design and supervision of the basic education school was signed only on April 2, 2011 – 15 months before the original closing date – which shows a weakness of the preparation of the civil works component. In addition to that, at the time of the MTR, the site for the construction of the upper secondary school had not been identified yet. 21. Assessment of legal framework and risks. Legal constraints restricted regional authorities from implementing education financing reforms. The Bank failed to assess this legal risk at appraisal, but the GoK also failed to identify the existing procurement law as a constraint. In addition to this shortcoming, the procurement risks were assessed as modest at appraisal, but the low capacity at the school level for undertaking procurement activities under the school development grants subcomponent delayed its implementation substantially. 2.2 Implementation Factors outside the Control of Government or Implementing Agencies 22. At early stages of the implementation cycle, the Project was affected by the political process of independence declaration, which contributed to divert the focus of the Government. Over the project’s lifetime, other political factors delayed implementation, like the collapse of the government in October 2010 and the subsequent leadership vacuum. The new government was established only at the end of February 2011, which led to the high turnover of MEST’s staff, including the Permanent Secretary, who played the role of Project Director, the Head of the Higher Education Department, the Head of the Donor Coordination Unit, and a few political advisers that supervised project activities. 23. At the project closing date, political instability was still affecting the implementation. The Northern part of the country was suffering from politically motivated attacks, including the use of explosives. Seven municipalities (18 percent of the total) with a majority of Serbian population had not allocated recurrent budget to schools according to the funding formula or participated in other reforms supported by the Project due to limited or non-existent cooperation with the GoK. 8 Factors Generally Subject to Government Control 24. The GoK and particularly the MEST remained highly committed to the objectives and implementation of the IDEP throughout the operational cycle. However, the Government’s inability to identify the school site and lack of funds delayed the launching of the construction of the upper secondary education school to such an extent that it made this activity untenable given the project’s duration. Factors Generally Subject to Implementing Agency Control 25. Although the MEST remained committed to the IDEP, changes in its leadership contributed to implementation delays. A high turnover of key staff involved in project management was observed, as follows: Project Director (five changes), Project Coordinator (two changes), Procurement Specialist (three changes) and Financial Management Specialist (two changes). These changes translated into loss of project-related knowledge. 26. The limited capacity within the MEST, the delayed establishment of the Project Coordination Team (PCT), and the lack of the POM at project launching resulted in implementation constraints and delays. For example, roles and responsibilities of relevant stakeholders were not clarified to individuals involved in project management and implementation until September 2010, when the POM was adopted. 27. Additionally, the first Project Procurement Plan was concluded one year after project effectiveness. The Financial Management Specialist and the Procurement Specialist were hired only in late 2008. The first Project Coordinator was not adequately skilled to manage the Project in the challenging early stages of implementation. These factors also explain significant implementation delays. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 28. M&E Design. The revised outcome indicators are adequate for measuring progress towards the achievement of the PDO. However, there are some disconnects between project components and outcome indicators, as described in Section 1.5, given that there are no PDO indicators related to Components 3 or 4. In other words, the results framework does not account for the interventions supported under Components 3 and 4 and the PDO. Nonetheless, the results framework does present intermediate level indicators for all project components 29. M&E Implementation. Although the M&E capacity has grown throughout the implementation period, the MEST has not fully implemented M&E activities as expected. The MEST did not begin to monitor project indicators until November 2009, almost two years after project effectiveness. Delays in finalizing and adopting the POM also affected M&E activities, given that the POM laid out the foundations for implementation, including the monitoring arrangements. Furthermore, the strengthened capacity of the MEST and Municipal Education Departments (MED) was expected to allow for the monitoring of education indicators beyond the project scope, as described in the PAD. However, the MEST has not been monitoring long-term education sector objectives. 30. M&E Utilization. The MEST has built some capacity towards utilizing data collected through M&E efforts. For example, the Education Management Information System (EMIS) is now feeding the new pre-university education financing mechanism introduced by the IDEP with data. The budgets of municipalities are allocated by the Ministry of Finance based on a per-pupil 9 formula calculated with inter alia EMIS data. In addition to that, in May 2013, a report on education statistics was prepared jointly by United Nations Children’s Fund (UNICEF) and the MEST using data produced by the EMIS. However, the EMIS only became operational in June 2011, one year before the original closing date. Additionally, the PAD describes provisions for conducting annual review meetings with key stakeholders to analyze and discuss progress towards project objectives and the implementation of the education strategies. These review meetings were intended to be the main forum for education sector planning and monitoring, although they never took place during the implementation of the IDEP. However, it bears mentioning that the MEST has been organizing joint annual review meetings with other line Ministries, municipalities and all education sector development, since 2012. 2.4 Safeguard and Fiduciary Compliance 31. Safeguards. The IDEP was classified as Category B because of the plan of building two model schools with project funds. The Project triggered the Environmental Assessment Policy (OP/BP 4.01) and an Environmental Management Plan was prepared and disclosed in October 2007. The Involuntary Resettlement Safeguard Policy (OP/BP 4.12) was not triggered as the Project was not expected to require land acquisition or resettlement for the construction of these schools. 32. The IDEP remained compliant with the sole safeguard policy triggered throughout its implementation lifecycle—the Environmental Assessment Policy (OP/BP 4.01). Nevertheless, the MEST failed to have the Municipality of Prishtina adopt measures to secure the abandoned brick building adjacent to the model school, like fencing the building off and boarding the broken windows to prevent glass falling in the schoolyard until the project closing date. 33. Financial Management. The financial management arrangements for the Project were found acceptable to the World Bank at preparation but the overall financial management risk was rated substantial. The Financial Management Specialist of the PCT was hired almost one year after project effectiveness, which did not contribute to speeding up the implementation at the earliest stage nor increasing disbursement—as of October 2010, only 18 percent of the grant had been disbursed. At early phases of implementation, the financial management and procurement arrangements for the implementation of the school development grants were made unnecessarily cumbersome, which significantly slowed down this process. The Task Team required that every purchase had written offers from at least three suppliers regardless of amount, and schools could not purchase goods because local providers were not willing to comply with those procedures for small purchases. Nonetheless, the financial management capacity within the MEST improved throughout the project implementation. The Task Team conducted regular assessment of financial management issues during project supervision and found generally satisfactory controls. 34. Procurement. The procurement risk was rated high at preparation and mitigating measures were adequately identified and stated in the PAD. The main procuring unit of the IDEP was the MEST, but MED and schools were also entitled to carry out procurement activities related to the implementation of municipal and school development grants respectively. Training activities on procurement according to the World Bank’s guidelines were provided to representatives of MEST, MED and schools. Additionally, full-time local and international Procurement Specialists were made available to support procurement activities at the central level. The full-time Procurement Specialist was hired almost one year after project effectiveness, like the Financial Management 10 Specialist. At the regional and local level, a Non-Governmental Organization 6 (NGO) was contracted to provide technical assistance to MED and schools throughout all phases of the grants programs. All these initiatives helped increase the procurement capacity at these three levels, but did not avoid slowing down the initial phase of project implementation, which was also caused by the financial management constraints. The procurement capacity at regional and local level was uneven, given the existence of staff familiar with procurement either according to the National Procurement Law or Bank’s guidelines in some but not all MED and schools. Issues related to procurement observed during project preparation and implementation include: (i) shortcomings in maintaining contract administration records; (ii) late preparation of Procurement Plans; (iii) MEST’s attempts to interfere with the selection of consultants at very early stages of implementation; (iv) lack of contract management capacity, which was a clear weakness noticed in the construction of the model school; and (v) inadequate assessment of the existing legal framework at preparation, given the limitations materialized during the implementation of the education financing reform. 35. Legal Covenants. The MEST complied with the two legal covenants in the second year of project implementation. The PSC was established and operational in June 2010, and the POM was adopted by this Committee only in September 2010. 2.5 Post-completion Operation/Next Phase 36. A follow-up operation is currently being prepared and expected to be submitted to the Board’s consideration in December 2014. The new education Project proposes to build on selected interventions financed by the IDEP, so as to continue strengthening the capacity and accountability mechanisms of institutions responsible for delivering pre-university education. The new Project would support activities on: (i) strengthening strategic and financial planning and monitoring capacity for the decentralized education system; (ii) implementing the teachers’ certification system; (iii) strengthening the student assessment and examination system; and (iv) creating mechanisms and capacities for evidence-based policymaking and system monitoring. Building on the success of the school development grants, the MEST has requested to include this as one of the priority activities, with a renewed and stronger focus on quality improvement and support to education reform implementation. These interventions are based on the Government’s priorities, as identified in the sector strategy. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 37. As explained below, the objective relevance rating is high, whereas the design relevance rating is modest. A combined assessment of these two dimensions leads to an overall relevance rating of substantial. 38. Relevance of Objectives - rating: high. Improving the quality of education continues to be a major priority for Kosovo. Kosovo’s Strategy for the Development of Pre-University Education 2007-2017 remains the central policy document for setting strategic directions in the sector. This strategy acknowledges that strengthening systems, institutions, and capacities are fundamental to improving education quality. The IDEP PDO directly addresses several strategic objectives laid 6 The Kosovo Education Center. 11 out explicitly in this strategy, including: (i) quality and efficient governance, leadership, and management of the education system, and (ii) a functional system of providing quality learning based on standards comparable with those of the developed countries. Continued investment in education and skills is also one of six action areas specifically mentioned in Kosovo’s Country Partnership Strategy (CPS) for FY12-FY15 to support its strategic objective on accelerating broad-based economic growth and employment generation. The PDO on strengthening systems, institutions and management capacities needed for education quality improvements is still highly relevant given the country’s priorities and objectives, articulated in both the Pre-University Education Strategy and the CPS. 39. A new education project 7 is being prepared for Kosovo, which aims to strengthen the capacity and accountability mechanisms of institutions responsible for pre-university education. It would build on selected activities that were financed by the IDEP, like (i) strengthening the strategic and financial planning capacities at central, municipal and school level, (ii) supporting school development grants, (iii) teacher certification and career advancement, (iv) strengthening capacities of key institutions for monitoring educational outcomes and assessment of student learning, and (v) enhancing education monitoring and evaluation capacity. This follow-up operation would roll-out some IDEP-supported activities, which shows how relevant the IDEP development objective remains. 40. Relevance of Design - rating: modest. Within the context of an unclear future status of Kosovo, the IDEP was prepared to help the government develop the legal framework, funding arrangements and administrative procedures clarifying roles and responsibilities at the central, regional and local levels of the education system. The project design included three components supporting activities in pre-university education and one, in tertiary education. While the current CPS has one strategic objective that involves investments in tertiary education, the 4th component of the IDEP limitedly financed technical support to amending the Law on Higher Education and one study on the size and shape of higher education in Kosovo. These facts reveal that the IDEP design remained modestly relevant. The implementation arrangements also remain at least modestly relevant, given that a decentralized implementation approach to strengthen capacity not only at the MEST level is still one of the needs to be addressed, as shown in the concept note of the new education project proposed for Kosovo. 3.2 Achievement of Project Development Objectives 41. As explained in the next paragraphs, the project’s efficacy rating is substantial. The targets of all five PDO level indicators were achieved. Additionally, of the 13 targets of intermediate results indicators, eight were fully met, and five were not achieved (see Annex 2). 42. Improvements were observed in some education indicators throughout the implementation of the Project. Transition rates between grades remained at almost universal levels in basic education, and improved significantly in upper secondary. Additionally the student-teacher ratios decreased, especially in basic education, putting Kosovo more in line with comparator countries (see Annex 3). However, it is important to highlight that several determinants are known to contribute to achieving improvements in education quality, which include but are not limited to those supported under the Project (i.e. strengthened systems, institutions and management capacity). Factors such as parental education, socioeconomic background, skills and experience of principals, and the quality and availability of teaching materials are also contributors to 7 Education System Improvement Project (P149005). 12 education quality. Thus, while project interventions were likely to have played an important role in the attainment of the PDO, this ICR acknowledges that there were other likely factors at play. 43. The assessment of the project’s efficacy is undertaken in this ICR considering that the three core aspects of the PDO (strengthened systems, institutions and management capacities) are intertwined so these elements of the objective are considered jointly rather than separately. The PAD does not define these aspects of the PDO as three separate sub-PDOs, and they are clearly interrelated and interdependent. Capacity building is a necessary step towards improved systems and institutions. Additionally, improved systems are frequently needed for developing new or changing existing institutions. For these reasons, this ICR assesses the PDO as one objective, rather than a composition of three separate sub-objectives. 44. While there are linkages between the outcome indicators and the PDO, some disconnects are observed between the project components and the development objective. As already stated, all five outcome indicators relate to the first two components (on financing reforms and quality improvement respectively). The results framework does not allow measuring the aggregated contributions of the component 3 (on school designs) nor the component 4 (on tertiary education) to the PDO. The outcomes of the interventions in each project component, i.e. contributions of each component to achieving the PDO, are presented below. Outcomes of Interventions in Education Financing (Component 1) 45. Component 1 activities directly contributed to the achievement of the PDO. The targets of the two PDO level indicators associated to this component were achieved. In addition to this, the targets of two out of four intermediate results indicators of this component were also met (see Annex 2). The Law on Pre-University Education and respective by-laws were developed, approved and are in place. This legal framework, which was adopted by the Parliament in June 2011, provides clear roles and responsibilities for delivering educational services as well as accountability. It lays out the foundation for reforms in relevant areas, including: school autonomy, financial decentralization, teachers’ qualifications and career development, selection of teachers and school managers, assessment and examination. 46. The per-capita financing formula developed before the project launching, in 2002, was revised and has been in use for four years, since 2010. This formula is now part of the regular budgeting exercise carried out by the MEST and the MFE, for all levels of pre-university education. The improvements to the budget allocation process include, inter alia, the use of update students-teacher ratios per level of education, a fairer allocation of budget of schools located in remote areas, and an allocation of non-teaching staff on a per-pupil basis. At present, EMIS data are used for the budget allocation. Out of 38 municipalities, 31 (82% of the total) are applying this formula; the other 7 are those with a majority of Serbian population, which do not cooperate with the Kosovar authorities for political reasons. School managers are now involved in their school budget preparation, which was not observed prior IDEP. As a result, in 2013 all schools were provided with a treasury account and had their recurrent budget allocated, which reduce their reliance on support from municipalities. It is important to note that prior to project interventions, the pre-university education budget preparation process was almost entirely led by the MFE. Now, MEST plays a leading role in this process, which is more reflective of education sector priorities. Additionally, the Project played a critical role in strengthening the capacities of regional authorities to administer the formula across their municipality. However, despite the successful rollout of the per-capita financing formula, about 50 percent of municipalities did not fully devolve financial autonomy to schools, specifically the authority for undertaking procurement activities, due to thresholds set in the existing National Procurement Law. Thus, the 13 schools of these municipalities had their budget allocated according to the revised formula, but were not always able to procure themselves the needed goods and services. 47. Concerning the school development grants, 207 schools benefited from this activity, which contributed to building the capacity of managers and teachers in planning, financial management, procurement and monitoring and evaluation. Principals of schools located in Gjakova, Klina, Prizren, Rahovec and Suhareka informed that they are now able to prepare proposals to raise funds for their schools from different donors, as a result of IDEP’s capacity building activities. School-based management activities like this one supported by IDEP allow schools to identify and address their own needs more effectively than centralized approaches, as evidenced in several experiences across the world. School managers and teachers know the challenges of their schools better than other actors of the education system, and are capable to make effective decisions on using scarce funds provided by central and regional authorities to address the needs of their schools, when adequate technical assistance is provided, like this case in Kosovo. Outcomes of Interventions in Pre-University Education Quality (Component 2) 48. Component 2 activities also contributed directly to the achievement of the PDO. The targets of the three PDO level indicators associated with this component were achieved. Additionally, the targets of three out of five intermediate results indicators of this component were met (see Annex 2). The State Council for Teacher Licensing (SCTL) was established and is operational with support from the IDEP. This council led the preparation of the legal framework for teachers’ licensing, professional development and performance assessment. The teacher salary structure has been revised and implemented so that salaries are differentiated by qualifications, compared to the same salary paid to teachers, in the previous system. Teachers’ licensing is the only one of these three activities that was implemented with IDEP support. The challenges for the implementation of teachers’ professional development and performance assessment have to do with the lack of human and financial resources within the MEST and municipalities. At the project closing date, 75 percent of teachers were licensed according to the newly established mechanism. Additionally, an information system with data on licensing of teachers is running and representatives of regional inspectorates were trained to operate this software. The future implementation of all three dimensions of the teachers’ development mechanism would help the MEST motivate, reward and retain good teachers. Improvements for the education system are expected and would derive from requirements for new teachers, as well as the in-service requirements that all teachers need to meet to stay and be promoted. 49. The EMIS is deployed and running in all schools of Kosovo, except those located in the 7 municipalities with a Serbian majority. This information system was improved and became operational with support from the IDEP. As a result, school management staff can enter the data into the EMIS database, and analyze information of their schools. Additionally, authorities at the central and regional levels now have access to quality information, and are using use this information to make decisions and better support school managers. Some improvements would strengthen the EMIS and its operational environment, like the need of avoiding repetitive data collection exercises, which are undertaken four times a year, and the integration with other important systems including the school mapping and the one on teachers’ licensing and career development. The data collection for the EMIS has replaced the multiplicity of previous processes based on electronic sheets, which was not totally accurate. In this regard, the Ministry of Finance has suspended its own school data collection process, and now uses EMIS’s data for budget allocation. In addition to this, a report on educational statistics was produced based on EMIS’s data, which was done in close collaboration with UNICEF. This show how the IDEP has 14 contributed to the use of quality educational information for decision-making in Kosovo, which is remarkable. 50. The IDEP also supported the development of a learning assessment system for grade 5, the first round of tests in 2010 (baseline), as well as the preparation of an assessment policy and reporting standards. The IDEP also supported the application of Kosovo for the Organization for Economic Co-Operation and Development’s (OECD’s) Programme for International Student Assessment (PISA) in 2015, as well as the improvements in administration of the Matura exam. The latter outcome was not anticipated in the PAD. Outcomes of Interventions in Developing Standard Designs for Schools (Component 3) 51. Although the results framework lacks outcome indicators for this IDEP component, it is reasonable to say that its activities also contributed to achieving the development objective. The targets of the two intermediate results indicators of this component were met (see Annex 2). The IDEP supported the development of the school mapping, which stores information on the physical conditions of all Kosovo’s schools, including needs for infrastructure improvements. This system has been used by the MEST to develop investment and maintenance plans. In addition, the Project financed the development of standards for schools design (the Manual for School Standards and Design for Pre-University School Facilities), which are now being used by the MEST as binding documents when making infrastructure investments. Under the Project, the MEST approved by-laws on ensuring that all new schools constructions are aligned with both investment plans and standards. The rationale for the construction of schools with project funds relates to the intention of building capacity within the MEST’s Infrastructure Unit on the use of school standards for future investments. In fact, the construction of the model basic education school financed by the Project was undertaken in accordance with the newly developed standards. The combination of the school mapping software and the new school design standards ensures that future infrastructure investments are both efficient and high-impact. Furthermore, professional certification for architects in Kosovo now requires a knowledge of these school standards, which further demonstrates how the IDEP has helped to institutionalize this initiative. Outcomes of Interventions in Capacity Building for Tertiary Education (Component 4) 52. As mentioned, the results framework does not contain outcome indicators for Component 4. Nonetheless, the implementation of its activities did contribute to achieving the PDO. The target of one of the two intermediate results indicators of this component was met (see Annex 2). The Law on Higher Education was revised, amended and approved by the Parliament. The amended law lays the foundation for some key initiatives required for quality improvement in higher education, including the accreditation of public and private providers of tertiary education. Additionally, the IDEP supported a study on the size and shape of tertiary education, which was used by the MEST for the development of its strategy on higher education. As a result, the Project helped to raise awareness on the importance of the accreditation process for tertiary education institutions, which can lead to an improved quality of higher education in Kosovo. 3.3 Efficiency 53. The project’s efficiency rating is modest. The economic and financial section of the PAD lacks any customary analysis carried out for Bank-supported projects, such as net present value (NPV), economic rate of return (ERR) and financial rate of return (FRR). It barely shows the increase in the education budget allocation over the previous seven years and some information on the grants and school design standards supported by the IDEP. Thus, this ICR does not 15 recalculate the NPV, ERR nor FRR, as there is no baseline for comparisons. Instead, this report assesses the project’s efficiency through aspects of the project design and implementation that either contributed to or reduced efficiency. See Annex 3 for more details. 54. The Project supported the revision of the per capita financing mechanisms, in order to better respond to school and municipal needs. Data collection exercises promoted by the Project have been crucial for informing policy makers (both at the municipal and at the central level) on teachers and student flows. School principals consulted for the preparation of this ICR highlighted the improvements in the budget allocation as well as the use of EMIS data for planning purposes. However, as the World Bank’s Public Finance Review of 2014 informed, the Project contributed to increased school autonomy and transparency, but the efficiency gains are still not visible enough, probably because of the complex design of the financing formula. 55. The school standards and school mapping have been used by the MEST to help ensure an efficient utilization of state budget and contributions of donor for schools infrastructure improvements and constructions. Nowadays, investments on infrastructure are informed by quality data on school needs, as well as are done based on standards developed for an efficient utilization of financial resources. 3.4 Justification of Overall Outcome Rating 56. The project’s overall relevance rating is substantial, the project’s efficacy rating is substantial, and the project’s efficiency rating is modest. The combined assessment of these dimensions yields an overall satisfactory rating for the achievement of the project’s outcome. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 57. Gender Aspects. Enrollment rates for girls in upper secondary increased from 80 percent in 2010 to 88 percent in 2012, while for boys the rate only increased from 92 percent in 2010 to 95 percent in 2012. The gender gap in enrollment has decreased in both primary and secondary education: full gender parity in basic education was achieved in 2013 (48.5 percent of girls in school, with girls comprising 48 percent of the school-age population). Substantial improvements in the gender gap were also achieved in upper secondary education (girls’ percentage in upper secondary went from 44.5 percent in 2007 to 46 percent in 2013). While these improvements were observed during project implementation, they cannot be directly attributed to the IDEP. 58. Social Development. The Pre-University Education and the Higher Education Laws developed or revised under the Project include requirements for protecting vulnerable groups of students, and for strengthening social inclusion by attending to the needs of special education students. The IDEP also contributed to increased participation of communities in school-based management activities through the implementation of school grants. (b) Institutional Change/Strengthening 59. The PDO is directly related to institutional development and all outcomes in this regard are mentioned in Section 3.2 above. 16 4. Assessment of Risk to Development Outcome 60. The risk that the IDEP development objective will not be maintained is moderate, despite the high relevance of this PDO to Kosovo’s current development priorities (see Section 3.1). Some systems strengthened throughout the project implementation may face challenges in the near future, such as the systems on teachers’ licensing, performance assessment and career development. While the legal framework of this system is in place, the challenges for the MEST rest on the needed provision of teachers’ training, which is required by the licensing mechanism, as well as on the availability of human and financial resources for rolling-out the activities. 61. The maintenance of other systems also poses some challenges for the MEST, such as the adequate administrative capacity of its assessment unit to keep the student assessment system operational. The assessment unit might lack human and financial resources to continue the progress observed so far. On the other hand, operation of the EMIS will require further improvements on integrating its database with others (e.g. school mapping and teachers’ licensing), and developing the students’ management module. 62. In terms of school autonomy, the risks rest on the limited financial autonomy currently devolved to schools, which can put this initiative in jeopardy in the short-term. Experiences in other countries show that stakeholders tend to claim higher levels of autonomy over the process, and if this does not happen, they usually avoid continued participation in school-based management activities. 63. The national elections foreseen for 2014 may also bring about leadership changes, which might not be favorable to the continued implementation of sensitive reforms like teachers’ performance assessment and per-pupil financing. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry – rating: moderately unsatisfactory 64. The PDO was aligned with the Interim Strategy Note for Kosovo 2007-2008, which had identified education as the main area of Bank’s support at that time. The IDEP was also designed to support the government in the implementation of its two strategies on pre-university and tertiary education. Additionally, lessons from the EPIP’s implementation were considered by the Bank at preparation, such as community participation and school board involvement in school- level interventions. The task team carefully considered the challenges of Kosovo’s post-conflict context at preparation, which was remarkable. However, there were significant shortcomings in the identification, preparation and appraisal of the IDEP, which lead the Bank’s performance in ensuring quality at entry to be rated as moderately unsatisfactory. These shortcomings include: • inadequate assessment of implementation readiness, as evidenced by the absence of the POM, Project Implementation Plan and Project Procurement Plans at project launching; • poor support to the counterparts for estimating project costs; • incomplete assessment of the existing legal framework at appraisal, which limited the level of autonomy devolved to schools during implementation; • disconnects in the results framework, which do not include any outcome indicators to measure the aggregated contributions of two out of four project components; 17 • ambitiousness of the PAD on monitoring long-term education sector objectives in a context of limited capacity; • complexity of project design vis-à-vis the existing capacity for project implementation; • failure to identify risks associated with planned civil works and respective mitigating measures, which contributed to delays in the identification of the model school site; • weaknesses of the economic and financial analysis, which did not include any customary analysis such as NPV, ERR and FRR. (b) Quality of Supervision – rating: moderately satisfactory 65. Bank supervision benefited from close collaboration with the PCT and implementing units, especially after the MTR, when a local Bank manager began to lead the IDEP. Having a local staff leading the project supervision from the Bank side helped the counterparts solve day-to-day technical and fiduciary problems, which contributed to increased disbursement (from the incredibly low level of 24 percent of the grant, basically 3.5 years after the effectiveness date) and achieve the PDO. In general, the Bank carried out two to three supervision missions per year, which never left the counterparts without technical support for long periods. Additionally, the Bank supported four project restructurings to correct for design weaknesses and implementation challenges (see Section 1.7), which indicates a strong effort to address constraints and remain focused on achieving the PDO. 66. There were moderate shortcomings in the Bank’s proactive identification of opportunities and resolution of threats which contribute to rating the quality of Bank’s supervision as moderately satisfactory. Several factors support this assessment including the high turnover of Bank staff. Four different Bank managers had led project supervision throughout the operational cycle, three of them in the first 20 months of implementation. At the end of the third month of implementation, the team leader who prepared the IDEP had already been replaced by the second of those four managers. Bank supervision failed to quickly fix one the biggest weaknesses of the preparation process: the lack of a POM, Project Implementation Plan and Project Procurement Plan in place by the project effectiveness date. The POM was not adopted until 33 months after this milestone. Although the Bank restructured the IDEP four times over its lifecycle, but the first substantial restructuring (the third of the four processes) took place very late, only seven months before the original closing date. The small changes promoted under the second restructuring could have been done later, together with other potential changes envisioned to address other problems related to the Project. (c) Justification of Rating for Overall Bank Performance – rating: moderately satisfactory 67. Overall Bank performance is considered to be moderately satisfactory. Despite significant shortcomings in ensuring quality at project entry, this rating is justified given the moderately satisfactory quality of Bank supervision and that substantial results were achieved under the Project, as reflected in the moderately satisfactory overall outcome rating. 5.2 Borrower Performance 8 (a) Government Performance – rating: moderately satisfactory 8 In this ICR, the word Recipient is replaced by Borrower to remain consistent with the ICR Guidelines. 18 68. Although the GoK remained committed to achieving the IDEP’s objective throughout implementation, there were moderate shortcomings in government performance over the project lifecycle such as: (i) inability to identify the site and lack of funds to finance the construction of the upper secondary education school, leading this activity to be dropped from the Project; (ii) weak coordination efforts with the Municipality of Prishtina to identify the site for the construction of the basic education school, the primary reason for the 18-month project extension; and (iii) a leadership vacuum between October 2010 and February 2011, which resulted in a high turnover of MEST staff. (b) Implementing Agency or Agencies Performance – rating: moderately satisfactory 69. The MEST exhibited a high level of ownership and active participation during project preparation, and remained committed to the Project throughout the implementation period. Capacity to manage, implement and monitor donor-funded operations has also improved in the MEST, MED and schools under the IDEP. Capacity building processes were facilitated by the Ministry’s decision to pair international consultants with local staff. Furthermore, there is evidence that the stronger capacity in the MEST has led the Ministry to proactively pursue quality improvements and reforms that were not envisioned at project preparation. For example, the IDEP supported the establishment of the MEST assessment unit, which has cultivated a greater appreciation and commitment to student assessment, demonstrated by the Ministry’s decision to prepare for and participate in PISA in 2015. The capacity to effectively plan and make informed decisions has also been built within the municipalities, which have begun using EMIS data on a regular basis for such purposes. 70. Despite these factors, there were moderate shortcomings in the performance of implementing agencies. Protracted delays in completing the POM, Implementation Plan, Procurement Plan, as well as delays in recruiting PCT staff, mainly fiduciary specialists, resulted in persistent implementation constraints and periods of non-compliance with the legal covenants. Costs of civil works were inaccurately estimated at preparation. Central and regional implementing agencies neglected to identify some aspects of the national procurement law as a constraint to the devolution of financial autonomy to schools, and this law was not changed throughout the implementation of the IDEP. Finally, monitoring of project indicators was delayed by almost two years following project effectiveness, and the MEST still does not monitor long-term education sector objectives, as expected in the PAD. (c) Justification of Rating for Overall Borrower Performance - rating: moderately satisfactory 71. Overall, the Borrower’s performance is deemed moderately satisfactory, given that both the government and implementing agencies performances are rated moderately satisfactory. 6. Lessons Learned 72. Close supervision of day-to-day project management makes a difference in contexts of limited capacity. Bank supervision positively impacted the implementation of IDEP and the achievement of its objective, especially after a local Bank Task Team Leader took over the project. This change allowed for a closer supervision of several aspects that had delayed project implementation and disbursement, which had reached only 24 percent of the grant after 3.5 years of implementation. The lesson learned from this experience is that in contexts of limited capacity, especially post-conflict environments like Kosovo, local staff should be closely involved in project implementation. 19 73. A balance between the levels of complexity and ambitiousness of project design and institutional and administrative capacities is essential. Early in the project’s lifecycle, implementation was affected by several factors, including the political process of independence declaration, changes in leadership within the MEST, high turnover of key staff involved in project management, and the lack of core project documents. On the other hand, the project design was overly complex and ambitious, considering the number of different and politically sensitive activities in areas like education financing, school autonomy, students’ assessment, teacher licensing and performance assessment, and regulation of tertiary education. Although institutional and administrative capacities have grown throughout the project, implementation was slow in the early years because of the mismatch between existing capacities and the complexity and ambitiousness of project design. 74. Building in flexibility in project design may help address challenges of projects prepared for countries in conflict-affected contexts. As mentioned above, the early stage of the implementation of the IDEP was affected by factors like those related to Kosovo’s post-conflict situation. The overly complex and ambitious design led to four restructurings, some of them very small in scope, to address problems observed during the project lifecycle. The lesson learned from this experience is that building some flexibility in designs of projects prepared for contexts like the one observed in Kosovo may help address problems throughout implementation. 75. An accurate assessment of both enabling conditions and implementation readiness is paramount. The relatively fast preparation of the IDEP (seven months between concept note preparation and appraisal) was justifiable, given the critical phase of Kosovo before the discussion of its status, under the administration of the UNMIK. However, the Bank did not carry out a comprehensive assessment of the necessary enabling conditions and implementation readiness, including an analysis of the existing legal framework and MEST capacity to prepare the required core project documents. As a result, legal constraints not correctly assessed at preparation limited the implementation of the education financing reform, and the lack of relevant documentation caused critical delays in the first several years of implementation. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 76. No comments. (b) Cofinanciers 77. Not applicable. (c) Other partners and stakeholders 78. No comments. 20 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Financing Latest Actual/Latest Percentage Components Estimate (*) Agreement Restructuring Estimate of Estimate (**) (***) Appraisal Component 1 - Strengthening the 4.7 4.0 3.5 3.43 74% Organization and Financing of the Education System in Kosovo Component 2 - Building Institutions 2.6 2.3 2.6 2.26 88% and Management Capacities to Promote Quality Improvements Component 3 - Creating Conditions to 3.3 2.2 3.7 4.09 127% Introduce Efficient Designs and Reduce Multiple Shifts in Kosovo’s Schools Component 4: Higher Education 0 0.5 0.2 0.28 0% Reform Total Baseline Cost (USD Millions) 10.6 9.0 9.9 10.06 97% Physical Contingencies (Unallocated) 0.4 1.0 0.01 0.0 0.% Price Contingencies 0 0.0 0.0 0.0 0 Total Project Costs 11.0 10.0 10.0 10.06 94% Front-end fee PPF 0 0.0 0.0 0.0 0 Front-end fee IBRD 0 0.0 0.0 0.0 0 Total Financing Required 11.0 10.0 10.0 10.06 94% (*) Source: PAD/Annex 5 – Project Costs. It is important to note that the Financing Agreement was signed in the amount of US$ 10 million. (**) 4th Restructuring (June 14, 2013) (***) Variations in total Costs are the result of exchange rate variations for the SDR, USD, and Euro. The actual/latest estimates were calculated by converting a final EUR 7.533 million into USD. The exchange rate used is the average exchange during the lifespan of the Project since the first disbursement in October 2010 (USD 1= EUR 0.748). (b) Financing Appraisal Actual Type of Percentage of Source of Funds Estimate Expenditure Cofinancing Appraisal (USD millions) (USD millions) Recipient 1.0 0.9 90% International Development Association (IDA) 10.0 10.06 100.6% 21 Annex 2. Outputs by Component Table 1. Project Outputs, by Component and Expected Outcomes Component 1 - Strengthening the Organization and Financing of the Education System with Particular Attention to the Legal Framework and Funding Formula Expected Outcomes: • MEST/MFE monitors, analysis and publishes capital and recurrent expenditures (by level of education, by municipality, per student, by spending composition) each year. • 60 percent of schools receiving School Grants produce school improvement plans and monitoring reports by the Project Closing Date Intermediate Outcome Indicators Project Outputs (2 out of 4 were achieved) 1. The amended law on primary and • A primary and secondary education law secondary education clarifies the roles which clarifies roles and responsibilities for and responsibility of each level of the delivery of education among various levels education system (MEST, municipality, of education system is developed and school). adopted by the parliament by June 2011. Output realized. The law was passed in Target achieved. This indicator was revised August 2011. in the 3rd restructuring process. 2. Funding mechanism which provides • Number of schools implementing SDGs transparent, formula-based and which are able to establish quality predictable transfers from municipalities. improvement indicators at school level and Output realized. The funding mechanism monitor them. is in place. However, seven municipalities Target not achieved (target = 181 (out of 38 Kosovo’s municipalities) with schools). The schools that implemented the Serbian majority are not allocating the SDG (207) are not monitoring indicators school budgets according to the formula. yet. This indicator was revised in the 3rd restructuring process. 3. 240 schools implemented the school development plans with quality focus. • Timely preparation/revision of per capita Output not realized. Only 207 schools funding formula that will direct resources to implemented the school development areas of need, and its implementation. plans with quality focus. Target achieved. This indicator was revised in the 3rd restructuring process. • The number of municipalities which allocate recurrent budget to schools according to funding formula. Target not achieved (target = 38 municipalities). Seven (out of 38) municipalities with a majority of Serbian population are not allocating the school budgets according to the formula. Component 2 - Building Institutions and Management Capacities to Promote Quality Improvements in Primary and Secondary Education Expected Outcomes: • MEST establishes baseline of assessment and examination instruments to measure the student learning outcomes at grade 5 by 2010. 22 • 60 percent of municipalities are able to report using EMIS data on the status of dropout and retention disaggregated by gender and community by 2012. • Percentage of teachers licensed as regular teachers. Intermediate Outcome Indicators Project Outputs (3 of 5 were achieved) 4. 70% of teachers are licensed as per • % of teachers who have completed the MEST’s administrative instruction of minimum in-service training requirements. 2007 by 2011. 9 Target achieved (target = 10%). As of Output realized. By February 2014, the 2013, 22.2 percent of teachers completed teachers licensing database stored data of the minimum of 100 hours of training. 80.3 percent of teachers (19,196 teachers). • The Council for Teaching Licensing is 5. Council for Curriculum, Textbooks and established, and teacher licensing, Assessment (CCTA) and National Teacher professional development and performance Education Board (NTEB) are established valuation systems are in place by project and functional. 10 closing date. Output realized. Target not achieved. The Council for Teacher Licensing was established and the 6. EMIS is used for decision making at teachers licensing aspects were functional, MEST and municipalities. while the other two elements of the system Output realized. The EMIS was are not in place yet. deployed to all schools (with the exception of those with Serb majority), • The availability and quality of annual EMIS and data collection has been done at the report, consisting of statistics and analysis school level. The data for the start of the of education trends. 2012/2013 school year was used by the Target achieved (target = report MFE for the financing formula. Moreover, disseminated). a report on Kosovo education statistics based on the 2012/2013 school data was • System for learning assessment at the produced in May 2013, with the support primary level (core indicator). of UNICEF. Target achieved (target = yes). • Number of additional qualified primary teachers resulting from project interventions (core indicator). Target not achieved (target = 17,666). 15,529 teachers were trained. Component 3 - Creating Conditions to Introduce Efficient and Appropriate Designs and Reduce Multiple Shifts in Kosovo’s Schools to Address the Current and Future Constraints in Learning Spaces Expected Outcomes: • No outcome was identified for this component. 9 The administrative Instruction was revised and licensing was done based on AI 16/2008 and then updated through AI 5/2010. 10 Following the MTR, the activities related to curriculum were dropped, and therefore the output related to the CCTA was also dropped. The NTEB became the State Teacher Licensing Council which was in fact established and is fully functional (see results framework after 2011 restructuring). 23 Intermediate Outcome Indicators Project Outputs (2 out of 2 were achieved) 7. School mapping which takes into • The availability of school mapping, and consideration population growth and costed investment plan and maintenance enrollment target completed. guidelines for school facilities by December Output realized. The school mapping 2011. was developed under the Project, and has Target achieved. The school mapping is been used to prepare investment and completed, and the investment plan and maintenance plans, which will guide maintenance guidelines are finalized, future interventions either by the approved and under implementation by the Government or donors. MEST. 8. Fiscally sustainable standards for school • Development of new school construction facilities developed. design standards. Output realized. The Manual for School Target achieved. The manual with school Standards and Design for Pre-University construction design standards was formally School Facilities was developed and adopted by the MEST in July 2011. adopted by the MEST. This is a binding document required to guide all new school infrastructure investments. The MEST has approved by-laws which elaborate the review mechanism ensuring that all new school constructions are in line with the: (i) school investment plan, and (ii) the mentioned manual. 9. Two model schools based on new standards built. 11 Output not realized. Only one school (for basic education) was built, as part of the 3rd restructuring process. Component 4 - Strengthening Management Capacity in Higher Education, at System and Institutional Levels to Support the Implementation of Higher Education Strategy 2005-2015 Expected Outcomes: • No outcome was identified for this component. Intermediate Outcome Indicators Project Outputs (1 out of 2 was achieved) 10. The results of the study on cost-effective • The Law on Higher Education is amended expansion and financing study on higher and adopted. education have been widely discussed and Target achieved. This indicator was policy choices have been made. revised in the 3rd restructuring process. Output realized. The Project supported the study on the size and shape of higher • The study on the size and shape of higher education which MEST reflected in their education in Kosovo is used by MEST to sector strategy. develop a concrete strategy which proposes 11 Due to financing constraints, the Project financed only one school (basic education) construction. This was reflected in the 3rd restructuring. 24 a cost-effective expansion of the system. Target not achieved (Target: a higher education financing strategy backed with budget projections is adopted by 2010). This study was carried out, its recommendations were incorporated in the higher education part of the sector strategy, but the strategy falls short of including a proposal for a cost-effective expansion of the system. 25 Annex 3. Economic and Financial Analysis 1. Although the PAD does not explicitly state the rationale for public investment, Kosovo’s Constitution states that "every person enjoys the right to free basic education”. Education in Kosovo is mostly provided by the public sector, with more than 95% of students attending public schools. Moreover, the provision of social services is generally accepted as a government obligation and private financing for it is mostly non-existent in Kosovo. Project interventions have therefore targeted public sector organizations. The World Bank’s involvement was based on its expertise in policy design and implementation following international best practices. This was clearly the fundamental added value from the institution. 2. Based on the PAD’s Economic and Financial Analysis and the broad definition of the PDO (strengthening systems, institutions and management capacities needed for education quality improvements), it is important to note that a quantitative assessment of the cost-effectiveness of project investments cannot be undertaken. The PAD discussed the alignment of the project with education sector needs and objectives, and focused on how effectively the project would support the implementation of the strategies within the overall assistance of development partners. It did not, however, calculate expected net present value or economic rates of return. Therefore, the remaining alternatives for economic and financial analysis include: (i) an analysis of the expected improvements in the quality of the system through teacher development and assessment systems; (ii) an analysis of the efficiency gains derived from the development of the new financing scheme, the EMIS progress, the new school standards, potential efficiency gains in the use of the financing formula, the derived allocation of resources, and the school development grants. 3. Education Indicators. During the lifespan of the Project, there were improvements in some education indicators. Transition rates between grades remained at almost universal levels in basic education, and improved significantly in upper secondary, especially for girls (see Figure 1). Enrollment rates for girls in upper secondary increased from 80 percent in 2010 to 88 percent in 2012, while for boys the rate only increased from 92 percent in 2010 to 95 percent in 2012. The gender gap in enrollment has decreased in both primary and secondary education: full gender parity in basic education was achieved in 2013 (48.5 percent of girls in school, with girls comprising 48 percent of the school-age population). There was also a moderate decrease in student-teacher ratios, especially in basic education, putting Kosovo more in line with comparator countries (see Figure 1). Figure 1. Transition Rates and Student-Teacher Ratios during IDEP Lifespan Average transition rates by education level Student Teacher Ratio by education level 25 1 Basic Education 0.98 23 0.96 Upper Secondary Primary 21 0.94 0.92 19 Lower 0.9 Secondary 17 0.88 Upper Secondary 15 0.86 2007/08 2008/09 2009/10 2010/11 2011/12 2012/13 2007/08 2008/09 2009/10 2010/11 2011/12 2012/13 Source: Ministry of Education, Science and Technology of Kosovo. 26 4. Donor Contributions. During project implementation, a number of international donor agencies increased their contributions to the education sector in Kosovo. Several international agencies increased their participation in education activities, especially Austrian Development Agency, European Union, GIZ, Denmark, Norway, Sweden, Switzerland, and the USAID. Disbursements from donors increased from €2 million in 2007 to over €20 million in 2011, excluding the World Bank’s support. Through the IDEP, the education system increased its capacity to absorb committed and planned disbursements, as the rate of disbursed/committed amounts increased, even more that what could have been planned by each donor per year (see next figure). This reflects an increased capacity of systems, institutions, and management to improve education in Kosovo, exceeding the initial expectations of other donors. In this sense, IDEP funds were efficiently leveraged to secure additional resources for the education sector. Figure 2. Donor Contributions during the IDEP Lifespan 25 1.40 Yearly Rate of Committment/Disbursement 1.20 20 Disbursements Disbursements (Million Euros) 1.00 15 0.80 Ratios 0.60 10 Disbursements/Commit 0.40 5 ments Rates 0.20 0 0.00 2007 2008 2009 2010 2011 2012 2013 Source: Ministry of European Integration of Kosovo. Efficiency Gains Attributed to the IDEP 5. Teacher Policies and Assessment. There is strong consensus in the education literature and research around the central role of teachers in the learning process. Evidence has shown that the impact of having a good teacher, as opposed to an average teacher, can be substantial. 12 The IDEP improved the average qualifications of the existing teacher workforce (which by 2007 was rather low) through: (i) the teacher licensing process, which completed the minimum in-service training; and (ii) the higher qualifications of young graduates entering the profession (see next figure). Strengthening the skills of the teacher workforce is a key factor for quality improvements of the system. In order to be effective, teacher quality improvements need to be tightly coupled 12 Rivkin, S. G., E. A. Hanushek, and J. F. Kain (2005) "Teachers, schools, and academic achievement", Econometrica, 73 (2), p. 417-458. 27 with assessment and curriculum policies. 13 The IDEP contributed substantially to this by developing the 5th grade assessment in 2010, which is paramount for monitoring quality improvements and will serve as a first step for rolling out PISA in 2015 for the first time in Kosovo. Figure 3. Teacher Qualifications during the IDEP Lifespan Number of teachers and qualifications 12,000 10,000 2008 8,000 2013 6,000 4,000 2,000 0 Unqualified / 5-year secondary 2-year Higher BA (3-year & 4-year) Masters and Ph.D Beginner teachers teacher school Pedagogical School degrees degrees Source: Kosovo Public Expenditure Review. World Bank (2014) 6. Financing and EMIS: Achievements and Challenges. The Project supported the development of the per capita financing mechanisms, in order to better respond to school and municipal needs. Due to the new municipal-to-school formula supported by the IDEP, municipalities have started to allocate recurrent budget. Data development at the central level has been crucial for this process, as it has informed policy makers (at both the municipal and central levels) on teachers and student flows. This has led to a more active approach, where funding follows needs, rather than the alternative. Informal conversations with school principals indicated large improvements in school allocation of resources and also the use of EMIS data for planning purposes. However, as noted in the World Bank’s 2014 Public Finance Review, the IDEP contributed to increased school autonomy and transparency, but the efficiency gains are still not very visible, perhaps due to a rather complex design of the formula, which may add rigidity and complexity for low capacity implementing agencies (MEST/MED). 14 The report proposed better monitoring by MEST, stronger implementation support for municipalities, and the facilitation of exchanges of experiences between the better performing municipalities. 7. School Development Grants (SDG). The PAD projected a US$4.0 million investment for the component of SDG, which would benefit around 92,000 students, implying an average of US$43 per student. Project reports show that: (i) 207 schools received grants, with 78 percent of them located in rural areas; (ii) 86 percent of the grants were provided to basic education institutions; and (iii) 124,154 students benefited from the school grants, with around $2.2 million spent in total, which leads to an average of $18 per student. These figures show that more 13 World Bank (2005). Expanding Opportunities and Building Competencies for Young People: A New Agenda for Secondary Education. 14 World Bank (2014). Kosovo Public Financing Review. Unpublished. 28 students benefited from the SDG than what was expected, with a lower amount per student, which reflects a more efficient implementation of this subcomponent. The assessment report of the SDG program conducted in 2013 showed that the “quality of the SDGs program was consistently high, and efficiency within the SDGs program was primarily achieved by having the IDEP staff, MEST, KEC and the ROSOs accomplish so much”. 15 Moreover, “the work of the IDEP employees was mostly legal and fiduciary, and this kept the schools on schedule to progress through to the final procurement of equipment and resources”. 8. School Map, School Standards, Shift Changes and Investments. The Infrastructure Unit of the MEST has reported extensive use of the school map, complemented with the construction standards (both supported by the IDEP), for subsequent investments both from MEST but also international donors willing to contribute to capital investments. The Project therefore has been fundamental in ensuring that public and donor funds are used efficiently by addressing the needs of overcrowded and deteriorated schools. Cost-Effectiveness of the IDEP 9. In the PAD, the following project alternatives were considered: (i) supporting school development grants only; (ii) conditional cash transfers to girls, poor, and ethnic minorities, to improve enrollment; (iii) school construction; and (iv) adaptable program financing. The alternatives are discussed below according to the project’s objectives, education sector developments during the project, and the project achievements: (i) Supporting School Development Grants only: although the SDG could provide schools and school boards with opportunities to start building their management capacities and mobilizing communities in school activities, the previous round of school grants under EPIP showed low capacity at the school and system level in terms of management and information systems. Therefore, the proposed Project supported an investment in school grants with capacity building at all levels of education management. The decision has proven to be effective as reported by school principals, which have stated that capacity building activities were crucial for making better use of SDG. (ii) Conditional Cash Transfers Targeted to Vulnerable Populations: these programs have been successful in many other countries. However, the education sector in Kosovo has witnessed a significant increase in upper secondary enrollment rates, which was the most problematic level in terms of dropouts and attendance for vulnerable populations. Enrollment rates for girls in upper secondary increased from 80 percent in 2009 to 88 percent to 2010, while for boys it only increased from 92 percent in 2010 to 95 percent in 2012. Therefore, a project aiming exclusively at improving access in upper secondary (and focusing on vulnerable groups) would not have added significant value to the system. (iii) School Construction: this intervention would not have addressed the quality or capacity and management constraints in the sector. Moreover, due to procurement issues, efficiency in spending would have faced several difficulties. (iv) Adaptable Program Financing: the unexpected events surrounding the Status of Kosovo, with well-founded doubts given the declared independence in 2008, were a major constraint to this approach. 15 Project Evaluation of School Development Grants Components. World Bank (2013). 29 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Nicholay Chistyakov Senior Finance Officer CTRLN Fin. Management Belita Manka Council LEGOP Proc. Specialist Flora Kelmendi Senior Operations Officer ECSH2 TTL Fadil Jakupi Driver ECCKO Driver Diomedes Berroa Lead Specialist (OPCS) OPSOR Operations Olav Rex Christensen Senior Public Finance Specialist CEURO Fin. Management Ruxandra Maria Floroiu Program Assistant HDNED HD Support Elona Gjika Financial Management Specialist Fin. Management Sachiko Kataoka Senior Education Economist ECSH2 Former TTL Plamen Stoyanov Kirov Senior Procurement Specialist LCSPT Proc. Specialist Jamil Salmi Consultant Educ. Specialist Maria E. Gratcheva Senior Operations Officer SASHD Educ. Specialist Keiko Miwa Country Manager EACLF Former TTL Imelda Mueller Operations Analyst ECSH2 HD Support Cristobal Ridao-Cano Country Sector Coordinator EASHS Former TTL Supervision/ICR Financial Nicholay Chistyakov Senior Finance Officer CTRLN Management Elona Gjika Financial Management Specialist ECSOQ Fin. Management Diomedes Berroa Lead Specialist (OPCS) OPSOR Proc. Specialist Sachiko Kataoka Senior Education Economist ECSH2 Former TTL Plamen Stoyanov Kirov Senior Procurement Specialist LCSPT Proc. Specialist Jamil Salmi Consultant ECSO2 Educ. Specialist Kashmira Daruwalla Senior Procurement Specialist ECSO2 Proc. Specialist Andrea C. Guedes Senior Operations Officer ECSH2 Educ. Specialist Janssen E. N. Teixeira Senior Education Specialist ECSH2 ICR TTL Lucas Gortazar Junior Professional Associate ECSO3 ICR co-author Ida N. Muhoho Financial Management Specialist EASHD Fin. Management Ireneusz M. Smolewski Program Assistant ECSHD HD Support Margaret M. Clarke Senior Education Specialist HDNED Educ. Specialist Esma Kreso Environmental Specialist ECSEN Envinm. Specialist Senior Financial Management Zeynep Lalik ECSO3 Fin. Management Specialist Imelda Mueller Program Assistant ECSH2 HD Support 30 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY07 0 0 FY08 0.8 5,522.88 Total: 0.8 5,522.88 Supervision/ICR FY08 6.55 21,966.47 FY09 41.29 89,163.44 FY10 32.28 47,773.63 FY11 37.54 54,302.37 FY12 29.50 44,702.44 FY13 27.98 39,600.71 FY14 23.75 57,404.78 Total: 199.70 357,436.72 31 Annex 5. Beneficiary Survey Results Not applicable. 32 Annex 6. Stakeholder Workshop Report and Results Not applicable. 33 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Project objective, design, and implementation 1. The Kosovo Institutional Development for Education Project (IDEP) aimed to support the government’s Strategy for the Development of Pre-University Education in Kosovo and the Strategy for the Development of Higher Education. The specific objective of the project was to “strengthen systems, institutions and management capacities needed for education quality improvements”. The Project supported four key elements of the government's Pre-University and Higher Education Strategies, which became the Project’s four key components: (i) strengthening the organization and financing of the education system in Kosovo, (ii) building institutions and management capacity to promote quality improvements, (iii) creating conditions to introduce efficient and appropriate designs and reduce multiple shifts in Kosovo's schools, and (iv) strengthening the management capacity at system and institutional levels for higher education. Each of these components included a number of subcomponent activities as described below. The project PDO was not revised. However, during the February 2011 mid-term review, some indicators were modified to reflect and address changes and new developments that emerged during implementation. A description of the four main components and subcomponents follows below: Component 1: Strengthening the organization and financing in the education system in Kosovo, including: • 1.1. Developing an appropriate legal framework and the management structures and processes for pre-university education which clarifies the roles and responsibilities of each level and the accountability mechanisms of the education system. • 1.2. Strengthening the leadership, decision making and resource management at the municipality and school levels through creating funding mechanism which provide transparent, formula based and predictable transfers from municipalities to schools. • 1.3. Providing school development grants which would strengthen the capacity of schools to plan and manage resources toward realizing their vision for developing effective learning environments and improve the quality of teaching-learning conditions and enhance inclusive education at school level. Component 2: Building institutions and management capacity to promote quality improvements in primary and secondary education, including: • 2.1. Strengthening the institutional capacities and management processes of the key department in the MEST concerning the quality of education, including teacher licensing, teachers’ professional development, and teacher’s performance assessment. • 2.2. Improve the quality and relevance of curriculum provision and processes so that it would positively impact on the quality of teaching and learning in Kosovo pre-university education. • 2.3. Implementing the assessment of students’ learning outcomes, and establishing the baselines to monitor learning outcomes in the long-term, and the review of the matura (high school leaving exam) and its full implementation. 34 • 2.4. Developing the Educational Management Information System (EMIS), providing training for key staff and enabling the use of the system for analysis of key policy issues and decision making by MEST. • 2.5. Providing project implementation support (PCT operational expenditures). Component 3: Creating conditions to introduce efficient and appropriate designs and reduce multiple shifts in Kosovo’s schools to address the current and future constraints in learning space , including: • 3.1. School mapping, as well as preparing the school facility investment and maintenance plans. • 3.2. Developing standard designs for classrooms and school facilities, which are fiscally sustainable, appropriate for new teaching methods, and energy efficient. • 3.3. Constructing two model schools based on the new design standards and school mapping. Component 4: Strengthening management capacity in higher education, at system and institutional levels to support the implementation of Higher Education Strategy 2005-2015, including: • 4.1. Providing technical support to amend and adopt the law on higher education and revising the existing by-laws and developing new ones. • 4.2. Feasibility study of the financial sustainability of the Kosovo higher education system (financing needs, cost-sharing, scholarships and student loans, budget allocation mechanisms). 2. The overall project objective was consistent with the sector issues identified and MEST policy reforms. Kosovo had just been declared a new state in 2008 and was in the early stages of building its own institutions. The project objectives reflected a wide range of challenges the Ministry was confronted with in reforming the education system. While the design was technically sound and well tailored to the needs of Kosovo, the interventions entailed politically sensitive processes that required more time and effort in preparing the various constituents and stakeholders. This risk was not discussed in the Risks Section of the Project Appraisal Document (Section E). In retrospect, the Project suffered from three fundamental weaknesses: (a) it was ambitious for the given circumstances of time and existing institutional capacity, (b) the funding was limited and (c) the four-year implementation period was too short. 3. Although many changes were introduced in the approach aiming to improve the final outcome, the Project was not subjected to major revision in its components, and most of the original activities have been successfully completed with two exceptions: (i) withdrawal of subcomponent 2.2 (curriculum) because of interest of many other donors to support this component (the UNICEF and EC are providing support towards this objective); and (ii) partial achievement of subcomponent 3.3 in which the design and construction of the second model pilot school (upper secondary) was also withdrawn because of the lack of funding on Government side. 4. During the life of the Project, there have been several personnel changes. The project dealt with five different Permanent Secretaries during its six-year implementation period. In addition there have been changes in the positions of key staff working on IDEP, namely, the Project Coordinator, the Procurement Specialist and later the Financial Manager, which resulted in 35 significant loss of institutional memory. Loss of momentum in project implementation resulted from delays in appointing replacements for these staff. Outcome of the Project against the agreed objectives (in terms of relevance, efficacy, and efficiency) 5. The project development objective (PDO). The Project substantially achieved the PDO and met or exceeded all seven expected outcomes: • The legal framework for education is amended in accordance with the resolution of the Status issues. • The Municipal Education Directorates (MEDs) are able to administer transparent and predictable funding formula for schools. • Schools are better equipped with the resources and capacities to improve the quality of education. • An adequate institutional framework is developed to address the quality of education issues. • MEST and the Municipal Education Directorates are able to monitor the implementation of strategies and to make informed decisions. • Improved planning and implementation capacities for the construction of school facilities which are cost efficient and appropriate for new curriculum. • The management procedures and capacities of the Higher Education Department of the MEST are improved. 6. Project achievements with respect to each subcomponent were as follows: Sub-1.1. Legal framework for pre-university education (i) The Law on primary and secondary education clarifying the roles and responsibility of each level of the education system (MEST, municipality, school) has been completed; (ii) Administrative Instructions (AIs) guiding the implementation of the Law have been finalized and approved; The Law on Pre-University Education also establishes the foundation for important reforms, such as the school autonomy, financial decentralization, teacher qualifications and career development, teacher and school management selections, role of assessment and examination, and role of school boards and parents in teaching process. Sub-1.2. School financial autonomy (i) The revised and complemented state formula for education-specific grants to municipalities has been in use for four years now, and is part of the regular budget allocation from central to municipal level for all levels of pre-university education; (ii) the municipal-to-school formula has been rolled out at a national level last year and was used by the municipal authorities for the school budget allocation for 2014. MEST has made several revisions to the formula on the basis of municipality responses to improve 36 its adequacy; to ensure transparency of the specific education grant formula, MEST and MoF have published the criteria on their websites to enable all interested parties to understand how each municipality’s specific education grant formula is determined; (iii) allocation of budget to the school level has meant an extended financial autonomy at school level. Schools, for the first time, received an allocation for recurrent expenditures which they could plan and prioritize according to their needs. However, the full delegation of financial autonomy for recurrent expenditures has faced considerable challenges with municipalities being hesitant to grant financial autonomy to schools and allow them to undertake procurement activities due to legislative constraints. The issue has been addressed and MEST has got assurance by the Kosovo Procurement Agency that the revisions of the public procurement law will enable schools to undertake minimum value procurement (EURO 1,000) which will ensure that the legal basis for the reform is there and can enable schools and municipalities which may be more advanced to take on full autonomy of their budgets when ready. Sub-1.3. School development grants (i) Development grants were awarded to 207 schools; (ii) all schools have received training on school development plan and monitoring implementation which was highly valued by school management and staff. Sub-2.1. Teacher Carrier System (i) Pre-service qualification verification and licensing of teachers has been implemented and the regional inspectorate departments have been trained and equipped with the skills, processes and tools to maintain and update the database; (ii) teacher licensing, professional development tools that are linked with career advancement and rewards have been developed; (iii) the instruments for teacher assessment have been piloted and approved; (iv) the final teacher assessment policy instruments and the AI on teacher performance assessment are by-legal documents; and (v) the Teachers Licensing Database has been completed 100 % of teachers who have applied. However, there is still about 20 percent of teachers missing, because unqualified teachers were excluded from the database. Sub-2.2. Curriculum (i) The revision of the existing curriculum has been initiated with the help of technical assistance by the project (ii) The plan and the supporting documents for the establishment of the Agency for Assessment and Evaluation have been drafted. Because the interest of many donors was great on this component the project has withdrawn by letting UNICEF and EC take over. This subcomponent was dropped from IDEP funding because it is supported by the EC and UNICEF. Sub-2.3. Assessment and Examination (i) The Assessment Policy and Assessment Reporting Standards have been developed; (ii) the Item Bank has been developed and accepted by MEST; (iii) the instruments and the baseline for grade 5 assessment have been developed; (iv) the Matura administration criteria and process have been improved; (v) overall capacities of the assessment unit have been strengthened, and (vi) support to application in the Program for International Student Assessment (PISA) has been provided. 37 Sub-2.4. Educational Management Information System (EMIS) (i) EMIS was developed, is operational and continues to help schools enter the required data for each academic year; (ii) data collected by EMIS is used for planning specific grants for municipalities, statistical analysis which will help in decision-making for effective budget planning, strategic planning and analysis of the current situation in education; (iii) MoF is using this data to determine the amount of the education-specific grant (per capita financing); and (v) a report of Kosovo Education statistics based on the 2012/2013 school data was produced in May 2013 with the support of UNICEF. Sub-2.5. Providing project implementation support (PCT operational expenditures) (i) Provide financial support for project management operations—salaries of PCT staff, procurement and FM training, equipment, consumable materials, etc. There was no such subcomponent mentioned in the PAD or the FA. However, the project drafted an Activities Plan and the sub-component 2.5 was introduced and renamed “Project Management Support” to reflect the activities and expenditures of the PCT. The 2.5 code was systematically used in the Procurement Plan and the FM database. Sub-3.1. School mapping (i) The school mapping system is fully operational, providing a complete database of all existing schools, including their physical condition and future improvement needs; (ii) all schools have been surveyed in accordance with inventory sheets designed by specialist consultants; (iii) the information collected is being integrated with EMIS; and (iv) a five- year Investment and Maintenance Plan (IMP) has been completed in 2012 covering MEST priorities for years 2013-2017. Sub-3.2. School building standards and norms (i) Developed guidelines for school standards and norms, which are being used. These guidelines comprised two volumes: General:—dealing with design concepts and general principles, facilities and inventory are designed based on the education levels and using anthropometrics of school-age children; and Specific:—dealing with interior and exterior spaces; (ii) MEST has approved AIs, making these guidelines legally binding to anyone designing schools in Kosovo. These guidelines have been applied to the model school built under subcomponent 3.3; and (iii) school designs conforming to these guidelines are expected to result in more efficient use of space, improved operation due to functional design, and reduced maintenance costs in the long term due to the use of durable materials and components. Sub-3.3. Construction of two pilot model schools (i) Construction of a basic education school (grades 0-9) with capacity for 720 students has been completed and is now fully furnished, equipped and ready operational. MEST is pleased with the special features of the building—quality of workmanship and building components, functionality, durability, aesthetics, efficient energy consumption, and low maintenance. The design and construction of an upper secondary school (grades 10-12) was dropped because the Government could not secure funds needed for this school and the project had limited funding for this component. 38 Sub-4.1. Supporting MEST in developing amendments to the Law on Higher Education (i) The new law on Higher Education and necessary by-laws have been amended and approved, following a broad consultative process with relevant stakeholders; (ii) key AIs required to guide the implementation of the law and ensure its harmonization with other education sector legislation have been finalized and approved; and (iii) key reforms, including the accreditation of public and private providers of higher education. Sub-4.2. Strengthening management capacities at MEST level (i) Completion of a study on the size and shape of higher education which MEST reflected in their higher education strategy. Efficacy 7. Despite its funding, IDEP has served as a strong catalyst in addressing a wide range of education policies with greater efficacy, positioning the project well in ensuring the coordination of efforts within the various sub-sectors that maximized the results, impact and project efficiency. The PDO remained highly relevant for Kosovo’s education sector and the Government’s development priorities for the sector. IDEP played an important role in sustaining the Kosovo Education Strategy Plan (KESP) 2011-2016. All the IDEP activities were in line with the initial KESP, including the revised one that developed during the life of the project. Many positive outcomes resulted from joint efforts to address many of KESP’s strategic objectives by launching the reforms in a number of sub-sectors. Direct connection of the IDEP activities with KESP include work carried out through the Project, namely: (a) the new law on pre-university education and higher education which created a mechanism to ensure the overall sector reform coordination; (b) school financial autonomy and its implementation; (c) building institutions and management capacities to promote quality improvements in primary and secondary education through the establishment of the Teachers’ Licensing mechanism and standards and supporting quality improvement at school level through the implementation of School Development Grants; (d) improvement of the educational management information system (EMIS); (e) reducing the shifts in schools and improving the quality of education; (f) developing school design standards and norms to promote appropriate design to minimize waste in facilities and enhance the “learning environment”; (g) demonstrating the new standards and norms by constructing a “model school” of improved design expected to result in more efficient operation and reduced school maintenance costs of schools; and (h) and supporting the creation of a better financing system in higher education. 8. While the overall relevance of this Project remains high, some of the interventions require considerable attention and efforts in order to insure sustainability. Many of these interventions encountered considerable challenges as described below. Recipient’s own performance during preparation and implementation, with special emphasis on lessons learned that may be helpful in the future 9. Project ownership. During preparation and later during implementation, the Project enjoyed strong commitment and support from the Minister of Education, Mr. Enver Hoxhaj, and the government in general. With the change of the government in 2011, the new Minister, Mr. Ramë Buja, also continued to provide his strong support. This strong relationship continued with the visits of the Bank’s high level officials, who expressed strong commitment to the reforms in education. The Project also contributed to the capacity building of the MEST staff involved in the 39 activities and helped them to become familiar with new developments in the field of education and to learn how to implement the reforms under the Project. 10. Administrative changes. During the project life, there were Government reshufflings with the change of Minister, Deputy Ministers and up to five changes of Permanent Secretaries who had the role of Project Director. However, as high level officials could not devote the time and energy needed to act in that capacity because of many reform activities happening at the same time, the Project suffered delays. The Project complexity compounded to the problem. Given that project implementation relied heavily on MEST programming and administration, the PCT staff introduced coordination meetings every two weeks with relevant MEST Departments aiming to track progress and address issues. Implementation Schedules and the Procurement Plan were regularly revised addressing necessary repackaging of contracts and often added new activities as the Project was being implemented. 11. Project Operational Manual (POM): The POM providing a detailed description of components, criteria and arrangements for the award and implementation of SDGs was not developed until the third year of implementation. Once in place, the POM served as an effective implementation tool as well as a management tool tracking project activities. 12. TA Consultancies. The Project relied on a large number of consultancies, foreign and local, requiring considerable effort in hiring them and managing their contracts, including the logistics of their activities during their visits. The PCT tried to use existing local technical expertise to the extent possible and used many local consultancies (about 25) and almost every subcomponent involved some foreign consultancy (about 20). 13. Disbursements. The disbursement rate was very low until late in the project implementation due to delays in two major components: SDGs and construction of the model schools to which about 50% of the total project funds were allocated. The Project leadership took measures to expedite the implementation of these components aiming to boost expenditures and disbursements. This proved difficult because original expenditure/disbursement estimates were based on unrealistic timelines. Implementation plans and disbursement schedules were revised to reflect an implementation pace in line with available resources. 14. Procurement. Since the start of the Project and throughout the project life, procurement experienced serious challenges, mainly due to lack of the procurement expertise and lack of understanding of the Bank’s procedures and requirements. The project has had three procurement officers, each of which have required significant external support in order to carry on the procurement processes. Over time, a close collaboration between the PCT and the Procurement Department of MEST developed, benefiting both parties and the Project. The project faced final challenge when the procurement specialist left at the final phase of the project when the efforts were being put in finalizing the construction works on school building and furnishing the school. 15. Financial Management. The task was particularly challenging with initial estimates in US Dollars converted in SDRs for the purpose of the Grant, but subsequently contracting was in Euros. The SDR balance was regularly checked to ensure that adequate funds were available to cover expenditures. The performance of the PCT in financial management was satisfactory in most respects. 40 Performance of the Bank during preparation and implementation, including the effectiveness of their relationships, with special emphasis on lessons learned Preparation 16. The project development objectives were clear and focused on outputs and short-term outcomes. The PAD was well structured with clear information on implementation arrangements and relationships between various groups. However, the information was provided in the condensed format of the PAD. The usual Project Implementation Plan (PIP) and package of working papers (project costs, implementation schedules, and a procurement plan) were not provided to the counterparts during or immediately after appraisal, and Annex 12 (Documents in the Project File) does not list such documents. 17. Project Implementation Plan (PIP). In June 2008, with participation from MEST and other stakeholders, the workshop was organized by the Bank team with support from PCT and Kosovo Education Center. This workshop formed the basis of a PIP identifying all activities under each subcomponent, together with their cost estimate, timing, and procurement implications. Similar efforts were directed towards setting up the procurement activities of the PCT and the Implementation Schedule for the key Project activities. 18. Support for Project Management. The PAD did not explicitly include a subcomponent in support of project management—salaries of the PCT staff and other operational expenditures for core management functions of the PCT. The Project prepared an Activity Plan and proposed a useful solution by substituting this subcomponent with another titled “Project Support”— essentially listing the PCT staffing, operational expenditures for the management of the Project and consultancies for core functions (coordination, procurement, financial management). The 2.5 subcomponent code was used in Financial Management (FM) as well as the Procurement Plan (PP) for any expenditures relating to project management. Implementation 19. As a donor and supervisor, the Bank team has always provided its close and intensive support to the PCT through monitoring, supervision missions, training courses, and on the job training. The Bank team has demonstrated strong commitment and involvement during the preparation of the Project and its implementation. The task team leader (TTL) for the Bank was supported by a local staff and consultants over the period of preparation. The Bank’s team was successful in gaining the trust of the Government and the Project was critical and extremely timely in addressing the need for educational reforms in Kosovo. 20. There were four Bank TTLs over the implementation period, all of whom were very supportive of the Project and made it a priority, whenever needed, to resolve problems and meet various challenges experienced during implementation. 21. The Bank has worked well with the PCT and the relevant government institutions to support this Project and find creative ways to improve implementation. The World Bank Country Office carried out regular project supervision and was involved and dedicated to this Project. The Bank management supported two extensions of the time period for this project to ensure that all activities would be completed by the project closing date. MEST worked closely with the MoF on school autonomy reform, and coordination with Municipality of Prishtina on the Model School construction to ensure provision of services agreed under an MOU with MEST. 41 Lessons learned and proposed arrangements for future projects (a) The Project was too complex with many subcomponents requiring specific knowledge of the subject matter and complex dialogue with MEST and other stakeholders. Simpler project design for new states like Kosovo is recommended. (b) Significant reprocessing of activities and recalculation of costs was necessary during project implementation because of the paucity of background material and working papers on the Project. Complete working papers on project costs, implementation schedules, activity breakdown, should be part of project preparation. (c) The proposed reforms were ambitious with complex political and financial implications in a constantly changing political environment that made implementation difficult, challenging, and time-consuming. Implementation of policy reforms should be programmed in a more comfortable timeframe. (d) The Steering Committee, requested in PAD to monitor the project success, although established did not proved to be effective and functioning. Instead the project organized by-weekly monitoring meetings with relevant MEST Departments involved in the project which proved to be much more functional and providing results. (e) Implementation that involves the local government to implement the certain project activities such as securing land in urban areas where large schools are needed is very difficult as well as time-consuming, especially if educational and other social aspects are to be met in the process of selection. The issue of site selection should have been addressed at the very start of the project and covered with appropriate covenants in the FA. (f) Reduce the scope of TA consultancies with more realistic outputs while providing adequate support and local counterparts to assist and also benefit from the consultancy. Management of as many TA consultancy contracts as demanded under this Project was a great burden for the PCT. 42 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 43 Annex 9. List of Supporting Documents Government of Kosovo (2007). Strategy for Development of Pre-University Education in Kosovo 2007-2017. Mar 2007, Prishtina. Government of Kosovo (2011). Law No. 04/L-037 on Higher Education in the Republic of Kosovo. Sep 9, 2011, Prishtina. Government of Kosovo (2011). Law No. 04/L-032 on Pre-University Education in the Republic of Kosovo. Sep 16, 2011, Prishtina. Government of Kosovo (2011). Kosovo Education Strategic Plan 2011-2016. Prishtina. Rivkin, S. G., E. A. Hanushek, and J. F. Kain (2005). Teachers, schools, and academic achievement, Econometrica, 73 (2). World Bank (2005). Expanding Opportunities and Building Competencies for Young People: A New Agenda for Secondary Education. Washington, DC. World Bank (2007-2013). Documents in Project’s Electronic File, including Financing Agreement, Aide-Memoires, Back-to-Office Reports, Audit Reports, Implementation Status and Results Reports, and Restructuring Papers. Washington, DC. World Bank (2007). Project Appraisal Document on a Proposed Grant in the Amount of SDR 6.4 Million (US$ 10 Million Equivalent) to United Nations Interim Administration Mission in Kosovo for the Benefit of Kosovo, Serbia, for a Institutional Development for Education Project. Report No. 41307-XK, Nov 13, 2007, Washington, DC. World Bank (2007). Implementation Completion and Results Report of the Kosovos’s Education Participation Improvement Project. Report No. ICR0000343, Jun 29, 2007, Washington, DC. World Bank (2012). Country Partnership Strategy for the Republic of Kosovo for the Period FY12-FY15. Report No. 66877-XK, May 1, 2012, Washington, DC. World Bank (2013). Project Evaluation of School Development Grants Component. Washington, DC. World Bank (2014). Kosovo Public Financing Review. Unpublished. Washington, DC. World Bank (2014). Project Concept Note of the Kosovo’s Education System Improvement Project. Washington, DC. 44 Map 45