Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005071 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF-A4096-001 ON A SMALL GRANT IN THE AMOUNT OF USD 0.34 MILLION TO Ukraine FOR Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) December 17, 2019 Education Global Practice Europe And Central Asia Region Regional Vice President: Cyril E Muller Country Director: Satu Kristiina J. Kahkonen Regional Director: Fadia M. Saadah Practice Manager: Harry Anthony Patrinos Task Team Leader(s): Lucia Casap ICR Main Contributor: James Gresham ABBREVIATIONS AND ACRONYMS AIСEM Automated Information Complex on Educational Management ECA Europe and Central Asia EMIS Education Management Information System FM Financial management GNI Gross National Income GSE General secondary education IA Implementing agency IEA Institute of Educational Analytics IRI Intermediate result indicator MoES Ministry of Education and Science ODP Open Data Portal OECD Organisation of Economic Cooperation and Development PDO Project Development Objective PI Preschool institutions PISA Programme for International Student Assessment SIDA Swedish International Development Agency STEP Systematic Tracking of Exchanges in Procurement TF Trust Fund TOR Terms of reference TTL Task team leader UCEQA Ukrainian Center for Education Quality Assessment UNICEF United Nations Children's Fund US United States VC Video conference TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME ...................................................................................................................... 8 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 11 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 12 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 13 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 14 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 20 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 22 ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 23 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Strengthening Evidence-Based Policymaking with P161312 Education Statistics and Analysis Country Financing Instrument Ukraine Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Ministry of Education and Science, Institute of Ukraine Educational Analytics Project Development Objective (PDO) Original PDO The development objective of this project is to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data and analytical methods to inform policymaking. Page 1 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A4096 340,000 340,000 336,624 Total 340,000 340,000 336,624 Other Financing Borrower/Recipient 60,000 0 0 Total 60,000 0 0 Total Project Cost 400,000 340,000 336,624 KEY DATES Approval Effectiveness Original Closing Actual Closing 16-Dec-2016 27-Jan-2017 24-Jan-2019 24-Jul-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 21-Jan-2019 0.17 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial Page 2 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 20-Apr-2017 Satisfactory Satisfactory 0.00 02 31-Jul-2018 Moderately Satisfactory Moderately Satisfactory 0.11 03 21-Aug-2019 Moderately Satisfactory Moderately Satisfactory 0.28 04 01-Dec-2019 Moderately Satisfactory Moderately Satisfactory 0.34 ADM STAFF Role At Approval At ICR Regional Vice President: Cyril E Muller Cyril E Muller Country Director: Satu Kristiina J. Kahkonen Satu Kristiina J. Kahkonen Director: Amit Dar Fadia M. Saadah Practice Manager: Mario Cristian Aedo Inostroza Harry Anthony Patrinos Task Team Leader(s): Igor Kheyfets Lucia Casap ICR Contributing Author: James Gresham Page 3 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Country context. Ukraine is a conflict-affected lower middle income country located in Eastern Europe. In 2014 it had a population of 45 million and a GNI per capita of US$ 3,560 (Atlas method, current US$) —a decline from US$ 3,760 in 2013 and well below the Europe and Central Asia (ECA) regional average of US$ 6,892. Despite moderate successes in reducing absolute poverty, average real incomes have stagnated and population has declined by 13 percent since 1991. A series of economic crises have been exacerbated by demographic decline and unstable political environment. More recently, a simmering conflict in the country’s east and loss of control over part of its territory contributed to a sharp contraction in GDP (by approximately 12 percent in 2015), as well as spiking inflation and sharp devaluation of the currency. 2. Political turmoil precipitated by the “Euromaidan� revolution in 2014 has led to a reorientation of Ukraine towards European integration and a rhetoric supportive of wide reaching social and economic reforms. A reform-minded government has initiated long overdue modernization of a number of sectors with help from the international community. Yet progress on reforms has been uneven and impeded by continued political and macroeconomic instability. Nevertheless, the window of opportunity for reform is currently open, as policymakers continue to express willingness to tackle tough challenges faced by their country. 3. Sector context. Ukraine’s education sector has long been in need of profound modernization and structural reform. The main challenges include (i) optimization of the school network and efficient use of budget resources, (ii) quality of education and its assessment, and (iii) management and transparency of the sector at all levels using modern evidence-based techniques. With respect to efficiency, the sizable demographic declines that Ukraine has experienced over the past two decades have led to a reduction in the number of children of school age, particularly in rural areas. Yet no commensurate reduction in the number of education institutions has taken place, leading to reduced school and class sizes and increasing unit costs of education provision. With regard to quality, Ukraine had previously not participated in internationally comparable systems of learning assessments, making it difficult to accurately evaluate the quality of education it provides. Despite high rates of secondary school completion and university enrollment, studies show large gaps between the skills possessed by graduates and those demanded by the labor market. Lastly, sector management still relies largely on outdated Soviet-era methods of prescriptive norms and lack of autonomy and accountability. These challenges were detailed in the recent Education Flagship Report prepared by the World Bank1. 4. The latter of these challenges is exacerbated by a lack of tradition of evidence-based policymaking, as well as weak statistical and analytical capacity at all levels of the education system. However, ambitious reforms have been initiated since 2014 to increase autonomy and accountability of local actors. Several pieces of legislation have been introduced to bring Ukraine’s education in line with international standards, reform the financing mechanisms within the sector, and decentralize decision making. The government also committed to Ukraine’s first ever participation in the Programme for International Student Assessment (PISA), a triennial measurement of learning among 15-year-old students conducted by the Organisation of Economic Cooperation and Development (OECD) in 2018. New bodies have been set up to enhance the analytical capacity of the education sector—including the Institute of Educational Analytics (IEA) under the Ministry of Education and Science (MoES)—and modernize the statistical practices to inform evidence- based policymaking. The task of modernizing the national information system in Ukraine aims to meet the demand 1 World Bank, 2019, Ukraine - Resume Flagship Report: Overview Page 4 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) from various governing bodies to improve the collection, processing, and analysis of statistical data in order to obtain a complete picture the functioning of the education sector and enhance its governance and management. Project Development Objectives (PDOs) 5. The PDO is to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data and analytical methods to inform policymaking. 6. The Project was supported through a small recipient-executed grant in the amount of $US 0.34 million financed out of the Multidonor Trust Fund for Statistical Capacity Development III (TF-A4096-001). It was approved on December 21, 2016 and became effective on January 27, 2017. The Project was planned initially to be implemented over 24 months, with the closing date on January 24, 2019. It was later extended with six months, having the new closing date on July 24, 2019. 7. The project recipient and implementing agency was the newly established State Scientific Institution "Institute of Educational Analytics" (IEA). The IEA is legally subordinated to the MoES and is responsible according to its charter for development and implementation of a system of education statistics and analysis in Ukraine through research, analytical, and innovative methods. Key Expected Outcomes and Outcome Indicators 8. The Project included the following key PDO-level indicators to measure progress towards the PDO: (a) Upgraded Education Management Information System (EMIS) is fully operational; (b) Open data portal is functional and allows stakeholders to access relevant EMIS data; (c) Qualified specialists are trained to prepare the national analytical report using PISA 2018 data. 9. Measurement of progress towards the PDO indicators was supported by the following six intermediate result indicators (IRIs): i. Percentage of general secondary education schools reporting in EMIS the latest year's data on students enrolled; ii. Percentage of general secondary education schools reporting in EMIS the latest year's data on teachers employed; iii. Percentage of preschool institutions reporting in EMIS the latest year's data on children enrolled; iv. Number of stakeholders trained in the use of the EMIS or the EMIS open data portal; v. Sampling of schools for the implementation of PISA 2018 is conducted using EMIS data; vi. PISA 2018 assessments are carried out in Ukraine. Components 10. The Project was designed around three components. Component 1: Development of an integrated EMIS 11. The objective of this component was to support the development of a modern EMIS that contributes to evidence- based policymaking based on modern education statistics. This component had four sub-components: 12. Sub-component 1.1. Integration and quality assurance of existing databases. This subcomponent was to support the modernization and upgrading of an existing EMIS that was recently acquired by the Ministry of Education and Science. Previously operated by a private company, the EMIS required significant quality assurance and integration Page 5 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) with existing data systems. Specifically, this required: (i) modernization of the existing data systems; (ii) support from programmers and database developers; (iii) consulting from international and domestic EMIS experts; (iv) knowledge exchange with other countries that had recently established a modern EMIS of their own; and (v) provision of hardware and software necessary to maintain and operate the EMIS. 13. Sub-component 1.2. Database security certification. This sub-component was designed to finance the necessary database security measures, as required by national legislation. Specifically, it was to include: (i) development of a Terms of Reference for the services of a firm engaged in database security certification (including assessment of security risks, development of a data protection policy, testing of the security system, and its implementation); and (ii) contracting of such firm to provide the necessary database security services. 14. Sub-component 1.3. Development of an open data portal. This sub-component had to support the construction of an open data portal that would make timely and relevant EMIS data available to all stakeholders in the education sector. Specifically, this included: (i) contracting programmers and data visualization specialists to develop a user- friendly portal linked to the EMIS; (ii) acquisition of statistical, database management, mapping, and data visualization software necessary to run the portal; and (iii) communications and outreach services necessary to inform the various stakeholders about the portal and its uses. 15. Sub-component 1.4. Training and analytical capacity building. This sub-component was to finance a range of capacity building activities to encourage effective use of education statistics, including training on the use of the EMIS and the open data portal developed under subcomponents 1.1 and 1.3. Specifically, this included: (i) training of the Institute of Educational Analytics staff in proper database maintenance and EMIS operation; (ii) training for representatives of local education authorities in proper data entry and submission techniques; (iii) training of other stakeholders and decision makers (including the MoES, local authorities, and others) in the effective use of education statistics for evidence based policymaking; (iv) instruction of key stakeholders in statistical and analytical methods that take advantage of data available in the EMIS. Component 2: Prepare for participation in the PISA 16. The objective of this component was to strengthen the capacity of Ukrainian authorities to participate in PISA 2018 and analyze the subsequent assessment results. This component was divided into two sub-components: 17. Sub-component 2.1. Preparation for PISA implementation. This sub-component was meant to finance selected activities to be undertaken by the Institute of Educational Analytics, the Ukrainian Center for Education Quality Assessment (UCEQA), and other agencies in preparation for PISA 2018 testing. Specifically supposed to include: (i) data collection and processing for the preparation of the PISA 2018 sample; (ii) preparation of the sample of schools to be included in PISA 2018 piloting and implementation; (iii) knowledge exchange and consulting support with local and international experts on topics relevant to PISA implementation; and (iv) informational activities for teachers and other education system stakeholders regarding international large-scale sample based student assessments. 18. Sub-component 2.2. Capacity building for the preparation of the PISA 2018 national report. This sub-component was to finance the training of Ukrainian experts in the accepted analytical techniques used in the preparation of PISA national reports based on international best practices. Specifically, this included: (i) training of the staff of Institute of Educational Analytics and other relevant agencies in statistical techniques used for analyzing PISA data; (ii) knowledge exchange with international organizations and other national agencies experienced in preparation of similar reports; (iii) consulting support from local and international experts on relevant topics. Page 6 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) Component 3: Institutional capacity building. 19. The objective of this component was to build institutional capacity of the IEA and ensure adequate support for Grant implementation. This component was divided into three sub-components: 20. Sub-component 3.1. Institutional capacity building. This sub-component was designed to finance the strengthening of institutional capacity of the Institute of Educational Analytics to become the leading education statistics and analysis organization in Ukraine. Specifically, this included: (i) the development of the Institute’s website; (ii) training of Institute staff in modern research methods, analytical report preparation, use of statistical software, and English language proficiency; (iii) knowledge exchange activities with domestic and international experts on topics relevant to the Institute’s mission; and (iv) provision of a small amount of furniture, hardware, and software. Training of IEA staff on sociological research methods was expected to be provided by the Recipient as part of the in-kind contribution toward the implementation of this Grant. 21. Sub-component 3.2. Grant implementation support. This sub-component was to finance the necessary support for the Institute of Educational Analytics to adequately implement the Grant in line with World Bank rules and procedures. Specifically, this included: (i) consulting services in the areas of procurement, financial management, translation, and others; (ii) operating costs; and (iii) other expenditures (e.g., training) required by the Project Management Team to successfully implement the Grant. Despite facing severe resource constraints, the Recipient was to provide an in-kind contribution in the form of covering administrative costs of the implementing agency (including staff time and operating expenses of the IEA). These were expected to amount to approximately US$ 60,000 over the life of the Grant. 22. Sub-component 3.3. Audit. This sub-component was to finance the required audit of Grant implementation activities. Restructuring 23. During the mid-term review of the Project which took place on September 17-21, 2018, several shortcomings were identified which resulted in a project restructuring. In particular, shortcomings included insufficient remaining implementation time, impossibility to access data for achievement of an indicator, and a need for redistribution of project resources across categories. Without restructuring, these shortcomings posed a risk to the achievement of the PDO. Thus, on January 16, 2019 the project restructuring was approved, allowing the: i. Extension of the project’s closing date by 6 months, from January 24, 2019 to July 24, 2019 to allow sustainable and reliable finalization of the works related to EMIS upgrade. These included several activities related to: (i) Database security certification; (ii) training of upgraded EMIS users meant to ensure system security and sustainability that could be performed only after system development; (iii) and creation of the Open Data Portal. ii. Renaming of the initial PDO indicator related to PISA, namely to change the formulation of the indicator “Qualified specialists are trained to prepare the national analytical report using PISA 2018 data�, dropping the reference to the year. As Ukraine didn’t participate in previous PISA waves and due to unavailability of PISA-2018 data for Ukraine within the Project timeline, the training of experts was done using PISA data of other countries from previous waves. The new indicator is “Qualified specialists are trained to prepare the national analytical report using PISA data�. This change had a minimal impact on achievement of the PDO, given that PISA data is highly structured and consistent across years and countries, meaning that analysis of data from past years of other countries would still provide qualified specialists with the skills needed to analyze PISA 2018 data for Ukraine once it became available. Page 7 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) iii. Reallocation of financing across the cost categories considering the slight changes in Project activities and significant difference between planned and actual cost for Upgraded EMIS (US$115,000 instead of US$46,000 planned, contract No. TFSCB-CQ-01). Namely, to compensate for this contract difference, the unused amount of US$23,800 was reallocated from the “Training� category to the “Goods and Consulting Services� category. Below is reflected the initial and restructured distribution of resources per categories: Table 1. Reallocation of financing across the cost categories at restructure Amount of the Grant Requested Cost funding Cost category Allocated (expressed in Amount (USD) percentage USD) (a) Goods and consulting services 273,500 297,300 100% (b) Training 58,500 34,700 100% (c) Operating Cost 8,000 8,000 100% TOTAL 340,000 340,000 II. OUTCOME Assessment of Achievement of Each Objective/Outcome 24. The objective of the project is: to strengthen the capacity of Ukrainian education authorities to utilize (i) administrative, statistical, and learning assessment data and (ii) analytical methods to inform policymaking. As the PDO statement has two parts, these are both assessed separately below. PDO Objective 1 - to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data to inform policymaking 25. The achievement of this objective is assessed moderately satisfactory given that PDO-level indicators were partially achieved with additional progress since project closing date, and all IRI targets were met or exceeded. The main activities supporting this objective were carried out under Components 1 – Development of an integrated Education Management Information System (EMIS). The results of this obective were measured by the following indicators: Indicator 1: Upgraded Education Management Information System (EMIS) is fully operational 26. The target for this indicator was partially achieved by the closing date, and has since been achieved. The baseline for it was an existing operational EMIS with incomplete coverage of general secondary education (GSE) institutions, and the target was to develop an upgraded EMIS that would cover all GSE schools and preschool institutions in at least 3 regions (oblasts). 27. At closing, the system successfully was updated and imported information on two previous years from DISO (State Education Information System – former EMIS) and was covering all general secondary education (GSE) schools (including vocational and higher education establishments where GSE could be completed) and all preschool institutions from all regions (oblasts). Also, the system was renamed from DISO to AIСEM – Automated Information Complex on Educational Management, so to avoid confusion, being accesible on the following link https://aikom.iea.gov.ua. The information collected and analyed in the new system is now being used for decision making processes, including in the calculation of the education funding formula which determines subsidies Page 8 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) (subventions) to be transferred from the State Budget to local authorities. Thus, this is a crucially important part of education system, given its decentralization and autonomous management process and the importance of an accurate funding formula for structuring the right financial incentives for school network consolidation and efficiency improvements. 28. Yet, due to dissonance in sequencing, as data shall be collected during the schooling year, avoiding other national statistics data collection processes, MoES and IEA decided to collect this year the data on pupils (ZNZ-1 form) in the DISO system and import it later in the Automated Information Complex on Educational Management (AICEM). To ensure AICEM operability and test system functionalities, during July 2019 was collected information on English textbooks use in schools. The results of this exercise showed some system bottlenecks and Consultant worked on their fixing. In November 2019 in the AICEM started to be collected the information on teachers (RVK-83 form). 29. All the IRIs used during the Project life to measure the progress of this indicator were met or surpassed their targets: i. IRI-1. Percentage of general secondary education schools reporting in EMIS the latest year's data on students enrolled had the baseline 95.11% and the target 100%. By the closing date the indicator was fully achieved. ii. IRI-2. Percentage of general secondary education schools reporting in EMIS the latest year's data on teachers employed had the baseline 82.31% and the target 100%. By the closing date 100% of the GES were reporting in EMIS the latest year's data on teachers employed. iii. IRI-3. Percentage of preschool institutions reporting in EMIS the latest year's data on children enrolled had the baseline 0.0% and the target 20%. By the closing date, due to the IEA efforts, all 100% preschool institutions were reporting in EMIS, the target being thus overachieved. 30. Also, to ensure the system security, that would allow in the future the EMIS use for datailed at the personal level data collection, if decided, under this component were also prepared the documents for Database security certification. The package was submitted for review and decision to the State Service of Special Communications and Information Protection of Ukraine, waiting their decision by the end of the 2019 year. Indicator 2: Open data portal is functional and allows stakeholders to access relevant EMIS data. 31. This indicator was partially achieved by the Project closing date, but has since been achieved. An Open Data Portal (ODP) was created for querying and generating custom reports in various formats (Microsoft Word, Excel, PDF) based on AICEM system, and it is available on http://opendata.iea.gov.ua/. At the closing date, the ODP was populated with data and in October 2019, after the completion of information uploading, the Portal was officially launched. 32. The IRI used during the Project life to measurement the progress of this indicator met its target. Thus, iv. IRI-4. The number of stakeholders trained in the use of the EMIS or the EMIS open data portal was at the Project closing - 204 persons, overpassing the initial target of 200 potential users. To assist EMIS users, were created user's video-tutorials and user's manual (in PDF format).The materials used for the training on EMIS and ODP content management and use are available on https://aikom.iea.gov.ua/help). All materials were adjusted in accordance with the recommendations received during the trainings. 33. In order to disseminate the information to stakeholders on the EMIS upgrade and OPD availability, in July 2019, a consultant was hired who produced: (i)a video on the EMIS update process that was promoted on the following Facebook page https://m.facebook.com/story.php?story_fbid=426285654889246&id =423292725188539; and (ii) a brochure presented at the following link within the MoES article on EMIS update and renaming Page 9 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) https://mon.gov.ua/ua/news/vilnij-dostup-do-osvitnoyi-informaciyi-ta-analiz-efektivnosti-roboti-zakladiv-institut- osvitnoyi-analitiki-mon-prezentuvav-novu-onlajnsistemu. Currently, the ODP has daily about 300 users and many more are expected after additional information that is to uploaded in December 2019. PDO Objective 2 - to strengthen the capacity of Ukrainian education authorities to utilize analytical methods to inform policymaking. 34. The achivement of this objective is assessed satisfactory with development indicator and all intermediate results fully achieved by the closing date. The main activities supporting this objective were carried out under Components 2 - Prepare for participation in the PISA. The results of this obective were measured by the following indicators: Indicator 3: Qualified specialists are trained to prepare the national analytical report using PISA data. 35. The indicator achieved its objective with 20 specialists trained to prepare the national analytical report using PISA data. On December 17-21, 2018, 20 IEA, UCEQA, as well as other stakeholders’ specialists participated in the training on PISA data analysis. The Consultant hired for this task (Daniel Caro) continued to provide assistance to Ukraine PISA team even after the contract finalization. 36. The IRI under this indicator also achieved their target. Thus: v. IRI-5. The, sampling of schools for the implementation of PISA 2018 was conducted using EMIS data. Also, in preparation for PISA 2018, from June 26 - 30, 2017, 3 IEA specialists and 1 representative of UCEQA participated to the 7th International Research Conference in Prague, Czech Republic. vi. IRI-6. PISA 2018 assessments were carried out in Ukraine and the results for Ukraine came out on December 3, 2019 in line with other countries. 37. In addition, out of the resources available from savings, the IEA staff received IT trainings on: Excel 2016 (IV): Power Pivot + Power View + Power Query + Power Map for 5 people; M20761 Querying Data with Transact-SQL for 2 people, M20762 Developing SQL Databases for 3 people, M20744 Securing Windows Server 2016 for 2 people. 38. The activities under Component 3 were to support the achievement of the above objectives from the perspective of institutional capacity building. Thus, during the Project, the IEA hired financial management (FM), procurement and translator Consultants, who provided their support in Project management by the closing date. To strenghten this specific competences, during the Project the IEA staff and the consultants benefited from the trainings in the are of FM, procurement and English courses. Also, under this component, a hired consultant developed the official IEA web-page that can be found at the following link: http://iea.gov.ua/ and the institution benefited from procured furniture and IT computer equipment that allows it to realize its functions and operate with big landscape of data, including from EMIS. 39. The Project financial audits funded from the third component were issued with a non-qualified opinion and it were published on the IEA’s website. The full description of the objectives, specific objectives, and the indicators and intermediary result indicators that contributed to reaching the objectives is presented in annex 1. Page 10 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) Overall Outcome Rating 40. The overall outcome rating of the project is assessed Moderate Satisfactory as of the project closing date, July 24, 2019. i. The objective to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data to inform policymaking was partially achieved as by its measured indicators which were partially met by the clossing date. Thus, PDO indicators 1 and 2 were partially achieved by the closing date, and the intermediate results indicators were fully achieved, one being overachieved. ii. The objective to strengthen the capacity of Ukrainian education authorities to utilize analytical methods to inform policymaking fully met as measured by Indicators 3 which was fully achieved reaching the target. All the IRIs under this objective achieved the targets. Other Outcomes and Impacts 41. As a result of the discussions with the Government during the grant implementation, currently, after ten years of hiatus, the Bank is preparing a new lending operation in the education sector in Ukraine. The project tentatively would have a component that would focus on expanding the current achievements of the grant in supporting EMIS. Specifically, it is expected to detail EMIS data from aggregate to individual level and expand the system coverage from pre-school to higher education level. This activity would build on the design of the EMIS supported under this Project. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 42. The project was running in a context of instability, followed by institutional leadership changes. Despite the volatile political environment further complicated by the change in the implementing agency (IA) team within the IEA and the election period in 2019, the IEA was very committed to implement the Project. Also, to ensure Project continuity, a larger group of stakeholders was engaged during the implementation. Thus, during the missions, meetings were held with both IEA and MoES teams. This helped to ensure that project implementation proceeded as planned in spite of changes in leadership and staffing. 43. The major challenge and delays were related to the lack of previous experience of the IA in managing Bank projects, as well as developing informational systems. The lack of experience of IEA to manage Bank project was addressed by close cooperation between the Bank team and IEA and frequent missions or video conferences (VC). Also, IEA capacity to implement a Bank-supported Project was enhanced by hiring procurement and FM consultants. To address the delays on informational system development, the Project closing date was extended with 6 months, from January 24, 2019 to July 24, 2019, with the procurement and FM consultants being kept during this period as well. 44. Another constraint to project implementation was insufficient cost estimation at the Project design stage. Thus, underestimation of the cost for the EMIS upgrade, represented another challenge during Project implementation. In this context, the timely reaction of the teams and support offered by the Trust Fund team and Country Management Unit was a good solution. This allowed to restructure the cost by category and extend the closing date with 6 months. Page 11 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) 45. At the Project implementation stage, the introduction of the new Bank procurement system (Systematic Tracking of Exchanges in Procurement - STEP) created occasional challenges which also affected implementation. Since the system was a new instrument, the lack of experience in its use from both sides, the Bank and IA, as well as some occasional glitches were bringing delays in timely approval of the terms of reference (TORs) or procurement plan changes. This was addressed by the Bank’s procurement team which delivered trainings on system use, as well as their close and continuous support to the Bank task team and the IA team. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME 46. Bank performance was satisfactory at entry. The Bank, IEA and MoES was closely collaborating from the beginning of the Project, prioritizing and designing it in support to the initiated reforms in educational sector. There were several moderate shortcomings, mostly related to the institutional leadership changes in the IEA. The continuous engagement and close collaboration of the Bank with IEA and MoES teams, as well as other development partners of Ukraine, ensured the Project smooth implementation. 47. Bank performance was satisfactory at supervision. There were regular supervision missions from the Bank sides, as well as frequent VC, quick turnaround for emails, or phone call discussions. The IEA counterparts repeatedly expressed their gratitude to the Bank for the prompt reaction to the requests, as well as for continuous support in implementation. Close and intense collaboration between the Bank, IEA and MoES teams contributed to the successful Project completion, despite delays for some indicator’s achievement. 48. Bank performance was satisfactory at completion. Over the two and a half years of implementation, the Project had three task team leaders (TTLs). Despite the change of task team leaders, it is worth mentioning that their collaboration, experience, and knowledge sharing even after the handover, plus Trust Fund (TF) accreditation, contributed to the smooth oversight of project implementation and its successful completion. 49. The project has been implemented with one extension of the closing date and spent 99 percent of the funds allocated. The PDO remained unchanged and has been rated Moderate Satisfactory. The risk in achieving sustainable results was addressed by a timely restructure and continuous support. Also, the new Bank engagement in the Ukraine educational sector would expand the achieved Project results. 50. There were no significant Procurement or Financial Management compliance issues. The Project have had an established simplified Procurement Plan. The flexible approach towards the activities included in the Procurement Plan contributed to an easy adjustment to the Project needs. Since the Project was initially planned for two years implementation and after the first year the spending was small, it was agreed a waved audit. However, at restructuring, since the requested extension of the closing date and changes in the disbursement categories, it was requested a Project audit. Thus, there were two audits conducted, both provided on time and unqualified. The final project audit was submitted in November 2019. 51. The supervision budget allocated for the project was estimated at US$178,000 and was exceeded by 40 percent in order to support additional analytical work which allowed a new engagement in the sector. Most of the funds were allocated to staff costs, followed by travels. 52. Risk to development outcome is rated as moderate. The updated EMIS serves as a main instrument for the Ukrainian education authorities to utilize administrative, statistical, and learning assessment data; and the conducted trainings on analytical methods are an important step in consolidating the authority’s capacity to inform policymaking. Yet, considering other countries experience, and developments at the international level, for a performant EMIS and Page 12 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) good data, the system needs a continuous update and a detailed at the personal level data is a must for targeted policy. Also, since human turnover in the institutions is a natural phenomenon, there is a need for continuous capacity building activities, including from the perspective of analytical method use. Thus, these efforts need to be continuously supported. V. LESSONS LEARNED AND RECOMMENDATIONS 53. During the two and a half years of Project implementation, significant experience was acquired and lessons for future engagements were learned which can be summarized in three main categories—general, national, and project levels. 54. At the general level, a key lesson learned is that informational systems upgrade, or development of a new system takes significantly longer time that the two years required by the TF criteria at the design stage for this Project. This is even more relevant in the cases of the countries or institutions that do not have experience in this area, or if the systems developed within the Project needs to follow different procedures, and there is no previous experience to apply them in the local context. Thus, a standard two and a half, or three years implementation time may be more realistic for such type of projects. 55. At the national level, a broader engagement, within the institution, as well as with other institutions and development partners were critical for implementation of the Project. Regular meetings during the Project implementation, beside those with the IEA, were critical for successful implementation. For example, the Project team met regularly with representatives of MoES, UCEQA, and Ukraine development partners, such as Swedish International Development Agency (SIDA), United Nations Children's Fund (UNICEF), Finland representatives and other donor partners contributed to smooth implementation of the project, despite the challenges. These other stakeholders and donor partners helped to facilitate implementation by reinforcing key design principles related to the new EMIS and to the management and implementation of PISA, for example. 56. It is important to note that in context of limited capacity and experience with Bank Project, frequent and regular communication, country presence of staff, especially for FM function, are critical to expedite the Project implementation and build capacity going forward. 57. At the Project level, a specific recommendation for the further work on EMIS is that having access to individual data and an integrated systems would allow the IEA and MoES to design better targeted policies and avoid data entry redundancy and administrative burden. Yet, this require, beside additional work on the system, amendments to the current laws, better equipped schools with computers, improved internet access, as well as IT trainings for those responsible for data input. IEA informed the Bank that the draft laws were developed and submitted to MoES for consideration. On the other aspects, there is a need for stakeholder’s discussions and decisions. This could be considerate a potential activity within the new Bank Projects. . . Page 13 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Upgraded Education Text Existing EMIS is EMIS is upgraded to EMIS is upgraded to The system is Management Information operational with cover all GSE schools cover all GSE schools developed and covers System (EMIS) is fully incomplete coverage and preschool and preschool all GSE schools operational of general secondary institutions in at least institutions in at least (including vocational education (GSE) 3 regions (oblasts) 3 regions (oblasts) and higher education institutions establishments where GSE could be completed) and all preschool institutions (PI). 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 Comments (achievements against targets): The indicator is partially achieved. The system is developed. Necessary training were conducted. System testing at the national level was performed. In November 2019 in the EMIS started to be collected the information on teachers (RVK-83 form). Page 14 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Open data portal is functional Text No open data portal Open data portal is Open data portal is Portal is created. and allows stakeholders to exists functional and allows functional and allows Processes of testing access relevant EMIS data stakeholders to access stakeholders to access and filling information relevant EMIS data relevant EMIS data are ongoing. 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 Comments (achievements against targets): The indicator is partially achieved. The Open data portal was developed and was lunched on October 18, 2019. Objective/Outcome: To strengthen the capacity of Ukrainian education authorities to utilize analytical methods to inform policymaking. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Qualified specialists are trained Number 0.00 20.00 20.00 20.00 to prepare the national analytical report using PISA 01-Nov-2016 24-Jan-2019 24-Jun-2019 24-Jul-2019 data Comments (achievements against targets): The indicator is achieved. Page 15 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) A.2 Intermediate Results Indicators Component: Components 1 – Development of an integrated Education Management Information System (EMIS) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of general Percentage 95.11 100.00 100.00 100.00 secondary education schools reporting in EMIS the latest 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 year's data on students enrolled Comments (achievements against targets): The indicator is achieved. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of general Percentage 82.31 100.00 100.00 100.00 secondary education schools reporting in EMIS the latest 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 year's data on teachers employed Comments (achievements against targets): The indicator is achieved. Page 16 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of preschool Percentage 0.00 20.00 20.00 100.00 institutions reporting in EMIS the latest year's data on 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 children enrolled Comments (achievements against targets): The indicator is overachieved. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of stakeholders Number 0.00 200.00 200.00 204.00 trained in the use of the EMIS or the EMIS open data portal 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 Comments (achievements against targets): The indicator is achieved. Component: Components 2 - Prepare for participation in the Programme for International Student Assessment (PISA) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Sampling of schools for the Text No sample of schools Sampling of schools Sampling of schools Sampling of schools Page 17 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) implementation of PISA 2018 is exists for PISA 2018 for the for the for the conducted using EMIS data implementation implementation of implementation of implementation of PISA 2018 is PISA 2018 is PISA 2018 is conducted using EMIS conducted using EMIS conducted using EMIS data data data 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 Comments (achievements against targets): The indicator is achieved. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion PISA 2018 assessments are Text No PISA assessments PISA 2018 PISA 2018 PISA 2018 carried out in Ukraine have been carried out assessments are assessments are assessments are in Ukraine to date carried out in Ukraine carried out in Ukraine carried out in Ukraine 01-Nov-2016 24-Jan-2019 24-Jul-2019 24-Jul-2019 Comments (achievements against targets): The indicator is achieved. Component: Component 3 - Institutional capacity building Page 18 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 - to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data 1. Upgraded Education Management Information System (EMIS) is fully operational (Indicator One in the Project Paper) Outcome Indicators 2. Open data portal is functional and allows stakeholders to access relevant EMIS data (Indicator Two in the Project Paper) 1. Percentage of general secondary education schools reporting in EMIS the latest year's data on students enrolled Intermediate Results 2. Percentage of general secondary education schools reporting in EMIS the latest year's data on teachers employed Indicators 3. Percentage of preschool institutions reporting in EMIS the latest year's data on children enrolled 4. Number of stakeholders trained in the use of the EMIS or the EMIS open data portal 1. An updated and tested EMIS with imported from the former system information on previous two years that covers all GSE schools (including vocational and higher education establishments where GSE could be completed) and all PI from all regions (oblasts). Data collection on teachers started in November 2019. Key Outputs by Component 2. A Database security certification that would allow in the future the EMIS use for datailed at the personal level data (linked to the achievement of collection, if decided, initiated. the Objective/Outcome 2) 3. An Open Data Portal (ODP) was created for querying and generating custom reports in various formats (Microsoft Word, Excel, PDF) based on updated EMIS. At the closing date was poluated with data and in October 2019, after the completion of information uploading, the Portal was officially launched. 4. 204 stakeholders were trained in the use of the EMIS or the EMIS open data portal. Objective/Outcome 2 - to strengthen the capacity of Ukrainian education authorities to utilize analytical methods to inform policymaking 1. Qualified specialists are trained to prepare the national analytical report using PISA data (Indicator Three in the Outcome Indicators Project Paper) Intermediate Results 1. Sampling of schools for the implementation of PISA 2018 is conducted using EMIS data Indicators 2. PISA 2018 assessments are carried out in Ukraine Key Outputs by Component 3. 20 specialists were trained to prepare the national analytical report using PISA data. 4. First PISA report with PISA 2018 data, collected on a sample created using EMIS data came out on December 3, 2019. (linked to the achievement of the Objective/Outcome 1) Page 19 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Components (US$M) Closing (US$M) Approval % Development of an integrated Education .17 .19 112 Management Information System (EMIS) Preparation for participation in the Programme for .07 .03 43 International Student Assessment (PISA) Institutional capacity building .10 .12 120 Total 0.34 0.34 100 58. Objecive 1 of the Project to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data to inform policymaking was achieved with funds allocated from Components 1, 63.3 percent of the expenses. Around 8 percent of the funds were allocated for the Component 2 which supported the achievement of the objective 2 to strengthen the capacity of Ukrainian education authorities to utilize administrative, statistical, and learning assessment data to inform policymaking. At the same time, 35.3 percent of the Project were allocated for the Component 3 - capacity building activities that contributed to achievement of both objectives. 59. As a category of expenses, consulting services is the one that leads the ranking with 83 percent, as shown in table 2.1. Table 2.1. Expenditure by Category of Costs Type of Costs Allocation Expenses Percentage Percentage spent of Spent of the Allocation Total (a) Goods and consulting 297,300 295,713 99.5 services 87.8 (b) Training 34,700 32,981 95.1 9.8 (c) Operating Cost 8,000 7,930 99.2 2.4 Total 340,000 336,624 99.0 99.0 60. The disbursement graph (figure 2.1) illustrates that IEA disbursement. As it can be seen, higher disbursement was registered by the end of the Project. Page 20 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) Figure 2.1. Project Disbursement Graph Source: Operations Portal 61. The selected auditor, performed the auditing services and provided a clean unqualified opinion covering all grant expenses. Page 21 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 62. There are no comments provided. An extract from the Recipient completion report is attached in annex 4. Page 22 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) Extract from Recipient Implementation Completion Report I. PROJECT BACKGROUND 1. Ukraine obtained financial assistance from the International Bank for Reconstruction and Development ("World Bank"), acting as the holder of the grant funds of different "Donors" within the framework of the Target Fund for Statistical Capacity Building III (TFSCB-III:TF070561) in the amount of USD 340,000.00 ("Grant") for the Strengthening Evidence-Based Policymaking with Education Statistics and Analysis Project ("Project"). Project general information: Country: Ukraine Project: Strengthening Evidence-Based Policymaking with Education Statistics and Analysis Project Grant No. TF0A4096 Project No. P161312 Total Amount: 340,000 USD Duration: 2.5 years (January 24, 2017 – July 24,2019) Project Extension: Project was extended once by 6 months in comparison with the initial Closing date Implementing agency: State Scientific Institution “Institute of Educational Analytics (IEA) Total number of main contracts: 29 Disbursed: USD 336,560.93 Undisbursed: USD 3,439.07 Categories: (1) Goods and consulting services – USD 297,300 (2) Training - USD 34,700 (3) Operating Costs - USD 8,000 II. PROJECT IMPLEMENTATION BY COMPONENTS 2. The project was implemented by the State Scientific Institution “Institute of Educational Analytics (IEA) under the overall supervision of the Ministry of Education and Science of Ukraine ("MoES"). 3. Implementation of the Grant was governed by the documents of the TFSCB Grant No. TF0A4096 and by the Resolution of the Cabinet of Ministers of Ukraine No. 153 "On Creation of a Unified System for Page 23 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) the Engagement, Use and Monitoring of International Technical Assistance" dated February 15, 2002, and other legal acts of the Cabinet of Ministers of Ukraine, IEA and MoES orders. 4. Beginning from the start of the Project the IEA signed 29 main contracts (excluding Operating Costs) in the total amount of USD 328.6 thousand. The amount of used funds within operating costs is USD 7.9 thousand. Component 1: Development of the integrated EMIS system 5. In order to achieve results under mentioned component 8 contracts were signed and successfully completed (1 CQ, 3 IC, 1 Shopping and 3 training contracts). 6. The national consultant for development of TOR for EMIS was selected and relevant contract was signed on October 27, 2017. For the compliance of TOR developed by national consultant with the best international practices the IEA also hired the international expert on April 24, 2018. 7. The main contract for development of EMIS and Open Data Portal (ODP) was signed on September 11, 2018 and completed in the end of June 2019. Initially the Project Concept envisaged two separate contracts for these activities but later it was reviewed and the IEA made decision to combine them and add scope of services for ODP development to the contract for the EMIS development in order to optimize time and avoid misunderstanding between contractors in case of signing separate contracts. 8. After EMIS and ODP were developed the IEA conducted training for 204 potential users and signed contract for Database security certification. The relevant contract was completed in July 2019. 9. For better functioning and operating EMIS the IEA also procured the high-performance server on February 2019. 10. During Project implementation the specialists of the IEA and the MoES visited Moldova (April 2-6, 2018) and Lithuania (March 12-17, 2018) for knowledge exchange on EMIS development. 11. The EMIS was developed on June 2019 by the Joint Venture of CIVITTA UAB ( Lithuania), MEDIAPARK UAB (Lithuania) and CIVITTA UKRAINE LLC (Ukraine) with the sub-consultant Centre for Information Technologies in Education (Lithuania). 12. After the System was developed the Ministry of education and science of Ukraine and SSI “Institute of Educational Analytics� decided to rename it and give the new title as AIСEM (Automated Information Complex Educational Management) - https://aikom.iea.gov.ua. 13. AICEM is an online system for collecting, storing, managing and using data in the decision-making process. AIСEM allows to improve resources and processes management in the education system to achieve greater system efficiency and better learning outcomes. 14. It generates and monitors statistics within the education system and has a multifaceted structure, in particular technological and institutional mechanisms for data collection, processing and dissemination. AIСEM plays a key role in tracking changes, ensuring data quality, timely provision of information, and facilitating the use of information in decision-making. The benefits of the new system are: • Parents. Free access to extensive educational information that allows to choose the best from schools to teach their child; Page 24 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) • Schools. Ability to analyze performance compared to previous periods and other schools, simplifying compilation and providing statistical reporting; • Managers in the field of education. Prompt and automated collection of any data in a convenient interface; • Society. Transparency and efficiency in the use of taxpayers' funds for education. Pilot testing of the AICEM 15. Pilot testing was started on June 2019. As a result of testing of the updated EMIS IEA collected information from about 10 thousand schools (about 65% from overall quantity). The level of school responding was lower due to the summer vacations period. IEA analyzed collected data using BI in the new system, for example (http://212.26.146.64:3000/public/dashboard/78e0f20f-0fca-4881-8cad- 3998a0f02efa) there are about 8700 schools who filled form #1 of monitoring (there were 2 different forms) and IEA could analyze what English textbooks are most and least popular. 16. Meanwhile, functionally system was successfully tested. Some minor adjustments were scheduled under development provided by the CIVITTA. Additionally, IEA listed all suggestions in letters to CIVITTA. Key features of the AICЕM: • Data collection, storage and management. Enrollment of over 30,000 schools and kindergartens, ability to connect from any computer through a web browser and without installing special software; • Form and report builder. Availability that allows to create, assign, process and submit information in any form without the involvement of the developer; • Business Intelligence (BI). Analytical module of work with statistical reporting; • Open Data Portal. For querying and generating custom reports in various formats (Microsoft Word, Excel, PDF) - http://opendata.iea.gov.ua/; • Indicators for assessing the state of education and comparing schools with specific indicators. Page 25 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) The differences between old System and AICEM: Old system AIСEM data that is currently available new additional data that will become available Functional Client program Web Solutions (Portal) Impossibility of forms Availability of form and report modification without builder involvement of the developer Availability of the BI analytics Absence of analytical module module and the open data and open data portal Audit data - Interaction with the databases of the State Service for Education Quality will allow to upload in the system information about the results of the regular school audit Data on students and teaching School level aggregated data The system is ready to deploy staff an individual data module, which provides the ability to automatically generate reports on the contingent of students and teaching staff Children performance data - Interaction with the databases of the Ukrainian Center for Educational Quality Assessment will allow to calculate indicators of school students' performance based on the results of national and international educational attestations, assessments Financial data - Interaction with the databases of the Ministry of Finance of Ukraine will allow to calculate expenditures per student, etc. New possibilities of using AICEM data: • Assessments: o Performance of schools; o Student performance; o Growth reports; o Diagnostic reports; o Effectiveness of teaching staff. Page 26 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) • Public administrations: o Public policy decisions; o Accountability; o Planning (based on the school level data). • Schools: o Academic success; o Effectiveness of teaching staff; o Comparison with other schools. • Users: o Access of parents to the education management system; o Access of communities to the education management system. • Government: o Cost effectiveness; o Remuneration of the teaching staff; o Data usage to set quality requirements. Functions of the AICEM • Data collection: o Collection and accounting of educational statistics according to approved forms of official statistical reporting; o Implementation of flexible forms collection due to the availability of the form builder (without the involvement of programmers); o Automation of control processes, cascading user registration. • Data analysis: o Automatic generation of official statistical reporting forms; o Ability to present information from database through user report (report builder); o The analytics module allows to work directly with the database in a user-friendly interface that does not require knowledge of the programmatic query language; o Ability to form random samples, visualize and place them in external resources. • Data verification: o The system has automatic checks for data integrity and validity; o The administrator can return the respondent's report with a comment to any field of input, as needed. • The system is ready for: o Deployment of the personal data module (students, teachers); o Interactions with other databases (education systems, Ministry of Finance, etc.). Page 27 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) • Data entry: o In digital form via a web interface (the system has functionality to pre-fill reports in case of availability of the information for the previous period); o By collecting using third-party software within interagency agreements on data exchange. • Data export: o Generate mandatory statistical reports (electronic and print forms); o Free access to the generated datasets and the ability to create individual searches through the Public Open Data Portal; o Export custom reports in different formats (Microsoft Word, Excel, PDF). Using data in managing school education (new possibilities): Existing practice Best practice (new additional data that will become available) Assessments Overall effectiveness of School performance education Student success Diagnostic reports Variation indicators (number of students per teacher, occupancy of classes, availability of computer equipment, etc.) Public administration Public policy decisions Accountability Planning Planning (allocation of funds and other resources at school level) Leadership Schools - Expanded Demographics Salaries Performance evaluation Certification training Society - Parents' access to AICEM Community access to AICEM Using data to make decisions Using data to set quality requirements Government Performance factors Cost effectiveness Infrastructure capacity Remuneration of the teaching Quality / performance staff metrics Gender equality 17. On the May 2019, AIKOM user trainings were successfully conducted. 204 participants from all regions took part in this event. Participants became familiar with new system, BI module and open data portal. Some feedbacks caused system adjustments by CIVITTA. 18. Bohdan Yarema (UNICEF) attended training and had possibility to ask local responsible how do they organize education enrollment monitoring process, etc. Page 28 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) 19. One of the training deliverables was possibility to discuss step by step user's video-tutorials and user's manual (PDF). All materials were adjusted in accordance with recommendations received and are available at https://aikom.iea.gov.ua/help. 20. In order to disseminate of information to stakeholders on the EMIS and data portal availability and uses promotional video was created - https://photos.app.goo.gl/wwQcptVKbu3EwpAG9. 21. Official information about new system on the MoES website is available at https://mon.gov.ua/ua/news/vilnij-dostup-do-osvitnoyi-informaciyi-ta-analiz-efektivnosti-roboti- zakladiv-institut-osvitnoyi-analitiki-mon-prezentuvav-novu-onlajnsistemu. Component 2: Preparation for Participation in PISA 22. In order to achieve results under mentioned component 2 contracts were signed and successfully completed (1 IC and 1 training contracts). 23. According to the letter from the Ukrainian Center for Education Quality Assessment dated 27.06.2018 No. 02-24-01/1514, the provision of services for preparation in PISA implementation envisaged initially was quite complicated, considering the tight deadlines for the Grant implementation and the possibility to obtain access to the outcomes of the Ukrainian researchers' work only in the last quarter of 2019. Taking into account this conclusion the IEA made decision to carry out the trainings on analysis of PISA data and report preparation only. The IEA carried relevant training during December 17-21, 2018. 20 IEA specialists and stakeholders participated this training. 24. During June 26-30, 2017 3 IEA specialists and 1 UCEQA representative participated 7th IEA International Research Conference in Prague under implementation of this Component. Component 3: Institutional Capacity Building 25. In order to achieve results under mentioned component 19 contracts were signed (1 CQ, 5 IC, 4 Shopping and 9 training contracts). 26. On April 18, 2017 the IEA signed 3 contracts with individual consultants for Project management for the following positions: (1) FM specialist, (2) Procurement specialist and (3) Translator. These consultants provided technical assistance till the end of the Project. 27. On September 21, 2017 the IEA signed and completed 3 contracts for procurement of furniture, computer equipment (10 computers, 3 laptops, 12 printers, 1 projector) and Private Automatic Branch Exchange (PABX). 28. In the end of December 2017 the selected consultant developed official web-site for IEA. Here is the link: http://iea.gov.ua/. 29. On July 2019 the IEA also procured Additional computer and office equipment as following: 6 computers, 2 laptops, 1 printer, 2 air conditioners in the server room, 1 server for IEA Enterprise Resource Planning system at the expense of the savings received during Project implementation. 30. In order to disseminate of information to stakeholders on the EMIS and data portal availability and uses the IEA signed relevant contract. It was completed in July 2019. Consultant produced: •a video on the EMIS update that can be found on https://photos.app.goo.gl/wwQcptVKbu3EwpAG9; and Page 29 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) • a brochure presented on the following link within the MoES article on EMIS updated and renaming https://mon.gov.ua/ua/news/vilnij-dostup-do-osvitnoyi-informaciyi-ta-analiz-efektivnosti-roboti- zakladiv-institut-osvitnoyi-analitiki-mon-prezentuvav-novu-onlajnsistemu. 31. Under this Component the IEA also organized a set of trainings for the IEA and MoES specialists, such as: • project management trainings (one on FM in Minsk in 2018, one on procurement in Kiev in 2018, and another one in Torino in 2019 specifically on Bank rules for procurement of Informational Systems), • English language trainings for 29 people; • and IT trainings on Excel 2016 (IV): Power Pivot + Power View + Power Query + Power Map for 5 people; M20761 Querying Data with Transact-SQL for 2 people, M20762 Developing SQL Databases for 3 people, M20744 Securing Windows Server 2016 for 2 people out of the resources available from savings. 32. According to WB’s FM rules and procedures the IEA also conducted the Project financial audit for 2017-2018. The audit opinion and management letter were sent to the WB in time. The audit report is published on the IEA’s website. For the conduction of the audit for 2019 the IEA signed the amendment to the contract with selected company which audited the Project for 2017-2018. The expected time period for audit for 2019 is September – October 2019. III. ENCOUNTERED ISSUES AND BOTTLENECKS 33. During the Project implementation the IEA faced the following problems: a. Lengthy process of preparation of TOR for integrated EMIS by national consultant. National consultant developed TOR about 6 months. As the result IEA couldn’t launch the selection of company for EMIS development. b. The absence of work with WB procurement application STEP. Due to absence of such experience sometimes the IEA has problems with uploading of relevant information and documents into the System. c. Impossibility to obtain access to the outcomes of the Ukrainian researchers' work within PISA 2018 during Grant implementation period. The IEA couldn’t select the Consultants for support during PISA 2018. IV. LESSONS LEARNED 34. During the project implementation the State Scientific Institution “Institute of Educational Analytics (IEA) as Project implementation unit (PUI) faced some problems and issues which were solved with great efforts and with some time-consuming procedures. In this part of the Implementation Completion and Results Report IEA will provide some information about mentioned facts what might be helpful for other PIUs in similar cases. a. Conflict of the national legislation in Ukraine and the procurement rules in World Bank . There are a lot of regulations, limits, restrictions in Ukrainian legislation which could make procurement within project implementation time-consuming, more complicated or even make them completely impossible. There was some information about prevailing World Bank procedures within the project implementation, but it required additional clarification from the Ukrainian side. It took about a month to receive such clarification from the Ministry of Finance of Ukraine. IEA was Page 30 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) enabled to use World Bank procedures instead of national ones finally and not to use Ukrainian procurement limits for goods. This document (clarification from the Ministry of Finance of Ukraine) was very helpful thing during the project implementation. b. It is not always possible to use the opportunity not to pay taxes in the procurement procedures. At the stage of negotiations, potential suppliers were offered to reduce the price of goods by the amount of VAT, taking into account that the Buyer (IEA) has the mentioned privilege indicated in the registration card No. 3597 dated 30.03.17. As a result of the negotiations, potential suppliers refused to conclude contracts on the such specified conditions for several reasons, including: the availability of the goods in stock (purchased with VAT) and the complexity of tax accounting for preferential transactions. c. Difficulties with the direct payments accounting. From the project accounts of the IEA (the foreign currency account in PJSC “Ukreximbank� and / or from the Treasury account with the funds of the Grant) in the respective years these payments were not made and in the accounting of the IEA there are no related mutual settlements. Meanwhile deliverables obtained as results of the contracts paid for using direct payments are the property of the IEA. The issue was resolved when IEA decided to reflect such payments as the financial values of the contract deliverables taken on the IEA balance sheet. d. The new World Bank procurement system STEP and the change in the composition of the PUI team was the cause of a short delay in the implementation process. Few months was needed for new PUI team to start all grant implementing procedures due to the necessity to become familiar with project strategy and new procurement system STEP. Some bugs in STEP caused situations when PUI was waiting for solving such bugs due to the inability to do some actions. e. Structure of the Project Plan (PP) was required adjustments during project implementation. At the initial version of the PP there was main contract for EMIS modernization and different contract for open data portal development. This split was not logical due to the interrelation of this components. So, IEA combined such contracts and converted the contract for open data portal development into the additional agreement to the main contract. It is very important to do such things while planning PP instead of while project implementation process when it could cause delays, additional procedures, etc. 35. IEA is ready to provide all necessary consultations in case of their importance for other PUIs. V. SUGGESTIONS on further cooperation with the World Bank on the development of educational statistics and analytics in Ukraine 36. In the field of education in Ukraine, a number of reforms are being implemented with the aim of increasing the efficiency of education in the context of the formation of human capital and its competitiveness in the global educational space. 37. The most significant goals of reforming are improving the quality of education at all levels, ensuring equal access of Ukrainian citizens to quality education, more efficient use of budgetary resources. However, the initiated reforms have not yet become irreversible and require significant further efforts to implement the planned educational policy measures. Page 31 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) 38. The success of the implementation is largely determined by the reliability of statistical and management information obtained in the framework of the collection of educational statistics, monitoring of educational processes, and conducting analytical studies on this basis. 39. The World Bank's project “Strengthening Evidence-Based Policymaking with Education Statistics and Analysis�, which was carried out by the State Scientific Institution “Institute of Educational Analytics� during 2016-2019, has significantly improved the Institute’s and Ministry of Education and Science of Ukraine’s capacity to collect statistical information, ensure its reliability, use it at making management decisions in the fields of general secondary education and to a certain degree of pre-school education. Modernized within the framework of the project, Ukrainian EMIS is a modern information educational system with new opportunities. 40. At the same time, the capacity of the Ministry of Education and Science of Ukraine to collect and use reliable statistical educational information remains limited. If depersonalized individual data on students and teaching staff is not available, there might be a risk of incorrect distribution of educational subsidies, wrong distribution of textbooks for schools, the process of the school network optimization could be more complicated, etc. 41. Therefore, the successful implementation of education reform requires the further development of educational statistics on the basis of depersonalized individual data and development of educational analytics on the basis of these statistics. 42. For this purpose, certain legislative preconditions are already being created in Ukraine. In particular, the possibility to use depersonalized individual data in the educational sphere is referred to in the current, recently adopted Law of Ukraine “On Professional Pre-Higher Education�. An electronic census of the population using ID is planned in Ukraine soon. This will give an additional basis to create a database of individual data on students and on the teaching staff. 43. Moreover, implementation of the grant P161312 included the procedure of database security certification in accordance with requirements of the State Service of Special Communications and Information Protection of Ukraine, which enables the new system to proceed personalized data. 44. Therefore, the next stage in the further development of a modernized Ukrainian EMIS should be in the configuration, automatization and adjusting depersonalized individual data modules. The best way to implement depersonalized individual data collection this is a step-by-step introduction: teachers’ data module for the beginning, and later – students’ module. International expert Jan Herczynski notes (SN90 “Collection of individual data on teachers in DISO�) that the Ministry of Education and Science urgently needs useful data on teachers for several reasons: • to develop an effective long-term teacher policy; • for making detailed forecasts of needs for new teachers, by area and by subject; • to analyze real numbers of teaching hours a week; • to assess whether all Ukrainian children receive the same teaching time; • to assess the teachers’ training needs, based on regional concentration and training required; • to create strategies for local education system development. Page 32 of 33 The World Bank Strengthening Evidence-Based Policymaking with Education Statistics and Analysis (P161312) 45. Another important component under the further improvement of the system is integration with the databases of other central authorities in Ukraine. The working group (Ministry of Finance, Ministry of Education, IEA) is currently working on developing ways to integrate the new system with the MoF databases to obtain information on each school's budget. 46. The elimination of these gaps is necessary for further development of information and analytical support for the reforms. 47. The implementation of individual and financial data collection in the new EMIS could provide additional opportunities for the development of educational institutions of various levels, additional indicators of education quality control for state regulators as State Service of Education Quality of Ukraine and National Agency for Higher Education Quality Assurance, etc. Page 33 of 33