RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE HAITI Can Preconditions for Results-Based Financing Be Established in Fragile States? SEPTEMBER 2018 REACH funded the development of a quality assurance system (QAS) to help the Government of Haiti establish the necessary preconditions for the adoption of RBF. The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. Results-based financing (RBF) measure the results. One of the most mechanisms such as conditional important preconditions of RBF is a Functional information cash transfer (CCT) programs, functioning education management systems can be difficult to establish in countries affected school grants, and teacher and information system (EMIS) so by fragility and conflict. school incentives are increasingly that results can be accurately being used around the world to monitored and reported.2 Without improve education results and an EMIS to precisely measure both outcomes and move away from learning outcomes and intermediate the procurement of “more of the indicators of educational quality, same” inputs.1 However, for RBF to it is impossible for education Yet, information systems work, certain preconditions must be ministries to effectively incentivize are important for measuring learning outcomes and met, the most important of which improvements. intermediate indicators of is the need to build the technical educational quality. A crucial question is whether these capacity of countries to define and implement RBF mechanisms and types of preconditions can be to develop monitoring systems that established in countries affected by Based on a note provided by Juan Baron and Melissa Adelman 2 RFB EDUCATION | EVIDENCE fragility, conflict, and violence (FCV) question is very relevant given the the most important dimensions of as a first step towards implementing fragility of institutions and regulatory educational quality in the country. RBF programs in these countries. frameworks in FCV countries as The idea was to include clear Using RBF may be an attractive well as the weaknesses in their indicators for each quality dimension option to consider in some FCV monitoring and governance systems. that would make it possible to countries given that donor and measure education results on the local exhaustion may have set in The Results in Education for All ground. The grant also funded a after decades of investments in Children (REACH) Trust Fund at the series of complementary activities the education sector that have not World Bank provided funding to the aimed at strengthening the technical produced the expected results. Government of Haiti with the goal capacity of MENFP staff to define The flexibility of RBF approaches of establishing the preconditions for and measure quality. By developing also means that they might be the the adoption of RBF in the Haitian a QAS for all primary schools in the financing modality that is best suited National Ministry of Education and country, the grant aimed to improve to the challenging and ever-changing Professional Training (Ministère de governance, enhance the data environment in FCV countries, l’Education National et de la Formation systems needed to measure results, allowing all stakeholders to keep an Professionnelle, MENFP). To this end, and establish the preconditions eye on desired results while adjusting the grant funded the development necessary to introduce an RBF the actions being taken to achieve of a quality assurance system (QAS) mechanism in the education sector those results when necessary. This based on specific standards for in Haiti. CONTEXT Although there have been Haiti considerable increases in enrollment With over 80 percent in recent years, the education sector of primary schools in Haiti still faces many challenges.3 operated privately, there is little oversight The net primary enrollment rate to maintain quality. rose from about 60 percent in the early 2000s to between 70 and 80 percent in 2012.4 Because there are not enough public schools to meet this increased demand, private providers have stepped in to fill the gap and now operate over 80 percent of primary schools.5 However, there is little oversight of these providers nor are they held accountable for the The Government of Haiti has been financing tuition waivers for students quality of their teaching or learning. able to raise enrollment rates by attending private schools through The public sector plays a similarly making increasing access to primary the donor-funded Education for All limited role in both the provision education one of its top funding (Éducation Pour Tous, EPT) Project and regulation of all other levels of priorities. The government has been and its Tuition Waiver Program since education as well. HAITI 3 2007 and through the government- funded Universal, Free, and Obligatory WHY WAS THE INTERVENTION CHOSEN? Education Program (Programme de Scolarisation Universelle, Gratuite, The idea behind the QAS was to Although the MENFP was already et Obligatoire, PSUGO) since 2011. help the MENFP to tackle some putting substantial emphasis on At their peak between 2011 and of the toughest challenges in the improving the quality of education, 2014, these two programs financed sector and to develop a system different actors had different the school fees of over 60 percent that would make it possible to link definitions of quality. It was critical of all primary students in Haiti, financing to results in the future. to establish a consensus on a with PSUGO accounting for about The QAS would be designed to definition of educational quality before 20 percent of domestic public cover both public and private specific learning standards could be spending on education.6 However, primary schools. The approach developed. Since multiple previous these gains in access are at risk as also included complementary attempts to develop accreditation these programs have not yet been activities to strengthen the systems for non-public schools had established as a permanent part of technical capacity of MENFP staff failed, the work under the REACH Haiti’s education system. to develop the QAS—in particular, grant focused on developing a to establish standards for learning definition of quality that would apply Despite these gains in enrollment, conditions and for student to both public and non-public schools. student achievement remains learning, as well as diagnostic very weak. The average Haitian assessments linked to those The final QAS design addressed child enters first grade nearly two standards. several issues, starting with the years late due to a combination of household financial constraints and restrictive school policies that require children to receive at least one year of pre-primary education in order to prepare for first grade. Once in school, 13 percent of first graders repeat the year, and only about half will reach the sixth grade. Student learning is also very weak. For example, a 2016 nationally representative early grade reading assessment found that 42 percent of students near the end of their second-grade year could not read a single word of Haitian Creole.7 Furthermore, a 2015 fourth grade math assessment by the MENFP in public and private schools found that student scores were only slightly better than random guessing. This shows that the learning crisis has also affected the Haitian education system.8 4 RFB EDUCATION | EVIDENCE definition of learning standards. and policymakers with the data that schools, to prioritize resource use, to Standards of learning were defined to they needed to measure progress and increase government accountability, prioritize the most important elements results and to target interventions. and to meet the local population’s of the curriculum and ensure that any most urgent needs. learning assessment that is developed To increase the capacity of the is aligned with the curriculum. The MENFP, the World Bank team in Haiti By developing a QAS for all primary QAS also established the necessary worked with the ministry to identify schools in the country, the REACH tools to measure the school learning and carry out specific tasks that Trust Fund hoped to improve conditions—outputs and intermediate would lead to the development of governance, enhance data systems outcomes—that are needed to improve the QAS. The process of building this to measure results, and establish the learning. By establishing systems capacity was intended to emphasize preconditions necessary to put in to collect data on learning inputs the MENFP’s ownership of each place an RBF mechanism to improve and outcomes, the QAS would be activity and to increase the likelihood the quality of education in Haiti. setting the stage for strengthening that it would sustain the QAS over the monitoring and evaluation of time. It also brought together diverse The QAS aimed to help interventions such as the tuition stakeholders and incentivized them the government improve waiver programs and to increase the to coordinate their work. The QAS accountability of both private and was also intended to empower local governance and enhance data public schools. The establishment of actors to get more involved in the systems by including both the QAS would also provide donors decisions that impact learning in public and private schools. HOW DID THE in schools and student learning outcomes; (b) the creation of of Planning and External Coordination (Direction de la Planification et de INTERVENTION validated measurement instruments for assessing the quality of learning; la Coordination Externe, DPCE). In particular, REACH is supporting a WORK? and (c) a nationally representative pilot test of these instruments that coordinated effort among donors and MENFP directorates to build The QAS was developed by will yield baseline measurements and sustain a functional EMIS. Using the MENFP through a series for the system. The QAS is now the frameworks developed by the World of workshops, meetings, and guiding framework for providing Bank’s Systems Approach for Better continuous technical assistance support to both public and Education Results (SABER) initiative, private primary schools under the REACH funds were used to carry out from the World Bank. The developers IDA-financed Providing an Education a diagnostic to identify strengths took into account relevant of Quality in Haiti (PEQH) Project and and weaknesses in data collection international experiences that is likely to be used by other donors to and information management in could be appropriately adapted to channel financing to schools as well. the Haitian education sector and to the Haitian context and built on support the MENFP’s design for an technical work done by the MENFP An important aspect of the REACH EMIS. A grant from the Trust Fund for and other partners. So far, what grant was to strengthen the capacity Statistical Capacity Building, as well has been accomplished includes: of the MENFP to collect and analyze as IDA funds under the PEQH Project, (a) the development of an officially education data. The REACH funds will be used to implement this design. endorsed framework for educational have been used to reinforce the quality with a holistic perspective statistical and information functions REACH funds have also been that includes conditions for learning executed by the ministry’s Directorate used to strengthen the capacity of HAITI 5 MENFP staff to define standards for An important goal of the QAS These learning standards were learning conditions and to develop was to build Haiti’s capacity to added to the QAS to provide clear assessments to measure conditions measure student learning in milestones for the development of in schools against these standards. primary education. To achieve this, reading skills in the third grade. These These standards define quality the REACH grant also supported standards can also be used to develop education in five dimensions: (a) the workshops attended by international assessments to strengthen the school principal’s leadership; (b) the and national experts, teachers, capacity of teachers to assess their school environment; (c) pedagogical and MENFD staff to develop own students, to measure learning in management and support; (d) the learning standards aligned with a transparent and technically sound management of human and physical the curriculum and funded an way, and to adjust their pedagogical resources; and (e) community accompanying assessment of third improvement plans. These standards participation. The standards for grade reading in Haitian Creole. were defined across five levels of each of these dimensions are learning as seen in Table 2. described in Table 1. Table 1. Standard for Each of the Dimensions of School Quality in Haiti Quality dimension Dimension description D1 School principals focus their practices towards student learning, thus requiring that they monitor results and School principal leadership carry out a strategic and planned vision. D2 The school, the administration, and teachers follow procedures and practices that are clear in terms of the School environment organization, preparation, implementation and evaluation of teaching, considering the needs of all students by taking concrete actions so that students achieve their learning objectives and reach their potential. D3 The school has policies, procedures, and strategies in place that favor an environment that is respectful, Pedagogical Management organized, safe, and appropriate for student learning and the school community. Additionally, the school supports and Support and promotes the spiritual, ethical, moral, emotional, and physical development of its students. D4 The school has policies, procedures, strategies, and practices in place for the management of its staff and Management of Human educational resources. and physical resources D5 The school has presents policies, procedures, strategies, and practices in place for creating and developing the Community Participation participation of stakeholders within the educational community. Table 2: Five Levels of Learning Levels of learning Description Advanced Pupils can make simple inferences: they are able to draw conclusions that are obvious, to find the meaning or subject of a theme or main idea of a text, to make inferences that an event is the cause of another one, to make simple inferences regarding the main characters, their attributes, feelings, or motivations, and are able to relate two pieces of information contained in two adjacent sentences. High Pupils can collect explicit information in a text: they are able to identify a word or idea already mentioned in a text and are able to retrieve information in a word or in a sentence. Average Pupils can read words and short sentences: they can recognize the image corresponding to the sentence or word that they read by themselves, identify the words read aloud by the teacher, and identify words written in uppercase or lowercase. Low Pupils can recognize letters and syllables: they are able to recognize images corresponding to short oral messages (a sentence or a word), to recognize syllables, to identify letters written in upper and lower case, to identify the first letter of a word and the initial sound of a word, as well as written characters among other symbols (such as numbers or signs). Below Average Pupils at this level do not demonstrate the skills measured by this test in Creole. These pupils have difficulties with the knowledge and skills at the low level. 6 RFB EDUCATION | EVIDENCE WHAT WERE THE RESULTS? School performance in Haiti QAS. Some of these improvements level of reading skill or lower (Figure 2). varies significantly across the five might actually be inputs, such as Only 2 percent of students in the pilot dimensions of educational quality. constructing proper facilities for were at the “Advanced” level, which is To assess progress towards the learning (as in Dimension 2. School defined as having the skills expected achievement of educational quality Environment). While school facilities by the official third grade curriculum, standards for each school, a set are often strictly a final outcome, the including being able to recognize of questionnaires were developed, lack of proper classrooms and latrines letters and syllables, to read words pre-piloted, and piloted in 2016 and can prevent students from being able and short sentences, to collect explicit 2017, one for school principals, one to effectively attend school and learn. information in a text, and to make for teachers, and one for school simple inferences. In other words, if characteristics. The data collected by The assessment of third grade third grade students were reading at these questionnaires was aggregated student learning in Haitian Creole the expected third grade level, most into a school’s profile consisting of was piloted in 20 schools in Port-au- students would have exhibited the the school’s scores on indicators that Prince in 2017. Almost 70 percent of “Advanced” level of reading in the provide an overview of its level of students in the pilot had an “Average” assessment. development in each dimension. The average school performance ranged from 28 percent in the “Management Figure 1: Average School Scores Figure 2: Students’ Competencies in in the Pre-Pilot Reading (Creole) (Pre-Pilot Data) of Human and Physical Resources” dimension to 62 percent in the “School 50 Environment” dimension (Figure 1). D1 40 Percentage of students (%) These school profiles, in combination D2 Quality dimension with measurements of learning, will 30 be valuable inputs into the school D3 improvement plans (SIPs) that each 20 school is required to develop with the D4 participation of their local community. 10 There is potential for policymakers D5 to use RBF to encourage schools to 0 10 20 30 40 50 60 70 80 0 Advanced High Average Low Below low achieve the goals set out in their plans Achievement of standard (%) Competency level as measured by the indicators in the HAITI 7 WHAT WERE the QAS is also acting as a guiding framework through which the and measuring educational quality, as the World Bank team in Haiti has THE LESSONS government can both improve the quality of public service delivery (by done over the past two years, and remaining persistent despite the LEARNED? prioritizing its spending according to inevitable setbacks and slowdowns clearly identified needs) and regulate that come from instability, shifting The QAS has established standards the private sector (by making public priorities, and emergencies on for what a high-quality school should financing conditional on schools the ground; (b) recognizing that look like in multiple dimensions. meeting high quality standards). “results” can often include what we This has helped to move the often dismiss as “inputs,” which dialogue away from focusing The continued interest and can actually matter a great deal for on enrollment numbers, inputs, engagement that the World Bank a school’s learning outcomes (for and vague pronouncements on team has experienced from example, having decent, functional the state of education towards a the MENFP and other sector latrines); and (c) recognizing that constructive discussion about what stakeholders demonstrates that many low-income countries may is working and what is not and how there was clearly a substantial latent already have policies that define improvements can be made. demand for a dialogue on results. fairly lofty ideals for standards The development of the QAS is but that having a dialogue on RBF It has also shifted the focus of the helping to legitimize the MENFP’s can actually be more useful than policy and technical dialogues in efforts to regulate the education introducing aspirational outcomes Haiti away from inputs and towards sector by creating clear, specific, and for bringing policy closer to reality. results and helped the MENFP realistic standards that will apply to to envision how to improve the both public and private schools. quality of education to the benefit of Haiti’s children. While development At the same time, moving towards organizations tend to focus on RBF is a process that will take time student learning as the ultimate and and multiple iterations. Creating most important outcome, it is very a sustained consensus about difficult to “move the needle” and quality standards despite shifting substantially boost learning in an priorities and political pressures enduring way without understanding will require ongoing technical and addressing the complex and assistance, dialogue, and knowledge interrelated aspects of what makes sharing. Moreover, the QAS is a a good school. The QAS assesses work in progress, as the realities of not only student learning outcomes implementation mean that the tools The QAS has been able to but also the intermediate indicators and approaches must be regularly shift the focus of policy that make up the learning conditions adjusted and refined to make them within each school, including the work on the ground. and technical dialogues in quality of the infrastructure, teaching, Haiti away from inputs and school leadership, and the overall Based on the experience of interpersonal climate. developing the QAS in Haiti, it seems towards results and helped that successfully implementing the Ministry understand Given the prevalence of private RBF in a low-income, fragile country schools in Haiti and the long history requires at least three elements: how to improve the quality of weak public sector management, (a) laying the groundwork for defining of education. CONCLUSION these learning conditions and outcomes. REACH also supported the it easier to target programs to address schools’ needs based on The set of activities funded by development of learning standards the best available data; (d) they have REACH successfully created a quality for reading in Haitian Creole. The QAS empowered local actors (such as assurance system (QAS) in Haiti data collection instruments have been school leaders, principals, teachers, that collects information on five piloted and revised and have become and parent associations) to become important tools for measuring more involved in school decisions; dimensions of learning conditions in educational quality in Haiti. and (e) they have established the schools, provides schools with clear standards to meet, and gives them foundation for introducing an RBF These activities have had several information about their progress benefits: (a) they have increased mechanism to incentivize school towards reaching these standards. the Haitian government’s capacity improvement. By establishing Now that this information is being to measure educational quality; (b) the QAS in Haiti as a reliable measured by the QAS, policymakers they have advanced the technical mechanism for measuring learning can decide to offer monetary design and piloting of accreditation and intermediate indicators of incentives to schools, teachers, or processes and improved the educational quality, REACH has other relevant stakeholders to make governance and regulation of the shown that the preconditions for RBF the necessary efforts to improve school system; (c) they have made can be established in FCV countries. 1 World Bank. 2015. The Rise of Results-Based Financing in Education. Washington, D.C.: World Bank Group. http://www.worldbank.org/content/dam/Worldbank/Brief/Education/RBF_ResultsBasedFinancing_v9_web.pdf 2 Holland, Peter A. and Jessica D. Lee. 2017. Results-Based Financing in Education: Financing Results to Strengthen Systems. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/715791489054110215/pdf/113265-REVISED-RBF-Approach-Final-Digital-Apr21.pdf 3 Adelman, Melissa A., Juan Baron, and David Evans, 2016. Why school enrollment is not enough: A look inside Haiti’s classrooms. Washington, D.C.: World Bank Group. https://medium.com/world-of-opportunity/why-school-enrollment-is-not-enough-a-look-inside-haitis-classrooms-34780e9614c7 4 World Bank. 2015. Better spending, better services—a review of public finances in Haiti (English). Public expenditure review (PER). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/246141467030212959/Better-spending-better-services-a-review-of-public-finances-in-Haiti 5 Observatoire National de la Pauvreté et de l’Exclusion Sociale. 2014. Investing in People to Fight Poverty in Haiti: Reflections for Evidence-based Policy Making. World Bank, Washington, .DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/21519 6 Adelman, Melissa A. and Peter A. Holland, 2015. Increasing access by waiving tuition: evidence from Haiti (English). Policy Research Working Paper No. WPS 7175. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/823841468035508642/Increasing-access-by-waiving-tuition-evidence-from-Haiti 7 Ministère de l’Éducation Nationale et de la Formation Professionnelle. 2016. Évaluation des apprentissages de base en lecture des élèves de la deuxième année fondamentale. 8 World Bank. 2018. World Development Report 2018: Learning to Realize Education’s Promise. Washington, D.C: World Bank. doi:10.1596/978-1-4648-1096-1. PHOTO CREDITS: Cover: “Education in Haiti” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 Page 3: “Source de la Grace Jumecourt Children’s Village, SDLG” by The Global Orphan Project, license: CC BY 2.0 Page 6: “Pre-primary section of the Ecole mixte de la Convention baptiste de Hinche” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 Page 7: “Education in Hati: Collège Mixte Joinvil in Port au Prince” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.