103712 Results in Education for All Children (REACH) Assessment of the Pilot Year REACH: Assessment of The Pilot Year is a companion piece to the REACH Annual Report 2015. Data used in the report is as of December 31, 2015. This report serves to reflect upon lessons learned thus far, and to provide recommendations to the Bank and to donors for the continued evolution of REACH. Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Serving Clients: Top Ten Things we Learned about RBF in 2015 . . . . . . . . . . . . . . . . 2 Strengthening WBG Capacity: Learning from early operations. . . . . . . . . . . . . . . . . 8 Program-for-Results: Learning from Tanzania’s Big Results Now . . . . . . . . . . . . . . 8 Early IPF DLIs: Testing hypotheses and debunking myths . . . . . . . . . . . . . . . . . . 8 Process Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Estimation of Expected Financing Gap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Appendix A. Annotated Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Appendix B. REACH Results Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 FIGURES 1 Relative Share of RBF, # Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2 Number of Projects, by Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Projected Demand 2017-2020 (US$). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 TABLES 1 A Short Glossary of RBF Terms, based on Musgrove, 2011 . . . . . . . . . . . . . . . . . 3 A.1 Results Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Acknowledgements: This Assessment was prepared by a team led by Peter A. Holland. Contributors include Melissa Adelman, Luis Benveniste, Jessica Lee, Diana Manevskaya, Wenna Price, Harriet Nannyonjo, Karthika Radhakrishnan-Nair, Shwetlena Sabarwal, Kavita Watsa, and Christina D. Wright. Photo credits: World Bank/Flickr REACH: ASSESSMENT OF THE PILOT YEAR REACH: Assessment of the Pilot Year Introduction Results-based financing (RBF) has the potential to transform how education systems operate, and the types of results they achieve for children the world over. As a financing instrument, RBF shows much promise for helping clients enroll the 121 million children still out of primary and lower secondary school, and teach the 250 million in school but still unable to read or write. Indeed, RBF has dramatically improved the delivery of health services in over 30 countries to date. Although more nascent in the education sector, RBF programs can foster the right incentive structures in education systems, overcoming challenges that impede additional and better education services for the most excluded children and youth. Generally speaking, RBF programs do this by rewarding the delivery of education outcomes through financial incentives, upon verification that the agreed-upon result has been achieved, and do so in a manner that can be credibly sustained over time. In 2015, the World Bank Group (WBG) launched REACH, a multi-donor trust fund that strives to support country clients in this space, and accelerate the RBF agenda within the institution. With generous financial support from the Governments of Norway, the United States, and Germany, REACH has just completed its pilot year. Highlights of activities during that year include the allocation of a Country Program Grant (CPG) to Nepal (US$4 M), and the approval of 19 Knowledge, Learning, and Innovation (KLI) Grants, in addition to training activities and knowledge and learning events with over 100 participants. This assessment reflects on lessons learned in 2015, and estimates the demand for future RBF in education. The lessons have been distilled from the early experiences with the 20 REACH-funded grants, as well as from the just-in-time support provided to Bank teams and country clients across the globe (about 20 countries in total). The assessment concludes with recommendations for the continued evolution of REACH. REACH: ASSESSMENT OF THE PILOT YEAR 1 Serving Clients: Top Ten Things we Learned about RBF in 2015 Lesson 1. RBF means different things to different people. Many international actors have developed away from input-based school financing a lexicon around RBF. A quick glance policies.1 reveals a true alphabet soup, distinguishing As we have been learning “how to speak between Results-Based Aid, Ouput-Based RBF”, we have chosen to adopt a Glossary Aid, Results-Based Budgeting, and so much of Terms from the health sector (see table more. The differences in terminology are 1). In short, RBF is the umbrella term that predicated on who is financing, who is we use for any program that rewards receiving, what results are sought, and under verifiable results. We believe that this what conditions. provides the most useful frame for thinking In the WBG’s Education Global Practice, about how our operations can help to RBF is viewed as a systems-oriented strengthen country systems. The source of financing approach that can transform the financing (e.g. whether government or how client governments achieve results donors) interests us much less than what the in the education sector. This is a different financing seeks to achieve. perspective from viewing RBF more narrowly In order to speak clearly with our clients as a means for generating more value-for- on this topic, and at the request of teams, money for donors, or as a primary strategy we’ve developed a short set of “RBF FAQs” for making aid “smarter”. While these may for clients. indeed be likely outcomes when donors embrace RBF approaches, they are not the 1 Using RBF to strengthen education systems is the theme of a main reason why governments should shift forthcoming WBG approach paper on this topic. 2 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) TABLE 1. A Short Glossary of RBF Terms, based on Musgrove, 20112 Term Definition Results-Based Any program that rewards the delivery of one or more outputs or Financing, RBF outcomes by one or more incentives, financial or otherwise, upon verification that the agreed-upon result has actually been delivered.   Incentives may be directed to service providers (supply side), program beneficiaries (demand side) or both. Payments or other rewards are not made unless and until results or performance are satisfactory. The definitions of results or objectives and rewards are embodied in contracts between one or more principals who provide the incentives and one or more agents who contract to deliver the specified results, outputs or outcomes. Pay for Performance Can all be considered synonyms. Performance in these labels means the (P4P), Performance- same thing as results, and payment means the same thing as financing. based Payment and These terms do not introduce any additional distinctions. Performance-Based Incentives (PBI) Performance-Based Incentives are directed only to providers, not beneficiaries; awards in Financing, PBF current programs are purely financial, although discussion in some countries contemplates provided non-financial rewards such as improved housing or transportation or the provision of schooling... Cash payment is by FFS for specified services; and payment depends explicitly on the degree to which services are of approved quality, as defined by protocols for processes or outcomes. Payments can be made to facilities or to individuals; “provider” includes both categories and can refer to any level of the health system, from community workers to hospitals. The relation between results and payments can be linear or non-linear. Performance-Based Setting a fixed price for a desired output and then adding a variable Contracting, PBC component that can reduce payment for poor performance or increase it for good performance compared to the standard defined in the basic   contract (Loevinsohn). The variable share at risk is often small, of the order of five percent of the base price in either direction, but it can be much larger... These are otherwise classical contracts that do not involve FFS or other output-related payments. They are usually applied to NGOs; the fixed price component leaves it to the provider to allocate funds among inputs. One may describe PBC as “contracting out” to distinguish it from PBF, which is a form of “contracting in”. Output-Based Aid, A subset of RBF […] includes only financial rewards. Output is used as a OBA synonym for results and does not usually include results better classified as outcomes. The distinguishing feature is that the principal is an aid   donor; the agent is therefore typically a recipient government or public agency, although it could be an NGO or private for-profit organization if external assistance is provided directly to such an entity rather than passing through a government. 2 http://www.rbfhealth.org/resource/rewards-good-performance-or-results-short-glossary-rbf-updated-march-2011 REACH: ASSESSMENT OF THE PILOT YEAR 3 TABLE 1: Continued Term Definition COD, Cash on Delivery Defined as “a new approach to foreign aid” it overlaps with OBA.   However, delivery may refer to outcomes rather than just outputs. It is distinguished by the maximal degree of autonomy for the agent in deciding how to produce and deliver the results. Once the objectives and the payment are contracted, the principal does not dictate or supervise the agent’s decisions or methods. This difference from RBF or OBA programs in general is procedural rather than referring to the objectives, the verification mechanism or the manner of payment. CCT, Conditional Describes demand-side programs where the incentives apply exclusively Cash Transfer or primarily directly to the program beneficiaries rather than to the agent(s) delivering services. Results are defined by the enrollment of beneficiaries in the program and their compliance with required behaviors such as consuming specific services. Incentives to recruit and enroll beneficiaries or to provide them with services may also apply on the supply side in these programs, as in RBF generally. For the name CCT to apply there must be a financial payment to the beneficiaries for compliance. CCTs typically offer non-financial rewards, such as food packages, as well. Lesson 2. Theories of change: a nudge instead of a lever. There exists a view that external financing outset, between donor and country or within can provide governments with an incentive the country system itself, success is unlikely, to undertake actions that they otherwise especially in the long term. might not, actions which are more important In contrast, rather than shifting objectives, to the financier than to the client. Based we believe that RBF serves more to shift on the principal-agent model, this theory the focus of attention and effort. By using argues that external financing can serve results as the starting point, and by using as a lever to shift the agent (e.g. recipient financing to sustain that attention over government) toward the principal’s (e.g. donor’s) objective, creating an alignment time, successful RBF works more to clarify of sorts (Clist and Verschoor, 2014). This and organize existing objectives and send presumes, however, that the agent does stronger signals about what matters. It not inherently value the result, or at least therefore nudges program actors – financiers, not to the same extent as the principal. Our recipient governments, service providers, experience to date has been different: If beneficiaries – to put resources towards the there is no alignment in objectives at the activities most likely to achieve those results. Lesson 3. Discretionary action is unlikely to drive solutions. One idea associated with RBF is that it must governments remain hands-off with schools or include discretionary action for recipients in teachers. It also suggests that the recipients order to succeed. This means that funders give have the knowledge and capacity to resolve full autonomy to recipient governments, or the problem at hand, whether increasing 4 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) learning, reducing dropouts, or getting the they see fit. Rather, improvements require remaining out-of-school children into school. new tactics, such as pedagogic interventions that help teachers teach at the level of In our experience, however, there are few students, training teachers in applying instances where such discretionary action is assessments, or other initiaves that usually the answer. This is true be it governments imply external support to front-line actors. or schools. Rarely have we seen schools or This is true for governments, too, that often teachers able to improve their students’ seek and appreciate working together learning simply by being left to their own to identify the critical pathways toward devices, and given the autonomy to act as achieving the desired results. Lesson 4. But discretionary spending might. A related idea is that discretionary is little reason to suggest that active spending – the freedom to choose when supervision regarding budget execution and how to spend the resources to achieve adds the same kind of value. As such, results – is a desirable feature of RBF. Here teams are encouraged to design RBF with we would tend to agree. While external discretionary spending for recipients— actors can add value in terms of technical whether governments, districts, firms or support toward achieving results, there schools—embedded therein. Lesson 5. RBF requires more than the usual level of client ownership. For RBF to work, the signals need to be to resolve bottlenecks in service delivery, strong. This means not only clarifying withholding payments to firms, schools, or incentives, but also following through teachers requires a much greater degree with integral program parameters such as of political will. When governments are withholding financing. This is particularly recipients in RBF modalities, they also take important early on, such that the rules of on greater risk, since non-performance the game are established at the start. Unlike could translate into not receiving project the usual problems that arise in project proceeds in their entirety. As such, client implementation, such as debarring firms ownership of an RBF scheme becomes an from competing for contracts, or working essential precondition for any RBF initiative. Lesson 6. The relationship between incentives and performance is not linear. Much of the RBF approach rests on availability of resources). If the incentive can an assumption that, properly applied, accelerate the motivation, then the desired incentives will enhance performance. This result will be achieved. is the theory underpinning teacher bonuses, However, research by Dan Ariely and others for instance, and other funding mechanisms such as those that link school grants to reveals that such incentives work best when student learning or test scores. the tasks conducted are mechanical in nature, The theory posits that motivation is the not cognitive (Ariely, Gneezy, Loewenstein, binding constraint (rather than ability, or and Mazar, 2005). What’s more, when the REACH: ASSESSMENT OF THE PILOT YEAR 5 stakes are high, the incentives can actually care workers are clear on how to achieve backfire. For some complex tasks requiring the result, and only require certain inputs high cognitive reasoning or creativity, the (vaccines, needles, refrigeration, etc.) to do higher the bonus, the worse the performance. so. Incentives can therefore be motivating, RBF has had much success in health, and the discretion over how to spend the increasing immunization rates and other resources can lead to better results. coverage indicators (World Bank and Gavi This has important applications to the Alliance, 20103). This is likely because education sector. For instance, perhaps administering vaccines is relatively enrolling previously out of school students is straightforward, regardless of the contextual more mechanical than teaching kids to read, conditions of the beneficiaries. Health in which case, incentives that involve financial stakes might work better for increasing 3 http://www.who.int/immunization/programmes_systems/financing/ analyses/Brief_19_Results_Based_Financing.pdf access than for improving learning. Lesson 7. To achieve more learning, aim for conditions that are conducive to it. It is difficult to improve learning without Instead, REACH has been exploring options knowing which students are struggling, and for other results indicators against which in what areas of specific subjects. Hence the financing can flow. What elements – reliably importance of good diagnostic testing, and measurable – are highly correlated, or even communicating those results back to teachers predictive, of learning? How can performance and schools in a timely fashion. However, using targets be reliably and reasonably set, not too these same tests to serve as accountability or high nor too low? REACH support to teams in incentive functions for teachers or schools risks Cameroon, Colombia, Democratic Republic of rendering obsolete the primary function of the Congo, Vietnam, and elsewhere are exploring test: to improve learning. (For a more complete this important question. discussion, see Neal, 2011). “Many accountability and performance-pay systems employ test scores from assessment systems that produce information used not only to determine rewards and punishments for educators but also to inform the public about secular progress in student learning. As long as education authorities keep trying to accomplish both of these tasks with one set of assessments, they will continue to fail at both tasks.” — Derek Neal, Designing Pay for Performance in Education 6 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Lesson 8. RBF tools that work to increase access may not also buy learning. When exploring the idea of using RBF that has worked for previous objectives to improve student outcomes such as of increasing access and improving learning, there is a natural tendency to look intermediate outcomes such as community at instruments and tools that have shown involvement, which might eventually lead success. To start to test this hypothesis, toward greater learning. Importantly, REACH did a deep dive on a good tool for we’re seeing many of these initiatives access – school grants – to see the extent trying to guarantee learning outcomes to which it is also a good instrument for by conditioning transfers to schools on improvements in student test scores. Given improving learning. After reviewing the the evidence, we are skeptical this can work. literature, visiting schools, discussing with Instead, we suggest teams consider binding teams, and hosting a debate, we are inclined on outcomes where there is evidence (e.g. to say that school grants, on their own, do not increasing enrollment/attendance), or buy learning. binding on conditions that are more fully As governments seek to redouble their within the discretion of school management efforts toward improving learning, there is teams (e.g. infrastructure conditions, proper a temptation to use the same instrument and clean school sanitation, etc.). Lesson 9. Invest in open data. RBF needs good indicators. But good systems. In time, this will serve as the basis for indicators do not appear overnight. Rather, a transparent RBF system that allows actors they require a significant investment of to align around the program objectives, and time and resources in order to build up the track progress. Such open data can then requisite robust monitoring systems. Even play a powerful role in shaping policy in the then, this may not be sufficient. Making education sector. In the United States, for data publicly available, and having the data example, the Open Education Data initiative field tested by, among others, researchers, will render much of the administrative data in is a great way for governments to revise education accessible to parents and students, and refine their data collection methods which is expected to have far reaching and education management information ramifications for the system. Lesson 10. The WBG’s support for RBF is a good way to introduce the concept to clients. As will be outlined in a forthcoming flow to governments using results-based approach paper, and as per the WBG’s modalities is an effective way to introduce Education Strategy 2020, the institution’s the RBF concept to clients. Building on promotion of RBF is as a mechanism this entry point, teams can then broaden to strengthen systems. “Smarter aid” the policy dialogue on how education is expected to be a byproduct of this systems, whether teacher policies, student systems strengthening rather than a assessments, quality assurance, and so on, primary objective. That said, having WBG, can move away from input-based financing GPE, or other international financing toward an RBF approach. REACH: ASSESSMENT OF THE PILOT YEAR 7 Strengthening WBG capacity: Learning from early operations In its role to strengthen WBG capacity to ■■ Butrequires clarity and specificity about undertake RBF operations, REACH has desired reforms, especially discrete, begun to document the operational lessons actionable steps to date. The two WBG RBF instruments most ■■ Main challenge: Not about capacity, widely used are the Program-for-Results but willingness: lending instrument, and Investment Project – for open data, and Financing (IPF) using Disbursement-Linked – to incentivize implementation level Indicators (DLIs). ■■ Teams should pay close attention to a. Program-for-Results: Learning from two things: Results chain, and the Tanzania’s Big Results Now4 implementation level actors The WBG’s first experience using the ■■ Teamsshould invest in three things: Program-for-Results lending instrument Open dialogue, open data, and open in education is in Tanzania. As part of a course-correction. REACH training session for staff in October, b. Early IPF DLIs: Testing hypotheses and the Tanzania team presented some early debunking myths lessons on the use of the instrument in education, and provided guidance To begin to document lessons for to teams on how to advise clients (and education project design using traditional manage internal processes) when using this Investment Project Financing, REACH instrument. Among the many takeaways visited Jamaica, where one of the Bank’s from the Tanzania case are the following: first education projects using DLIs has ■■ Effective been operating since 2008. The team instrument for reform, as participated in an implementation it generates political will, and helps support mission, and conducted semi- address multiple bottlenecks; structured interviews in the ministries of 4 Adapted from Sabarwal, S. “Implementation Realities: PforR in finance, planning, education, and health, Education”. Presentation given at PforR training in October, 2015. 8 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) as well as within the commission that out. Interviews were also conducted with served as project implementation agency, task team leaders from each stage of in order to assess the gap between the project. The highlights of the early design and implementation, several years lessons are as follows: Myth 1: DLI operations result in undue pressure on teams to disburse, irrespective of results achieved. A commonly perceived risk of RBF In the case of Jamaica, the DLIs were operations is that of teams coming under largely met on time. In the few instances pressure, whether internally or by clients, to when they slipped into future years, disburse resources regardless of whether they were offset by other DLIs that were the indicators have been met. Rather than met ahead of schedule. The WBG team withhold payment, the perception is that had never been pressured to disburse. financiers will have to re-calibrate the What’s more, the DLIs provided a strong indicators/targets, or otherwise take a more incentive in the crucial early stages of “flexible” approach that would facilitate strategy implementation, and achieving disbursement. them created momentum. Myth 2: Too many DLIs are always a bad thing. Constructing indicator frameworks is among On the surface, this seems excessive. the most important tasks during project However, given the numerous stakeholders preparation, especially for RBF operations. involved, and the need for a holistic WBG teams have tried to be careful not to cross-sectoral view of what needed overload the results framework, to limit the to be done when and by whom, this key outcome indicators to 5, and the total complete monitoring table was universally appreciated. The original 45 indicators number of indicators to 15. meant roughly 9 targets per year for 5 However, it seems that an abundance years; 12 more indicators were added in of indicators (45!) may be appropriate 2014, as part of an additional financing. depending on the context. In addition to The spacing of these indicators was also having 45 DLIs, the Jamaica project had appreciated, as it ensured that momentum an additional 30 monitoring indicators. in implementation never let up. Myth 3: DLIs need to be “heavy” (that is, highly valued in US$) to have their intended effect. There is a view that one of the theories of It was the increased attention to the desired change behind RBF relates to the size of results and the imposition of a unifying results the incentive to shift government priorities. framework that made all actors focus on what In this case, however, the DLIs weighed in actually mattered. at a modest US$180,000 each. As such, the This echoes other findings that it is not really money per se did not represent the incentive. the money that matters (see Lesson 2 on page 4). REACH: ASSESSMENT OF THE PILOT YEAR 9 c. Process Evaluation innovation while mitigating financial In addition to exploring the early lessons and reputational risk by creating from WBG-wide RBF initiatives, REACH conditional tranches, or providing has also reflected on opportunities for small one-time grants for teams to improving its performance, and the further refine their proposals. overall impact of the trust fund. ■■ Being careful not to exceed our Some examples of adjustments made capacity for providing just-in-time support. The REACH core team to KLI calls: adopted a first come, first served ■■ Casting the net wider. To ensure that approach to taking on just-in-time we support the best/most worthwhile support requests. This has worked initiatives regardless of their affiliation well, as it has allowed REACH to serve (NGO, recipient government, teams most ready for support, and academics, WBG teams, or other), we has encouraged regional managers opened the second call to agencies to prioritize operations to work with with no WBG affiliation. REACH. However, demand is much ■■ Involving WBG teams upstream. higher than REACH capacity, and so a In the case of outside agencies (e.g. more formal prioritization/selection of Cordaid) being shortlisted, we have teams will need to be introduced been reaching out to operational in 2016. teams working in those countries to ■■ Intentionally supporting diverse review the proposals and participate activities. As REACH works with teams in the interview panel. We have also interested in applying for financing, sought management support for these we have been encouraging teams to projects upstream. This has led to explore new areas for RBF, such as better collaboration, more productive performance-based contracting with discussions and stronger working teacher training agencies in China. relationships. ■■ Being explicit about the F in RBF. ■■ Promising ideas, but not ready We have noted that, while teams for full funding: conditional are very comfortable discussing how tranches and seed money. Grants they or their clients need to become of US$200,000 are not always more “results-based”, the “F” part – appropriate, especially when the financing – is not always present. activity is highly innovative, or being Unleashing the full potential of RBF implemented in an unpredictable means more discussion about how environment (such as during an donors, and governments, are going election year). We increased flexibility to pay, or not pay, based on previously in our funding for high-risk, high- agreed upon and independently reward activities to encourage verifiable outputs and outcomes. 10 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Estimation of Expected Financing Gap Demand from clients for lending (including in 6 years. Whether in terms of number of IDA grants) that use RBF modalities projects, or whether we consider the dollar has been growing quickly. This section value of the portfolio, the trends are similar. undertakes an analysis of the education portfolio, identifying trends since 2010, by Regional trends region. Working from the trends, the analysis Since the beginning, South Asia has been projects what expected demand might look leading the way for the WBG’s RBF work, like in the coming years. Note that these whether in project terms (figure 2) or numbers do not reflect actual demand as (especially) in absolute dollar terms. This is a expressed by clients. reflection of the degree to which large clients such as India and Pakistan recognize the value Global trends: RBF as a proportion of focusing more on results sought, and not of the portfolio merely on the processes and fiduciary aspects As shown in figure 1, in 2010, only 1 of the for how to acquire the requisite inputs. It is 36 new education projects approved by the also worth highlighting that in recent years WBG Board used RBF elements. By 2015, this Sub-Saharan Africa has witnessed a rapid had grown to 9 (out of 38), representing an increase in demand in this area. With the increase in proportion of the portfolio from projected direction of the GPE, this number is less than 3 percent to more than 20 percent expected to continue to increase. FIGURE 1. Relative Share of RBF, FIGURE 2. Number of Projects, by Region # Projects 50 12 40 10 30 8 20 6 10 4 0 2 2010 2011 2012 2013 2014 2015 2016 0 Total RBF Total Portfolio 2010 2011 2012 2013 2014 2015 2016 Africa MENA Latin America and the Caribbean Europe/Central Asia East Asia South Asia REACH: ASSESSMENT OF THE PILOT YEAR 11 Projected future investments million above the WBG’s commitment at Looking forward, as clients become more the World Education Forum in Incheon, familiar with the advantages of RBF, and as Republic of Korea in May 2015 to double its teams learn lessons from the first generation support for RBF by 2020. of projects, it is expected that demand will continue to rise. FIGURE 3. Projected Demand 2017-2020 (US$) Figure 3 estimates what demand for WBG 1600 financing (including 1400 GPE) in this area will look like. These rough 1200 projections are not a 1000 reflection of the actual 800 pipeline, but rather a simple projection of 600 trend lines, based on 400 the recent past. 200 The total amount of projected demand for 0 2016-2020 is US$5.59 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 billion, or US$590 12 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Recommendations As the REACH trust fund wraps up its pilot Strengthen global partnerships. For 2016, year, it is important to reflect on progress REACH will be increasing its engagement made to ensure that lessons learned with partners through initiatives such as are incorporated into future activities the Global Book Fund. Similarly, REACH and to make course correction where will formalize the collaboration with GPE, warranted. Working from a variety of to ensure that country teams are best sources and using different methods, this positioned to advise clients on how to take preliminary assessment serves to inform advantage of the GPE variable financing future technical, financial, and operational component. The Global Partnership for directions in the short term. The concluding Output-Based Aid represents another recommendations are as follows: opportunity for closer partnership. Support more Country Programs that are Plan ahead for REACH staffing needs. The ready for scale-up. Due to limited financing, REACH team is already close to capacity. REACH’s pilot year has only supported Looking ahead, the team will continue one Country Program Grant: Nepal. Yet to need a mix of in-house skill sets. This many countries are demonstrating both a includes program management, technical willingness and a readiness to move toward advisory skills, operational advisory skills, RBF. The KLIs Grants5 are serving to conduct and economic analysis/impact evaluation the groundwork for large-scale country skills. The coming year will likely bring more programs in future rounds of support. Larger demands on the core REACH team in all programmatic support will be instrumental of these areas, especially with the launch in bringing about more systemic change and of REACH for Reading, and the expected shifting culture to focus on results. increase in demand for just-in-time support Use REACH to leverage funds. In the and learning events, given the success coming years, we expect a significant of such activities in 2015. The REACH increase in demand from clients for RBF core team should continue to hone its (US$5.59 billion).  The WBG has made a technical skills, and continue to draw on the pledge to support such increase in demand operational experience from regional teams (US$5 billion).  Targeted REACH grants and wherever possible. technical assistance can be instrumental to These recommendations, along with the “crowd in” RBF resources to IDA countries, lessons presented in this Assessment, will where the needs are greatest. guide the continued evolution of REACH. Looking ahead, as activities financed by REACH come to fruition, it will be important 5 19 Knowledge, Learning, and Innovation Grants were awarded in 2015. Read about them at http://www.worldbank.org/en/programs/ to conduct a thorough external evaluation of reach/brief/reach-grantees-2015 For more details on the REACH work program for 2016, please the results achieved, to maximize REACH’s see the REACH annual report. reach over the long term. REACH: ASSESSMENT OF THE PILOT YEAR 13 Appendix A. Annotated Bibliography or Reviews of RBF in Education: Our Top Ten There are a number of important reviews achievement. This paper provides a review and other influential papers in the realm of the US and international evidence on the of results-based financing. Below we’ve effectiveness of such input policies. It then included some that have us talking and contrasts the impact of resources with that thinking about how best to advise clients of variations in teacher quality that are not and design operations that help to move systematically related to school resources. education financing away from traditional Finally, alternative performance incentive input-based models to those that are policies are described. more likely to generate results. The ideas expressed herein generally provide the 2. Steven Kerr: On the Folly or Rewarding conceptual basis for RBF in education, and A, and Hoping for B try to collate available evidence. Some http://www.ou.edu/russell/UGcomp/Kerr.pdf provide important words of caution. Whether dealing with monkeys, rats, or Although this literature dates back to 1975 human beings, it is hardly controversial to (and earlier), in many ways it is still nascent, state that most organisms seek information as is the evidence-base that provides the concerning what activities are rewarded, and analytical underpinnings for RBF. From our then seek to do (or at least pretend to do) myriad experience with client governments, those things, often to the virtual exclusion what is still as yet unexplored is how of activities not rewarded. The extent to RBF can play a transformational role in which this occurs of course will depend on education systems. By aligning actors, the perceived attractiveness of the rewards activities, objectives, and financing around offered, but neither operant nor expectancy the ultimate results that we seek, we can theorists would quarrel with the essence strengthen the performance of the various of this notion. Nevertheless, numerous pieces that, taken as a whole, constitute examples exist of reward systems that are education systems. The forthcoming fouled up in that the types of behavior approach paper for RBF in education will rewarded are those which the rewarder is explore this topic. trying to discourage, while the behavior desired is not being rewarded at all. 1. Eric Hanushek: The Failure of Input- Based School Finance Policies 3. Derek Neal: Pay for Performance http://onlinelibrary.wiley.com/ in Education doi/10.1111/1468-0297.00099/pdf http://www.nber.org/papers/w16710.pdf In an effort to improve the quality of This chapter analyzes the design of incentive schools, governments around the world schemes in education while reviewing have dramatically increased the resources empirical studies that evaluate performance devoted to them. By concentrating on pay programs for educators. Several themes inputs and ignoring the incentives within emerge. First, it is difficult to use one schools, the resources have yielded little in assessment system to create both educator the way of general improvement in student performance metrics and measures of student 14 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) achievement. To mitigate incentives for do the benefits of the performance based coaching, incentive systems should employ contract outweigh the costs, relative to other assessments that vary in both format and item forms of aid? Six headings are used to group content. Separate no-stakes assessments factors which will affect the likely costs and provide more reliable information about benefits. This provides a viable framework student achievement because they create to consider the appropriateness of a no incentives for educators to take hidden results-based contract in any given setting, actions that contaminate student test scores. underpinned by the relevant conceptual Second, relative performance schemes and theoretical research. Several examples are rare in education even though they are are given of how the framework could be more difficult to manipulate than systems implemented, and three main research gaps built around psychometric or subjective are identified. performance standards. Third, assessment- based incentive schemes are mechanisms 5. DFID’s report on Payment by Results that complement rather than substitute for https://www.gov.uk/government/uploads/ systems that promote parental choice, e.g. system/uploads/attachment_data/ vouchers and charter schools. file/213938/payment-results-current- approaches-future-needs.pdf 4. Clist and Verschoor: The Conceptual With a very few exceptions, almost all research Basis of Payment by Results and evaluation studies of PBR have been in http://r4d.dfid.gov.uk/pdf/outputs/misc_ the health sector. Almost all the studies are infocomm/61214-The_Conceptual_Basis_of_ of Results-Based Finance (RBF) initiatives Payment_by_Results_FinalReport_P1.pdf (incentives to service provider organisations and individuals) rather than of Results-Based Two economic models are used to examine Aid (RBA) to governments. The importance of the costs and benefits of Payment by an outcome (or results) orientation, focusing on Results, relative to other forms of aid. the actual benefits arising rather than on inputs First, the principal agent model provides a and services provided, is largely uncontested. framework to consider the most important Nevertheless, the evidence regarding the factors, with a focus on linking the agent’s potential of incentives to change professional payoff to an outcome which the principal practice is weak. Perhaps the most optimistic is concerned with. Much of the promised conclusion that can be drawn from available benefit of Payment by Results is related to evidence is that contracting out may increase the efficiency improvement that comes by access and use of health services in the short linking an agent’s payment to an outcome of term rather than broader health outcomes. the principal’s interest. Second, the multitask Unintended effects are quite possible, and model draws attention to the requisite there is limited evidence to date to date that characteristics of a good measure: it is not PBR approaches offer value-added compared enough to be correlated with the desired to other modalities. Actual implementation outcome. The actions needed to improve of PBR approaches has encountered a performance measure should be similar significant challenges and difficulties. There to those needed to improve the actual has been limited attention to some basic outcome that motivates the principal. questions about PBR approaches, including The two models, and various second order the mechanisms by which incentives may effects, are summarised by a single question: work or not, cost effectiveness, comparison REACH: ASSESSMENT OF THE PILOT YEAR 15 with other potential approaches, impact on effort and enhancing productivity relative equity, and sustainability. What does emerge to non-contingent pay schemes. However, strongly from the evidence base is that psychological research suggests that excessive PBR needs to be implemented as part of a rewards can in some cases produce supra- package that includes other forms of supports optimal motivation, resulting in a decline and services. The underlying complexity in performance. To test whether very high of each intervention presents a serious monetary rewards can decrease performance, challenge to implementation and evaluation, we conducted a set of experiments at MIT, the inhibiting meaningful generalisation without University of Chicago, and rural India. Subjects identification of the specific mechanisms at in our experiment worked on different tasks play. and received performance-contingent payments that varied in amount from small to 6. USAID’s Incentives and Accountability large relative to their typical levels of pay. With in Education: A Review some important exceptions, we observed that https://www.eddataglobal.org/reading/ high reward levels can have detrimental effects index.cfm?fuseaction=pubDetail&ID=722 on performance. The experience to date shows that some 8. German Development Institute’s incentives work and some do not, and success Improving Education Outcomes by is highly specific to the school environment. Linking Payments to Results Monetary incentives seemed to work well when teachers and learning inputs were https://www.die-gdi.de/uploads/media/ aligned, but the magnitude of the results DP_2.2015.pdf varied greatly. Some interventions had In results-based approaches, funding is positive effects on attendance and retention, linked to pre-agreed results that are defined and others positively affected learning in the form of indicators. Disbursements outcomes. However, for incentives to be only take place once progress toward scaled up beyond the level of randomized the indicators has been verified. This controlled trials, the following general issues places high requirements on the quality must be taken into account: i) Align all the of indicators used. Different development stakeholders with power; ii) Be aware that actors have started implementing results- some stakeholders may extract benefits based approaches, yet little attention has from the education system; iii) Fix as much been paid to potential advantages and as possible the deficiencies in the school disadvantages of the specific indicators support infrastructure; iv) Fix the misalignment that are used. The paper addresses this between policy and politics; v) Make sure that gap by first conceptualizing a typology of performance measures are sustainable. indicators and devising criteria for assessing the quality of indicators. The typology and 7. Ariely and others: Large Stakes criteria are then applied to five results-based and Big Mistakes pilot programmes in the education sector https://www.bostonfed.org/economic/wp/ in developing countries (Ethiopia, Rwanda, wp2005/wp0511.pdf Sri Lanka, Tanzania). A comparison of the Most upper-management and sales force indicators used across these programmes personnel, as well as workers in many other provides insights into how indicators for jobs, are paid based on performance, results-based approaches can be selected in which is widely perceived as motivating a more informed manner in the future. 16 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) 9. Center for Global Development’s Cash 10. World Bank Education Sector on Delivery Program as it relates to Strategy: Learning for All education http://web.worldbank.org/WBSITE/ http://www.cgdev.org/page/application- EXTERNAL/TOPICS/EXTEDUCATION/ education 0,,contentMDK:22474207~menuPK:28240 CGD developed a proposal in which donors 2~pagePK:210058~piPK:210062~theSite could commit to pay US$200 for each PK:282386,00.html additional assessed completer, that is, each Improving education systems means moving additional child who takes a standardized beyond simply providing inputs. There is no competency test in the final year of primary question that providing adequate levels of school. Defining the target as the number schooling inputs—whether these are school of assessed completers, rather than as the buildings, trained teachers, or textbooks—is achievement of certain test scores, minimizes crucial to a nation’s educational progress. incentives progress to misreport progress. Indeed, the increase in inputs in recent The country would report the number of years has made it possible to enroll millions additional assessed completers each year more children in school; this effort must and the donor would pay for retesting in continue wherever levels of inputs remain a random sample of schools to verify the inadequate. But improving systems also numbers, after which the COD Aid payment requires ensuring that inputs are used more would be made. The country could choose effectively to accelerate learning. While to use the new funds for any purpose: to past strategies have recognized this goal, build schools, train teachers, partner with the new strategy gives it more emphasis, the private sector on education, pay for setting it in a context of education system conditional cash transfers, or for that matter assessment and reform. build roads or implement early nutrition programs. This innovative approach would place full decision-making about the use of funds in the hands of developing country governments, letting them determine the best way to achieve the outcome that recipient and donor both want: a quality education for all. REACH: ASSESSMENT OF THE PILOT YEAR 17 Appendix B. Results Framework 2016 2017 2018 2019 Indicator Baseline 2015 Planned Planned Planned Planned Result 1. Children benefiting 0 12,000 72,000 72,000 60,000 from REACHa 1.1 Increase in the number of girls and boys 12,000 36,000 36,000 20,000 who have access to education 1.1a Boys/Girls 50%/50% 50%/50% 50%/50% 50%/50% Outcomes 1.2 Increase in the number of girls and boys who 0 18,000 18,000 20,000 complete education 1.2a Boys/Girls 50%/50% 50%/50% 50%/50% 50%/50% 1.3 Increase in number of children who pass 0 18,000 18,000 20,000 national tests in primary and lower secondary school, or who pass equivalent tests in informal education programmes (gender disaggregated) 1.3a Boys/Girls 50%/50% 50%/50% 50%/50% 50%/50% Result 2. Country systems and capacity 0 14 19 11 17 18 for RBF strengthened 2.1 Sustained support to Country Programs 1 6 6 6 6 and Pilots 2.2 Financial support to activities for Knowledge, 19 15 5 9 9 Learning, and Innovation 2.3 Just-in-time support to WBG program teamsb 4 5 6 7 8 Result 3. Global evidence base for RBF is developed and made publicly available Outputs/Process 3.1 Number of policy notes on RBF approaches 0 0 19 15 5 prepared and disseminated 3.2 Number of impact evaluations approved 6 6 2 3 3 3.3 External REACH website designed and 1 1 1 1 1 operating Result 4. WBG RBF agenda strengthened 4.1 WBG operations incorporating 8 8 11 14 12 14 RBF approaches 4.2 WBG staff certified through 2 40 40 40 40 40 Program-for-Results training 4.3 WBG staff mentored and participating 0 70 110 110 110 110 in RBF learning events 4.4 WBG strategic document on RBF 0 1 1 1 1 1 approved and implemented a This indicator is primarily for reporting purposes and will change over time as a function of the nature of projects supported. The indicator is calculated by dividing the total REACH investment, assumed at US$40m in 2016, by the average government expenditure per primary student (US$) of countries benefitting from REACH (if that country has reported data as part of the World Development Indicators - WDI database). Given that average expenditure is US$297 but REACH CPGs (the largest investment of REACH) are only open to IDA countries where average expenditure is US$97, US$200 is used. Data as of Jan 7, 2016 comes from the 2012 WDI, which was the year that had the largest number of countries reporting data. The US$40m investment is assumed to be additional to existing commitments, where US$36m is allocated to 6 CPGs in equal amounts and US$4m to KLI grants between 2018 and 2019. In the future, CPG grants will be required to choose from a menu of indicators that will be aligned with global IDA indicators. Actuals will be added as data become available. b Countries that are not receiving CPG or KLI Grants 18 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Results in Education for All Children (REACH) Education Global Practice World Bank 1818 H Street, NW Washington DC, 20433 USA Website: www.worldbank.org/reach Email: reach@worldbank.org