Report No: AUS0001422 . DEMAND-DRIVEN SKILLS TRAINING AND RESULTS-BASED CONTRACTING Lessons for Youth Employment Programs . March 2020 . SPL . 1 . © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2020. DEMAND-DRIVEN SKILLS TRAINING AND RESULTS-BASED CONTRACTING Lessons for Youth Employment Programs. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. 2 Contents Contents ..................................................................................................................................................................... 3 Diagrams ................................................................................................................................................................... 6 Acronyms .................................................................................................................................................................. 7 Acknowledgments .................................................................................................................................................. 8 EXECUTIVE SUMMARY ................................................................................................................................. 9 Introduction ............................................................................................................................................................................ 9 Demand-driven training (DDT) ....................................................................................................................................... 9 Results-based contracting (RBC) in skills training ................................................................................................. 11 1. INTRODUCTION ..................................................................................................................................... 15 1.1 Background and rationale for the study ...................................................................................................... 15 1.2 Objectives, scope and audience ..................................................................................................................... 16 1.3 Methodology and database .............................................................................................................................. 16 2. DEMAND-DRIVEN TRAINING FOR YOUTH ............................................................................ 19 2.1 What is demand driven training (DDT)? .................................................................................................... 19 2.2 Main characteristics of short-term demand-driven youth training...................................................... 22 2.3 The results of demand-driven training ......................................................................................................... 25 2.4 Success factors .................................................................................................................................................... 28 2.5 Challenges identified ......................................................................................................................................... 32 3. RESULTS-BASED CONTRACTING IN SKILLS TRAINING ................................................ 36 3.1 What are the main characteristics of RBC? ............................................................................................... 36 3.2 The results-chain in employment and skills training .............................................................................. 37 3.3 Results and impacts of RBC ........................................................................................................................... 39 3.4 Findings and lessons on the design of RBC............................................................................................... 43 3.5 Findings on monitoring and evaluation ....................................................................................................... 47 3.6 Challenges identified ......................................................................................................................................... 49 4. OPERATIONAL ADVICE ON RESULTS-BASED CONTRACTING IN DEMAND- DRIVEN TRAINING ......................................................................................................................................... 51 4.1 Initial considerations on RBC ........................................................................................................................ 51 4.2 Steps in the design of results-based contracts ........................................................................................... 51 5. CONCLUSION AND LESSONS ......................................................................................................... 59 5.1 Demand-driven training ................................................................................................................................... 59 5.2 Results-based contracting ................................................................................................................................ 60 3 BIBLIOGRAPHY ............................................................................................................................................... 63 Annex A: Information on Employment and Training Programs............................................................ 74 A.1. Kenya Youth Empowerment Project (KYEP) ................................................................................................. 74 A.2 Jóvenes Programs in Chile and Argentina ......................................................................................................... 81 A.3. Nepal Employment Fund using performance-based incentives ................................................................. 86 A.4. NEPAL- Enhanced Vocational Education and Training Project (EVENT I) ........................................ 94 A.5. Liberia- Economic Empowerment of Adolescent Girls and Young Women (EPAG) ....................... 99 A.6. Chile- Lifelong Learning and Training Project (Chile Califica) ............................................................ 104 A.7. Colombia “Jovenes en Accion” (Youth in Action) ..................................................................................... 106 A.8. Bangladesh SEIP I ................................................................................................................................................. 107 A.9. Grenada- Skills for Inclusive Growth ............................................................................................................. 109 Annex B: Overview of Results-Based Financing (RBF) – with application to Skills Training . 110 B.1. Introduction: Features of results-based financing........................................................................................ 110 B.2. Advantages of results-based financing............................................................................................................ 114 B.3. Limitations and risks in results-based financing .......................................................................................... 115 B.4. Findings..................................................................................................................................................................... 116 B.5. Elements of good practice ................................................................................................................................... 120 B.6. Funding mechanisms for vocational training ................................................................................................ 121 Annex C - RBF and the Use of Disbursement-Linked Indicators (DLIs) in Skills Training ...... 123 C.1. Overview of results-based schemes using DLIs .......................................................................................... 123 C.2. DLI examples in skills development programs ............................................................................................ 125 C.4. Findings in the specific operations reviewed ................................................................................................ 132 Annex D: “How to” guidelines on RBC ..................................................................................................... 134 D.1. How to apply a results-based contracting system........................................................................................ 134 D.2 Lessons from NEF on how to apply RBC to an employment and training program ......................... 135 D.3. Step-by-step guide to monitoring of RBC ..................................................................................................... 138 D.4. Model contract for short-term training using RBC ..................................................................................... 140 Annex E: List of additional relevant project-related sources ................................................................ 145 4 Tables Table 1: Characteristics and Evidence of Selected DDT/RBC youth training programs in three continents ............................................................................................................................................................... 17 Table 2: Moving from Supply to Demand-Driven Training Programs ................................................ 21 Table 3: Means and mechanisms to build demand-driven training....................................................... 22 Table 4: Comparison of DLI-type operations with RBC ......................................................................... 37 Table 5: Detailed results chain for interventions offering skills training ............................................ 38 Table 6: Assumptions for achievements in the results chain .................................................................. 39 Table 7: Summary of findings on RBC in reviewed skills development projects ............................ 40 Table 8: RBC Payment schedule for short-term skills training programs (% of total) ................... 44 Table 9: Step 1- Defining expected results................................................................................................... 52 Table10: Step 2 - Establishing organizational and administrative structures and procedures....... 52 Table 11: Steps 3 and 4 - Setting the price and payment schedule ....................................................... 53 Table 12: Step 5 - Targeting and selecting trainees ................................................................................... 54 Table 13: Step 6 - Contracting- procuring training providers ................................................................. 54 Table 14: Step 7 - Monitoring progress and verifying results ................................................................ 55 Table 15: Step 8 - Minimizing risks and avoiding unintended consequences ................................... 56 Table A.1- Overview of Cases ......................................................................................................................... 74 Table A.2: Project results .................................................................................................................................. 77 Table A.3: Costs and impact of Joven Programs in Argentina and Chile ........................................... 83 Table A.4: Summary Findings: Labor Market Impacts of Job Training Programs .......................... 85 Table A.5: Jóvenes Programs in Latin America and the Caribbean: Impact and Cost Effectiveness ......................................................................................................................................................... 85 Table A.6: NEF support for training content ............................................................................................... 87 Table A.7: NEF support for capacity building of training providers by type and year of implementation ..................................................................................................................................................... 88 Table A.8 - Number trained compared with annual targets..................................................................... 89 Table A.9: Tranche 1achievements in relation to targets ....................................................................... 108 Table A.10: Achievements compared with targets .................................................................................. 109 Table A.11: Classification of schemes by recipient of the funds ......................................................... 112 Table A.12: Overview of funding mechanisms in vocational training .............................................. 121 Table A.13: Number of DLIs in five skills training programs by subject ......................................... 126 Table A.14: Summary Table- Number of DLIs by project and category .......................................... 127 5 Table A.15: Characteristics of the DLIs...................................................................................................... 128 Table A.16: Advantages and disadvantages of DLIs according to scale on the results chain ..... 129 Table A.17: Advantages and disadvantages of scaled and target indicators .................................... 130 Table A.18: Checklist for Performance-based Contracting (in the Health Sector) ........................ 134 Table A.19: How to apply a results-based financing system: a step-by-step guide from the NEF ................................................................................................................................................................................. 135 Table A.20: How to conduct monitoring in a results-based contract: Step-by-step guide from the NEF ........................................................................................................................................................................ 139 Table A.21: Outcome budget and installment plan.................................................................................. 142 Diagrams Diagram 1: Demand driven training programs: Main actors .................................................................. 11 Diagram 2: Operational recommendations based on best practices ...................................................... 14 Diagram 3: Steps in Results-based Contracting .......................................................................................... 37 Diagram A.1: Terms applied to results-based financing ........................................................................ 110 Diagram A.2: Results Chain ........................................................................................................................... 111 6 Acronyms ADB Asian Development Bank AGI Adolescent Girls Initiative B-H Bosnia and Herzegovina CCT Conditional cash transfer DDT Demand-driven training DFID Department for International Development DLI Disbursement-linked indicator EPAG Economic Empowerment of Adolescent Girls and Young Women ESP Employment Support Project EVENT Enhancement of Vocational Education and Training ICR Implementation Completion and Results Report IDA International Development Association IE Impact Evaluation IFAD International Fund for Agricultural Development ITT Intention-to-Treat Estimates KEPSA Kenya Private Sector Alliance KYEP Kenya Youth Empowerment Project KYEOP Kenya Youth Employment and Opportunities Project M&E Monitoring and evaluation NEF Nepal Employment Fund NGO Non-governmental organization NPR Nepalese Rupee PAD Project appraisal document PDO Project or program development objective PESP Punjab Education Sector Program PforR Payment for Results PSDP Punjab Skills Development Project PSM Propensity-score matching RBC Results-based contracting RBF Results-based financing RCT Randomized Control Trial RMA Rapid market appraisal SD Skills development SDP Skills Development Program SEIP Skills for Employment Investment Project SIMO Skill India Mission Operation SSDP Skills Sector Development Program SSEP Skills Sector Enhancement Project STRIVE Skills Strengthening for Industrial Value Enhancement Operation T&E Training and employment service provider TVET Technical-vocational education and training WB World Bank 7 Acknowledgments This report is produced under the task From Safety Net to Comprehensive Social Protection and Jobs Services (P164812). This task is led by Abla Safir and Yulia Smolyar, with Abla Safir leading the component on “Operationalizing innovations to increase earning opportunities through youth employment and livelihoods”. The report was written by Richard Johanson, with extensive comments provided by Abla Safir, Arvil Adams, and Kevin Heraniah. The report was edited by Carmen de Paz and Moulik Shrivastava. The report was produced under the guidance of Rob Chase and Paolo Belli and received comments from peer reviewers Indhira Santos and Victoria Strokova, as well as comments from Hilari Asasira, Evarist Baimu, Alastair Haynes, and Utz Pape. We acknowledge financing from the Rapid Social Response Trust Fund (TF0A5804). 8 EXECUTIVE SUMMARY Introduction Demand-driven training (DDT) and results-based contracting (RBC) are two mechanisms used to enhance the effectiveness and impact of skill training programs. DDT links skills development explicitly with the job market, typically through employer engagement. Private sector employers define what type of training is necessary, either directly or indirectly through training providers . RBC is based on an agreement with training providers by which the disbursement of funding is conditional on the achievement of defined targets – most often verified gainful employment. Although these two mechanisms are independent, they are frequently combined. Much has been written about various approaches to DDT. However, relatively little is known about the application of RBC in skills training. This report reviews the literature, identifies project examples and derives lessons for the des ign and implementation of both DDT and RBC. A summary of the existing literature on international experiences with both DDT and RBC, including good practices, is presented here. This review aims to identify the most effective ways to deliver these programs and provide general lessons on their design and implementation. The audience for this review includes World Bank teams and other development and policy practitioners interested in results-based contracting in the context of demand- driven training. Demand-driven training (DDT) Contrary to “supply-driven” training, demand-driven training ensures that training responds to labor market needs as its foremost objective is placement in employment. At the core of demand-driven training is that actual labor market demand determines the scope of training offered and how it is organized and delivered. The assumption is that skills training aligned with the needs of the job market enhances the chances that trainees will be productively employed at the completion of their training, for the benefit of both the individual and the economy. The existing evidence shows that demand-driven training can be effective in improving youth employment, especially among women. A specific kind of demand-driven approach, the Jóvenes (or youth) model pioneered in Latin America, has been replicated worldwide in short-term training for disadvantaged and unemployed youth. According to multiple project evaluations of these programs, this model has been successful for the most part in raising employment, earnings, and job quality. Short-term training has been shown to be especially fruitful for women. The review of the literature examining these and other similar programs across countries (see below), allows to derive lessons, both regarding success factors and challenges in the design and implementation of effective demand- driven youth training programs. Diagram 1 shows the main actors in DDT, challenges, and success factors. Successful programs point to one central feature: the key role of employer leadership in defining the skills needs. The aim of these programs is to align the training offered with market needs. For that purpose, it is necessary to make sure that a proper market analysis is conducted beforehand, and that employers are fully involved in the process. The close involvement of employers, directly by overseeing the process and providing work practice, and indirectly by advising on occupational 9 demand, is particularly central for program effectiveness. However, it is not always possible to achieve close employer involvement in the training process given their interests and constraints. An alternative solution is to have training providers act as intermediary where they identify promising occupational demand and arrange internships as well as employment for beneficiaries. This altern ative works best when service providers are paid on the basis of results. Other challenges persist. For instance, employers may not be engaged enough when they are not in the driver seat. In addition, poorly planned and supervised internships are common, leading to graduates not entering the labor market, and therefore reducing the impact of investments in training. Note that from a theoretical perspective, employers should be leading technical training given that it responds to their needs for skilled labor and youth should pursue training as it increases their income. However, because of market failures, employers and youth may stay away from training, thus creating the need for public intervention. Employers can typically be worried about poaching: they may be afraid of investing in training staff who then leave. Employers may also face credit constraints that prevent them from paying for training when benefits can take time to materialize. This is particularly true of micro and small enterprises. Youth may also shy away from paying for training because of credit constraints or because of lack of guarantee of hiring once they invest in these skills. For both reasons, financing of skills training may be too low, especially for those skills that are easily transferable between firms, such as general skills and sectoral skills. Governments typically step in to resolve these market failures. 1 Demand-driven training addresses the above market failure but also ensures that, contrary to “supply-driven” training, training responds to labor market needs. One challenge with “classic” vocational-technical training as was often provided by government institutions is that it lacked market orientation and did not respond to labor market needs, thus resulting in low returns for individuals and for the economy. Demand-driven training aims to address this gap by having training that focuses on employer needs. Hence, good practices in building DDT are logically related to establishing strong linkages with employers in order for training to be relevant and conducive to improvements in em ployment outcomes. These include: (1) Focus on job placement; (2) identify demand early, in part through rapid market assessments; (3) involve employers closely in defining specific competencies and skill sets , possibly let employers lead the training; (4) use training providers to establish linkages with employers; (5) integrate workplace practice as an integral part of the training; and (6) provide post - training support especially for women in non-traditional jobs, and those in self-employment, as both may need support in part to increase their self-confidence. 1 A nuance to the point above is that other market failures, such as labor market imperfections may, on the other hand, encourage training by firms, as workers may not be able to change firm easily (Acemoglu and Pischke 1999). 10 Diagram 1: Demand driven training programs: Main actors *combined institutional training with work experience. In effective programs, governments focus on particular strategic tasks, including monitoring and evaluation, and delegate the provision to private training institutions. It is clear from the literature that the government role should focus on planning, financing, and oversight, instead of on the direct provision of training. The government would act as the main standard-setting agent and monitor the quality and results of the programs. The competitive outsourcing of training provision offers special advantages, as it seems to fuel private training markets. Howeve r, inefficiencies can arise in competitive bidding. This can happen, for instance, in connection with the identification of employment opportunities leading to small lots per trade and inefficient class sizes. With regards to the targeting and the content of the training, certain features also appear to be related to program effectiveness. Adequately selecting program participants is important for success. It must for instance be ensured that they actively search for a job upon completion, which is not always the case. Comprehensive training with life skills is generally endorsed. Finally, the literature indicates that skill signaling, as done through assessment and certification, raises the marketability of beneficiaries and can thus help to improve their employment prospects. Results-based contracting (RBC) in skills training RBC entails that service providers implement activities and receive payments based on the achievement of milestones. RBC promotes and reinforces DDT. The two mechanisms of DDT and 11 RBC are highly compatible and were used in conjunction in almost all the projects reviewed. Joint use in all probability increased synergy, effects and impact. Indeed, the inclusion of incentives for training institutions in DDT appears to be an additional success factor. RBC raises effectiveness through motivation: the amount paid to training providers is conditional on results being attained, with no payment for those milestones that are not achieved. When results are defined as employment, this gives impetus for better linkages between the training and the job market , focusing attention on the actual placement of graduates in employment. From the training-specific examples, it is clear that the results-orientation enhanced accountability among training providers. In addition, attaching payments to skills certification helped ensure quality thresholds. RBC seems particularly appropriate for short-term skills training because outcomes can materialize relatively soon and be measured fairly precisely. RBC aims to handle the principal-agent problem in training whereby the entity commissioning the training (employer or government) needs to ensure that the training providers are focused on employment outcomes and not only on supplying training . Indeed, RBC gives the principal (which can be the employers directly or the Government) a way to provide the right incentives to the agent (training providers) to deliver high quality training that is aligned with labor demand, in particular by making payments conditional on employment outcomes. This helps increase the demand- driven nature of training, away from focus on outputs such as delivering the training. A potential key challenge with RBC that focuses on employment outcomes is that it transfers some risk to the training provider who cannot completely control the outcome. Employment outcomes cannot be entirely determined by the training providers, given that employment is dependent on the trainee’s effort, which is difficult to observe, and on market conditions, which are beyond the control of a given project. Projects with RBC need to put in place mechanisms to manage this risk. It is important to ensure that training providers are well rewarded for the risks taken and that their payment, in particular, takes into account that some trainees will inevitably fail to secure employment. Rigorous evidence is limited about the effects of RBC in skills development. One challenge in examining the effect of RBC is that it constitutes only one element among many to achieve project outputs and outcomes. It is therefore difficult to single out its unique contribution to the results. Second, this review was able to identify only nine completed skills development projects that used RBC. Consequently, the findings and lessons on training are derived from a limited set of project- cases (although supplemented by findings on RBC in the general literature). A second challenge is that even fewer projects report on results. Finally, none of the cases evaluated the RBC instrument itself. However, the reviewed cases show that with appropriate planning and systems in place, RBC can be instrumental in achieving the intended outcomes. Evaluated projects with RBC were successful in attaining desired outcomes, as demonstrated in Chile, Colombia, Grenada, Liberia, and Nepal. The evaluations included experimental impact evaluations with randomized control trials (RCT) in Colombia and Liberia, impact evaluations with counterfactuals for Argentina, Chile, the Nepal Employment Fund (NEF), and completion process reports for the Nepal Enhancement of Vocational Education and Training Project (EVENT I) and Grenada. Overall, the RBC mechanism has been endorsed strongly in the evaluation of the projects where it was used. The findings also support the 12 hypothesis that demand-driven training can be made more effective through RBC. The literature points to several key design elements that help manage the risk that is transferred to training providers. First, pricing the incentive adequately is key in order to ensure that training providers do not take on excessive risks and end up dropping from the program. Second, additional financial incentives have boosted the inclusion of important disadvantaged groups that may otherwise be excluded from participation. Third, an adequate payment schedule is necessary. Training projects with RBC used widely different disbursement schemes, all of them entailing some form of deferred payments. Performance risks may deter participation by providers or inflate costs. However, the experiences in Nepal show that performance risk can be managed through agreement on results, effective two-way communication and joint verification of results. Transparent information was indeed instrumental in overcoming the initial reticence by providers. Fourth, the providers were offered complementary support to build their implementation capacity. Sanctions were also important motivators. It is as important to ensure that RBC does not create unintended negative effects and that trainee attrition is prevented. The incentives for the training institutions within the RBC system may be adverse, leading them to select the candidates that are more likely to find jobs – instead of those that need the training the most. Those who do not really need the training may drop out early if they find better options. Gaming (false reporting and collusion with employers) is a particular risk with RBC. This possibility needs to be anticipated and steps taken to moderate it, including explicit payment conditions and robust verification and sanctions. On the other hand, when targeting is indeed successful in reaching individuals who do struggle to find a job, these may also drop out of the program if they become discouraged or if their credit and liquidity constraints are not covered by the training stipend. Dropout preventive measures included career counseling, systematic interaction with trainees and careful monitoring at the different stages of the training cycle. Accompanying measures to address related constraints are also necessary (e.g. transport, childcare , etc.). The findings suggest that incentive payments should be introduced gradually and after careful assessment of pre-requisites. Those considering the use of RBC should ask four initial questions distinctive to the concept. These are: 1) What result is sought; 2) how it will be measured and verified; 3) how much will be paid for the result and when; and 4) how will fraud be managed? In addition, careful consideration needs to be given to readiness for the use of RBC. Pre -requisites for entering into RBC include: 1) Minimum threshold administrative capacity to manage the process; 2) adequate information, available or collectible, by which to measure results; 3) a sufficient pool of willing and able service providers, i.e. able to pre-finance operations and manage the training and monitoring processes; and 4) reasonable understanding of the concept by all parties. In addition, strong monitoring and verification systems should be in place. Monitoring needs to encompass the whole process of training and employment. Otherwise, opportunities will be missed for the early detection and solution of problems. Sample surveys are typically used to verify employment. Good practices in validation involve joint teams that include all main actors. However, the costs of monitoring and verification can be substantial. Therefore, thoroughness needs to be balanced with affordability. For this reason, efficiencies need to be sought. Self -monitoring by training providers and the employment of seasonal field monitors can help to optimize resources. 13 Finally, online data monitoring systems raised the efficiency and effectiveness of RBC implementation. Good practices can be highlighted along the eight main steps in the design of RBC (see Diagram 2). In defining the expected results, it is recommended for instance to focus on outcomes and outputs, and make sure that they are clear, measurable, realistic, and agreed by all parties. Establishing organization and administrative structures and procedures will require a clear definition of roles and responsibilities, collaborative mechanisms and making sure that the required capacity is in place. Setting an adequate price in turn requires collecting information, giving price ceilings and factor ing in direct costs, among other measures. Establishing the right payment schedule demands a focus on outcomes, avoiding complicated systems, and minimizing the number of payment tranches. With regards to targeting and selecting trainees/beneficiaries, it is important to introduce measures aimed at hard-to-reach clients, offer facilities that are close, provide child-care arrangements, and minimize political interference. Contracting service providers will call for avoiding complexity in contracts, defining minimum quality criteria for the bid, incorporating result indicators, and handling non - achievement, among other measures. In the area of monitoring progress and verifying results it is recommended to establish a baseline, use country data systems and online monitoring, and minimize the pressure on Monitoring and Evaluation (M&E) staff. Finally, minimizing performance risks and gaming in payments, and avoiding corruption, distortions, and cherry-picking in trainee selection would be required. Diagram 2: Operational recommendations based on best practices 14 1. INTRODUCTION This report provides a summary of relevant lessons extracted from experiences of demand -driven training programs and results-based contracting - which often come hand-in-hand. The report aims to inform the design and implementation of one of such programs in Kenya. It is based on a review of the literature as well as on project information regarding various Jóvenes programs in Latin America and the Caribbean (LAC) and similar schemes in Kenya, Nepal and Liberia. The following sections (2 and 3) review some of the key features of both types of programs, their history, evidence of their impacts and both success factors and potential challenges identified in the literature. The last section offers some central conclusions, including operational lessons for the design and implementation of RBC that could serve as an input for the Kenyan Youth Employment and Opportunities project. 1.1 Background and rationale for the study Demand-driven training and results-based contracting are two different mechanisms used in training projects to increase effectiveness and impact. “Demand-driven training” (DDT) links skills development explicitly with the employment market, typically through employer engagement .2 “Results-based contracting” (RBC) is based on an agreement with service providers to deliver funding conditional on the achievement of defined targets - often verified gainful employment. Each concept is defined more fully in the respective sections. The two mechanisms are independent but frequently combined in the same operation.3 Much has been written about various approaches to DDT. However, relatively little is known about the application of RBC in skills training. This study was commissioned to provide operational advice on the design and implementation of RBC in the Kenya Youth Employment and Opportunities Project (KYEOP, 2016 -2021). One of the sub-components of this project finances short-term training for youth through recognized accreditation training providers under results-based contracts. In an earlier project, the Kenya Youth Empowerment Project (KYEP 2010-2016), employers played a direct role in implementing a demand- driven approach for short term youth training. KYEP did this effectively by using an employers ’ organization for the implementation. The follow-on KYEOP project, in contrast, uses a government agency. The KYEOP strategy has been to achieve demand orientation by paying training providers based on RBC, where performance is determined by the employment of the beneficiaries. This approach requires the providers to establish close relations with local employers. However, pay-for- performance methods have had not been tried on a large scale in youth training in Kenya. It was felt that the Kenyan authorities could benefit from a study identifying the most effective ways to deliver RBC based on the existing international experiences of RBC in DDT. This paper provides a summary of the lessons that can be drawn from the literature on the topic. 2 Parents and youths who make training choices with their spending may also part of demand-driven training when they have good information about employment prospects and freedom to choose institutions and occupations. 3 Results-based contracting can also be used for supply-driven training projects. As an example, training providers could be paid a bonus for enrolling certain target groups or for expanding the capacity of training institutions. Similarly, demand-driven projects do not necessarily involve results-based contracting, as in the KYEP. In short, results-based contracting can be used for either demand-based or supply-based programs. 15 1.2 Objectives, scope and audience The study identifies and analyzes literature and project documentation on the use of DDT and RBC in training unemployed youth. It reports on findings, lessons and provides operational advice. The review focuses on short-term training programs (i.e. usually less than six months in duration) for jobs mostly in the formal sector of the economy. 4 The targets of the training are young adults (typically 18-35 years of age) in low- and middle-income countries who are not in school or gainfully employed.5 The main questions addressed are: a. DDT: What is demand-driven training? Where has it been applied in training vulnerable youth? Is it effective? What are the factors that can contribute to its success or failure, and what the most effective delivery approaches? What are the main issues and concerns? b. RBC: What has been the scope of results-based contracting in training for youth employment? What are the main characteristics and results? What factors facilitate their success? What are the most effective delivery approaches? What are the main issues and concerns? c. What are the main lessons based on the existing experience with DDT and RBC? What are good practices in the design and implementation of RBC? What “how to” guidance can be extracted from the literature? The study is aimed at both external and internal audiences. The primary audience of this review are the KYEOP implementation and supervision teams. The secondary audience includes other World Bank teams and other development practitioners interested in results-based contracting in the context of demand-driven training. 1.3 Methodology and database The study combines a review of the literature as well as project documents on DDT and on RBC in skills training, which were sometimes followed up with clarification exchanges. The study is based almost solely on a review of available documentation. It is thus limited in scope by the methodology. For some programs, there is a good body of evidence available. However, this is not the case for all projects reviewed. The review included the project documents of ten completed youth employment projects that featured demand-driven training, nine of which also used financial incentives for training providers through RBC. For a Kenya perspective, the Kenya Youth Empowerment Project (KYEP) was included in the review for its demand-driven approach even though it did not use RBC. The cases vary widely in scope, from a few thousand beneficiaries in the Liberia and Ethiopia pilot projects to over a hundred thousand in Chile, Argentina and Bangladesh. 4 While some programs allow self-employment after the training, including as a milestone for payment of training providers, most programs are focused on training and internship in the formal sector. 5 With income below a minimum threshold. 16 The cases also vary widely in evidence and available documentation. Several include impact evaluations: RCT of the Colombia Jóvenes en Acción, KYEP, Liberia Economic Empowerment of Adolescent Girls and Young Women (EPAG); difference-in-difference estimates and/or propensity score matching (PSM) for the Nepal Employment Fund (NEF), Chile Califica, Chile Joven, and Argentina Proyecto Joven. Others have process evaluations or completion reports: Nepal Enhanced Vocational Education and Training Project (EVENT I), Grenada, and Bangladesh SEIP I. Annex A summarizes the characteristics and results of these 10 programs. Table 1 summarizes characteristics of demand-driven training and results-based contracting among six of the ten reviewed projects.6 Table 1: Characteristics and Evidence of Selected DDT and/or RBC youth training programs in three continents Latin South Sub- America Asia Saharan Africa Arg. Proyecto Joven Nepal EVENT I Level of Evidence 7 Liberia EPAG Characteristic Kenya KYEP Chile Joven Nepal NEF OBJECTIVES X X X X X X Strong evidence from rigorous impact evaluations of the Employability of positive impact that all disadvantaged youth interventions had on the employment prospects of disadvantaged youth. PROVIDERS - DDT X X X X Limited process evaluations and Require collaboration with project documents show that employers in identifying linking providers with market needs and arranging employers pays off in relevance work practice for trainees of skills and employability. FINANCING X X X X X Several process evaluations and Separate public financing project documents indicate from provision and allocate effectiveness in establishing resources through training markets (Colombia, competition Chile, Nepal). 6 Evidence is available on a topic that is close to RBC for skills development, and that is contracting for employment services. See Finn and Johnson (2014), which includes a discussion of the principal-agent model in contracting and the challenges it faces. A lot of findings on contracting for PES are similar to the findings discussed here. 7 The hierarchy of evidence goes from experimental to quasi-experimental to non-experimental evaluation with a counterfactual. Process evaluations are also considered carefully given that, while they do not allow measuring impacts, they provide highly valuable details on implementation. The best evidence combines rigorous impact evaluations with process evaluation, such as in the case of the Nepal NEF. 17 EMPLOYERS X X X X X Strong evidence. Experiences show it is essential to engage Provide content advice and private sector in developing work experience for course content to ensure that trainees graduates learn useful skills for the workplace. FINANCING - RBC X X X X X Limited but favorable process evaluations and project Use financial incentives documents. Positive impact but through results-based as part of impact evaluation (IE) contracting of overall DDT. CONTENT X X X X X X Strong evidence, including meta- Combine institutional analysis, particularly combined training with work with intermediation. experience (dual training) CONTENT X X X X X Limited but positive evidence Include soft skills in showing effectiveness of soft program, i.e. skills training on employability, comprehensive training formality and earnings. Source: Author’s compilation using projects’ documents, and OECD/ECLAC/CAF: 184. 18 2. DEMAND-DRIVEN TRAINING FOR YOUTH Demand driven training refers to skill development programs where the content is determined by the labor market demand – i.e., by employers and their existing needs. Employers therefore play a key role, most often indirectly through training providers. These systems are also characterized by the existence of training markets and competitive procedures, the provision of on -the-job training, and the inclusion of soft-skills training. They often include financial incentives. There is wide evidence of the positive impacts that demand-driven short-term training can have on a range of labor market outcomes, and especially so among women; the evidence comes from various evaluations of different Jóvenes programs in Latin America and the Caribbean (LAC), or of similar projects in Kenya, Nepal or Liberia. When conducted, cost-benefit analyses show that they are also cost-effective. Some elements of success include ensuring a proper market analysis, as well as the adequate engagement of the private sector and the existence of competitive bidding processes for training institutions or the provision of on-the-job training through internships. However, it must be noted that expectations should be realistic and that problems can emerge to engage training providers, ensuring that internships are effective, or that they lead into employment, among other challenges. This section will provide a more detailed account of all these and other relevant issues. 2.1 What is demand driven training (DDT)? Demand-driven training (DDT) is defined as skills development aiming at placement in employment. At the core of demand-driven is that actual labor market demand determines the scope of training offered, how it is organized and delivered. The assumption is that skills training aligned with the needs of the job market enhances the chances that trainees will be productively employed at the completion of their training, for the benefit of both the individual and the economy. More specifically, this review uses the following definitions: • “Demand-driven” training implies that jobs and the employers that use the skills play a leading role in shaping the training process, including the definition of occupations, content, standards and certification of results. 8 • “Demand” means effective labor market requirements as articulated by employers, or by the requirements for gainful self-employment. In some circumstances, requests by youth for training in specific occupations can be a proxy for labor market demand. 9 • “Training” is job preparation aligned with knowledge, skills and attitudes needed for employment or self-employment. The focus of this review is on short-term training, rather than the general system of technical-vocational education and training. The focus of DDT is generally wage employment, even if some programs have evolved to include self-employment. DDT, including as inspired by the dual training model (see below), is traditionally focused on wage employment. Recognizing the reality of employment in low- and middle-income 8 In this review “employers” has a broad scope and includes employers in the informal sector of the economy, although the focus of a lot of these programs is on training and internships in the formal sector. 9 These circumstances are where trainees have ample information about employment prospects and have the freedom to choose the most promising training providers and occupational areas. 19 countries, DDT has evolved to allow self-employment. In Liberia, the training included explicit support. However, DDT in its core form falls short from providing the additional support needed to start a business, including business grants, training in management skills or in socioemotional skills that matter for entrepreneurship. Based on these definitions, DDT implies first and foremost the engagement of employers. The engagement of employers is key to make requirements known and to provide jobs or internships. Employers also allow the identification and inclusion in training programs of the skills that are in demand, another key feature of DDT; and, finally, only they can allow achieving a third dimension of DDT: final placement in an internship or a job. Close employer involvement is for instance a key factor in the success of the traditional apprenticeship (“dual system”) in Germany, Austria, Switzerland and Denmark. One of the greatest challenges in skills development is how to engage employers outside of settings where they have engaged on their own. Employers normally do not consider training to be a priority, except perhaps where there are skill shortages. Market share, profit and growth are the predominant motives. Their time is in short supply. The Jóvenes (Youth) training model (see Box 1) has a unique way to engage employers. Training providers themselves forge the necessary linkages with employers and arrange placement in internships or jobs often before the training begins. This model has been selected for the study because it was the progenitor of youth training programs including those in Kenya, although the Kenya Youth Empowerment Project (KYEP) was led by a private sector organization. Box 1: The Jóvenes training model Formal vocational training systems, in Latin America as elsewhere, typically prepared workers for jobs in the formal sector of the economy, mainly in manufacturing. They cater primarily to school age youth at the end of their formal education and are center based. It was not a central objective to train unemployed first-time jobseekers with low levels of formal education, or to address the needs of vulnerable groups such as women. In the 1990s the demand for skilled and semi -skilled workers in the manufacturing sector declined with consequent rising unemployment. This gave rise to a new training model, initially in Latin America, that became known as the Jóvenes (or “youth”) programs. The aim was to train disadvantaged youth for their insertion into the labor market. 10 The Jóvenes programs differed from traditional training in several innovative respects: They focused on the final placement in an internship or a job; they had a greater demand orientation – attained through the involvement of private enterprises; offered greater diversification of service providers; used a new training model and used new instruments for public funding; and targeted more diverse groups. 11 The following section elaborates on these innovations. Note that from a theoretical perspective, employers should be leading vocational and technical training in the first place, given that it responds to their needs for skilled labor but, because of market failures, employers stay away from training, thus creating a financing gap and the need for public intervention. Indeed, this is because employers can typically be worried about poaching: they may be afraid of investing of training staff who then leave. Employers may also face credit 10 Eichhorst, et al. 2012: 21-22; WB 2015: 25; Kluve, et al. 2017: 150 11 Ibid. 20 constraints that prevent them from paying for training when benefits can take time to materialize. 12 Youth may also shy away from paying for training because of credit constraints or because of lack of guarantee of hiring once they invest in these skills.13 For both reasons, financing of skills training may be too low, especially for those skills that are easily transferable between firms, such as general skills and sectoral skills. Governments typically step in to resolve these market failures. Demand-driven training addresses the above market failure but also ensures that, contrary to “supply-driven” training, training responds to labor market needs and is not driven by the availability of training. One challenge with “classic” vocational-technical training as was often provided by government institutions is that it lacked market orientation and did not respond to labor market needs, thus resulting in low returns for individuals and for the economy. DDT contrasts with “supply-driven” training, where the availability of training dictates what is produced. Supply orientation means that training providers produce the same types of graduates, year in and year out, with little regard for the needs of enterprises and the labor market. Within this model, training c an easily and unintentionally lose touch with the needs of the labor market and therefore with employment prospects. If the courses that are being offered are irrelevant to the existing needs and demand, this results in a “skills gap” between supply and demand. Table 2 contrasts the two different approaches. There is little dispute in theory and from evidence that demand -driven training is more conducive to employment for beneficiaries than supply-driven programs. Table 2: Moving from Supply-Driven to Demand-Driven Training Programs Area From Supply-Driven To Demand-Driven Market feedback Slow to identify and adapt to market Relies on market information, e.g. tracer studies, employer views Governance Bureaucratic, government dominated Participation by end users – employers Management of Often centralized; little, if any, Typically, semi-autonomous; substantial institutions accountability for results accountability for results Training Typically, government owned and Competitive market of service providers providers operated Training Often determined centrally, tend to be Often determined locally, frequent programs the same year-to-year changes according to market signals Instructors Full time, often tenured or with long Short-term contracts, use of part-time term contracts, low turnover, work staff, job requires frequent visits to exclusively at the training institution enterprises Trainees Trained in the institution only Undertake work practice, internships in industry Facilities Fixed Adaptable, possibly rented Budget Assured, regardless of performance Has to be earned, based on performance Incentives Obey rules, please hierarchy Raise revenue by selling services in the marketplace Source: adapted from author’s contribution to ADB 2009: 15 See also ELLA:4. 12 Acemoglu and Pischke, 1998, 1999. 13 Becker, 1962. 21 Demand-driven training can be achieved by various means and mechanisms. Table 3 summarizes some of them. Table 3: Means and mechanisms to build demand-driven training Means Types of mechanisms Identify employment needs Elicit information from employers about their hiring plans. Labor market information, surveys, rapid market appraisal, tracer studies .14 Capturing employer views on Sector councils, local bodies to identify needs, employer training content participation in defining standards (certification, competencies required), selection and final assessment of trainees. Linking providers to employers Employer role in provider selection; joint proposals with employers, e.g. training plans, letters of intent; provider placement of trainees in workplace trials; Providing work experience to On-the-job training, apprenticeships, dual apprenticeship, trainees internships. Providing financial incentives (a) for employers- reducing costs, e.g. relief from social insurance or tax payments, including training levies;15 wage subsidies, vouchers that pay for training (prospective) workers, levy-grant, sharing training costs; (b) for providers- results-based financing where performance is defined in labor market terms; (c) for trainees- vouchers to train for occupations that have reasonable employment prospects (assuming availability of good information about employment prospects, and freedom to choose training institutions and programs.) Developing training markets Separation of financing from provision; market mechanisms to allocate resources; competitive bidding on equitable basis. Addressing trainee needs with Comprehensive training, 16 including soft skills, job search, support services counseling, placement assistance. Source: author’s compilation. See also KfW 2008 for an overview of innovative instruments for training and employment. 2.2 Main characteristics of short-term demand-driven youth training a) Objective The ultimate objective of DDT is the productive and well-remunerated employment of the 14 Honorati and McArdle 2013: 54 15 Ibid.: 44. 16 “Comprehensive” training can be defined as the combination of in -classroom and workplace training plus other services, including counseling, mentoring and monitoring, job search and placement assistance, training on soft and life skills, and financial incentives. (Fares and Puerto 2009: 9). 22 beneficiaries through training directed by the employers. Demand-driven training by definition entails the involvement of employers, since they create jobs and provide employment. Employers may not be able to anticipate skill requirements more than a few months ahead, but they still know best the kinds and number of worker skills they need, and which ones are most difficult to recruit in the labor market. The Jóvenes programs involved employers in two ways: (i) employers, not government, set the training content; and (ii) employers provided workplace internships and jobs. These provisions were expected to ensure the relevance of the content and increase the chances of job placement. 17 b) The role of public and private actors A central feature of these programs is the changed role of public authorities. One of the problems with supply-driven training, as indicated above, is the centralization of training decisions far from local labor markets. The administrative systems of traditional training placed responsibility on the central authorities for planning, financing, regulating, executing delivery and controlling results. The main weaknesses were the low ability to respond to employer demand and poor levels of efficiency. The Jóvenes programs sought to solve these problems by (a) devolving management to the regional, local and often sectorial levels; and (b) separating finance from provision. The government ’s role focused on planning, financing and oversight. The programs were financed with public funds, but training was delegated largely to non-government training providers. 18 Private providers also play a key role within this model. In supply-driven models, public institutions are the default mode for training services. However, the demand-led model supported the functioning of a training market through competition among training entities. A market mechanism, competitive bidding, became the instrument to select providers and allocate r esources for training. In fact, the establishment of a training market was an explicit objective in many early J óvenes-type programs. 19, 20 Providers were also required to identify the skills in demand to be eligible for funds. In some cases, the prospective trainer first had to appraise the local labor market. 21 In addition, providers were required to link with employers and arrange placements for trainees in internships. As proof, training providers typically were required to submit letters of commitment from firms stating they will provide internships for at least 80 percent of the course enrolment. Such written commitments were intended to ensure that the content of the course was effectively demanded by the private sector. In total, 15 out of 20 of youth training programs in Latin America included the provision of internships by the provider. 22 One pre-condition here is that the markets for training providers need to be large enough. c) Incentives Some of the Jóvenes programs involved incentive payment schemes. Based on these schemes, training providers received incentive payments depending on trainee outcomes. The first program in 17 Ibarrarán and Rosas Shady 2009: 12; Honorati and McArdle 2013: 25-26; OECD/ECLAC/CAF 2016: 177; Castro 1999: 2,12. 18 UNEVOC: 2; Ibarrarán and Rosas Shady 2009: 2; Castro 1999: 2,4,10; OECD/ECLAC/CAF 2017: 177. 19 Including Chile Joven, Proyecto Joven in Argentina, Jovenes en Accion in Colombia, ProCaJoven in Panama and Projoven in Peru. (OECD/ECLAC/CAF 2017: 178-9.) 20 ELLA: 3; UNEVOC: 2; Honorati and McArdle 2013: 35; Almeida, et al. 2012: 147; Kluve, et al. 2017: 150. 21 Almeida, et al. 2012: 147; Employment Fund Secretariat 2015: 11. 22 OECD/ECLAC/CAF 2017: 178-79; ELLA: 3,8; UNEVOC: 3; Ibarrarán and Rosas Shady 2009: 12. 23 Chile tied incentive payments to actual employment. Argentina Joven, implemented during more challenging economic conditions, made a share of the payments contingent on completion of the internship as a proxy for employment. The Colombia Jóvenes en Acción program paid training institutions upon completion of training by the participants of the program. 23 Training institutions also had to guarantee apprenticeship in legally registered (formal sector) companies in order to be paid by the government for the provision of training. This pay-for performance element created incentives for training institutions to offer participants skills that were relevant and in -demand. 24 The incentive aspects are the focus of section 3 on results-based contracting in youth training, and therefore will be further examined in that section. d) The substance of the training Demand driven training models – for both general and technical skills – are characterized by broadened content, including socioemotional skills and job readiness skills training. Rather than just job-specific technical training, these programs often offer basic job readiness skills, life skills and entrepreneurship – i.e. “comprehensive” training. Job readiness skills include job search strategies, search information, interview techniques, and the preparation of personal work history forms. It may also include Information and Communication Technologies (ICT) skills like word processing and spreadsheets. Life skills, or “soft” skills, aim to develop personal and social attributes such as self-esteem, communications, leadership, decision making, conflict resolution and leadership. It also includes training on sexual education, hygiene and how to avoid occupational hazards. Nine of 20 youth training projects in Latin America included life skills. 25 Training for entrepreneurship may be added, covering such topics as financial literacy, basic accounting and preparat ion of business plans.26 Demand-driven training models for technical skills also combine initial classroom training with workplace training, usually in the form of an internship in a firm. This so-called “dual-training” mode had its distant roots in the German system of apprenticeship training where institution -based training is complemented by structured workplace experience. 27 Internships serve as a proxy for the availability of jobs. The assumption is that willingness of firms to offer internships correlates with the possible availability of jobs, on the basis of the premise that employers would be unlikely to absorb the costs of taking interns if there was no possibility of a job being offered.28 23 Attanasio et al. 2011: 192. 24 Kugler at al 2019: 9. 25 OECD/ECLAC/CAF 2017: 178-179. 26 Honorati and McArdle 2013: 6-7; Ibarrarán and Rosas Shady 2009: 12; Adams 2018: 9; OECD/ECLAC/CAF 2017: 177. 27 See Eichorst et al. for an explanation of the pre-conditions necessary for the German dual system, who also explains that these conditions cannot easily be transplanted to a different institutional and historical context. (p.24) See also van Eekelen: 69 and Deissinger: 70 for challenges in transplanting the condition. One area of success seems to have been in Egypt (Adams 2010: 66). “The dual model worked reasonably well in Vietnam but it was confined to a small number of mostly German-funded training centers. South Africa has tried to apply the model, but not without difficulty. Some sector-specialized training centers in Morocco have success with the model, and it appears to be working well in Singapore and Malaysia. The main lesson here is that the model requires a well-functioning and quite large modern sector” (Private communication from Jorgen Billetoft, October 2018). 28 Ibarrarán and Rosas Shady 2009: 2,12,31; OECD/ECLAC/CAF 2017: 177; ELLA: 5; Castro 1999: 6, 25. 24 2.3 The results of demand-driven training An additional characteristic of the Jóvenes approach is the rigorous evaluation of results. This has provided a wealth of information. 29 Overall, the results on employment and wages are mixed, but generally positive. General youth training programs are found to improve the labor market outcomes of participants across countries. In particular, the Jóvenes programs in LAC have shown positive results for employment, and especially for the quality of such employment – formality and earnings. The effects were larger among women, in large cities and in the long term. Evidence from the reviewed programs in Kenya, Liberia and Nepal also shows a positive impact of participation on employability, engagement in income generation activities and earnings. In addition, when available, cost -benefit analyses indicate that these programs tend to be cost-effective. This section will examine these results in further detail. a) Evidence of impact There is evidence that the labor market outcomes of youth are improved by skills training . Impact evidence is usually determined on (a) increased rates of employment (often termed employability), (b) improved job quality (formality), and/or (c) increased earnings. A recent meta - analysis of 97 active labor market programs targeted at youth in countries of all income ranges showed that investing in youth through skills training pays off and leads to positive changes in labor market outcomes. Generally, the effects of training for youth are better in low- and middle-income countries compared with higher income countries. 30 There is however research indicating that effectiveness largely depends on design, country and beneficiaries´ characteristics. While agreeing that studies show the promise of vocational training’s impact on employment, McKenzie pointed out that these impacts were modest. Only three of nine cases reviewed had a significant impact on employment. Kluve et al. (2017) found a similar proportion of studies on youth employment programs with statistically significant positive effects from a much larger sample (113 studies.) The conclusion drawn was not that most youth employment programs do not work, but that “much of the difference in performance seems to be related to design and implementation factors, as well as country characteristics and beneficiary profile ”. In Latin America the Jóvenes projects showed generally positive returns. Most of these programs have been successful with sustained effects. In each country evaluations showed positive and sometimes significant effects on employment and earnings six-12 months after participation. In Chile Joven, employment rates were 18-22 percent higher compared to a control group, formality increased by 15-23 percent and wages rose by 22-25 percent. Peru Projoven also demonstrated large impacts on employment, formality and wages. Proyecto Joven in Argentina was less successful, reflecting the stagnant labor market in the country, but still recorded a 10 percent increase in employment and earnings.31 Effects were stronger for employment quality than for employment rates, not surprisingly in 29 See summary tables in (1) Ibarrarán and Rosas Shady 2009: 27; and (2) Fares and Puerto 2009: 16. 30 Kluve, et al. 2017: 148. 31 OECD/ECLAC/CAF 2017: 188; Adams 2018: 9; OECD/ECLAC/CAF 2017: 185,186; ELLA: 7; Ibarrarán and Rosas Shady 2009: 12. 25 low- and middle-income countries. The evidence suggests that the impacts of youth training are not necessarily manifested in employment rates, but more likely on the quality of employment, i.e. on formality (e.g. Mexico Probecat), and on wages (e.g. Colombia Jóvenes en Acción). This is not surprising, considering that, per ILO definitions, employment rates tend to be high in low - and middle- income countries. Half of the programs reviewed showed improvements in employability. For example, in the Dominican Republic the program Juventud y Empleo had a positive impact on job formality for men of about 17 percent even though its impact on employment was negligible. 32 The size of effects appears to vary with timeframe, sex and location. The training effects may be small in the short term, but larger in the medium to long term (2-3 years) after completion. 33 In Colombia evidence indicated that the employment and earnings impacts of training decrease over time, mostly because the control group catches up with the treatment group. However, following trainees over 8 to 11 years allowed registering other impacts, including on schooling of their relatives.34 Impacts among women were both more frequent and larger than for men, especially with regards to employability and earnings. 35 This was the case in Panama (ProCaJoven), where the impact on wages was large for women (+38 percent) and in Colombia (Jóvenes en Acción), where wages increased 22 percent for women compared to only 10 percent for men. In Peru (Projoven) the employment rates were 20 percent for women, but the impact was negligible for men, while in Colombia the employment rate increase was 5 percent for women (compared to zero for men). In addition, both in Panama and the Dominican Republic, the programs had little impact outside the big cities, largely because those areas lack fully functioning labor markets. 36 Findings from Nepal reinforce the positive results from Latin America . The Nepal Employment Fund (NEF) had a strong impact on employment rates. Individuals selected for NEF training programs experienced an increase in employment rates of 50 percent; in particular, non-farm employment increased by 15-16 percentage points. Gains were higher for women (13 compared to 2 percent overall and 19 compared to 10 percent in non-farm employment). A World Bank evaluation found an increase in average monthly earnings of approximately 72 percent. The gains for individuals who ac tually complete the training (not just those who were selected) were even larger. The impacts were among the largest for training programs that have been evaluated around the world. 37 The impact evaluation of the Nepal Enhanced Vocational Education and Training Project (EVENT) project reported that the probability of employment increased by 20.1 percentage points for graduates of the training compared to the control group. The increase was higher for females (21.7 percentage points) compared to males (19.7 percentage points). 38 The program in Kenya also appeared to be successful in improving the chances of employment of participants. Results from the impact evaluation of the second cycle in Kenya´s KYEP showed that 80 percent of men, and 50 percent of women who were placed in internships were in paid work 14 months after internship completion, compared with an average of 69 percent of men and 43 percent 32 ELLA: 7-8; Kluve, et al. 2016: 37; OECD/ECLAC/CAF 2017: 181, 183. 33 Card 2015; Kluve and Weber 2015; Kluve ILO 2016 in OECD/ECLAC/CAF 2017: 187; Vooren 2017 in Adams 2018: 9. 34 Kugler et al. 2019. 35 OECD/ECLAC/CAF 2017: 188. 36 ELLA: 8; Adams 2018: 17; World Bank 2015; OECD/ECLAC/CAF 2017: 185-186; Adams 2018: 17. 37 Chakravarty et al. 2016: 21-22. 38 World Bank 2017b, EVENT I ICR:18. 26 of women in the control group. This was equivalent to a 10.8 and 6.7 percentage points increase in the probability of being employed among male and female participants, respectively. This gain increased to 14.2 percent among men completing the full training program, 8.7 percent among women.39 Similar results are observed in the case of Liberia. The number of trainees engaged in income earning activities increased by 18 percentage points relative to the control group. This translated into a 47 percent increase in employment from the baseline of 38 percent. Positive employment outcomes were stronger among the business skills trainees. By the end -line 70 percent of business skills graduates were engaged in at least one income-generating activity (wage or self-employment) compared to fewer than 60 percent of job skills graduates. The project Economic Empowerment of Adolescent Girls and Young Women (EPAG) in Liberia increased average weekly incomes by 80 percent relative to the control group, i.e. starting from a small base this amounted to an additional $8 per week.40 b) Cost-benefit analyses Despite the costly nature of youth training programs, there is evidence of their cost-effectiveness. According to Blattman and Ralston, it is hard to find a skills training program that passes a simple cost-benefit test. Allegedly programs that have positive impacts are often so expensive that costs far outweigh benefits. Not many youth training programs appear to have attempted to conduct cost- benefit analysis. However, some did and found positive returns. The program in Colombia, for instance, showed a 21.6 percent return even under a “pessimistic” scenario.41 The KYEP evaluation found that the net earnings gain after six months would pay for the cost of training in nine months. 42 The cost-benefit analysis for short term training in Nepal EVENT I project calculated an internal rate of return of 23.1 percent.43 In Liberia, average monthly income increases meant that business training participants would need just three years to cover the costs.44 A review of Jóvenes youth training programs in LAC conclude that most of them have a positive ratio of benefits to costs even without considering positive externalities. 45 However, in some cases results are not so positive and require adjustments. It is the case of the program in Liberia, where average monthly income increases entailed that that job skills participants would need 12 years to cover costs. It was also the case of the Argentina Proyecto Joven. These results underscore the importance of having data on which to redesign interventions. Armed with appropriate data, steps can be taken to alter the design and improve results. In the Liberia case, the low returns of the wage employment training prompted project management to make changes: In view of the existing limited opportunities for wage employment, the proportion of trainees in the job training stream was reduced in later rounds from 35 to 18 percent, and prospects for wage employment were more 39 Honorati 2015. 40 Adoho et al. 2014. 41 Attanasio et. al 2011: 209. 42 Honorati 2015. 43 World Bank 2017b, EVENT I ICR: 20. 44 Adoho et al. 2014. 45 Almeida et al: 148-149. 27 carefully scrutinized.46 2.4 Success factors The literature generally supports the value of the innovations introduced under the Jóvenes programs and similar demand-driven training programs. Factors contributing to success, which are explained in further detail below, include: (1) labor market analysis and early detection of skills demanded, for instance using tools such as the rapid market appraisal – e.g., Nepal Employment Fund and EVENT project; (2) private sector engagement is key for effectiveness; (3) changing the role of Government from provider to strategic actor in charge and competitively outsourcing training is overall a positive factor, although public providers may need to be reached through specific mechanisms; (4) the dual training model offers special advantages; (5) comprehensive skills are endorsed, but “life skills” has yet to demonstrate impact; (6) assessment and certification raise the marketability of beneficiaries; and (7) short-term training can be particularly effective for young women. a) Labor market analysis is essential It is well established that good knowledge of labor market conditions is fundamental for choosing skills for training and improving the employability of trainees.47 This requires early identification of the skills in demand in the locality. Despite the importance of information on labor market, supply-oriented training programs base course offerings on what is available without much regard for likely employment.48 Labor market information can come from labor market statistics and reporting, skills gap analyses, tracer studies and feedback from employer associations. The more control given to employers, the less one needs to make a separate assessment of skills demand – since employers would provide that information. The Nepal Employment Fund and EVENT successfully used a technique known as “rapid market appraisal” (RMA) to identify occupations and skill opportunities in various localities. 49, 50 Plans for placement in employment began early in the procurement cycle. RMA was required as part of the bidding process and was instrumental for good results. The instrument involved the collection of information through interviews with local employers, focus group discussions and observations. The NEF Secretariat provided training for potential bidders in RMA techniques, conducted random spot checks on RMA assessments by service providers and published detailed 46 AGI e 2015: 13, 16. 47 By putting money in the hands of the employer to buy training services (such as with vouchers), it may not be necessary to define results. The employer presumably will do this. However, the employer may define results in narrow terms and society may prefer a much broader definition of results. 48 One review found this to be the case for rural training programs in India. “The insufficient use of labor market analysis and attention to market demand led to the selection of providers and course offerings based mostly on availability of trainers and existing infrastructure, rather than on job availability. This also explains why there was a concentration of offerings over a limited number of trades across all programs. (WB 2015: 28). 49 See the RMA manual produced by the Employment Fund (Employment Fund Secretariat-b); also, an ILO 2016-sponsored RMA handbook). For more exhaustive labor market survey advice see International Youth Foundation and USAID, n.d. 50 The NEF also conducted a “Market Opportunity Survey” at the start of its current project. This macro-level survey was intended to recommend promising occupations for training providers and potential areas for apprenticeship training. It provided guidance to training providers that had successfully passed the expression of interest stage to conduct micro-level RMA. – Email communication from Bhanu Pandit, Monitoring and Evaluation Manager, NEF- Helvetas Swiss Intercooperation on June 7, 2019. 28 guidelines and forms on its application. 51 Some of the issues encountered included the lack of information about the training that other organizations and projects were planning so as to avoid market saturation. While instrumental, RMA was insufficient by itself. Training providers also had to establish continuous relationships with employers to ensure placement. 52 b) Private sector engagement is central for effectiveness Numerous studies have concluded that short-term skills building programs are more effective when the private sector is involved in the selection of training contents and the provision of workplace internships. Strong collaboration with the private sector in the selection of training contents and the provision of workplace internships was a critical factor of success in the J óvenes programs. Employer involvement helps ensure that graduates have learned skills useful in the workplace. 53 The Kenya KYEP is a good example. It was implemented by an employer association and used sectoral associations to help to identify required work competencies as well as to arrange internships. Peru Projoven and Mexico Probecat exemplified close employer involvement in the Jóvenes programs. 54 Not only formally committing to provide internships but also sharing the costs of training appear to be the most effective mechanisms to ensure engagement. A review of Jóvenes projects found that simply requiring employers to issue letters of intent about providing internships was not sufficient. Such letters did not prevent employers from later cancelling internships – cancelations increased from 7 to 24 percent in Argentina’s Proyecto Joven. Instead, the private sector should share the costs of training. Two successful programs did this. The Mexico Probecat program used an on - the-job training modality in which the firms covered the direct costs of training. Firms paid a minimum wage to participants during the internship phase in the successful Peru Projoven program. 55 Having employer associations arrange the training, as in KYEP, was both unusual and effective. c) From government´s provision to competitive outsourcing of training The Jóvenes programs in Latin America changed the government role in the provision of labor training, making it more strategic. Instead of being the main supplier of training, as in supply- driven models, the government’s role shifted to providing training through market mechanisms. The competitive outsourcing of training provision increased the opportunity for the government to focus on roles that it is uniquely placed to carry out, such as planning, evaluation, standard setting, financing, and regulation. In addition, the Jóvenes programs substantially changed the prevailing managerial culture into a decentralized and more flexible mechanism. Thanks to decentralization, the programs were able to cover large geographical areas relatively quickly. 56 51 To counter the tendency of training providers to break out of gender stereotypes for occupations, the NEF emphasized the need to identify at least one new occupational area in the RMA process. Contemporary Development Center 2014: 30. 52 Contemporary Development Center 2014: 25-33; Honorati and McArdle 2013: 15, 55; Honorati 2015: 3; World Bank 2015: 28; Employment Fund Secretariat-a: 1; AGI a: 1; Johanson 2015: 22. 53 Fares and Puerto 2009 and Betcherman et al. 2007 in WB 2013: 44; OECD/ECLAC/CAF 2017: 184; Honorati and McArdle 2013: 11; Adams 2018: 9-10. 54 Ibarrarán and Rosas Shady 2009: 31-2. 55 Ibarrarán and Rosas Shady 2009: 3-4; Castro 1999: 3-4. 56 Inter-Regional Inequality Facility 2006: 3; ELLA: 8; Loevinsohn 2008: 83, 92. 29 Private providers responded enthusiastically to open, competitive selection. This gave impetus to the development of private training markets. In addition, requiring providers to arrange internships and jobs motivated them to align training programs with private sector demand. However, the experience indicated that the organization managing the bidding needs extensive knowledge about prices to minimize costs (see section on operational lessons for results-based contracting - RBC).57 However, competitive selection of providers has not, by and large, attracted public training institutions. The NEF, in particular, made repeated efforts to attract public providers through open bidding, but to no avail. The incentives are not as strong for public training institutions as for private providers. Prima facie one could assume that public providers would have a built-in advantage in bidding against private providers. After all, their basic costs are covered through publicly -financed budgets. This has not proved to be the case, as in Nepal. The reasons are not clear, but may include that public training institutions were not geared to short term training, were prevented by regulations from keeping at the institution any rewards earned, could not handle the extra workload on top of existing training programs, or were not accustomed to dealing with employers. Different mechanisms may be required to reach public providers, e.g. application for grants in competitive applications limited to providers in the public sector. Other components in EVENT addressed the needs of public providers. d) The dual training model works The literature on impact evaluation strongly confirms the effectiveness of combining classroom training with workplace experience. The dual training model has positive results for employability, formality and earnings. Fares and Puerto, in a review of 345 evaluations of training programs, found that the interaction of classroom and workplace training combined with other supplementary service s increased the probability of favorable outcomes compared with classroom training only. Comprehensive training programs increased the probability of employment by 21-37 percentage points.58 In addition to developing job-specific skills, internships provide non-cognitive benefits to trainees (e.g. discipline, dealing with authority) and enterprises the chance to screen and select the best trainees.59 On-the-job training is also an effective way to break down information asymmetry in the workplace between employer and worker, i.e. the employer gets to know the worker and vice versa.60 Program length may be an important design consideration. The total length of the combined training must be sufficient – programs under four months do not have much impact. 61 e) Comprehensive skills are endorsed Comprehensive, integrated training, combining technical skills with job readiness skills and socioemotional skills, received wide endorsement as effective in promoting employability, but 57 Inter-Regional Inequality Facility 2006: 3,4; ELLA: 8; Hirschliefer 2016 in McKenzie 2017; Almeida, et al. 2012: 149. 58 Fares and Puerto 2009: 25; Adams 2018: 9-10; Kluve et al. 2017: 150; OECD/ECLAC/CAF 2017: 184; Ibarrarán and Rosas Shady 2009; Betcherman et al. 2004 and Betcherman et al. 2007; Kugler 2015 in OECD/ECLAC/CAF 2017: 186; Honorati and McArdle. 2013: 10; WB 2013: 44; Honorati 2015: 3; Castro 1999: 8. 59 Castro 1999: 8, 13. 60 Adams 2018: 10. 61 Kluve et al. 2016: 30. 30 the evidence is limited. 62,63 Aggregated study results have not systematically indicated better outcomes.64 Soft skills, when offered alone, were found to have no impact on the employment outcomes studied in the KYEP impact evaluation. There is no treatment arm however that offered vocational skills without life skills and the treatment arm that offered them in combination d id find a positive impact, therefore we cannot reject that the combination of life skills and vocational skills was effective. 65 The challenge with life skills remains that while there is ample evidence that shows that they matter for labor market outcomes, there is still limited evidence regarding the effectiveness of socioemotional skills interventions. Experiences in several programs validated the importance of delivering training for self- employment in contexts where wage employment is limited. In Liberia most trainees ended up self- employed regardless of the training track and the returns of business skills training were compe lling.66 In addition, multi-skilling, i.e. training in more than one occupation, proved effective particularly for those especially in rural areas with seasonal variations. 67, 68 Similarly, extended post-placement support was also endorsed, in part to cope with labor churning. 69 f) Assessment and certification raise the marketability of beneficiaries Good design provides for some kind of qualification and certification for those who succeed. Skills testing proved to be an effective check on quality in the Nepal Employment Fund. In fact, Nepalese training graduates ended up valuing the certification they received more than the income. 70 In Uganda, recent research found that vocational training had a much larger impact on long term earnings than on-the-job training. It hypothesized that this was because beneficiaries of vocational training obtain transferable skills, get certified and are better able to signal their skills. 71 There is need for more work to better understand the mechanisms through which certification influences labor market outcomes, and in particular whether it modifies them because it helps workers update their beliefs about themselves and thus modifies their behavior, or because it is actually valued by employers. An experiment in South Africa looks more into the mechanisms through which skill certification influences job search behavior and firm behavior. 72 g) Short-term training can be particularly effective for young women Experiences with short term training demonstrate that such programs can work for young 62 “Comprehensive” training can be defined as the combination of in -classroom and workplace training plus other services, including counseling, mentoring and monitoring, job search and placement assistance, training on soft and life skills, and financial incentives. (Fares and Puerto 2009: 9). 63 Ibarraran 2009, Betcherman et. al. 2004, 2007; Honorati and McArdle. 2013: 10. 64 Kluve, et al. 2016: 34; Kluve, et al. 2017: 149 65 Honorati 2015: 25. 66 AGI e 2015: 5, 12,13; Adams 2018: 12; OECD/ECLAC/CAF 2016: 183. 67 According to a tracer study in 2012, training on multiple skills provided nearly one-third additional income for the training graduates in Nepal. (Contemporary Development Center 2014: 23) 68 Johanson 2015: 23; Employment Fund Secretariat 2015: 9, 12. 69 WB 2015: 6, 33; AGI f 2015: 2,6. 70 Hollenbach 2015: 59; See also Adams 2018: 9,10; Honorati and McArdle. 2013: 10; Johanson: 22; Betcherman et. al 2004. 71 Alfonsi, et. al. 2017: 34. 72 Carranza et al. 2019 31 women, and in many cases may be more successful than those for men. The successful experiences of Liberia and Nepal, for instance, stress the importance of incorporating already in the design an analysis of the special constraints that women face.73 Special recruitment strategies may be necessary to reach the desired targets. Guidance and counseling take on special importance for unemployed women who often lack self-confidence Coping strategies may include the formation of group enterprises. An important aim should be to channel young women into non-traditional occupations. Ensuring participation by women in non-traditional trades requires extra effort beyond the monetary incentives provided to training providers. This led to the introduction of differential pricing and a special women’s window in the Nepal programs. 74 2.5 Challenges identified The same evaluations allowed to identify some key concerns and issues in project design and implementation, including: (1) expectations for youth training are often unrealistic , since these programs do not create employment and need to be part of a more comprehensive labor market strategy; (2) employers are not always able to represent their interests given for instance time constraints; (3) trainee selection can be challenging due to reliance on self-reported information and, in RBC, the existing incentives to reach out the candidates most likely to succeed in finding a job for providers; (4) internships may not be properly structured or supervised to maximize learning; (5) graduates may not seek to enter the labor market, reducing the returns on investment; (6) trained graduates may merely crowd out others in the labor market; (7) competitive bidding may lead to inefficiencies – e.g. the micro-identification of employment opportunities can lead to small lots per trade and inefficient class sizes; and (8) in conclusion, design – issues related to analysis of demand, targeting, pricing, etc. – matters for effectiveness, even more so at times than the specific content of these programs. 75 a) Expectations for youth training are often unrealistic Youth unemployment is structural and pervasive, particularly in low income countries. The implicit assumption in skills training programs is that jobs are available, but the youth do not have the skills necessary to do them, i.e. a skills mismatch. This can be true in some countries, markets, or for specific occupations in high demand. However, programs are implemented in contexts of low labor demand, in which case they may simply displace workers. Except for entrepreneurship and self- employment, training does not create employment and even then, additional support may be needed. As stated by Ibarrarán and Rosas Shady (2009) these programs do not tackle the root causes of unemployment and success depends heavily on a positive macroeconomic context. Besides, the relatively small investments in these programs cannot be expected to make seismic impacts. These programs have to be part of a more comprehensive labor market strategy and complemented by other types of interventions. A limited investment of funds and a few short 73 These include cultural restrictions that limit women’s ability to travel and work outside the home; women’s triple burden of income generation, household duties and childrearing; and stereotypes about male/female trades. (AGI a: 1) 74 AGI a: 1, WB 2015: 13; AGI f 2015: 1; Employment Fund Secretariat 2015: 14; AGI f 2015: 3-4; Johanson 2015: 24; AGI g: 13; World Bank 2017b, EVENT I ICR: 32. 75 These concerns may apply widely to all training programs, not just demand-oriented. 32 months in training for relatively simple occupations will only dent youth unemployment at bes t.76 As stated at the conclusion of the project in Grenada, “in the design of (similar) skills development projects, it is important to realize that at best these programs can enhance the supply of labor skills within the existing, exogenously determined demand for labor skills.”77 b) Employers are not always able to represent their interests Engaging employers can be difficult because of their time constraints. Participation in consultative forums may be lackluster or resisted altogether. Employer associations can have the same bureaucratic tendencies as other organizations and may include staff removed from recent contact with enterprises. Representatives of sectoral employers’ groups may not always be attuned to the demand, as was sometimes the case in KYEP and Bangladesh SEIP I. All these considerations must be taken into account when defining the way employers will be involved in DDT programs. c) Trainee selection can be challenging Difficulties were experienced in weeding out the “wrong” candidates (i.e. those that did not meet eligibility requirements) in some projects. Self-reported information (e.g. not in school or gainful employment) is difficult to verify. The Liberia project had to rely primarily on self -reported data in the absence of other forms of identification and documentation. To counter possible false information, it did not publicize eligibility criteria on age or educational attainment. KYEP was successful in using a lottery to avoid political interference in selection but ended up with a higher than intended percentage of trainees with some tertiary education, and a high proportion that continued on to further education. In India the profile of trainees in five programs called for better selection methods to weed out candidates who were either too young to benefit from the training or had no intention of ent ering the labor market. 78 Ensuring that the “right” (i.e. targeted) parties are reached can also be challenging. Under RBC training providers will tend to select the trainees most likely to succeed so they can receive full payment.79 The participants most likely to succeed may be those with the highest levels of educati on and coming from relatively higher income families. Moreover, the intended disadvantaged beneficiaries may not even seek to participate. Special outreach may be needed for these groups. Several projects offered attractive financial rewards for reaching high priority beneficiaries, e.g. women, very poor, or socially discriminated groups. This “differential pricing” technique is discussed in the next section under RBC. d) Internships may be ineffective The internship may not be properly structured or supervised to maximize learning. This is a common problem across countries and is made more difficult by the short duration of internships in the Jóvenes approach. Low cost may create an incentive to exploit the interns as inexpensive labor. 76 Ibarrarán 2008: 32; Castro 1999: 7. 77 World Bank 2014, Grenada ICR: 17. 78 WB 2015: 6-7,21. 79 The “Generation Kenya” program reportedly has an 84% placement rate for graduates and 78% remained in the same job after six months. However, entrance is highly selective. Only 20% of applicants are admitted to the program. https://kenya.generation.org/ 33 As stated, obtaining a firm commitment by employers to internships can be difficult and may not in any case lead into long term employment. 80 Requiring the employer to share training costs can help to make their commitment stronger. e) Graduates may not enter the labor market, reducing the returns on investment Training for employment by definition cannot be effective if the graduates do not seek work. Yet in several cases training places have been occupied by youth who never intended to enter the labor market. In Argentina and Chile about one fifth of the graduates went on to further education. In KYEP a beneficiary assessment found that 13 percent of the trainees had returned to school to continue their education. In the India programs 28 percent of trainees indicated that they did not intend to work after training. This does not mean the investment made in trainees was wasted entirely. The training may have increased self-esteem, as in Liberia. In addition, adding to the trainee’s knowledge and skills eventually will lead to better outcomes for the individual and presumably the economy. However, it is costly and does not contribute to the rate of return of the project if the objective is immediate ly increased employment and incomes. The investments could have instead been directed towards those in need of, and willing to take, jobs. In India, low employment interest called for improved trainee selection to screen out those with no intention to enter the labor market. 81 f) Trained graduates may merely crowd out others in the labor market Training may increase the chances that a recipient will obtain employment, but the graduate may just replace another worker. In this case the training investment achieves no net gain in the number of jobs. This is termed a “substitution” or “displacement” effect. The counter argument is that the trained graduate, even if supplanting a less-well trained employee, will be more productive for the employer and, by extension, the economy as a whole. 82 Existing evidence on displacement effects in low- and middle-income countries is limited. One paper in fact finds limited displacement effects from a dual training program in Cote d’Ivoire with about 0.75 new position created per subsidized trainee.83 The authors take the limited substitution effect as a sign that credit constraints are likely widespread in low- and middle-income countries. They also mention unmet job vacancies in firms. This may explain why displacement effects are limited, as opposed to when implemented in hig h- income settings during times of economic crisis. 84 g) Competitive bidding may lead to inefficiencies Complex bidding procedures can cause delays. This occurred in the initial stages of most projects, including Chile, Argentina and Kenya. Pricing potential contracts to preserve interest yet avoid waste can be challenging. It may be subject to information asymmetry—bidders may have better information about how to deliver the outcome with less effort and cost than the results -based allotment assumes thereby wasting resources. Micro-identification of employment opportunities can lead to small lots per trade and inefficient class sizes. Each provider may be expected to tailor-make the training 80 Ibarrarán and Rosas Shady 2009: 31-2. 81 WB 2015: 5; Adams 2014: 11. 82 Castro 1999: 7; Ziderman 2016, Adams 2018. 83 Crépon and Premand, 2019. 84 Crépon et al, 2013. 34 program and to prepare related materials. Tailor-made courses tend to have high costs. The micro- identification of demand may lead to unnecessarily high transaction costs with employers. 85 In short, it can evolve into a piecemeal approach that is both inefficient and fails to create anything lasting. In Argentina small lots resulted in over half the training providers being individuals. 86 The Nepal Employment Fund countered the tendency to proliferation by insisting on minimum class sizes, and by supporting the development of training curricula and materials that were made available to all training providers. 85 The KYEP project appraisal document stated that allowing training providers to develop internships with private employers would tighten the link between the provider and employer improving the relevance of the training offered. However, it explicitly rejected the approach due to the difficulty of having large numbers of training providers contacting individual employers. Instead, an employer organization approached individual employers to create internships. (KYEP PAD). 86 UNESCO 2018: 7; Inter-Regional Inequality Facility 2006: 4; Castro 1999: 12, 17-18, 26. 35 3. RESULTS-BASED CONTRACTING IN SKILLS TRAINING RBC in skills training refers to arrangements where training providers are paid upon the attainment of pre-defined and verified results. The available evidence on its use and impact for training is limited. Indeed, this review only identified nine such projects. Most projects, however, endorse the use of RBC in this type of training, as it appears to be associated with better training and employment outcomes. The existing literature allows isolating several factors for success including careful attention to pricing and payment schedule. Performance risks need, however, to be minimized, for instance through transparent communication and technical assistance to training institutions, while trainee attrition can be prevented with career counseling, systematic interaction with trainees, and careful monitoring. 3.1 What are the main characteristics of RBC? Results-based financing (RBF) includes a wide array of terms. In this report, the two main terms used are: (1) results-based financing (RBF), an umbrella term which covers any program that rewards delivery of verified pre-defined results with a financial incentive; and (2) results-based contracting (RBC), a sub-set of RBF that pertains to a formal agreement between a funder and an implementing entity for payment on delivery of pre-agreed results. In practice few, if any, pure RBF programs exist. Most use a wide range of points on the results chain. Annex B presents an extensive review of the literature, including features, advantages, risks and elements of good practice. RBF has five common features and presents certain advantages and challenges. The five common traits are: 1) achievement of results, 2) transfer of performance risk from the funder to the implementer, 3) incentives, 4) payment conditional on achievement, and 5) verification of achievements. The main advantages of RBF, at least in theory, are greater effectiveness, increased innovation through provider autonomy, clarity and transparency, enhanced accountability and raised efficiency. The main concerns and limitations, on the other hand, include that performance risks may limit the number of providers and increase costs, verification requirements may increa se costs, and incentives may lead to unintended consequences, such as distortions, fraud (gaming) and cream - skimming. RBF models used in skills development projects can be classified into two types: National programs using disbursement-linked indicators (DLIs) and narrower projects using RBC.87 RBC differ from programs using DLIs in several important ways (see Table 4). Under DLI-type programs the government or sub-government entities carry out development activities and receive the funds. Annex C provides an extended and detailed examination of DLI-type operations, including with application to skills training. Indeed, RBF is not new in vocational training. It has been used for decades in guiding financial allocations to training institutions as well to private provider of employment placement services. This review, however, concentrates on RBC, the instrument used in Kenya Youth Employment and Opportunities Project (KYEOP). 87In the Asian Development Bank, this instrument is termed “Results-based Lending.” An example is the Sri Lanka- Skills Sector Enhancement Program (SSEP). 36 Table 4: Comparison of DLI-type operations with RBC Subject DLI-type programs Results-based contracts Objectives Some increased service delivery, but Concentrates on service delivery focuses mainly on systemic reforms and improvements Implementers Works through government Works through service providers, implementing agencies typically non-government Scope Broad, system-wide interventions More limited in scope Stage in results chain Concentrates on inputs and processes Focus on outputs and outcomes Share of total Encompasses virtually all project May be component of larger project program financing financing 3.2 The results-chain in employment and skills training Before proceeding to describing the findings of RBC, this section presents the results -chain in employment and skills training. Diagram 3 depicts the linear steps in results-based contracting. Outputs in training are typically defined as training completed with qualifications obtained, and outcomes as jobs obtained and earnings levels (see Table 5). Payments are usually based on the achievements of the individual trainee. 88 Diagram 3: Steps in Results-based Contracting Results -based Contracting Step 1 Define outcomes, set pricing Funder Step 2 Select providers, Payment finalize contracts Verification Step 3 Service Provider of Results Implement services Step 4 Monitor, collect data, Targeted verify results Beneficiaries Step 5 Payment Source: based on USAID:6 with minor modification on steps. 89 88Honorati and McArdle 2013: 44-46. 89Of course, this linear approach to payment ignores that contracts can and often do provide payments for interim inputs and some outputs, while focusing on outcomes like employment. This is particularly true where providers need up front financing to deliver the service. Such interim payments can provide “quick wins” and reduce provider risks. 37 Table 5: Detailed results chain for interventions offering skills training Activities Outputs Outcomes Technical and business skills a. Completion of training with a. Increased employment training; Literacy or numeracy additional skills. (initial; sustained), as skills training; and behavioral, b. Improved technical indicated by job life skills for soft skills training competencies, e.g. in a placement and its length. Offered through: specific trade. b. Reduced time to find job; a. Provision of skills training c. Skills certification obtained. shorter unemployment (e.g. distance or classroom d. Improved management duration; greater training). skills and understanding of efficiency in the job b. Placement in workplace business mechanisms. search. training (e.g. internships, e. Improved financial literacy. c. Increased ability to retain on-the-job training f. Improved reading, writing job; longer job duration schemes). and mathematical skills. (hours worked). c. Placement in apprenticeship g. Improved psychosocial, d. Better quality of schemes. decision-making, employment (contract d. Provision of financial communication and type, job type). incentives to young teamwork skills. e. Increased earnings or apprentices and employers h. Increased self-management consumption. providing apprenticeship and self-esteem. f. Increased business training. i. In sum, having performance (efficiency, employability skills. profits, investments, output of entrepreneurs). Source: Kluve et al. 2017: 147, with adaptations: (the addition of training completion and increased employability as outputs, and “increased employment…” as the first outcome, rather than “increased probability of employment.”). See also Honorati and McArdle 2013: 10. For each dimension of the results chain to be effective, a number of assumptions needs to be met. The pattern is complex and progress depends on multiple factors in design and delivery, such as (i) the skills and competencies sought (technical, soft), (ii) the training program content or curriculum, (iii) competence of the training provider, (iv) participation of employers in identification of occupations and competencies required, (v) training models (classroom, on -the-job or combination), (vi) incentives for youth and employer participation, (vii) methods of targeting and recruiting participants, (viii) methods of selecting training providers, (ix) methods of paying providers and participants, (x) monitoring, reporting of implementation, and (xi) verification of results. Table 6 shows how a skills training program leads to outputs of trained graduates and then to increased employment, earnings and business performance. The progression includes some intermediate short - term results that lead to improved labor market outcomes. The progression is important for defining results in incentive-based financing. “Results” are defined as outputs and outcomes. The results in the above table rest on a range of assumptions. 38 Table 6: Assumptions for achievements in the results chain Assumptions Activities Outputs Outcomes a. Target group a. Participants attend and complete a. Existing labor demand for participates in the training. skilled labor. training (there is b. Training addresses participants’ b. Learned skills match labor awareness about the constraints (e.g. existing skill market needs and demand existence of the shortages) as well as the c. No stigmatizing effects. program). constraints of the labor market. d. Training completion and b. Contracted training c. Participants learn in training; related certificate signals institutions conduct training increases skill level; acquisition of increased training and link training is well matched to level of skill and higher participants to interests and abilities of (expected, observed) employers (if participants. productivity. conditionality d. Training induces expected e. Employers value certified exists). behavioral and attitudinal training. c. Employers train change. f. Participants gain recognized young people on- and valued qualifications. the-job and offer g. Adequate economic, social, placement (if institutional and conditionality administrative conditions exists). are in place. Source: Ibid. 3.3 Results and impacts of RBC A meta-analysis of skills training programs finds that programs that include incentives for providers have larger effects, the range of incentives examined is, however, wide. The range of incentives covered also lacks detail.90 Regarding the types of incentives providers in programs such as the Jovenes programs, Kluve et al. 2016 note that “While these are design features that have been observed in successful programs, such as some of the Jovenes program in Latin America and the Caribbean, empirical evidence on whether these features systematically improve youth employment performance, to our knowledge, does not exist.”91 A general caveat pertains to attribution as RBC is only one feature of the skills training programs reviewed. Results-based contracting is only one element among many others to achieve project outputs and outcomes. It is difficult to single out its unique contribution to the results. Indeed, none of the cases evaluated the RBC instrument itself. None tested the results of different financing parameters for the success of the incentives. 92 90 Kluve et al. 2016: “unfortunately, the information about contracting and payment system in our sample of programs is quite sparse.” 91 Kluve et. al. 2016: 33, 38. 92 This could be done with a study using a control group (i.e. the control having all the same elements except incentive payments to providers). However, this does not appear to have been tried thus far. 39 Nine examples were found in this survey of results-based contracting in skills development (Table 7). The evaluations included experimental impact evaluations with randomized control trials (RCT) in Colombia and Liberia, impact evaluations with counterfactuals for Argentina, Chile, the Nepal Employment Fund (NEF), and completion process reports for the Nepal Enhancement of Vocational Education and Training Project (EVENT I) and Grenada. In this review, a lot of prominence is given to the Nepal NEF because it has been covered with a quasi -experimental impact evaluation as well as with a process evaluations and tracer studies. The reviewed cases show that with appropriate planning and systems in place RBC can be instrumental in achieving the intended outcomes. Evaluated projects with RBC were successful in attaining desired outcomes, as demonstrated in Chile, Colombia, Grenada, Liberia, and Nepal. Overall, the RBC mechanism has been endorsed in the evaluation of the projects where it was used. The findings also support the hypothesis that demand-driven training can be made more effective through RBC. In the Nepal NEF, payment incentives were a major success factor and led to not only training of young people but also to their employment. Positive statements around the potential of RBC to incentivize the employment of trainees are also found in the case of Liberia EPAG and Bangladesh SEIP I, while in the case of the Chile Califica, Grenada, and Colombia Jóvenes en Acción projects RBC appears to play a central role in ensuring the best quality and most relevant training offer. Documentation consulted on Chile Joven and Proyecto Joven does not mention the effects of results-based contracting. Similarly, the Nepal Enhanced Vocational Education and Training Project (EVENT I) completion report says little about the effects of RBC. Table 7: Summary of findings on RBC in reviewed skills development projects Skills Type of Results on Remarks and findings on RBC Development evaluation Employment and/or Project Skills 1. Nepal- NEF IE with Positive. Trainees RBC considered instrumental for difference-in- experienced an increase project success. The outcome-based difference and of 50% in non-farm payment system effectively assisted propensity score employment and an the program in meeting its main goals: matching (PSM). average increase in relevant training of good quality and And two process monthly earnings of gainful employment.93 The use of evaluations and 72% Average gainful results-based payments proved to be two tracer studies. employment rates were highly effective in ensuring gainful 75% between 2008 to employment for graduates.94 Payment 2015. incentives was a major success factor explaining the large impact. Outcomes-based payments fostered an incentive not just to train young people, but to find jobs for them.95 93 NEF 2015: 5. 94 Johanson 2015: 13. 95 Chakravarty et al. 2016: 22. 40 2. Colombia IE with an RCT Positive. In the short- Positive statements on effects of RBC. Jóvenes en with intention-to- term, women trained “This pay-for-performance element Acción treat (ITT) had 7 percentage points of the program created incentives estimates. more of formal paid for training firms to offer training Results, therefore, employment and that was relevant to employers’ account for drop- earned 19.6% more. needs.” outs. Men trained had 6 “Our results indicate that a vocational percentage points more training program such as Colombia’s of formal paid YiA that was designed to combine employment but saw no classroom, apprenticeships and non- significant increase in cognitive skills training, and has pay- earnings.96 Effects on for-performance incentives built-in, is employment are an attractive social investment and a smaller in the long- potential avenue for social mobility”98 term, especially for men, but other benefits are registered on secondary and tertiary schooling, including among trainees’ 97 relatives. 3. Liberia EPAG IE with an RCT, Positive, especially for Positive statements about RBC in AGI with ITT. informal sector Liberia documentation: “Results- estimates. Results employment: 47% based payments conditional on account for drop- increase in employment placement can improve the outs before and and 80% from small employment of graduates.”100 RBC after the base in average weekly mentioned as one of the key features of program’s start. incomes. 95% the program in the impact completion rate.99 evaluation.101 4. Chile Lifelong IE with a Positive impact on Positive conclusion about the effects of Learning and randomly selected income (10% increase). RBC: “A sound system of incentives Training treatment group No significant impact was crucial to the program’s success. Project (Chile and a control on employability. The incentives system motivated Califica)102 group formed ex- providers to reach competency post using thresholds, minimize absenteeism and 103 PSM. Plus completion 96 Attanasio et al. 2011. 97 Kugler et al. 2019. 98 Kugler et. al. 2019: 9, 36. 99 Adoho et al. 2014. 100 AGIa: 1. 101 Adoho et al. 2014 : 5. 102 The Chile Califica program did not aim at employment, but only at increasing qualifications. 103 The IE may suffer from small sample size 41 process dropout rates and maximize evaluation. learning.” 104 5. Nepal Completion Positive results. RBC mentioned in the Project EVENT I process Internal Rate of Return Appraisal Document as a mechanism evaluation, with (IRR) estimated to 23% to align training with market needs. No results of a tracer for short term training. reference to RBC in the study and an Implementation Completion and impact evaluation. Results Report (ICR). 6. Chile Joven Positive for both men IE with cross- No mention found in available sectional and women. 21-26% PSM documentation on effects of RBC. increase in employment and before-after analysis. and earnings, respectively. 7. Argentina IE with cross- Mixed. Overall 10% No mention found in available Proyecto sectional PSM. increase on documentation on effects of RBC. Joven105 employment and wages for women, but not for men. 8. Bangladesh Interim project Positive interim results: Performance-based contracts provided Skills for monitoring. 80% certification rate inducements to achieve certification Employment and 68% employment and job placement, but small Investment rate among 228,000 percentage of final payment (20%) Tranche I trained graduates. limited the influence. (SEIP I) 9. Grenada Completion Positive results on Positive statement on RBC: “By Organisation process employer participation requiring training providers to of Eastern evaluation and trainee demonstrate employers’ commitment Caribbean participation: to provide on-the-job internships as States Overachievement of part of their proposal, the Project trainee target by 80%, ensured that employers played a role in (OECS) Skills 93% completion rate determining the allocation of training for Inclusive and five times the resources among the various Growth expected participation occupations and economic sectors, Project by enterprises with thus ensuring the relevance of the internships. 106 technical skills developed under the Project.”107 Source: Author’s compilation. Note: The projects are ranked following the strength of the evidence. While Nepal NEF is not an RCT, it is ranked first because it has rich process evaluation. 104 World Bank 2010, Chile Califica ICR: 28. 105 Lack of job opportunities led to changes in the payment schedule: final payment changed to completion of internship instead of payment on employment. 106 The Grenada indicator on employment was dropped during the Project’s restructuring. 107 World Bank 2014, Grenada ICR: 12. 42 3.4 Findings and lessons on the design of RBC Based on the available evidence and project documents, the following lessons can be highlighted with regards to RBC design : (1) RBC reinforces demand-based training; (2) the results orientation raises the accountability of training providers; (3) setting the right level of incentives requires thorough knowledge of delivery costs; (4) projects use widely different payment schemes; (5) incentive payments should be introduced gradually; (6) despite initial reluctance, training providers can be convinced to accept the performance risks thanks to open communication and technical assistance; (7) extra financial incentives can boost the inclusion of disadvantaged groups; (8) avoidance of trainee attrition requires careful selection and counseling; (9) sanctions were important motivators. a) RBC reinforces demand-based training The pay-for-performance incentives reinforce demand-driven objectives by ensuring that the courses offered were those that the employers needed. Linking funding to desired outcomes improves efficiency in training. This practice obliges training providers and program managers to shift the focus from inputs toward outcomes. It motivates training providers to a nalyze local labor markets, establish links with potential employers and offer courses that truly match the labor market demand. 108 Requiring training providers to guarantee apprenticeships in enterprises helped ensure the relevance of the technical skills. b) The results orientation raises the accountability of training providers A clear definition of outputs and outcomes enhance accountability among training providers. Within these programs, job placement becomes the goal of skills training and the driver for its quality. Linking training with employment required a paradigm shift for many training providers, away from an activities-based approach towards results. 109 Accountability to beneficiaries (or “downward accountability”) also improved. 110 That is, training providers took a keen interest in the trainees and their circumstances, as noted below. c) Setting the right level of incentive requires a thorough knowledge of delivery costs The price for a contract should be the combination of the direct and indirect costs of delivery plus an incentive for the provider as well as a compensation for the additional risks. 111 The implementation of the Proyecto Joven in Argentina suffered because of the lack of knowledge on the part of project staff about the real costs of delivering training under different circumstances. 112 Without extensive knowledge about delivery costs, accepted bids may be too low (discouraging bidders and risking provider withdrawal) or too high (providing a windfall to providers, undercutting 108 WB 2015: 7. 109 Employment Fund Secretariat-a; Johanson 2015: 12, 14. 110 Ibid. 111 For a clear explanation of cost components, see pp. 26-7 of Helvetas, “Implementation and Monitoring Manual,” Rural Enterprise and Remittance Project (IFAD). 112 Castro 1999: 21. 43 value for money). 113 d) Projects use widely different payment schemes RBC has been applied in substantially different ways; payments vary by stage in the results chain (see Table 8). At one extreme, the Nepal Employment Fund (NEF) allowed no payments to training and employment service providers until completion of training and testing. This required the providers to fully pre-finance the training and assume most of the performance risk. Other projects , however, provided payments earlier in the results chain for mobilization, enrollment, completion and placement in internships. This approach minimized risks to providers but lessened the momentum for the achievement of outcomes. The Nepal Enhanced Vocational Education and Training Project (EVENT) also has 50% of payment conditional on employment. The 50% covered in the first two instalments that cover enrolment and graduation do, however, cover most of the training costs. As explained by the World Bank’s task team leader for the project, “Most of the training costs are covered by the time of the second installment with the remaining installments working as a bonus/incentive for the training providers. This has reduced the risk for training providers of poor results but has not been adverse for the project with respect to employment results, as the average employment has been above 70 percent.”114 Table 8: RBC Payment schedule for short-term skills training programs (% of total) Program Results Stage Milestone Initial Enrollment Graduation, Testing or Placement in Certification employment After 3 After 6 months months Bangladesh SEIP I 40% 40% 20% Chile Califica115 75% 25% Nepal EVENT I 10% 40% 25% 25% Grenada 40% 30% 30% Nepal NEF 40% 25% 35% Source: See Annex A e) Incentive payments should be introduced gradually The introduction of incentives needs to be gradual in order to allow training providers to learn and adapt. In general, results-based contracting requires several prior rounds of training so that providers can gain experience with regards to the costs and risks involved. 116 This was clearly the case in Nepal, given the decades-long prior involvement of Swiss financing in short term skills training. f) Training providers can accept the performance risks with adequate communication and assistance Although training providers can be reluctant to accept performance risks, experience shows that they end up accepting them. As an example, many potential training providers were initially 113 Employment Fund Secretariat 2015: 10, 14. 114 Email communication from Sangeeta Goyal, World Bank task team leader, on May 29, 2019. 115 The Chile Califica program did not aim at employment, but only at increasing qualifications. 116Honorati and McArdle. 2013: 59. 44 reluctant to bid under the NEF, given the existing risks. They argued that the government should be responsible for linking training outputs with jobs. Their main concerns were delayed payment until completion of training and the specter of non-payment for non-employment at the end. However, in 2015, the NEF was working with 57 service providers, most participating year-after-year. This indicates their acceptance of the risks and endorsement of the approach. Transparent information is instrumental in overcoming the initial reticence by providers: To convince training providers to participate, there is need for a clear definition of results, open communication and collaboration in their verification.117 Training providers compete with one another, but collaborative approaches ensured that results, indicators and monitoring systems are jointly agreed and accepted by all partners. In the Nepal NEF, semi-annual meetings were held with providers and field and headquarters staff to identify and resolve issues. Project documentation also points to the importance of having a procedure for complaints and grievances. The NEF Secretariat established a complaint system and “hot-line” to receive feedback and early warnings about problems in the field.118 Training providers should not be abandoned to their own devices after winning the bids. The Colombia program Jóvenes en Accion provided help to build the capacity of training providers. 119 The NEF Secretariat also provided extensive capacity development for training providers, including in- service training for managers and for trainers, development of instructional programs and training in the use of rapid appraisal techniques. 120 Nepalese training providers specifically attributed their success to the guidance, technical support and professional development provided by the Fund Secretariat.121 g) Extra financial incentives can boost the inclusion of disadvantaged groups and decrease “cherry- picking” Initial trainee selection, identifying those able to learn and willing to work, is essential for success. As concluded for the Liberia program, equitable access and participation of disadvantaged groups can be enhanced through financial incentives for private training providers. However, incentives can be a two-edged sword: they can motivate training providers to select those genuinely interested in working after training, but they may also lead providers to neglect important groups that are difficult to reach, or to discourage poor performers so they drop out. To counter this “cherry- picking” the NEF introduced rewards for hard-to-reach youth (“differential pricing”).122 This mechanism worked well and has been used again under EVENT I and II. “Differential pricing” helped to increase access to this program to women and other discriminated youth. For example, 20 percent 117 Contemporary Development Center 2014: 23; AGI f 2015: 3. 118 Honorati and McArdle 2013: 47; Employment Fund Secretariat 2015: 12. 119 Government of Colombia: Chapter 7. 120 The NEF helped to define 45 occupational profiles and over 30 new curricula and new trades with associated training manuals. It provided training to over 3,800 staff, including training of trainers and managers of service providers. In addition, field staff of the Secretariat provided technical support and advice. 121 Hollenbach et al. 2015: 14; Employment Fund Secretariat 2015: 8,9; Johanson 2015: 15, 20. 122 Cherry-picking, also called “cream-skimming” can be defined as: “focusing on those near performance thresholds at the expense of those below it” (Castro 1999: 33) or “selection and concentration on those most likely to succeed rather than those most in need” (Holzapfel and Janus 2015: 14). 45 of the enrollments in later NEF programs were Dalits – above the national population of 12 percent.123 h) Avoidance of trainee attrition requires careful selection and counseling Since payments are based on individual trainee achievement of milestones, dropouts reduce the income of training providers. As a result, training providers under RBC are highly motivated to minimize dropout, as was seen in Chile Califica.124 In Nepal, trainee attrition was the main factor affecting provider earnings. 125 As reported by one service provider, “Once the trainees are selected, we cannot sleep well. We frequently call them by telephone to motivate them to continue in the training leading to job opportunities.” According to another provider, “We do not want to lose any profit. So, we are very aware about our participants. We do screening from the very beginning so that they won’t leave the training.” 126 Dropout preventive measures include career counseling, systematic interaction with trainees and careful monitoring at the different stages, as well as possibly additional measures for women. While the programs should avoid cherry-picking, it is also important to enroll individuals who are interested in completing the program. Linking funding to desired outcomes needs to have accompanying measures to allow training providers to manage the risks involved. 127 The NEF also allowed training providers to enroll 22 participants per group even though payments would be made for only 20 beneficiaries. This allowed a modest dropout of two participants per class .128 Allowing for group payments creates, however, complications in program monitoring, and can only be implemented if a robust monitoring system is in place. i) Sanctions are important motivators Performance assessment and sanctions to providers who do not meet the necessary thresholds can motivate them to keep higher quality standards, as well as requirements for skills certifications. Under the NEF training providers were graded on various performance indicators, including timely compliance with data entry and minimum employment rates. Those that did not meet minimum performance were disqualified from further participation. The Nepal Employment Fund barred an average of five providers per year. Providers were also subject to an annual performance review to qualify to be able to submit new bids each year. 129 Linking payments for graduates to competency tests motivated providers to meet minimum quality standards in Chile Califica and the two Nepal programs. 130 123 AGI a: 2; AGI b: 3; Johanson 2015: 13; Employment Fund Secretariat-a: 1; Karki et al. 2014: 21; Employment Fund Secretariat 2015: 14. 124 World Bank 2010b, Chile Califica ICR. 125 Employment Fund Secretariat 2015: 12. 126 Karki et al. 2014: 13, 18. 127 Employment Fund Secretariat 2015: 12; Honorati and McArdle. 2013: 46; World Bank 2015: 7. 128 Helvetas, 2019, Implementation and Monitoring Manual: 13. 129 Karki et al. 2014: 15; Employment Fund Secretariat 2015: 11; Johanson 2015: 14. 130 The NEF only required that the trainees undergo the certification process. Funding was not attached to passing the test, as this could invite pressure for graft. The requirement was that 95% of graduates had to take the test and 90% of those should pass it. (Helvetas 2019: 16). 46 3.5 Findings on monitoring and evaluation There is need for more evaluations, and preferably impact evaluations of different dimensions of RBC in skills training. As is described in Table 1, most rigorous impact evaluations on RBC are available on them as a whole, and as part of DDT. There is lack of rigorous evidence on particular dimensions of RBC, in particular on the impact of different payment schemes on trainee selection and on results. More projects need to be subjected to rigorous impact evaluations in order to improve project design in the future. Still, several lessons on monitoring can be derived from the literature reviewed . These include: (1) monitoring and verification proves to be especially important for RBF/C; (2) these activities cannot be restricted to results, and need to cover the whole training and employment process to avoid inefficiencies early or late in the process; (3) they require the time of qualified staff; (4) sample surveys are normally used for verification; (5) it is important to involve all the main actors; (6) self - monitoring by providers and part-time staff can promote efficiency; and (7) online data systems help to improve efficiency. a) All sources stressed the fundamental importance of strong monitoring and verification Monitoring is central for the effective functioning of RBF/C programs. Since achievements trigger payments, results-based financing/contracting needs a reliable and accurate, transparent system that is trusted by providers to generate and verify actual events. Monitoring and sharing of results keep all partners on their toes. For instance, in Liberia, monthly quality ratings helped maintain a healthy competition between providers. 131 Moreover, the monitoring system needs to be fully explained and agreed with training providers to reduce disagreements on the results.132 b) Monitoring has to encompass the entire training and employment process133 Monitoring should span pre-training, training and post-training, i.e. from pre-qualification of providers and selection of participants through verification of employment after several months. Just monitoring results is insufficient. It misses the opportunity for early detection of problems and taking corrective action. Thus, monitoring only outcomes can yield lower results than possible with wider monitoring. For example, without checking the selection process, the wrong participants may be enrolled, while without reviewing the standards of training imparted, the necessary skills may not be imparted and acquired. The NEF Secretariat and EVENT project staff provided specific and detailed guidelines for implementation that helped providers to follow each step in training management. Service providers welcomed the close monitoring by the NEF Secretariat since it helped to raise chances for success and profits. 134 131 AGI b. 132 Employment Fund Secretariat-a: 1; 5b; AGI b: 1. The AGI website provides a resource guide on RBC design, monitoring and verification. < https://www.s4ye.org/agi/index.html. Materials include monitoring and verification forms and contract templates. 133 NEF monitoring officers had to visit the training venues twice during a training cycle. 134 Karki et al. 2014: 13, 20. 47 c) Monitoring and verification require ample resources Resources are required to fulfill monitoring and verification needs. Under the EVENT project monitoring teams visited each training stage a total of six times, including verification of training venue, twice during the training, during the skills testing of candidates, and twice for employment and income verification (at three and six months after training.) 135 As stated by the qualitative evaluation of the NEF, the result-based payment system is indeed only possible due to the labor and time-intensive monitoring. Four out of every five NEF secretariat staff were devoted to monitoring and verification.136 d) Sample surveys are typically used for verification of results The use of sample surveys is unavoidable to rationalize the use of resources. Sample sizes were 30 percent in Nepal and 25 percent in Liberia. The results of these surveys were extrapolated to the entire population. Outcomes were verified among the random sample by field interviews and accessing records. Inaccurate claims reduced payments. 137 e) Involve all main actors in validation (i.e. joint monitoring) It is important to engage all relevant actors in monitoring. Joint monitoring of the activities under the EVENT I project included district-level ministry staff, representatives of the vocational training council and third-party independent monitors from an international NGO. The tripartite arrangement provided credible checks and balances and strengthened the certification process and results. The project showed that rigorous monitoring is possible even on a large scale with appropriate coordination, human capital and capacity .138 f) Self-monitoring by providers and part-time field staff promote efficiency Self-monitoring appears to be helpful to promote efficiency. Self-monitoring was developed in training providers through clear process guidelines, measuring tools and training provided by the NEF Secretariat. Self-monitoring was complemented by spot checks by project staff during implementation and random sample verification of the outcomes. Part time field monitors can compensate for seasonal variation in workload . Field monitors needed to be well informed, familiar with project activities and kept up to date on changes. 139 135 Email from Sangeeta Goyal, World Bank task team leader, on May 29, 2019. 136 Hollenbach et al. 2015: 14; Johanson 2015: 16. 137 Employment Fund Secretariat 2015; AGI b: 3. 138 World Bank 2017b, EVENT I, ICR: 33. 139 Employment Fund Secretariat 2015: 10,12. 48 3.6 Challenges identified The issues and challenges identified in the review of literature regarding results-based contracting are virtually the same in short-term skills training as in RBF in general (see Annex B). These include: (1) performance risks may limit provider participation and increase costs; (2) verification requirements can increase costs; and (3) results-based incentives might distort and undermine program goals. a) Performance risks may limit provider participation and increase costs Excessive risk can discourage providers and limit their number. Agreed upon results may not be reached because of factors outside the control of the provider, thereby placing the provider at a disproportionate level of risk. Moreover, under PBF, most if not all activities have to be pre-financed regardless of results. 140 Smaller actors may be particularly affected.141 Recipients must have a large degree of control over outcomes. Otherwise risks may inflate prices. 142 b) Verification requirements can increase costs RBC entails considerable efforts in initial design and additional costs for monitoring and verification. An extra layer of bureaucracy sometimes has to be created to ensure adequate monitoring and evaluation of service delivery. Third parties are often used to conduct assessments. Transaction costs can be extremely high. As an example, in a Rwanda health program a team of five district health professionals used an average of five hours each per month just to validate in one health center. 143 Parallel M&E systems increase costs substantially often without building country capacity. 144 The question must be asked whether the additional administrative costs of RBC are justified by its additional benefits. A question of validity also surrounds the use of self-reported data on employment and income during validation. Information on employment and income in the NEF and EVENT projects is gathered during interviews with the beneficiary, usually at the workplace. Information provided may come from documents, but often is based on recall. Attempts are made, where possible, to corroborate the data through third parties (e.g. employers, community leaders, neighbors, others in similar employment). However, this “triangulation” is not feasible in most cases. Ascertaining employment results is challenging in the informal sector, be it among wage workers or among self -employed, given that all parties have an incentive to collude. Verifying income is particularly difficult for those who are self-employed in the informal sector. The standard should be net income, but this requires information on both gross receipts and operating costs. Given the challenge in accounting details and the difficulties of calculating net income, one project is experimenting with using only gross income as the basis for performance payments. 145 140 IDA/GPOBA 2009: 18. 141 IDA/GPOBA 2009: 18; WB 2007: 27; USAID: 2. 142 Clist and Verschoor 2014: 2. 143 Pearson, et al. 2010: 39. 144 USAID: 2; UNESCO 2018: 7; Eldridge and Palmer 2009: 165; Pearson, et al. 2010: 38. 145 Email communication from Bhanu Pandit, Monitoring and Evaluation Manager, NEF-Helvetas Swiss Intercooperation, on June 7, 2019. 49 c) Results-based incentives might distort and undermine program goals Grittner laid out seven types of unintended negative consequences of results-based incentives. 146 These include: (1) unrewarded activities may be ignored; (2) providers can inflate services to maximize payments; (3) a short-term focus may undercut institutional development and sustainability (4) incentives for quantity may overwhelm delivery quality; (5) RBF offers opportunities for fraud because incentives can be “gamed; (6) “cream skimming” or “cherry-picking” may occur; and (7) incentives may dilute intrinsic professional motivation. The final section presents ways to counteract these adverse consequences. 147 146 Grittner 2013: 19-21. See also Klingebiel 2012; Clist and Dercon 2014: 1; Pearson, et al. 2010: 4,33,36,; SIDA 201511,13; Eldridge and Palmer 2009: 164; Oxman and Fretheim: 79; Carnavan, et al. 2008: 20; USAID: 2. 147 These terms can be defined as focusing on those near performance thresholds at the expense of those below it; (Castro 1999: 33) or selection and concentration on those most likely to succeed rather than those most in need (Holzapfel and Janus 2015: 14). 50 4. OPERATIONAL ADVICE ON RESULTS-BASED CONTRACTING IN DEMAND-DRIVEN TRAINING This section outlines some initial pre-requisites for results-based contracting (RBC) systems, such as having the necessary capacity into place, thinking about what results are sought, how much to pay for their attainment and when, and considering fraud prevention from the outset. The section also summarizes potentially useful best practices in setting up this type of systems along the seven main steps in their design: (1) Defining clear results based on outputs and outcomes that are measurable, realistic, and agreed by all parties; (2) establishing the required administrative struct ures and procedures with clear responsibilities and coordination mechanisms; (3) setting and adequate price that is based on market information and considers the costs of monitoring, and establishing a payment schedule that is simple and allows dealing with performance risks; (4) targeting and selecting the trainees, considering hard-to reach beneficiaries and establishing mechanisms to avoid external interferences; (5) contracting and procuring training providers, establishing accountability measures as well as those that allow the feasibility of implementation; (6) monitoring progress and verifying results, possibly through country data systems, online mechanisms and initial baselines; and (7) minimizing performance risk and gaming, and avoiding distortions and corruption in contractor selection. 4.1 Initial considerations on RBC Anyone considering the use of results-based contracting (RBC) should ask four questions distinctive to the concept: (1) what is the result sought; (2) how will it be measured and verified; (3) how much will be paid for the result and when; and (4) how will fraud be managed. These four questions present special challenges for design, including the need to sharply define intended results, to choose results that are measurable and verifiable at a reasonable cost, to calibrate the incentive price in the appropriate zone (not too high or too low), and to find ways to minimize fraud. In addition, careful consideration should be given to readiness for the use of RBC. Pre-requisites for entering into RBC include these conditions: (1) the minimum administrative capacity to manage the process needs to be in place; “RB approaches should only be implemented if good measures can be identified”, and therefore if there is adequate information by which to measure the results available; (3) there should be a sufficient pool of willing and able service providers, i.e. able to pre -finance operations and manage the training and monitoring processes; and (4) a reasonable unde rstanding of the concept by all parties should exist. 148,149,150 4.2 Steps in the design of results-based contracts RBC requires a careful design based on seven differentiated steps. These include: 1) defining expected results; 2) establishing organization and administrative structures and procedures; 3) setting the price and payment schedule; 4) targeting and selecting trainees/beneficiaries; 5) contracting service providers; 6) monitoring progress and verifying results; and 7) avoiding risks and unintended 148 DFID 2014: 8. 149 Holzapfel and Janus 2015: 11, 48; Clist and Verschoor 2014: 2. 150 Employment Fund Secretariat-a: 1; Carnavan et al. 2008: 19. 51 consequences. Tables 9-15 summarize operational lessons and advice for the seven steps in the design of RBC. Table 9: Step 1- Defining expected results Design Factor Good Practices 1. Defining expected • Concentrate to the extent possible on outputs and outcomes results in skills development. a. Which results are • Since the outputs and outcomes determine payments, they sought? Have the right need to be clearly defined. 152 Apply SMART criteria to results been chosen? results indicators- i.e. specific, measurable, attributable, Where are the results realistic, time-related.153 placed in the chain of • Results also need to be realistic and achievable; taking into causation? account that attainment depends not only on provider b. Which indicators will performance, but also external factors. 154 trigger payments? • The chosen results need to be agreed by all parties at the c. How will the triggers be start. measured? Are the targeted results well defined, measurable and achievable?151 Table10: Step 2 - Establishing organizational and administrative structures and procedures Design Factor Good Practices 2. Establishing organizational • Clearly allocate roles and establish collaborative and administrative structures arrangements. Delineate which units are responsible for and procedures procurement of recipients and providers, implementing, monitoring and validation, managing finances. 156 a. Are roles clearly allocated and • Ensure that the scale of the intended intervention is feasible, collaborative and that sufficient capacity exists for implementation. 157 arrangements established? b. Is the scale of the intended intervention organizationally and administratively feasible for management of the 151 Pearson, et al. 2010: 1; USAID: 9. 152 AGI a: 3; Employment Fund Secretariat 2015: 10. 153 Employment Fund Secretariat 2015: 10. 154 AGI a: 3; AGI b: 1. 156 AGI h. 157 Pearson, et al. 2010: 45; Macharia 2015: 6; DFID 2014: 5, 8. 52 process (e.g. procurement and financial administration)?155 Table 11: Steps 3 and 4 - Setting the price and payment schedule Design Factor Good Practices 3. Setting the price Setting an appropriate incentive price and distributing it along the results chain requires thorough understanding of provider a. What are the costs of costs and needs. It also requires fair terms and thorough service provision to understanding by implementers: achieve intended results? • Collect information on market prices to avoid excessive quotes. Factor in direct costs, including pre-training, during b. How much incentive is training (such as costs of monitoring and reporting) and appropriate above post-training (such as costs of testing and assessment), and service costs? add the incentive. 158, 159 • Give price ceilings to providers before asking for a quote. • Require training providers to disaggregate the proposed costs.160 c. 4. Setting the payment schedule • To the extent possible, link the larger part of the payment with the achievement of the outcomes, and the smaller part a. How should payments to outputs.161 be distributed along stages in the results • Allow advance payments if necessary, to reduce chain? performance risk to providers. b. How much (what % of • Keep the number of payment tranches to the minimum. Too the total payment per many tranches can (i) add to the workload of verification, trainee) for each step? and (ii) dilute incentives by fragmenting the size of payments. • Do not obsess about the exact numbers in the payment schedule. The important thing is to place the overall weight on outputs, to agree with contractors on the output indicators and their measurement, and to establish an effective system of verification. • Discuss in particular how informal employment and self- 155 AGI e 2015: 1. 158 Helvetas 2019 - “Implementation and Monitoring Manual”: 26-7. 159 Feeny: 2 160 Employment Fund Secretariat 2015: 10. 161 Employment Fund Secretariat 2015: 11. 53 employment will be measured. • Define how non-achievement will be handled (i.e. reductions in payments, and/or cancellation of contract.) Table 12: Step 5 - Targeting and selecting trainees Design Factor Good Practices 5. Targeting and selecting • Consider using a lottery, as KYEP did, to minimize the trainees/beneficiaries problem of political interference in selection of trainees. It has the additional benefit of making impact evaluation a. Have the right recipients easier even if decided ex-post. been identified? • Include additional measures for hard-to-reach clients, e.g. b. How to verify self- initial counseling, community support. 162 reported information and ensure that eligibility • Consider providing training venues close to home and child criteria are met? care (e.g. in Liberia EPAG). c. Are additional • Disaggregate indicators and triggers for payment to the incentives necessary to level of specific groups. include hard-to-reach groups by gender, ethnicity or region? Table 13: Step 6 - Contracting- procuring training providers Design Factor Good Practices 6. Contracting-procuring If prices are reasonably known, it is not necessary to have training providers competitive bidding based on lowest price. Instead, base the competition on technical evaluation of merits. a. What method (e.g. competitive bidding, 1. Accountability measures in contracts: based on technical merit • Incorporate results indicators and measurement system and/or price) should be clearly in contracts.163 Ensure to agree on measurement used? of informal employment as it is challenging. b. How can accountability • Enforcement of a contract requires a clear process for be ensured? termination, use of other sanctions and a practical c. How unproductive mechanism for resolving disputes. 164 barriers to entry by • Avoid complexity.165 providers be avoided? • Set out minimum quality criteria for bid evaluation, e.g. 162 Johanson 2015: 18; World Bank 2017b, EVENT I ICR. 163 Employment Fund Secretariat 2015: 11. 164 Loevinsohn 2008: 6. 165AGI 2014: 3; AGI b: 1. 54 d. Feasibility- Can instructor qualifications. sufficient interested • Explain how non-achievement will be handled (See providers be found with Annex D - the model contract incorporates this in two (a) strong labor market ways: payment only for achievements so that under- linkages, and (b) achievement results in lower payment, and possible adequate delivery contract cancellation for non-performance.) capacity (including self- monitoring); and (c) 2. Measures to ensure feasible implementation: sufficient capacity to • Nepal- poor results led to limiting the size of each pre-finance activities? provider contract to no more than 10% of annual e. How can the capacity of enrollment target. 166 service providers be • Target numbers allocated to a provider must be in improved during relation to its implementation capacity. 167 implementation? • The implementation process and quality of provider’s services need to be discussed and agreed with the partners. Neglecting this endangers the achievement of the desired result. 168 • Avoid franchising if possible - Direct communication proved impossible in the franchising model in Nepal. The more distant the training provider is from the project management, the more difficult it is to collaborate closely. 169 • Administrative capacity risks can be mitigated through prior or parallel assistance. 170 • It may be productive to have some “quick wins” (early payments) among the measured results to reduce the risk of RBF for providers. Table 14: Step 7 - Monitoring progress and verifying results Design Factor Good Practices 7. Monitoring progress and • Establish an initial baseline from which to compare progress verifying results and identify benefits. This may involve interviewing a. What steps in the selected participants at the start to determine whether they process will be were employed, in what occupation, type of employer, monitored? number of hours worked and incomes. b. What data are required? • Use of country data systems is best in the long run (and in Are data sources the short run if there is confidence.) 166 Johanson 2015: 15. 167 Employment Fund Secretariat-a: 1 col 2. 168 Employment Fund Secretariat 2015: 11. 169 Employment Fund Secretariat 2015: 12; Johanson 2015: 20. 170 WB 2017: 26. 55 (information systems) • Recognize the tradeoff between thoroughness in data adequate? collection and verification, and costs. Costs of data c. How is provider to generation and verification can be substantial and must be report progress? anticipated and budgeted. d. How will results be • Use an online system for monitoring (as in Nepal and verified? (Options Bangladesh).171 include using/building • Support to verifiers with coping strategies to handle country data systems, pressures from providers. 172 setting up parallel M&E • Minimize pressure on M&E field staff by fully briefing systems specific to the them, organizing them in teams, and transferring field staff program, impact from one region to another so as to remain more objective evaluations, and surveys (NEF practices.) by independent • Use independent third-party verification if necessary but verifiers). recognize that parallel systems will not help resolve e. Are staffing and capacity constraints in the long run. 173 financial resources adequate for this function? f. How can monitoring and verification be made more efficient? Table 15: Step 8 - Minimizing risks and avoiding unintended consequences Design Factor Good Practices 8. Minimizing risks and avoiding unintended consequences in RBC a. Minimizing performance • Consultation and a participatory approach remain important risk to ensure that all parties understand and concur with the approach.174 Ensure full understanding of contract terms; spend ample time familiarizing the training providers with the entire verification process including random sampling/verification with terms explicitly articulated in the contracts.175 • Financial risks can be mitigated by choices in the payment scheme- cash advances, partial payments for partial results and payouts for late achievements. 176 171 Eldridge and Palmer 2009: 164. 172 Employment Fund Secretariat 2015: 12. 173 WB 2017: 26 174 Employment Fund Secretariat-a: 1; Employment Fund Secretariat 2015: 10-11. 175 AGI 2014: 3; Employment Fund Secretariat 2015: 11; Johanson 2015: 21. 176 WB 2017: 27. 56 • Ensure that achievement is fully or mostly under the control of the provider, and that the incentive covers the performance risk, i.e. takes into account possible external factors. • Ensure the contract has dispute resolution mechanisms that are not entirely dependent on the legal system. 177 • As preventive measures (to avoid dropout), use career counseling and ensure systematic interactions with trainees.178 b. Avoiding unintended Incentive payments give an incentive to provide false consequences information and collude. i. Avoiding distortions • Strong verification and biting sanctions help minimize it. Regular and rigorous monitoring at different stages is essential • Use direct rather than indirect indicators. Ensure good measures exist (avoid proxy measures). • Ensure that actors can significantly influence results. 179 ii. Avoiding corruption • Use competitive selection procedures, in contractor selection • Have large contract “lots” that attract more competition and larger service providers, • Make the bid evaluation process more transparent by having clear selection criteria and involving multiple stakeholders.180 Measures to minimize gaming in processing payments: c. Minimizing gaming (e.g. • Nepal EVENT II World Bank supervision team staff reported false reporting, collusion that collusion is minimized by relying on decentralized with employers) monitoring teams and by tracking the employment status of trainees at three and six months. “While this does not remove the problem of collusion, it helps mitigate it because of its relatively long duration.” 181 • Have clear and explicit conditions for payment of invoices.182 177 Loevinsohn 2008: 94. 178 Employment Fund Secretariat 2015: 11-12. 179 Holzapfel and Janus 2015: 48. 180 Loevinsohn 2008: 90. 181 Email communication from Sangeeta Goyal, May 29, 2019. 182 Ibid. 57 • Ensure robust verification of the results that trigger payments.183 • Other mitigation measures pertain to sanctions, e.g. • Reduce payments for discrepancies between self- reported results and verified results. 184 • Poor performers should not be eligible to continue (not allowed to participate in future bidding) or terminate mid contract.185 • Provide termination clause in contract. d. Avoiding cherry-picking • Either do not allow provider to select beneficiaries (e.g. use lottery if there is an excess number of applications, as in KYEP), or supervise carefully the process (as in NEF). • Disaggregate targets to explicitly identify specific types of recipients. • Track individual trainees. Collect data on individuals, not just groups or classes, so as to identify dropouts and hold providers accountable for training/placing those originally recruited.186 • Increase rewards payments for reaching disadvantaged groups or remote regions. 187 An example is the use of differential pricing in Nepal. 188 The combination of a result- based system with financial incentives ensured that the service providers gave priority access to women and other discriminated youth. 189 Source: Author’s compilation. 183 Holzapfel and Janus 2015: 48. 184 AGI 2014: 3,4; AGI b: 3. 185 Employment Fund Secretariat 2015: 11. 186 AGI b: 3. 187 Holzapfel and Janus 2015: 14. 188 AGI 2014: 2. AGI b: 3. 189 Employment Fund Secretariat-a.: 1. 58 5. CONCLUSION AND LESSONS This section lays out some concluding observations on DDT and RBC. Demand driven training relies on employers to identify the existing demands, either directly or indirectly. This model, originated in the Jóvenes programs in LAC, offers many innovations that have been widely valued and incorporated throughout countries. There is ample evidence of the positive impacts that DDT can have on labor market outcomes. Some good practices extracted from the literature include a focus on job placement, the need to pre-identify demand, the close involvement of employers in the process, and the combined provision of theoretical and workplace training. Several evaluations confirm the benefits of RBC. Its results orientation and the capacity to raise motivation through financial incentives , in particular, make it unique. It is most often used in combination with DDT. It can also be useful to determine disbursement in training funds. However, it must be noted that it is not suitable in all circumstances, and that it entails performance and gaming risks that need to be prevented. 5.1 Demand-driven training Demand-driven training, by definition, relies first and foremost on employers who generate the demand. This entails the close involvement of employers in providing leadership to the training, either directly by overseeing the process or indirectly by advising on occupational demand and providing work experiences. However, it is not always possible to achieve close employer involvement in the training process given their interests and constraints. An alternative solution is to have training providers act as intermediaries, identifying promising occupational demand and arranging internships as well as employment for beneficiaries. This alternative works best when service providers are paid on the basis of results. This specific kind of demand-driven approach, pioneered in Latin America and widely adopted afterwards, promoted several innovations. Market mechanisms were applied to overcome the mismatch between the supply and demand for skills.190 First, training was linked with jobs by combined classroom work with work practice. Training providers were required to collaborate with employers in pre-arranging internships or jobs placement for trainees. Second, public financing of training was separated from provision. Public resources were allocated mainly to private training providers based on relevance, quality and cost. This helped achieve efficiency as well as building private training markets. Third, as discussed below, providers were paid for results defined as acquisition of market-relevant skills and gainful employment. According to multiple project evaluations the evidence suggests that the demand-driven interventions model have been successful for the most part in raising employment, earnings and job quality. One comprehensive review of the Jóvenes evaluations concluded that demand-driven training contributed to greater employment, formality and earnings while supply-driven interventions did not.191 Short-term training has been shown to be especially fruitful for women. Salient findings on demand-based training include the following points: • Good labor market analysis is essential to point training in the right direction; rapid market 190 ELLA, “From Supply- to Demand-Led: Labor Training in Latin America”: 3. 191 OECD/ECLAC/CAF: 184. 59 appraisal (RMA) helps. • Employer involvement enhances labor market orientation when they identify needed skills, specify their content, and provide workplace experience. Effecting their involvement, however, is difficult. • Making providers responsible for conducting initial market analysis and then arranging workplace internships made them more keenly attuned to market prospects. • Competition helps build private training markets, but public providers have been reluctant to participate. • Dual training works and comprehensive training with life skills is logically attractive (though currently lacking supportive evidence). Good practices in DDT follow logically from these findings: 1. The focus should be on job placement, with employment or internships arranged in advance. 2. Demand need to be pre-identified. This can be done through labor market surveys or consultation with employers about the knowledge, skills and attributes they seek. 3. Employers must be closely involved in program design and/or implementation - not only for their input on the skills required, but also allaying their natural fears about hiring first -time workers. 4. Training should be provided for the specific competencies and skill sets that are in demand; new training content would be created, or an existing curriculum is modified, based on intense discussions and feedback from employers interested in hiring program graduates. 5. Training providers are instrumental in establishing linkages with employers (the J óvenes model). 6. The most effective training involves workplace practice or simulates the workplace environment of employers. 7. In addition, trainers make regular visits to the workplace to help ensure relevant training content. 8. Post-training support is provided, especially for women in non-traditional occupations, and for those in self-employment who may lack confidence. 9. Other program elements, such as recruitment and the selection of candidates, need to be aligned with employer expectations. 10. Service delivery should emphasize agility and adaptability to each employer's hiring situation and each cohort of students. 192 5.2 Results-based contracting Although evidence on the use of RBC is limited, most of the cases show that with appropriate planning and systems in place it can be instrumental in achieving intended outcomes. Compared with DDT, the evidence on the use of RBC in skills development is limited. This identified only nine completed skills development projects that used RBC. The level of evidence varied greatly among them, with the most extensive documentation pertaining to Nepal. However, the skills development projects in which RBC was employed were successful – not only in producing intermediate outcomes (e.g. training completion and number certified) but more importantly in employment impact, as attested by 192 See, for example, Rockefeller a: 17. 60 rigorous evaluations of the NEF, Liberia EPAG and Colombia, as well as process evaluations in EVENT I, Chile Califica and Grenada. The RBC mechanism has been endorsed, often strongly, in the evaluation of projects where it was used. Documentation on six of the cases pointed to the effectiveness of the RBC mechanism (NEF, Liberia, Colombia, Chile Califica, Grenada and Bangladesh.) Even in the cases where no explicit assessment was made of RBC, one can reasonably assume that the mechanism was effective because (1) they generated positive outcomes and (2) these successes depended importantly on RBC incentives, among other factors. It is unlikely that ineffective RBC could have yielded positive project results. Therefore, it is reasonable to conclude that the RBC mechanism augmented the achievement of program goals in the evaluated skills development projects where it was employed. What makes RBC unique is also what makes it effective: the results-orientation and motivation through incentives. Payment incentives helped propel training providers to achieve intended results. RBC helps change the mindset of “traditional” training providers, focusing them on employment and giving them a strong incentive to engage with employers. A results orientation focuses time and attention laser-like on what matters. When results are defined as employment, this gives impetus for better linkages between the training and the job market. The results orientation concentrates on jobs from the start through market assessment (e.g. RMA) and employer links. It focuses attention on recruiting participants who are likely to want jobs. It focuses provider attention on the quality of t raining so that graduates can pass skills assessments. It focuses attention on the actual placement of graduates in employment since that is what triggers final payments. RBC raises effectiveness through motivation. The financial incentive of being paid for results (or, not being paid without the results) motivates training providers, especially those that focus on profit. Motivation leads to better results. The private training providers in the Nepal projects would hardly have remained active in the progra m if it were unprofitable to them. The two mechanisms of demand driven training and results-based financing are highly compatible and were used in conjunction in almost all the projects reviewed. The findings support the hypothesis that demand-driven training can be made more effective through RBC. Joint use in all probability increased synergy, effects and impact. As stated, pay-for-performance incentives reinforced demand-driven objectives. Requiring training providers to guarantee at least apprenticeships in enterprises helped ensure the relevance of the technical skills acquired, for instance in Colombia and Grenada. RBC is not appropriate in all circumstances. RBC seems particularly appropriate for short-term skills training because outcomes can materialize relatively soon and be measured fairly precisely. However, reforms or training that takes a long time to come to fruition would not be suitable. Longer term results increase performance risk for agents. RBC may be best suited to increasing market relevance and expansion of service delivery, but it does not seem to have a comparative advantage in raising quality of public training systems (compared with operations using disbursement-linked indicators or DLIs – see Annex C to the extent that it operates through non-government providers). However, RBC may be an appropriate mechanism for use in disbursement in training funds. 193 The World Bank-financed Ghana Skills Development Fund (SDF) did not use results-based 193 See Johanson 2009. 61 contracting. However, one source makes a convincing case that RBC could have improved the fund’s implementation and outcomes. The SDF would enter into agreements with grantees on the specific targets for training outcomes (e.g. trade certificates obtained by graduates, increased em ployment or improved productivities of firms), and the disbursement of the grant would be dependent on their achievement at the end of the period. Theoretically, this approach would simplify the bidding process and allow more qualified applicants to receive funding. It would also minimize the leakage of funds being used for unintended purposes. It would speed disbursements, increase accountability by grantees and ease difficulties in evaluating proposals. Hence, RBC potentially could enhance the effectiveness of training funds. 194 However, it could also crowd out some small providers who were unable to pre-finance the training. RBC carries two main risks that may make it unworkable in some circumstances. • Performance risk encompasses three types: financial, operational, and demand risk. 195 Performance risk may deter participation by providers or inflate costs. The experience of NEF and EVENT, however, shows that performance risk can be managed. Some 57 providers participated under NEF, and over 80 in EVENT I and II, even though the first su bstantial payment was only made after training completion and testing. • Incentives bring unintended consequences that may not be completely controllable, but that can be moderated. Gaming (false reporting and collusion with employers) is a particular risk with RBC. This possibility needs to be anticipated and steps taken to control it, including explicit payment conditions, robust verification and sanctions. 196 Strong monitoring and evaluation are also key in the early phases of a program, to ensure that ultimate impacts on employment are achieved. 194 Fu and Tu 2013: 29-3, 38. 195 IDA/GPOBA 2009: 13-15. 196 Perrin 2013: ii. 62 BIBLIOGRAPHY 1. Acemoglu, Daron. and Jörn-Steffen Pischke. 1998. “Why do firms train? Theory and evidence,” The Quarterly Journal of Economics, 113, 79–119. 2. Acemoglu, Daron. and Jörn-Steffen Pischke. 1999. “The structure of wages and investment in general training,” Journal of Political Economy, 107, 539–572. 3. Adams, Arvil V. 2014. “The Kenya Youth Empowerment Project: Lessons from Implementation for Scaling Up.” Mid-term Review Report, World Bank. 4. Adams, Arvil V. 2018. “A Global Review of Labor Activation Programs and their Effectiveness”. Asian Development Bank. 5. Adams, Arvil Van. 2010. “The Mubarak Kohl Initiative- Dual System in Egypt. An Assessment of Its Impact on the School-to-Work Transition.” GTZ/German Technical Cooperation. http://www.urban-project.lviv.ua/php_uploads/data/articles/ArticleFiles_51.pdf. 6. Asian Development Bank. 2009. “Good Practice in Technical and Vocational Education and Training.” Focus on Education. Mandaluyong City. https://www.adb.org/sites/default/files/publication/28624/good -practice-education-training.pdf. 7. Asian Development Bank. 2016. Bangladesh Skills for Employment Investment Program Periodic Financing Request Report (Tranche 2) and Minor Change to the Facility. Project Number: 42466-016 MFF Number: 0081 November 2016. https://www.adb.org/sites/default/files/project- document/213966/42466-016-pfrr.pdf. 8. Asian Development Bank. 2017a. “Results-based Lending at the Asian Development Bank- An Early Assessment.” Reference SES: REG 2017‐13, https://www.adb.org/sites/default/files/evaluation-document/317151/files/ce-rbl.pdf. 9. Asian Development Bank. 2017b. “Education Sector RBL Programs” Appendix, Linked Document 3 to “Results-based Lending at the Asian Development Bank- An Early Assessment. https://www.adb.org/sites/default/files/linked-documents/3-Education-Sector-RBL- Programs.pdf. 10. Asian Development Bank. 2018. SEIP Tranche 1: Aide Memoire Joint Review Mission. October, Manila. 11. Adoho, Franck, Shubha Chakravarty, Dala T. Korkoyah Jr., Mattias Lundberg, and Afia Tasneem. 2014. “The Impact of an Adolescent Girls Employment Program: The EPAG Project in Liberia.” Policy Research Working Paper 6832, World Bank. http://documents.worldbank.org/curated/en/610391468299085610/pdf/WPS6832.pdf . 12. Adolescent Girls Initiative (AGI) a. 2014. “Results-based approaches to Improve Inclusion and Job Placement.” Jan 2014 https://www.s4ye.org/agi/pdf/Implementation/AGI%20Learning%20from%20Practice%20Result s%20Based%20Contracting.pdf. 13. AGI b. 2014. “Implementing Results-based Contracts”. AGI, A Resource Guide. https://www.s4ye.org/agi/html/Implementation_Implementing_Results-Based_Contracts.html. 14. AGI c. 2014. “Monitoring and Evaluation- Verifying Employment.” AGI, A Resource Guide https://www.s4ye.org/agi/html/Monitoring_and_Evaluation_Verifying_Employment.html . 15. AGI d. 2014. “Design: AGI Lessons Learned.” Promoting Employment and Economic Empowerment, AGI Resource Guide. https://www.s4ye.org/agi/html/Project_Design_Promoting_Employment_and_Economic_Empo 63 werment.html. 16. AGI e. 2015. Adolescent Girls Initiative (AGI). 2015.“Can skills training increase employment for young women? The Case of Liberia” World Bank. http://documents.worldbank.org/curated/en/343811468266115466/pdf/900690BRI0P11000Box3 85299B00PUBLIC0.pdf. 17. AGI f. 2015. “The Role of Skills Training for Youth Employment in Nepal: An Impact Evaluation of The Employment Fund.” AGI, Results Series 2015 http://documents.worldbank.org/curated/en/172771468060558339/pdf/941570BRI00PUB00AGI 00results0series.pdf. 18. AGI h. Implementation Determining Institutional Roles and Responsibilities. https://www.s4ye.org/agi/html/Implementation_Determining_Institutional_Roles_and_Responsibiliti es.html 19. Chakravarty, Shuba, , Mattias Lundberg, Plamen Nikolov. 2015. “The role of training programs for youth employment in Nepal: Impact Evaluation Report on the Employment Fund.” AGI Results Brief. http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2015/02/09/000477144_2015 0209083940/Rendered/PDF/941570BRI00PUB00AGI00results0series.pdf. 20. Alfonsi, Livia, Oriana Bandiera, Vittorio Bassi, Robin Burgess, Imran Rasul, Munshi Sulaiman, Anna Vitali. 2017. “Tackling Youth Unemployment: Evidence from a Labor Market Experiment in Uganda.” STICERD - Development Economics Papers - From 2008 this series has been superseded by Economic Organisation and Public Policy Discussion Papers eopp64, Suntory and Toyota International Centres for Economics and Related Disciplines, LSE. 21. Almeida, Rita, Jere Berhman, and David Robalino, eds. 2012. “The Right Skills for the Job?: Rethinking Training Policies for Workers.” Human Development Perspectives 70908, World Bank. http://documents.worldbank.org/curated/en/535251468156871924/pdf/709080PUB0EPI0067869 B09780821387146.pdf. 22. Attanasio, Orazio, Costas Meghir, and Adriana Kugler. 2011. “Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial.” American Economic Journal: Applied Economics . 3. 188-220. 10.1257/app.3.3.188. 23. Barder, Owen, Rita Perakis, William Savedoff and Theodore Talbot. August 27, 2014. “12 Principles for Payment by Results (PbR) in the Real World”. Center for Global Development. https://www.cgdev.org/blog/12-principles-payment-results-pbr-real-world-0 . 24. Becker, Gary. 1962. “Investment in human capital: a theoretical analysis,” Journal of Political Economy, 70, 9–49. 25. Barder, Owen, Rita Perakis, William Savedoff and Theodore Talbot. 2014. “12 Principles for Payment by Results (PbR) in the Real World”. Center for Global Development.https://www.cgdev.org/blog/12-principles-payment-results-pbr-real-world-0 . 26. Betcherman, Gordon, Karina Olivas and Amit Dar. 2004. “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries.” Social Protection Discussion Paper Series No 0402. World Bank, Washington. http://documents.worldbank.org/curated/en/426901468779104515/pdf/29142.pdf. 27. Betcherman, Gordon, Martin Godfrey, Susana Puerto, Friederike Rother, and Antoneta Stavreska. 2007. “A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory.” Social Protection Discussion Paper No. 0715. World Bank, Washington. 64 http://documents.worldbank.org/curated/en/498381468779418249/pdf/414120Youth0workers0SP007 1501PUBLIC1.pdf. 28. Brenzel, Logan. 2009. “Taking Stock: World Bank Experience with Results-Based Financing (Rbf) For Health”. 54034. World Bank. http://documents.worldbank.org/curated/en/998271468331022932/pdf/540340WP0RBF1T10Bo x345635B01PUBLIC1.pdf. 29. Carnavan, Anne and Jurriën Toonen and Riku Elovainio. December 2008. “Performance Based Financing An international review of the literature.” http://www.bibalex.org/Search4Dev/files/305434/135297.pdf. 30. Carranza, Eliana , Robert Garlick, Kate Orkin, and Neil Rankin. 2019. Job Search, Matching, and Hiring with Two-Sided Limited Information about Workseekers' Skills. Mimeo. 31. Castro, Claudio de Moura. 1999. "Proyecto Joven- New Solutions and Some Surprises." Inter- American Development Bank, Washington, D.C.https://publications.iadb.org/bitstream/handle/11319/4729/Proyecto%20Joven%3a%20New %20Solutions%20and%20Some%20Surprises.pdf?sequence=1&isAllowed=y>>. 32. Chakravarty Shubha, Mattias Lundberg, Plamen Nikolov, and Juliane Zenker. 2016.“The Role of Training Programs for Youth Employment in Nepal Impact Evaluation Report on the Employment Fund.” Social Protection and Labor Global Practice Group and Gender Cross-Cutting Solutions Area, Policy Research Working Paper 7656, World Bank http://documents.worldbank.org/curated/en/766871468001182488/pdf/WPS7656.pdf. 33. Clist, Paul. 2016. “Payment by Results in Development Aid: All That Glitters Is Not Gold. ” The World Bank Research Observer, Volume 31, Issue 2, August 2016, Pages 290– 319. https://doi.org/10.1093/wbro/lkw005. 34. Clist, Paul and Arjan Verschoor. 2014. “The Conceptual Basis of Payment by Results” University of East Anglia. https://assets.publishing.service.gov.uk/media/57a089bb40f0b64974000230/61214- The_Conceptual_Basis_of_Payment_by_Results_FinalReport_P1.pdf. 35. Clist, Paul and Stefan Dercon. 2014b. “12 Principles for Payment by Results (PbR) In International Development”. https://assets.publishing.service.gov.uk/media/57a089d2e5274a27b20002a5/clist -dercon- PbR.pdf. 36. Contemporary Development Center. 2014. “Documentation of Processes, Tools and Approaches Applied by the Employment Fund for Its Collaborative Interventions.” Submitted to Employment Fund Secreatriat, Kathmandu. 37. Crépon, Bruno, Esther Duflo, Marc Gurgand, Roland Rathelot, and Philippe Zamora . 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 128(2), pages 531-580. 38. Crépon, Bruno and Patrick Premand. 2019. "Direct and Indirect Effects of Subsidized Dual Apprenticeships," IZA Discussion Papers 12793, Institute of Labor Economics (IZA). 39. Card, David, Kluve, Jochen and Andrea Weber. 2018. "What Works? A Meta-Analysis of Recent Active Labor Market Program Evaluations." Journal of the European Economic Association, vol 16(3), pages 894-931. 40. Deissinger, Thomas. 2015. “The German dual vocational education and training system as ‘good practice’?” Local Economy. Vol. 30 (5) 557–567. 65 https://pdfs.semanticscholar.org/2d58/05292a4288faa641c2e90648439be286f7be.pdf. 41. Department for International Development (DFID). 2014. “Designing and Delivering Payment by Results Programmes: A DFID Smart Guide.” https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/fil e/352519/Designing-Delivering-PbR-Programmes.pdf. 42. Department for International Development (DFID). 2014b. “Sharpening incentives to perform: DFID’s Strategy for Payment by Results” https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/fil e/323868/Sharpening_incentives_to_perform_DFIDs_Strategy_on_Payment_by_Results.pdf. 43. Department for International Development (DFID). 2015. “Support to the Employment Fund- Annual Review- Summary Sheet.” Kathmandu, Nepal. 44. Development Vision - Nepal. 2012. “Tracer Study of Technical Skills Development Training Graduates, 2008/2009.” May. Prepared for Employment Fund Secretariat. Kathmandu. 45. Development Vision- Nepal. 2013. “Tracer Study of Employment Fund Supported Training Graduates of 2011.” August. Prepared for Employment Fund Secretariat. Kathmandu. 46. Eichhorst, Werner, et al. 2012. “A Roadmap of Vocational Education and Training Systems Around the World”. IZA Discussion Papers, No. 7110, Institute of Labor Economics, Bonn. https://www.econstor.eu/bitstream/10419/69486/1/734002173.pdf. 47. Eldridge, Cynthia and Natasha Palmer. 2009. “Performance-based payment: some reflections on the Discourse, evidence and unanswered questions.” Health Policy Plan. 24:160-6. 48. ELLA (Evidence and Lessons from Latin America). “From Supply- to Demand-Led: Labour Training in Latin America”. Policy Brief- Active Labor Market Policies. http://ella.practicalaction.org/wp- content/uploads/files/130607_ECO_ActLabMarPol_BRIEF1.pdf. 49. Employment Fund Secretariat. 2012. “Employment Fund Operating Guideline”. Helvetas, Kathmandu. 50. Employment Fund Secretariat. 2013. “Implementation Guideline for Micro Enterprise Development for Job Creation.” Helvetas Swiss Intercooperation, Kathmandu. 51. Employment Fund Secretariat. 2014. “Annual Report 2013.” Helvetas Swiss Intercooperation Nepal, Kathmandu. 52. Employment Fund Secretariat. 2015. “Results-based Financing in Technical and Vocational Training- a Step-by-Step Implementation Guide.” Helvetas, Kathmandu. http://www.employmentfund.org.np/wp-content/uploads/2015/05/Results-based- Financing_Implementation-Guide.pdf. 53. Employment Fund Secretariat-a. N.d. “It’s the Employment that Pays Off.” Helvetas Kathmandu https://www.shareweb.ch/site/EI/Documents/VSD/Helvetas%20at%20al.%20 - %20Result%20Based%20Payment%20Systems%20in%20VSD%20-%20Factsheet%20- %20en.pdf and https://www.s4ye.org/agi/pdf/Implementation/OutcomeBasedPayment.pdf . 54. Employment Fund Secretariat-b. N.d. “Rapid Market Appraisal- Handbook for RMA Practitioners.” Helvetas, Kathmandu. https://www.s4ye.org/agi/pdf/Targeting_And_Diagnosis/RMA%20handbook%20Manual.pdf . 55. Fares, Jean and Puerto, Olga Susana. 2009. Towards comprehensive training, No 52188, Social Protection Discussion Papers and Notes, The World Bank, https://EconPapers.repec.org/RePEc:wbk:hdnspu:52188. 66 56. Feeny, Emma. “Implementing WASH Programmes in a Payment by Results Context” SWIFT Consortium, Oxfam, Oxford. https://assets.publishing.service.gov.uk/media/57a3610ae5274a31e0001b61/PbR_Learning_Brie f_web.pdf. 57. Felstead, Alan. 1998. Output-related Funding in Vocational Education and Training: A Discussion Paper and Case Studies. Thessaloniki, Greece: European Center for the Development of Vocational Training (CEDEFOP). http://www.cedefop.europa.eu/files/5080_en.pdf. 58. Finn, Dan and Richard Johnson. 2014. Experience of OECD Countries in Contracting for Employment Services: Lessons for the Kingdom of Saudi Arabia. 59. Forcier Consultants. 2013. “Manual for Conducting Market Assessments: South Sudan.” http://www.mdgfund.org/sites/default/files/YEM_MANUAL_South%20Sudan_ConductingMark etAssessments.pdf. 60. Fritsche, Gyorgy Bela and Robert Soeters and Bruno Messen. 2014. “Performance-based Financing Toolkit.” 85412, World Bank. http://documents.worldbank.org/curated/en/369941468325159289/pdf/854120PUB0978100Box 382147B00PUBLIC0.pdf. 61. Fu, Ning and Shunde Tu. 2013. Innovations in Workforce Training Programs in Ghana: Using Pay for Performance Contracts. Harvard Kennedy School. https://www.hks.harvard.edu/sites/default/files/centers/mrcbg/files/Fu_Tu_Final.pdf . 62. Gelb, Alan and Nabil Hashmi. 2014. “The Anatomy of Program-for-Results: An Approach to Results-Based Aid.” Policy Paper 374, Center for Global Development. https://www.cgdev.org/sites/default/files/anatomy-program-for-results.pdf. 63. Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2011. “Impact Evaluation in Practice”. World Bank. https://www.s4ye.org/agi/pdf/Monitoring_and_Evaluation/Impact_Evaluation_in_Practice.pdf. 64. Global Partnership on Output-Based Aid (GPOBA). 2015. “Paying for Performance: An Analysis of Output-Based Aid in Education.” Results for Development Institute. https://www.gpoba.org/sites/gpoba/files/Docs/Paying_for_Performance_- _An_Analysis_of_Output-Based_Aid_in_Education_R4D_Final.pdf. 65. Global Partnership on Output-Based Aid (GPOBA). 2016. “Output-Based Aid in the Results- Based Financing Universe,” https://www.gpoba.org/sites/gpoba/files/Output- Based%20Aid%20in%20the%20Results-Based%20Financing%20Universe_0.pdf . 66. Global Partnership on Output-Based Aid (GPOBA). 2016b. “Output-Based Aid in Education: What Have We Learned So Far?” Note Number 49. Results for Development Institute. https://www.gpoba.org/sites/www.gpoba.org/files/publication/downloads/OBA49_Educat ion.pdf 67. Global Partnership on Output-Based Aid (GPOBA). 2010. “Output-Based Aid in Education: A Solution for Quality Education.” Note No. 22, Inga Murariu and Lars Johannes, Results for Development Institute. https://www.gpoba.org/sites/www.gpoba.org/files/publication/downloads/OBApproaches -on- Education-3-23-10screen.pdf. 68. Government of Colombia. 2008. “Subprograma Jóvenes en Acción: Consultoría para la evaluación de impacto del Subprograma Jóvenes en Acción.” Evaluación de Políticas Públicas No.9. Departamento Nacional de Planeación; Dirección de Evaluación de Políticas Públicas. Bogota. http://guia.oitcinterfor.org/sites/default/files/experiencias/Colombia_SENA_Subprog_jov_acc.pdf . 69. Grittner, Amanda Melina. 2013. “Results-based Financing-Evidence from performance-based 67 financing in the health sector.” Discussion Paper 6/2013, German Development Institute, Bonn. https://www.die-gdi.de/uploads/media/DP_6.2013.pdf. 70. Helvetas Nepal. 2019. “Implementation and Monitoring Manual.” IFAD Rural Enterprise and Remittance Project, SAMRIDDHI; Sub-component 1.3- Decent Jobs. Kathmandu. 71. Hollenbach, Pia, Annabelle Jaggi, Ulrike Müller-Böker, Sunna Seithel, Jeevan Lohani, Barsha Bhetwal, Uttam Prasad Upadhya and Kamana Upreti. 2015. Employment Fund NEPAL: Qualitative impact evaluation report. https://www.zora.uzh.ch/id/eprint/139680/1/2015_QualitativeImpactStudy.pdf . 72. Holzapfel, Sarah and Heiner Janus. 2015. “Improving Education Outcomes by Linking Payments to Results”. Bonn, German Development Institute. https://pdfs.semanticscholar.org/d44f/0c90d80548635590da22cd6f260f0f7731c4.pdf. 73. Honorati, Maddalena and Thomas P. McArdle. 2013. “The Nuts and Bolts of Designing and Implementing Training Programs in Developing Countries”. Social Protection and Labor Discussion paper No. 1304. World Bank, Washington, DC. 74. Honorati, Maddalena. 2015. “The Impact of Private Sector Internship and Training on Urban Youth in Kenya”. World Bank Social Protection and Labor Global Practice Group. Policy Research Working Paper 7404. World Bank. http://documents.worldbank.org/curated/en/468501468185042536/pdf/WPS7404.pdf . 75. IDA and GPOBA. November 2009. “A Review of the Use of Output-based Approaches,” IDA 15 Mid-Term Review. https://ida.worldbank.org/sites/default/files/pdfs/oba_ida15mtr.pdf . 76. ILO CINTERFOR. n.d. “Jóvenes en acción (Youth in action). SENA. Colombia”. https://www.oitcinterfor.org/en/experiencia/j%C3%B3venes-acci%C3%B3n-youth-action-sena- colombia. 77. ILO (International Labor Organization). 2006. “Rapid Market Appraisal- A Manual for Trainers.” December, Banda Acheh, Indonesia. https://www.ilo.org/wcmsp5/groups/public/---asia/---ro- bangkok/---ilo-jakarta/documents/publication/wcms_124488.pdf. 78. Ibarrarán, Pablo and David Rosas Shady. 2009. “Evaluating the impact of job-training programs in Latin America: evidence from IDB funded operations.” Journal of Development Effectiveness 1: 2, 195-216. https://doi.org/10.1080/19439340902918094. 79. Inoue, Keiko, Emanuela di Gropello, Yesim Sayin Taylor, and James Gresham. 2015. Out -of- School Youth in Sub-Saharan Africa: A Policy Perspective. Directions in Development. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/293771468004845225/pdf/947410PUB0978100Box3854 16B00PUBLIC0.pdf. 80. Inter-Regional Inequality Facility. 2006. “Youth Training- Chile and Argentina.” Policy Brief 8. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/4068.pdf. 81. International Youth Foundation and USAID. N.d. “Ensuring Demand-Driven Youth Training Programs How to Conduct an Effective Labor Market Assessment”. https://www.iyfnet.org/sites/default/files/library/Labor_Mkt_Assess.pdf . 82. Johanson, Richard. 2009. “A Review of National Training Funds.” SP Discussion Paper 922. The World Bank. http://documents.worldbank.org/curated/en/893311468178131914/pdf/521870NWP0Box345554 B001PUBLIC100922.pdf. 83. Johanson, Richard and Alesha Sharma. 2013. “Review of Employment Fund Project”. May, Swiss 68 Development Corporation, Kathmandu. 84. Johanson, Richard. 2015. “Capitalising on Experiences of the Employment Fund: Phase Capitalisation Report,” 29 September. Helvetas, Kathmandu. 85. Karki, Vishnu and Laxman Acharya, Sagar Mani Neupane. 2014. “Employment Fund- Good Practices and Critical Factors in Monitoring and Online Database System. ” National Institute for Research and Training, Kathmandu. 86. KfW. 2008. “Innovative approaches to promoting vocational training and employment.” Executive summary Guidelines for German Financial Cooperation projects and programmes. https://www.kfw-entwicklungsbank.de/Download-Center/PDF-Dokumente-Sektoren- Berichte/2008_06_Berufsbildung_kurz_E.pdf. 87. Klingebiel, S. 2012. “Results-based Aid: New aid approaches, limitations and application of good governance.” Bonn, German Development Institute. https://www.die- gdi.de/uploads/media/DP_14.2012.pdf . 88. Kluve, Jochen. 2016. “A review of the effectiveness of active labour market programmes with a focus on Latin America”, ILO Research Department Working Paper, No. 9, International Labour Organization, Geneva. 89. Kluve Jochen, David Card, and Andrea Weber. 2015. “What works? A meta-analysis of recent active labour market programme evaluations”, NBER Working Paper, No. 21431, National Bureau of Economic Research, Cambridge, US. 90. Kluve Jochen, Puerto Susan, Robalino David, Rother Friederike, Weidenkaff, Felix, Stoeterau, Jonathan, Romero, Jose Manuel, & Witte Marc. 2016. “Do Youth Employment Programs Improve Labor Market Outcomes? A Systematic Review. IZA Discussion Paper No. 10263. 91. Kluve Jochen, Puerto Susana, David Robalino, Jose Romero, Rother Friederike, Stöterau Jonathan, Weidenkaff Felix. and Witte Marc. 2017. “Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions.” Campbell Systematic Reviews, 13: 1-288. doi:10.4073/csr.2017.12. 92. Kugler, Adriana, Maurice Kugler, Juan Saavedra, and Luis Omar Herrera Prada. 2019 (forthcoming Journal of Human Resources ). “Long-Term Direct and Spillover Effects of Job Training: Experimental Evidence from Colombia.” Working Paper 20617, National Bureau of Economic Research, Cambridge, MA. https://www.nber.org/papers/w21607.pdf. 93. Loevinsohn, Benjamin. 2008. “Performance-based Contracting for Health Services in Developing Countries: A Toolkit,” Health, Population and Nutrition Series, 44821, World Bank Institute https://openknowledge.worldbank.org/bitstream/handle/10986/6481/448210PUB0Box0101offici al0use0only1.pdf?sequence=1&isAllowed=y. 94. Lohani, Jeevan Raj. 2017. Employment Fund: Skills for Reconstruction: Endline Study Report. RIDA International, Kathmandu; Supplement to Project Document: Employment Fund Phase 2, Skills for Reconstruction. 95. Macharia, Cynthia. 2015. “Learning from Successful Models: Output-based Financing, Experiences from Kenya and Asia”. GIZ. Powerpoint Presentation 12/3/2015. https://www.slideshare.net/achapkenya/output-based-financing-by-cynthia-macharia-giz . 96. Macaulay, Fiona. 2016. “Why is demand-driven training like a long-term marriage?” November 14. https://www.devex.com/news/opinion-why-is-demand-driven-training-like-a-long-term- marriage-89141 . 69 97. Martin, John P. and David Grubb. 2001. “What Works and for Whom: A Review of OECD Countries' Experiences with Active Labour Market Policies”. Swedish Economic Policy Review, Vol. 8, No. 2, Fall 2001, pp. 9-56. . Available at SSRN: https://ssrn.com/abstract=348621 or http://dx.doi.org/10.2139/ssrn.348621. 98. Meessen, Bruno, and Agnes Soucat, Claude Sekabaraga. 2010 “Performance-based financing: just a donor fad, or a catalyst towards comprehensive health-care reform?” Bulletin of the World Health Organization. 89:153-156. 99. Mumssen, Yogita, Lars Johannes, and Geeta Kumar. 2010. “Output-based Aid- Lessons Learned and Best Practices.” Directions in Development- Finance- 53644. World Bank, Washington, D.C. http://documents.worldbank.org/curated/en/206041468337170198/pdf/536440PUB0outp101Offi cial0Use0Only1.pdf. 100. Musgrove, Philip. 2011. “Financial and Other Rewards for Good Performance or Results: A Guided Tour of Concepts and Terms and a Short Glossary.” World Bank. https://www.rbfhealth.org/sites/rbf/files/RBFglossarylongrevised_0.pdf . 101. OECD/ECLAC/CAF. 2016. “Latin American Economic Outlook 2017: Youth, Skills and Entrepreneurship.” OECD Publishing. Paris. https://www.oecd.org/dev/americas/E- Book_LEO2017.pdf. 102. O’Brien, Thomas and Ravi Kanbur. 2013. “The operational dimension of results-based financing”. Cornell, Working Paper 2013-15. https://ecommons.cornell.edu/bitstream/handle/1813/57773/Cornell_Dyson_wp1315.pdf?sequen ce=1&isAllowed=y Also: Public Administration and Development. Vol 34, Issue 5, pp 345-358. December 2014. 103. Oxman, Andy D. and Atle Fretheim. 2009. “Can paying for results help to achieve the Millennium Development Goals? Overview of the effectiveness of results -based financing. Journal of Evidence-Based Medicine. 2009; 2:70-83. https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1756-5391.2009.01020.x. 104. Pearson, Mark and Martin Johnson and Robin Ellison. March 2010. “Review of major results- based aid (RBA) and results-based financing (RBF) schemes”. DfID http://www.oecd.org/dac/peer-reviews/Review-of-MajorRBA-and-RBF-Schemes.pdf 105. Perakis, Rita and William Savedoff. 2014. “An Introduction to Cash on Delivery Aid for Funders.” CGD Policy Brief, Center for Global Development, Washington, D.C. https://www.cgdev.org/sites/default/files/introduction-cash-on-delivery-aid-funders.pdf. 106. Perakis, Rita and William Savedoff. 2015. “Does results-based aid change anything? The roles of pecuniary interests, visibility, accountability and discretion in four case studies.” (CGD Policy Paper 052). Washington, DC: Center for Global Development https://www.cgdev.org/sites/default/files/CGD-Policy-Paper-52-Perakis-Savedoff-Does-Results- Based-Aid-Change-Anything.pdf. 107. Perrin, Burt. 2013. “Evaluation of Payment by Results (PBR): Current Approaches, Future Needs.” Working Paper 39, DFID, London. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/fil e/213938/payment-results-current-approaches-future-needs.pdf. 108. Puerto, Olga Susana. 2007. “International Experience on Youth Employment Interventions: The Youth Employment Inventory.” World Bank, Washington, D.C. http://siteresources.worldbank.org/INTLM/Resources/390041- 1188243960961/YEI_LessonsLearned_rev07-06.pdf 70 109. Results for Development Institute. 2015. “Paying for Performance: An Analysis of Output- Based Aid in Education” Study Commissioned by Global Partnership on Output-Based Aid (GPOBA), https://www.r4d.org/wp-content/uploads/Paying-for-Performance-An-Analysis-of- Output-Based-Aid-in-Education-.pdf. 110. Rockefeller Foundation and Making Cents International. a. 2017. “Scaling Demand-Driven Training Programs: A Framework.” http://docs.wixstatic.com/ugd/cf5fc8_f06c8b6b14ce4371a7c64acac1bd183f.pdf. 111. Rockefeller Foundation and Making Cents International. b. 2017. “Demand-driven Training For Youth Employment toolkit”. https://youtheconomicopportunities.org/sites/default/files/contentupload/73868/Making_Cents_1 01717_Interactive_V3s.pdf. 112. Savedoff, William. 2016. “A Global Offer for Learning (GOL): Based on Experiences with Paying for Results. Center for Global Development, Washington, D.C. https://www.cgdev.org/sites/default/files/CGDGlobalOfferforLearningFinal.pdf . 113. SIDA. 2015. “Results-based Financing Approaches (RBFA)- What Are They?” https://www.sida.se/contentassets/1b13c3b7a75947a2a4487e2b0f61267c/18235.pdf . 114. UNESCO. 2018. “Walk before you run: the challenges of results-based payments in aid to education.” Policy Paper 33, Global Education Monitoring Report. http://unesdoc.unesco.org/images/0026/002611/261149E.pdf. 115. UNEVOC. “Chile Joven, Job Training Programs in Latin America.” https://unevoc.unesco.org/fileadmin/user_upload/pubs/Chile%20Joven,%20Job%20Training%20 Programs%20in%20Latin%20America.pdf. 116. USAID and Palladium. n.d. “Pay for Results in Development A Primer for Practitioners”. https://www.usaid.gov/sites/default/files/documents/1865/Pay_for_Performance_Primer_Final.p df. 117. Vooren, Melvin, Carla Haelermans, Wim Groot., and Henriette Maasen van den Brink. 2017. “The Effectiveness of Active Labor Market Policies: a Meta Analysis,” Tier Working Paper Series WP 17/01, Amsterdam, https://www.researchgate.net/publication/313925698_The_Effectiveness_Of_Active_Labor_Mar ket_Po licies_A_Meta-Analysis 118. Women’s Refugee Commission. 2008. “Market Assessment Toolkit for Vocational Training Providers and Youth: Linking Vocational Training Programs to Market Opportunities”. Prepared by the School for International and Public Affairs, Columbia University. https://www.womensrefugeecommission.org/images/zdocs/Market_Assessment_Toolkit_rev._20 13.pdf. 119. World Bank. 2002. “Lifelong Learning and Training Project (Chile Califica) - Project Appraisal Document.” Report No. 23632-CH . http://documents.worldbank.org/curated/en/275461468769919343/pdf/multi0page.pdf . 120. World Bank. 2006. World Development Report 2007: Development and the Next Generation. https://openknowledge.worldbank.org/handle/10986/5989. 121. World Bank. 2008. OECS Skills for Inclusive Growth Project - in Support of the Second Phase of the OECS Skills for Inclusive Growth Program - Grenada - Project Appraisal Document. Report No. 43942-GD. http://documents.worldbank.org/curated/en/658121468250308490/text/439420PAD0P0951y100I DASecM200810731.txt. 71 122. World Bank. 2010. Kenya Youth Empowerment Project (KYEP) – Project Appraisal Document (PAD). Report: 53090-KE. 2010. documents.worldbank.org/curated/en/973531468285056714/pdf/530900PAD0revi1IC10IDA1R2 0101005912.pdf . 123. World Bank. 2010. “Empowering girls and young women in Liberia”. Document 81677, Adolescent Girls Initiative (AGI) gender equality as smart economics series. Washington DC ; World Bank. http://documents.worldbank.org/curated/en/598561468272721702/Empowering- girls-and-young-women-in-Liberia. 124. World Bank. 2010b. Lifelong Learning and Training Project (Chile Califica) - Implementation Completion and Results Report. Report No: ICR1390. http://documents.worldbank.org/curated/en/106131468236083494/pdf/ICR13900P068271Official0us e0only191.pdf 125. World Bank. 2011b. Nepal. Enhanced Vocational Education and Training Project (EVENT)- Project Appraisal Document (PAD). Report: 59308-NP. March 28, 2011. http://documents.worldbank.org/curated/en/962331468053350208/pdf/593080PAD0P1041OFFICIA L0USE0ONLY191.pdf . 126. World Bank. 2012a. “Designing a Results Framework for Achieving Results: a How-to Guide”. Independent Evaluation Group. https://www.s4ye.org/agi/pdf/Further_Reading/designing_results_framework.pdf . 127. World Bank. 2012b. “Punjab Education Sector Project (PESP) - Implementation Completion and Results Report.” Report No: ICR2432. http://documents.worldbank.org/curated/en/249571468327380807/pdf/NonAsciiFileName0.pdf . 128. World Bank. 2014. “Implementation Completion and Results Report. OECS Skills For Inclusive Growth Project.” Report No. ICR2872 http://documents.worldbank.org/curated/en/150251468037574347/pdf/ICR28720P095680C0disclose d030100140.pdf. 129. World Bank. 2014b. “Project Appraisal Document- Sri Lanka- Skills Development Project.” Report No. PAD 888. May 27, 2014. http://documents.worldbank.org/curated/en/661811468169771793/pdf/PAD8880PAD0P13010Box38 5222B00OUO090.pdf. 130. World Bank. 2015. “The Rise of Results-Based Financing in Education.” 2015. http://www.worldbank.org/content/dam/Worldbank/Brief/Education/RBF_ResultsBasedFinancin g_v9_web.pdf. 131. World Bank. 2015b. Project Appraisal Document- Punjab Skills Development Project. Report No. PAD1010. April 30, 2015 http://documents.worldbank.org/curated/en/634231468096559464/pdf/PAD10100PAD0ID08101 0P13019300OUO090.pdf . 132. World Bank. 2016. Kenya Youth Empowerment Project - Implementation Completion and Results Report. http://documents.worldbank.org/curated/en/710301471883981006/pdf/ICR00003730-08042016.pdf. 133. World Bank. 2016b. Project Appraisal Document- Bosnia and Herzegovina- Employment Support Project.” Report No; PAD 1339. World Bank. 72 http://documents.worldbank.org/curated/en/457221482159012451/pdf/1482159025082- 000A10458-PAD-December-12-2016-12152016.pdf. 134. World Bank. 2017 “Results-Based Financing in Education: Financing Results to Strengthen Systems.” http://documents.worldbank.org/curated/en/715791489054110215/pdf/113265- REVISED-RBF-Approach-Final-Digital-Apr21.pdf. 135. World Bank. 2017b. Nepal - Vocational Education and Training Project. Implementation Completion and Results Report. (P104015) Report No: ICR00004135, December 28, 2017 http://documents.worldbank.org/curated/en/458241514911318490/pdf/ICR00004135-Nepal- 12282017.pdf. 136. Ziderman, Adrian. 2003. “Financing Vocational Training in Sub-Saharan Africa.” Africa Region Human Development Series, World Bank. http://siteresources.worldbank.org/INTAFRICA/Resources/Financing_VET_SSA.pdf . 137. Ziderman, Adrian. 2016. “Funding Mechanisms for Financing Vocational Training- An Analytical Framework,” IZA Policy Paper 110, 2016. http://ftp.iza.org/pp110.pdf. 73 Annex A: Information on Employment and Training Programs Table A.1- Overview of Cases Country Project Name RBC DDT Duration Number trained Target Actual 1. Kenya Kenya Youth Empowerment Project X 2010-16 10,000 13,300 (KYEP) 2. Chile Chile Joven X X 1991-02 100,000 110,000 3. Argentina Proyecto Joven X197 X 1994-00 100,000 112,000 4. Nepal Employment Fund (NEF) X X 2008-15 105,400 102,932 5. Nepal Enhancement of Vocational X X 2011-17 65,500 71,500 Education and Training (EVENT I) 6. Liberia Economic Empowerment of X X 2010-12 2500 2100 Adolescent Girls and Young Women (EPAG) 7. Chile Lifelong Learning and Training X198 X 2002-09 120,000 148,350 Project (Chile Califica) 8. Colombia Jovenes en Accion (JeA) X X 2001-05 n.a. 81,600 9. Bangladesh Skills for Employment Investment X X 2014-19 254,700 227,600 Project Tranche I (SEIP I) 10. Grenada199 Skills for Inclusive Growth X X 2009-13 500 886 A.1. Kenya Youth Empowerment Project (KYEP) In 2010, the Government of Kenya, with funding support from the World Bank, launched the Kenya Youth Empowerment Project (KYEP). The program was designed in response to the issue raised by Kenyan employers that many youths who come out of schools and training centers lack the relevant work experience and competencies needed for employment. The KYEP program was designed to incorporate the success factors that have been identified in the international evidence, such as the Jovenes programs in Latin America. However, there was limited evidence from similar youth interventions, particularly from low-income countries. KYEP provided an opportunity to evaluate the effectiveness of the approach in a pilot. It called for a comprehensive impact evaluation to create specific evidence from Kenya as a guide for future possible expansion. Objective The main objective of KYEP was to improve youth employability, by providing youth with work experience and skills demanded by employers. The project development objective was “to support the Government of Kenya (GoK) efforts to increase access to youth-targeted temporary employment programs and to improve youth employability”. The intermediate outcome is to provide youth with work experience and skills through the creation of internship and relevant training in the formal and informal sector with priority given to growing industries. 197 Lack of job opportunities led to changes in the payment schedule: final payment changed to completion of internship instead of payment on employment. 198 The Chile Califica program did not aim at employment, but only at increasing qualifications. 199 The Grenada indicator on employment was dropped during the Project’s restructuring. 74 Approach Design features included: (1) a combination of classroom-based and on-the-job training, (2) the demand- driven design of skills training curricula, (3) the key role of the private sector in the implementation of the program, and the (4) inclusion of like skills as increasingly important in modern job markets. The pilot training and internship program was designed and implemented to address key constraints on both demand and supply in the labor market. On the demand side, the program mobilized private employers, gave them incentives to accept interns and teach them job‐relevant skills, and encouraged them to create jobs (and retain interns who successfully completed the program). On the supply side, the program aimed to reduce skills constraints in the six sectors where growth is expected according to Vision 2030, Kenya’s national development strategy, by providing youths with job‐relevant skills (technical and life skills) as well as workplace experience. To ensure private sector ownership of the program and the effectiveness of the program’s demand‐ driven design, the intervention operated as a private‐public partnership. The Government of Kenya set the strategic vision for youth employment and financed the program while the Kenya Private Sector Alliance (KEPSA), a policy advocacy organization for private employers, defined the competencies needed by employers in the six growth sectors, mobilized and motivated employers to participate in the program, recruited training providers on a competitive basis, and managed and implemented the overall program. To ensure that the program was demand‐driven, employers’ organizations in each sector were invited to identify their skill needs and to develop sector‐specific training plans. Activities supported Private Sector Internships and Training. These activities were conducted under KYEP component 2 and supported four activities: (i) developing and implementing a youth internship program; (ii) providing training and technical assistance to unemployed youth to obtain skills in potential growth sub-sectors identified in the Vision 2030; (iii) training and upgrading the skills of selected master craftsmen in areas aligned with or relevant to youth employment; and (iv) monitoring and evaluating lessons learnt from the pilot for integration into existing and future youth employment programs. Target groups The program targeted vulnerable youths defined as males and females between ages 15 and 29 years who have a minimum of eight years of schooling, have been out of school for at least one year, and are not in employment at the time of their application to the program (and therefore no one was investing in their skills). The failure to invest in skills at this critical stage of the lifecycle subjects these youths to later problems with finding meaningful work and increases the risks of long-term unemployment and falling into poverty. While not easy to verify, being out of school and being without a permanent job at the application stage were the main characteristics used to identify vulnerable youths or those at risk of longer‐term unemployment or of becoming stuck in low‐productivity jobs. The minimum of eight years of schooling was included as a way to ensure that the participants would have had a basic education and thus would be able to benefit from further training. The share of interns with tertiary education (i.e. university graduates) 75 was to be capped at 40 percent to avoid program capture by this education group and provide an opportunity to compare program results for different levels of education. The selection process was to balance participation by gender. The selection process started with advertising the program in the news media, on KEPSA’s website and by District Youth Officers. Given extensive youth unemployment demand for the training and internships exceeded spaces available. After determining eligibility, selection was done by lottery to ensure transparency and fairness. Training providers Training providers developed modular, competency-based skill training for trainees. The program financed third-party training requested by employers through their KEPSA Sector Boards as part of a sector training plan prepared with KEPSA. This third-party training is distinguished from that offered on the job by employers. Employers through their Sector Boards contributed to development of a Sector Training Plan that was used by KEPSA in procuring training services. The competencies developed with training were established by KEPSA Sector Boards in consultation with their employer members and expressed as part of the Sector Training Plan. Training plans included an assessment of sector training needs, the type of training and competencies to be delivered, and how the training was to be delivered and its duration. KEPSA prepared requests for procurement of training services, based on individual Sector Training Plans, stating the quantity and type of training services to be procured, location of delivery and other conditions of service. KEPSA advertised for expressions of interest from qualified training organizations, as defined above, to provide services for individual Sector Training Plans. KEPSA reviewed and validated these expressions of interest and developed a roster of pre-qualified training organizations based on pre-agreed criteria. Training content The program focused on training in the six growth sectors identified in the government’s Vision 2030 – energy, finance, ICT, manufacturing, and tourism, in the formal sector, and micro and small enterprises in the informal sector. Training and internships were to be offered in six cycles of six months duration over the four-year pilot. Classroom-based training courses included three phases: • Life skills training (two weeks) aimed at strengthening life and other non‐cognitive skills ( such as conflict resolution and communication skills). • Core business training (five weeks) focused on communication, customer care, entrepreneurship, basic computer use, and office practices, among others. Interns placed in firms operating in the five formal sectors received the five weeks of training while interns placed in the informal sector received three weeks of training coupled with a two‐week module on entrepreneurship skills training. • Sector‐specific training to promote specific technical skills (five weeks) in each of the five formal sectors. For interns placed in the Jua Kali sector, the training plan consisted of three weeks of sector‐specific training supplemented by an additional two weeks of work experience. Work experience (12 weeks) in selected private firms. After the training, youth were provided with 12 weeks of work experience in the formal and informal sectors. (Employers were expected to provide on-the- 76 job training and mentorship. In the informal “Jua Kali” sector, internships consisted of apprenticeship training by master craftsmen.) Stipends High training costs in Kenya are a barrier that prevents many young people from accessing skills development programs. The project offered a small monthly stipend of KES 6,000 to youths (equivalent to US$70) to compensate them for their transportation and food costs during the six months of the program. Employers were also given monthly compensation of KES 3,000 (equivalent to US$35) to offset the time spent overseeing the interns and to reimburse them for the costs of materials during the 12 weeks of the internships. Implementation The program was carried out in six cycles of six months each. The Kenya Private Sector Alliance (KEPSA), a private sector led apex organization, was responsible for implementation and a project unit (PMU) was created within it to handle the work. The KEPSA PMU was responsible for overall implementation of the program, including selection participants, procuring the third-party training providers, mobilizing firms in the private sector to participate in the program, matching youths with employers for the internships, paying the stipends, and monitoring the training and internship experiences. Implementation of the component started in Nairobi, and Mombasa was added after the first six-month program cycle. It was to expand to one rural area in the second year of project implementation. Instead, the component was expanded to Kisumu due to difficulties in finding a large base of employers to offer internships in rural areas. Tracer studies were conducted after each cycle to determine the extent to which graduates were finding employment. Project results and outcomes Table A.2: Project results Indicators Original Actual value target value achieved at completion Project Development Objective (PDO) - To support government efforts to improve youth employability • Percentage of interns who complete the internship and are 35% 78% immediately employed by their internship employer or who have found employment with a new employer or are starting a business (disaggregated by gender • Percentage of interns employed or self-employed six months after 50% 76% internship completion (disaggregated by gender) 77 Intermediate outcome- Provide youth with work experience and skills through the creation of internship and relevant training in the formal and informal sector with priority given to growing industries. • Number of youth participating in the internship and training 10,000 13,305 program. • Total number of internship weeks provided. 200,000 289,781 • Total number training weeks provided by third party providers 100,00 148,273 • Total number of youth completing life skills training 15,000 19,532 • Completion rate of internship 95% 86% • Percentage of participants satisfied with internship and training 95% 86% program • Percentage of employers satisfied with internship and training 95% 84% Source: World Bank, 2010, KYEP PAD: 47-48; and World Bank, 2016, KYEP ICR: viii-xi. The PDO to support the government’s efforts to improve youth employability was met, as evidenced by the average overall employment rate of 75 percent against the target of 50 percent. According to results from Cycle 5, 84 percent of young men and 73 percent of young women were employed six months after completion of their internships. The average of 76 percent exceeds the PDO target of 50 percent. The employment rate has increased in each cycle, beginning with 24 percent in Cycle 1. The Cycle 5 employment rate increased sharply from a 53 percent average in Cycle 4. Based on the implementation experience after each cycle, important adjustments were made to the length, type of training, and duration of internships, which proved effective and resulted in improved targets. With regard to the project outputs set, the Project succeeded at exceeding its targets for training and internships. The initial target of 10,000 youth participating was exceeded with 13,305 (6,542 female)200 completing the training and internships. The targeted number of weeks of training and internships was also exceeded. In addition, the total of 19,532 (9,235 female) youth who successfully completed the life skills training exceeded the set target of 15,000 youth. However, with regard to internship completion, the rate of 85 percent fell slightly below the target of 95 percent.201 According to the project tracer surveys, the project was successful at improving the average overall employment rate of its participants: 75 percent of participants secured employment within six months of completing the internship. The Project exceeded the set target of 50 percent employment. Additionally, with each subsequent cycle, there was an increase observed in the proportion of the interns completing the program and being employed immediately after their internship completion. This happened at an average of 52 percent employment between Cycles 1 and 6, against the set target of 30 percent for years 1 and 2 and 35 percent in years 3 and 4 of the Project. The beneficiary assessments conducted in Cycles 1, 2, and 4 revealed that the majority of youth and employers participating in the training and internships expressed satisfaction with the program. 200 At the time of the mid-term review it was observed that the pool of eligible applicants had not been stratified and the selection from this pool by lottery resulted in 58% males and 42% females (Adams 2014: 12). 201 World Bank 2016: 32. 78 According to these assessments conducted for Cycles 1, 2, and 4, the majority of the youth participating in the training and internships expressed satisfaction with the program, with the percentage in Cycle 4 reaching 86 percent satisfaction compared to 79 percent in Cycle 1 and 85 percent in Cycle 2. The beneficiary assessments also revealed that employers’ satisfaction with the program rose steadily, from 64 percent in Cycle 1 to 78 percent in Cycle 2 and 84 percent in Cycle 4. However, the overall satisfaction rate among youth and employers participating in Component 2 fell slightly below the 95 percent target.202 Additional accomplishments • The internship pilot demonstrated the benefits of public-private partnerships for addressing youth employment. Evidence of the positive employer response is reflected in the majority that participated in more than once cycle. • Life skills training produced an unusually positive response from interns and employers. Youths and employers valued the life skills training even above sector-specific technical training. • The selection process by lottery proved to be transparent and fair. However, failure to stratify applicants by gender initially produced under-representation of young women.203 • The project was largely successful in getting youths to open bank accounts, although initial verification problems led to delays in problems.204 Overview of findings from the impact evaluation Educational background of beneficiaries.205 The program benefitted more educated youths than expected. Overall, the program seems to have reached mostly tertiary dropouts rather than secondary dropouts, as 18 percent of the sample had not completed secondary education, almost 29 percent had completed secondary school, and 53 percent had some tertiary education, having either started or completed college and/or university. Overall employment. The impact evaluation carried out on participants and a control group in Cycle 2 demonstrated an important positive impact. Results from the impact evaluation of the Cycle 2 show that 80 percent of men, and 50 percent of women, who were placed in internships were in paid work 14 months after internship completion, compared with an average of 69 percent of men and 43 percent of women in the control group, equivalent to 10.8 and 6.7 percentage points increase in the probability of being employed among male and female participants respectively. This gain increased to 14.2 percent for those completing the full training program. Among young women, there was a 6.7 percent increase in employment and an 8.7 percent increase for those who completed the full program. This impact evaluation finds that the full program has had significant positive results, specifically an increase of 15 percent in current employment among both male and female participants. Male employment. The KYEP program had a substantial positive impact for males on the probability of having paid wage employment with a written contract, which also translated into a significant increase in the number of hours worked per week. Given that 64 percent of males in the control group had paid 202 World Bank 2016: 31. 203 Adams 2014:13-14. 204 Adams 2014: 16. 205 All of this section draws on Honorati 2015. 79 employment at the time of the endline survey the ATT estimates for interns were equivalent to a 15 percent increase in paid employment in the previous month. Descriptive statistics show that, on average, wage employment among male interns in the finance, energy, and tourism sectors increased by 20, 18, and 18 percent respectively compared to control group males. Albeit positive, the change in wage earnings was not significantly different from zero among males overall but was significant for older males (those older than 24 years old). Wages increased by about KES 5,000 among males placed in internships, which corresponds to 26 percent of the wage earnings of control group males and by KES 12,000 among older males. Female employment. The outcomes in terms of both employment and earnings for females were positive and significant, but because of the high attrition rate for female participants in the endline survey, they need to be interpreted with caution. A Lee bounding approach was used to test the robustness of the results for females. While the program contributed to a 6.1 percent increase in paid employment, mostly in the tourism sector, and an increase in earnings of KES 7,500 for participating females compared to the control group females. However, it has to be noted that those estimates may be biased, and the possibility cannot be excluded that the program had zero impact on participating females. Self-employment. Overall no effects were found on the probability of starting up a new business, being self‐employed, or working for a family business. However, the program had a significant impact on self‐ employment among lower educated youths (those for whom primary was the highest level of education attained) and in Mombasa. In Mombasa, the program significantly increased the probability that program interns would work for themselves and/or for their own family’s business by 4 percentage points relative to the young people in the control group, which represents a 20 percent increase in self‐employment. Life skills training. The life skills training, when conducted alone, component was found to have had no significant impact on any of the main variables. The impact evaluation estimated that impact of the two weeks of life skills training were not significant for any outcome in the endline survey carried out 18 months and 22 months after the participants had completed their life skills training. However, qualitative evidence collected through focus groups and individual interviews with both program beneficiaries and employers yielded enthusiastic positive feedback on the usefulness and relevance of the life skills component of the classroom‐based training (compared to the technical sector‐ specific and core business training), which proved successful in building confidence and improving communication and behavior in the workplace. Consistent with KEPSA’s efforts to get youths to open bank accounts in order to access their stipends, the evaluation found positive and significant results among program participants in the probability of having a bank account and of being less risk averse. Cost-benefit analysis. The results are at the basis of a cost‐benefit analysis. With a total estimated cost of KES 97,000 per beneficiary and an estimated KES 6,768 monthly gain for males (although this is statistically insignificant) and a KES 9,623 monthly gain for females, the benefits of the program exceed the costs for both males and females if we assume the gains were sustained for at least 14 months, which does not seem to be a unrealistic assumption. Thus, if we ignore any potential displacement effects to program non‐participants, the pilot program proved to be cost‐effective. However, possible spillover effects need to be taken into account and may prove be particularly important should the KYEP program be scaled up to be large enough relative to the labor market.206 206 Honorati, 2015: 4-6 and 24-26; and World Bank 2016: 31. 80 A.2 Jóvenes Programs in Chile and Argentina The Jóvenes projects (Chile 1991-2002, Argentina 1994-2000) aimed to increase the employability of disadvantaged youth and featured several innovations. The programs were explicitly market led. They relied on the market to reveal the demand for training. The participation of the private sector by providing internships helped to assure the market relevance of the courses and were an indication of employers’ possible interest in the training. Training providers were responsible for arranging internships for trainees. Course provision was determined competitively through a training market where providers competed for contracts. Two main characteristics of the se programs were the provision of work experience through internships following training and the inclusion of socioemotional skills or job readiness training. In addition, training providers were paid in part for the achievement of outputs and outcomes. Both programs were sizeable, training over 100,000 youth each.207 In the early 1990s, in response to structural adjustments in the economies, innovative training programs were conceived to address the growing problem of youth unemployment. The model has been replicated throughout the region. Chile Joven (“Chile Youth”) was the first demand-driven training programs in South America. It started in 1991 and lasted until 1996. A second stage then operated until 2002. Proyecto Joven (“Youth Project”) started in Argentina 1994 and was completed in 2000. Each program aimed massively at training approximately 100,000 unemployed and disadvantaged youth (age 15-24) over four years. Features • The purpose was to increase the employability of participants. Two salient characteristics in this regard were inclusion of soft-skills or job readiness training, and provision of work experience through internships following training. • The programs were explicitly market led. They relied on the market to reveal demand for training (participation of the private sector by providing internships helped assure market relevance of the courses), and course provision was determined competitively through functioning of a training market where providers compete to deliver courses.208 • The programs focused on disadvantaged youth with low chances of insertion into the formal labor market. Assumptions and pre-conditions These programs are based on several assumptions. The main underlying assumption is that the targeted segment of the labor market is not able to find a job because of lack of skills that are demanded by the productive sector, and/or because relevant labor market experience is lacking. The programs also assumed the existence of training providers responsive to, and with good working relations with, the private sector. This is not a minor point, because in some countries the new programs have been used as an instrument to develop a competitive, responsive, and efficient 207 Ibarrarán and Rosas Shady 2009; Inter-Regional Inequality Facility 2006; UNEVOC; Evidence and Lessons from Latin America (ELLA); WB 2006 2007: 117. 208 Ibarrarán and Rosas Shady 2009 : 12. 81 market of training providers. 209 They aimed to address specific challenges. Chile Joven was conceived as a one-time intervention to address a specific situation, the existence of a delimited group of young people that abandoned the formal schooling path during the crisis of the mid-1980s. The diagnosis made by the Chilean government indicated that this was a stock problem. The Chile program was only a temporary intervention designed to tackle the problems faced by an accumulated stock of youngsters who had left school at an early age. It was not intended as a substitute for the formal education and training system, or a permanent solution for the problem of young people from poor backgrounds dropping out of school. In Argentina, in contrast, the nature of the problem addressed is not cyclica l but structural, i.e. continuous flow of unemployed youth. 210 Training model Although Chile Joven had four training modalities, one of them served as a model for other countries: training with practical experience at firms. It was designed to develop skills for work as the employee of a company. The training would be offered in two successive stages of similar length: a stage of classroom instruction conducted by a training institution (about 200 hours); and a stage involving on the-job-learning in business (a full-time internship in a firm for a period of 3-6 months). Private training institutions provided the training classes and supervised the placements in the second stage. The internships at the firm had been previously located and contacted by the training provider.211 Training providers Training providers are selected through a competitive market where a public bidding system ensures quality and fosters private sector participation. Training institutions coordinate courses and internships, balancing the needs of the productive sector with the skills taught in the program. (Puerto 2007: 10) The selection process of training providers is through public calls where firms can submit offers to be evaluated by the program. The training centers are responsible for attracting the participants, designing and administering the training courses, and arranging placement by identifying the firms in which the trainees will do the internships. To guarantee that the content of the course was effectively demanded by the private sector, the course proposals submitted for this subprogram must be accompanied by letters of intent or some other equivalent written commitments with businesses or business associations that cover at least 80 percent of the course enrolment. The purpose of these "letters of intent" was to show that the content of the course was effectively demanded by the private sector, i.e. the service provider had contacted where students can be placed for their on -the-job learning experience. 212 Participants and their selection Eligibility in both programs was based on the logic of self-targeting: the aim was to provide an 209 Ibid.: 14. 210 Ibarrarán and Rosas Shady 2009 : 13; Inter-Regional Inequality Facility 2006: 1. 211 Ibarrarán and Rosas Shady 2009 : 12; Inter-Regional Inequality Facility 2006: 2; UNEVOC: 2. 212 Inter-Regional Inequality Facility 2006: 2-3; UNEVOC: 2; Ibarrarán and Rosas Shady 2009: 12; Puerto 2007: 10. 82 offer only attractive to the target population. For example, a person with a higher educational background would not feel attracted to programs that offered training at a semi-skilled level. In the same way, adults were not attracted by the amount of money offered as the scholarship grant covered only a bit more than traveling expenses. Publicity material stressed that it was destined for poor young dropouts. Each program was advertised through the mass media (e.g. printed press, radio, television, notices, leaflets), with messages specially addressed to reach extremely poor young people. Potential beneficiaries were invited to attend local employment offices. Neither program allowed the training institutions to carry out the recruitment of beneficiaries themselves. 213 Costs to participants and employers Courses were completely cost free in terms of fees and study materials, and there was a scholarship for transportation and other minor expenses that individuals faced when attending. In this way, both programs attempted to eliminate the usual accessibility barriers that arise for extremely poor young people. Women were encouraged to participate by an extra f inancial amount for childcare. For companies, participation was also free of cost. This was considered necessary to secure companies’ participation, since there are some risks for companies providing labor placements (e.g. risks of theft or damage), and some subjective barriers (e.g. discrimination), which often discourage job placements for the disadvantaged. 214 Results Both programs successfully targeted youth from low income families who were unemployed. About 75-80% were under 25 years of age; 80-95% were both under age 25 and from low income households; 75-90% were unemployed or inactive. 215 Both programs surpassed their numerical targets (Chile 12% more than 100,000, in 4056 courses and in over 10,000 companies) and the vast majority of participants were from low-income households.216 Table A.3: Costs and impact of Joven Programs in Argentina and Chile Argentina Proyecto Chile Joven Joven Coverage (number of trainees) 116,000 165,000 Cost per trainee ($) 2,000 730-830 Private benefits Impact on employment (percentage point increase) 10 21 Impact on earnings (percentage point increase) 10 26 Source: WB 2006:117. In terms of the employment possibilities of beneficiaries (finding a job or a job of better salary or in the formal sector), impacts varied depending on beneficiary characteristics and the country in which the program was carried out. In Chile, labor insertion improved on average by 26%, but it increased by more among women than among men, and also by more at the end of the program than at the beginning. By contrast, in Argentina there was no significant increase in average labor insertion, 213 Inter-Regional Inequality Facility 2006: 2-3. 214 Inter-Regional Inequality Facility 2006: 2. 215 Ibid.: 3; Evidence and Lessons from Latin America (ELLA). 216 Inter-Regional Inequality Facility 2006: 3. 83 but there was a significant increase for young men and, in particular, adult women. These differences may be explained by the different economic context. A significant number of beneficiaries decided to return to formal education after finishing the program. Measurements for the Argentine program show that, after the training, 21% of beneficiaries went back to the educational system. The figure was similar for Chile. The programs had a positive and statistically significant impact on the labor earnings of young males and adult females, although not on those of adult males or young females. 217 Success factors and lessons • Youth training programs of this type can be highly effective in a favorable context, involving sustained economic growth and flexible labor regulations that facilitate job creation. In such cases, they are appropriate instruments to tackle low labor market participation among disadvantaged groups. • The mechanism of periodic public tenders, where training institutions interact with companies to define the required profiles and then design the courses, turned out to be highly effective. Competitive tenders gave transparency to the use of public resources and drove the private sector’s efforts to follow the changing requirements of the labor market. • The Chile Joven program was useful in terms of updating ideas about how to be innovative in the operation of training systems. The market-driven design, in which private training institutions identify companies’ demand for skills, define courses based on this demand, and are then paid based on results, has been incorporated into its current training system for formal companies and labor intermediation. (Note: payment by results is not explained in consulted documentation. However, in Chile Joven the intention was to pay training providers on the basis of employment rates. It is not clear from available documents whether this was done. In the case of Argentina, however, the notion of payment upon employment was not feasible, and an output- completion of internship-- was taken as a proxy for outcome and the basis for part of the payment). Issues Nevertheless, implementation was not entirely free of problems. One failure was the unstable relationship between the State and the training institutions, which meant that the latter experienced uncertainty in relation to their financial flows. These two new training programs drastically changed this managerial culture to a more decentralized and flexible mechanism, but they were kept highly autonomous and had low interaction with the pre-existing bureaucratic structure. Another problem was that the time taken by the tendering process was long, in relation to labor market requirements. Strict administrative procedures, justified to ensure compliance with transparency in the use of public resources, are often in conflict with flexibility and pertinence. 218 Transferring the Joven model to low and middle-income countries requires strong institutions to manage a decentralized program and to coordinate the goals and operations of training institutions and participating companies. It also requires continual evaluation of the quality of the courses and internships and the performance of training providers.219 217 Ibid.: 3. 218 Ibid.: 3,4. 219 World Bank 2006, World Development Report 2007: 117. 84 Table A.4: Summary Findings: Labor Market Impacts of Job Training Programs Country Employment rate Formality Wages Dominican None, higher (5-6%) but Health insurance 9% higher for 17% (marginally Republic not significant in the East men (43% vs 34%) significant), larger & Santo Domingo for males under 19 Colombia 5% for women, none for 6-7% for women, 5-9% for men 22% for women, men 10% for men Panama Overall not significant Overall not significant, Overall negligible, 10-20% for women and in probably higher outside Panama large for women Panama City City (38%) and in Panama (25%) Peru Large, 13% (much higher Large: overall 11%, 14% 12-30% for women- 20% than for women, 5% men men- negligible) Argentina 0-11%, 10-30% for 0%-3%, 6%-9% for youngest in No significant youngest (21) one cohort pattern Mexico Overall, no clear pattern; Positive effects (10-20%) since No consistent on-the-job training robust 2002 patterns, at best positive effects (12-30%) small and mostly not significant Chile 18-22% larger for 15-23% larger for youngest 22-25%, imprecisely youngest groups groups estimated Source: Ibarrarán and Rosas Shady 2009. Note: employment figures show differences in percentage points vis -à-vis the comparison group, while the wages refer to percentage differences. Significance refers to statistical confidence, significant denoting 5% and marginally significant denoting 10%. Table A.5: Jóvenes Programs in Latin America and the Caribbean: Impact and Cost Effectiveness Country Impact on Impact on Earnings Cost-benefit Analysis Employment Argentina 10% 10% NPV 0 if 12 years of Proyecto Joven (women) (monthly wages) positive benefits (DR = 5%) Chile 21% 26% NA Chile Joven (men under 21, women) Colombia 5% 18-25% IRR= 4.5%- 13.5% Jóvenes en Accion (women) (men-women) (men- women) Dominican Republic Not significant 10% NPV 0 if 2 years of Juventud y Empleo positive benefits (DR = infl.) Peru 6% 18% NPV 0 if 7 years of positive ProJoven (placement (hourly) benefits (DR= 5%). IRR. 4% Source: Fares and Puerto 2009:6. Note: NPV: Net Present Value; DR: Discount Rate; IRR: Internal Rate of Return. 85 A.3. Nepal Employment Fund using performance-based incentives The Nepal- Employment Fund (NEF 2008-2015) and Enhanced Vocational Education and Training Project (EVENT I 2011-2017) aimed at raising the incomes and living standards of disadvantaged youth aged 16-35 by providing them with quality skills training and by facilitating their entry into gainful employment. A central innovation was the competitive selection of training providers who were responsible for both the local market assessment of skills and arranging placement of trainees in employment with firms. The training model combined three months of classroom training with follow-up support in actual employment (not internships.) Life skills became an integral part of the training. The program featured two types of financial incentives: payment contingent on results (60% of the payments to providers were contingent on employment at a minimum earnings threshold), and a differential payments mechanism for high priorit y disadvantaged groups. The program trained about 90,000 people by the end of 2015. Part of the World Bank-financed EVENT I project continued the NEF’s short-term skills training program with minor modifications. Over the life of the first EVENT project, more than 73,000 youth were trained, exceeding its target. 220 Background The Nepal Employment Fund (NEF-2008-2016) aimed at raising the incomes and living standards of disadvantaged youth aged 16-35 by providing them with quality skills training and by facilitating their entry into gainful employment. The Employment Fund was established in 2008 in a joint effort between the Government of Nepal, the Swiss Agency for Development and Cooperation (SDC) and Helvetas Swiss Inter-cooperation. Helvetas operated the Fund under a steering committee with representatives of the government and donors. The Employment Fund was received financial support from DFID starting in 2010 and by the World Bank in 2014. The NEF approach has been continued for short-term training in the World Bank-financed EVENT I and EVENT II projects (see Annex A). Training model Training and employment service providers were selected annually through a competitive procurement process based on the record of accomplishment concerning previous training outcomes and the established business network of each provider. The training mode was classroom and workshop training for three-four months followed by placement in employment (not in an internship). After the call for proposals each Training and employment service provider (T&E) completed a Rapid Market Assessment (RMA) outlining viable and potential local employment opportunities. They were also responsible for arranging placement of trainees in employment after training. After issuance of a contract to the winning bidders the T&E was then free to recruit and select their own trainees for each of their training events, according t o guidelines established by NEF.221 220 Chakravarty et al. 2016: 6; Johanson 2015: 13; Employment Fund Secretariat 2015: 1; Johanson and Sharma 2013; AGI 2014: 1-2; Hollenbach, et al. 2015: 11; AGI f 2015; World Bank 2011b, EVENT I PAD; World Bank 2017b, EVENT I ICR. 221 Employment Fund Secretariat 2015: 11. 86 Training Content Training was provided in over 60 trades, following national standards for content and certification. Those opting for self-employment could attend a basic business skills training and were linked with financial services. Apart from technical skills training, all trainees received an orientation on reproductive health, HIV/AIDS and labor rights. Women were provided 40 hours of life skills training beginning in 2011. Target groups The main target group was divided into three categories and used a differential pricing mechanism. The three categories were: (A) disadvantaged groups; (B) economically poor and socially discriminated (e.g. Janajati and women); and (C) economically poor from other castes. The Fund uses a differential pricing mechanism that pays training providers a higher rate for the poorest an d most discriminated. The Employment Fund used a differential pricing mechanism that awarded a higher incentive to service providers who trained (and placed) more disadvantaged groups, e.g. highly vulnerable women, discriminated social groups, disabled) according to established vulnerability criteria. The incentives were gradually lowered for less prioritized groups. Training providers that were able to cater to these higher priority target groups were therefore eligible to receive a higher outcome price, but they also faced a higher risk of failing to achieve the outcome of gainful employment. 222 Costs per trainee and per graduate A program evaluation in 2013 estimated average costs per trainee of Rps 26,000 and per gainfully employed graduate of Rps 36,500. Assuming an exchange rate of 95 Rps/US$ in 2013, this is the equivalent of US$ 275 per trainee and $385 per employed graduate. These costs included total expenditure on training and employment, linked expenditures on monitoring and evaluation, and a pro-rata share of overhead expenditures, but exclude opportunity costs (which likely would be low).223 Shifting from an activities-based approach to a results-based approach meant a paradigm shift for many training providers. When the NEF introduced its results-based working approach, it was confronted with resistance. By 2015, however, 57 collaborating providers demonstrated that the approach was feasible and functional. This required continuous capacity-building and backstopping support by field staff. Assistance included development of occupational skill standards and training curricula where none existed, and provision of training for T&E trainers as well as managers. 224 NEF support to vocational training and Training and Employment (T&E) service providers is shown in the following tables. Table A.6: NEF support for training content Products 2009 2010 2011 2012 2013 2014 Total developed 222 Chakravarty et al. 2016: 6; Johanson 2015: 13. 223 Johanson and Sharma 2013: 24. 224 Ibid.: 8-9. 87 Occupational 7 9 10 8 5 5 44 profiles New curricula 5 11 7 11 34 Curricula updated & 9 12 29 50 translated New trades 7 9 14 30 Training manuals 4 3 5 13 4 29 Flip files225 & 10 10 handbooks Quality indicators 226 14 14 Total 18 27 44 24 54 44 211 Source: EFS as presented in Johanson 2015: 9. Table A.7: NEF support for capacity building of training providers by type and year of implementation 2008 2009 2010 2011 2012 2013 2014 Total RMA 16 37 78 58 56 245 Training of trainers 112 110 445 556 640 221 2084 T&E management 12 19 43 20 18 15 127 Life skills 40 46 25 84 85 280 Business skills 240 81 35 17 44 417 Career counselling 36 46 82 Skills testing 100 159 67 153 117 596 Total 228 329 366 682 730 968 528 3831 Source: EFS Annual Reports 2008- 2014, as presented in Johanson 2015: 11. Note: RMA and training of trainers include basic, refresher and advanced training. Sanctions for poor performance and assessment of T&Es Sanctions for performance and assessment of the T&Es were also key features. The organizations needed to pass a procurement process to continue working. T&Es are subject to annual performance review.227 Annual competitive procurement helped ensure that only the best performing providers were contracted. T&Es were assessed based on their performance. Poorly performing providers were not eligible to work again, or contracts were terminated mid-way for non-compliance. About five to seven T&Es were disqualified each year. Even successful partners had to pass the procurement process annually to continue their work. 228 Monitoring and evaluation The success of the program, inter alia, depended on careful monitoring and evaluation of training processes and employment results. In 2015 the Fund Secretariat devoted 85% of its staff to monitoring and evaluation. Field officers of the Fund regularly monitored the performance and training quality of the T&Es. Final employment and income levels were verified by ran domly sampling 30 percent of graduates for face-to-face interviews. An online database was used for real 225 Flip files are visual aids to demonstrate the use of tools and equipment in short term training. They are designed to make the instruction effective. 226 Quality indicators set out minimum criteria for training provision. They are used by monitors to assess the operations of T&E service providers. The indicators cover physical facilities, trainer qualifications and skills, training delivery, on-the-job training, and placement and counseling support. 227 Johanson 2015: 13-14. 228 AGI e 2015: 1; Employment Fund Secretariat 2015: 11. 88 time entry of monitoring observations, easy tracking of individuals and creation of automated tables for processing financial claims and reporting. The data were shared with other projects to avoid duplication. Periodic meetings were held with T&Es to identify and resolve problems. 229 A unique feature of the program was financing based on outcomes. The outcome was defined as “gainful” employment, i.e. earnings of at least NRP 4,600 for a minimum of three and six months. Payments were in three installments in relation to results achieved: 40 per cent on completion of training and taking national skill tests; 25 percent after three months based on verificatio n of employment; and 35 per cent after six months of employment above the pre-defined minimum income. No payments were made for activities, such as the training itself. As providers had to pre -finance the skills training, they bore the risk of losing the investment if the graduates were not working after training. Trainees who enrolled but did not complete the training and/or did not sit for the skills test were not counted for payment. Similarly, graduates with an income below the threshold of NPR 4,600 were not considered gainfully employed and payments were not made. 230 Performance Overall, the program achieved 98 per cent of its training targets, 54% of whom were female and 75 per cent of whom were gainfully employed. More than 100,000 graduates were produced between 2008-2015. In 2011 82 potential T&Es applied for contracts, 36 were pre-selected and 32 were finally contracted; 6-8 were eventually weeded out for non-performance. The Fund started with 4,000 trainees in 2008, increasing to 17,000 in 2013with an accumulated total from 2008-15 of nearly 111,000 trainees. In 2010 81 percent of the 11,400 trainees, 5 percent female, were gainfully employed after six months at an average cost of NPR 21,200 in 55 districts and 65 occupations. 87 per cent of the trainees passed the NSTB skill tests. Most (68 per cent) of trainees came from category B, with 17 per cent from category A and 15 per cent from category C. Table A.8 - Number trained compared with annual targets Year Target Achievement 2008 6800 4003 2009 13300 14470 2010 11000 11418 2011 13500 13222 2012 15000 14255 2013 17000 16450 2014 15800 16008 2015 13000 13106 Total 105400 102932 Source: NEF Annual Reports 229 Johanson and Sharma 2013; also AGI 2014: 2-3; Johanson 2015: 16; Employment Fund Secretariat 2015: 11. 230 AGI 2014: 1-2 with diagrams; Johanson 2015: 13, Employment Fund Secretariat 2015: 7. 89 Results The Quality Impact Evaluation Study found that most of NEF’s targets were reached . Indeed, the program provided over 50% female and disadvantaged participants with access to vocational training or to bring graduates into gainful employment. Eye-catching is that the proportion of graduates becoming self-employed increased continuously over the years. 231 The training financed through the NEF has a direct impact on the lives of its direct beneficiaries, their families and dependents. However, reaching the outreach to most disadvantaged groups has not noticeably improved. The key change at the personal level is improved self-confidence, at the family level increase in household income. The incentive-based payment system … encouraged T&Es to increase the intakes of trainees from disadvantaged groups and the poor. The NEF was not able to cover the ultra-poor, in part because T&Es were reluctant to take the risk of financial loss… as they rate the long-term employment options for disadvantaged and ultra-poor to be limited. The NEF helped to improve, promote and strengthen the private training sector to deliver high quality skills training and to widen its outreach. … T&Es have expanded the variety of trades they cover and have developed innovative schemes to deliver high quality training courses. 232 The World Bank conducted an impact evaluation of the Employment Fund in 2015 as part of the Adolescent Girls Employment Initiative. To understand the effects of the NEF program, the study compared the experience of NEF program trainees with the experience of comparable non- trainees, approximately one year after the program. The sample included both male and female trainees, with a particular focus on the AGEI population (i.e., females aged 16 -24). Approximately one year after each baseline survey, a follow-up survey was conducted. The response rates were quite high for both follow-up surveys. Using a combination of difference-in-difference and propensity score matching methods, large, positive and statistically significant effects were found on labor market outcomes for NEF trainees, as follows: • Employment rates and earnings. Individuals selected for NEF training programs experienced an increase in non-farm employment of 15 to 16 percentage points for an overall gain of 50 percent. The study detected an increase in average monthly earnings of approximately 72 percent. The gains for individuals who actually complete the training (not just those who are selected for training) were even larger. • Empowerment. The study found strong impacts of NEF training programs on economic and psychological empowerment. Trainees had more control over economic resources and stronger self-assessed self-confidence. Alongside the sizable general impacts on employment outcomes, the study found that training courses in electronics, beautician services, and tailoring underpinned most of the NEF program’s impacts. These three categories of training were much more effective in consistently increasing employment and earnings than construction, poultry rearing, handicrafts, and food preparation and hospitality. • Women. The study also found larger impacts on employment for women than for men. Women selected for training in 2010 to 2012 experience overall and non-farm employment gains of 13 and 19 percentage points respectively, while the corresponding impacts for men are 2 and 10 percent. 231 Hollenbach, et al. 2015: 11. 232 Ibid.: 11-13. 90 The impacts for young women aged 16 to 24 were not significantly different from those for older women aged 25 to 35. This finding highlights the suitability of TEVT programs for younger women, although it may require specialized outreach strategies to recruit them. 233 Success Factors The World Bank impact evaluation posited two factors as possible explanations of these large impacts. First, the NEF had time to become established and to develop systems prior to the launch of the impact evaluation in 2010. The program was already operating at scale, and the service delivery processes already road-tested, which surely contribute to the large impacts. Second, the NEF program was designed around employment outcomes. For instance, training providers must complete market assessments as part of their proposals to ensure future employability in the trades in which they propose to train individuals. Training providers were rewarded with outcome-based payments that are higher if they can demonstrate that their graduates have found work, a policy which fosters an incentive to not just train young people, but to find jobs for them. The fact that many training providers returned year after year to work with the NEF despite this unusual contracting arrangement speaks to the feasibility of this approach .234 According to the Phase Capitalization Report (2015) six main factors contributed to the impact of the Employment Fund, including: • The core principles of the Fund: market orientation, results orientation, focus on disadvantaged groups. • Incentives: differential pricing, results-based payments, competitive procurement, accountability for service providers. • Effective project management, including a consultative approach and emphasis on field staff. • Support to training providers and TVET institutions: curricula development, management training, training of trainers. • Strong monitoring and evaluation, including online database; process and results M&E. • Opportunity to transfer the Fund’s modalities to major government investment projects ( Nepal EVENT I and II) (Johanson 2015: 5).235 Lessons Based on the Adolescent Girls Initiative (AGI), the following lessons can be highlighted: • Training programs should be designed around employment outcomes. In Nepal providers must complete market assessments as part of their proposals to ensure future employability in the trades in which they propose to train individuals. Training and employment are also incentivized to support graduates into employment through results-based financing. The majority of payment (60%) is paid only if the graduate is gainfully employed. This policy encourages providers to take the extra step beyond training youth to actually help find jobs for them. 233 Chakravarty et al. 2016: 21-22. 234 Ibid.: 22. 235 Johanson 2015: 5. 91 • Skills training programs can work for young women, and in some cases may be more successful than for young men. • Skills programs should educate young women (and their families) about the relative returns of different trades and encourage them into growing and higher-paying fields. • Specialized outreach strategies may be needed to recruit young women into skills training programs.236 According to the Phase Capitalization Report, eleven lessons can be derived from the experiences of the Employment Fund, as follows: • Incentives worked: differential pricing can increase the inclusion of disadvantaged groups; performance payments can be effective. • Besides incentives, additional efforts are needed to access the hard-to-reach, particularly women. • Service providers needed continuous and multi-faceted support for capacity building. • Early identification of employable skills through rapid market appraisal (RMA) was a key to successful job placement. • Careful trainee selection facilitated employment. • Skills testing proved to be one of the most successful quality checks. • Rigorous monitoring of the process and outcomes was essential for the results-based payments. • Training in vocational skills has to be supplemented with soft skills and life skills. • Multi-skilling can increase incomes, particularly for seasonal workers. • Efforts at creating micro-enterprises can increase gainful employment appreciably, especially for women. • Not all initiatives of the Employment Fund succeeded, but piloting helped avoid major mistakes.237 The qualitative study of the Employment Fund (2015) added the following lessons: • T &Es consider the guidance and governance that the NEF program provides to secure the effective implementation of set modalities to be the key of the NEF’s success. • The result-based payment system is only possible because of a labor- and time intensive monitoring system. • Uniform program implementation and approaches do not fit into all socio-economic, cultural, - political contexts within Nepal. • The drivers and barriers, as well as the perception of change and achievement differ among individuals, locations and socio-economic, -cultural, -political settings. • Multiple skills trainings (directly or indirectly connected to core skills) secure long-term employment, enable the expansion to multiple working fields and increase the confidence of trainees to become entrepreneurs by informing them about their rights and possibilities. • Acquired skills are rated higher than levels of income. Graduates highly value their certification and hence their recognition as skilled workers.238 Recommendations included: 236 AGI f 2015: 2-4. 237 Johanson 2015: 5. 238 Hollenbach et al. 2015: 14. 92 • broadening the single national measure of “gainful employment” to take into account local differences and other employment benefits that include job quality. • extend beyond six months further support to graduates to increase long term employability. • work with employers to create more professionalized and decent working environments for graduates.239, 240 Subsequent Projects by the Employment Fund: 1) Skills for Reconstruction (NEF-SR, 2016- 2020) The Nepal Employment Fund continued in a modified form from 2016 to the present and is funded by Helvetas until 2020. Its website has the following statement: “Skills for Reconstruction (extended phase) is a project funded by Swiss Agency for Development a nd Cooperation (SDC) and implemented by HELVETAS Swiss Intercooperation Nepal so as to address the burning need of reconstruction of the country after the earthquake of 2015. It mainly supports the Government of Nepal in producing construction sector human resources with earthquake resilient skills. Skills for Reconstruction also aims to provide technical assistance (engineering support) to the affected homeowners and facilitate NRA Engineers for inspection of reconstructed houses and release tranches in time.” 241 As of December 2017 NEF-SR trained 10,850 construction workers were trained and 3,580 houses were constructed. Just over 60 percent of participants came from disadvantaged and marginalized communities and one-fourth of participants were women. An evaluation at the end of the first phase of NEF-SR found that almost three fourths of trained graduates were gainfully employed. The skill training was the main contributor to employment. In addition, more than half the training beneficiaries were able to reconstruct their own houses. 242 However, employment was low among women, only 16 percent of total employment compared with a quarter of total enrollments. In addition, participation in skills testing and certification was low. All participants reported to have received inadequate support from the training providers to link with employers. Women received little support to build their confidence, ease their household responsibilities and link them with employers. Recommendations included: • The development of an enabling environment for women to learn and utilize construction skills; • Greater support and backstopping for women before, during and after training to overcome barriers; • Additional support for certification through skill testing. 243 239 Ibid. 240 Additional Sources: Powerpoint Presentation, “Skills Leading to Employment,” delivered by Siroco Messerli to HELVETAS Swiss Intercooperation Pakistan, September 19, 2011; and author’s interview with Siroco Messerli, Kathmandu, October 2011. 241 http://www.employmentfund.org.np/. 242 Supplement to Project Document: Employment Fund Phase 2, Skills for Reconstruction, December 2017. 243 Lohani 2017: 4-6. 93 A subsequent study examined barriers to women’s participation and employment in the program. Community respondents linked the low enrollment of women with the profit prospects for training and employment providers. That is, the providers probably did not select women since they would not get employment and the provider would not receive the full performance payment. The study found the following types of barriers for employment of women: (i) at the individual level, low level of initial basic skills and experience in the trade, lack of confidence and absence of support and guidance; (ii) at the family level family care responsibilities, lack of freedom to make decisions, restrictions on mobility; (iii) at the community level gender stereotypes about female participation in the construction trades; and (iv) at the policy level lack of provisions and incentives to employ women in non-traditional trades. Among the positive findings were that women who were part of mixed or women only construction groups had easier access to employment. Recommendations included gender sensitive training; counseling and support throughout the process; special financial incentives for selecting women in training; and more assistance in forming construction groups for women. 244 2) HELVETAS IFAD Rural Enterprises and Remittances Project (RERP, 2017-2021) This is a joint initiative of the Government of Nepal and the International Fund for Agricultural Development (IFAD) and implemented by the Ministry of Industry, Commerce and Supplies. HELVETAS Swiss Intercooperation Nepal provides implementation support for sub -component 1.3, Decent Jobs. The expected outcome of the sub-component is gainful jobs or skills-based self- employment (minimum NPR 10,000/mo.) for at least 24,900 poor individuals (half women) for at least six months after completing training. The outputs to deliver the outcomes are 6,000 people completing 12 months apprenticeships with local business (of whom 95% are employed after completing the apprenticeships), and 24,000 completing technical and vocational training and gaining training certificates (80% of whom are employed or gainfully self-employed). 245 Helvetas has produced a comprehensive “how to” guide on the Employment Fund approach. 246 It includes clear guidance on all stages in the process of results-based contracting along with instruments for each step, e.g. trainee application forms, trainee selection instruments, training session plan, instructor’s logbook, pre-training monitoring checklist, during training monitoring checklist, skill test observation check list and employment income verification forms. A.4. NEPAL- Enhanced Vocational Education and Training Project (EVENT I) 247 PDO The PDO was to expand the supply of skilled and employable labor by increasing access to quality training programs. Project Outcome Indicator: Employment rate of short-term training graduates 3 and 6 months after completing training in supported programs. Target: 80%, increasing from 70% in year 1 to 80% in year four.248 Intermediate targets included: Average monthly earnings 244 Lohani Ibid.: 5-8. 245 IFAD, “Terms of Reference- HELVETAS Implementation Support to RERP”. 246 Helvetas, 2019. “Implementation and Monitoring Manual” on Sub -Component 1.3 Decent Jobs, for the Rural Enterprise and Remittance Project, SAMRIDDHI.” Kathmandu. 247 This annex refers only to the short-term training components of EVENT I. 248 World Bank 2011b, EVENT PAD: 24. 94 of graduates: 2000 Nepalese Rupees (NPR) (yr1) to NPR 2500 (yr4); Number of short-term graduates completing training tests: 10,000 (yr1) to 37,500 (yr4). 249 Detailed project description Sub-Component 3.1 Results-based financing of short-term training: (US$23 million with US$21 million IDA financing). Short-term training will also be supported using a results-based financing approach whereby payments will be made directly to the training providers, and where training providers will be paid for performance, and not for inputs per se. Selection of training providers and beneficiaries for this activity will be carried out on a semi -annual basis. Each six-monthly cycle will include: (i) identification of areas for training, (ii) selection of training provide rs, (iii) selection of trainees by training providers, (iv) training of selected trainees, and (v) placement of the trainees. Districts where training could be delivered under this modality in this project are likely to include Kanchanpur, Kailali, Surkhet, Banke, Dang, Rupandehi, Palpa, Chitwan, Makwanpur, Parsa, Sarlahi, Dhanusa, Sunsari, Morang, and Jhapa. The beneficiaries, however, could come from other surrounding districts as well. Given the project’s special focus on lagging regions, the Secretariat and training providers will make a particularly concerted effort to reach out to potential beneficiaries in the hill districts of the mid and far western development regions. A selection of training curricula will include counseling and training on non-cognitive skills; outcomes for trainees will be compared across curricula with and without these elements. Selection of training providers The training providers selected under the project were to respond to advertisements published by the Secretariat inviting expressions of interest. The Secretariat made the final selection of award of contracts to those training providers who will offer courses that are in demand either in the local or non-local domestic markets or internationally based on rapid market assessments conducted by the providers. Other criteria used for the selection of the training providers included the details of training with respect to number of hours, training facilities, instructors, curricula used, and assessment and certification systems were proposed to be used which will be verified through 100% site visits of locations where training will be provided. Selection of trainees The short-term training providers were responsible for recruiting trainees. The project focuses on proactively providing training to young women and youth belonging to disadvantaged social groups. The outreach strategy of the training providers will include provision of information to potential trainees on likely job prospects. Incentive payments Re-imbursement was provided to training providers in installments, distributed between the start and completion of the course (completion of the training and taking the skills test), and against the placement of trainees. For both results-based and voucher-based financing, incentives were be provided to training providers to train women and individuals from disadvantaged population 249 Ibid.: 26-27. 95 groups and regions. The incentives and payment modalities are presented below. 250 The rate of incentives on the basic training costs and target category was: • Target group A: (100 % of the training cost) Poor Dalit men and women; Special groups: widows, men and women with disabilities, ex-Kamaiyas and Kamlaris. • Target group B: (80 % of the training cost) Poor women of all castes; poor Madheshi and disadvantaged Janajati men; poor men from disadvantaged districts. • Target Group C: (60 % of the training cost) Poor men from other castes. Payment modalities Payment for skills training and employment services were outcome based. The outcome price includes direct training costs, placement and follow up costs, and incentives. In exceptional cases, additional costs can be paid for youths coming from very remote areas. While the final size of re- imbursement to a training provider will depend on final results delivered, training providers will be able to recoup their costs of training when they become eligible to receive the second installment. Payment will be made in the following installments: • First Installment: 10% of the agreed total outcome price is paid the training provider one week after the beginning of the training upon submission of necessary information on a report on trainees enrolled to the Secretariat’s information management system. • Second Installment: 40% of the agreed total outcome price is paid after submission of the training completion and skills test report by the training provider to the EVENT Secretariat. • Third Installment: 25% of the agreed outcome price is paid after submission of the three months trainee employment verification report by the training provider. The EVENT Secretariat will verify outcomes claims made by the training provider. • Fourth Installment: 25% of the agreed outcome price is paid after the submission of the six months trainee employment verification reports by the training provider. The submitted results must correspond with the results of the verification done by the Secretariat, based on a random sample which includes at least 15% of the employed trainees. 251 Project performance • Overall enrollment in short term training. 73,392 youth enrolled in short-term training, surpassing the target of 70,000. Of these, 71,486 youth completed the training, surpassing the target of 65,500, with 82% passing the national skills test. • Enrollment by target group. Enrollment by specific subgroups (girls, Dalits, and marginalized Janajatis) increased from 46.4% to 55.1%, surpassing the target of 50%. Disaggregated results were as follows: enrollment of girls increased from 26.4% to 40.8% (target: 33.0%); enrollment of Dalits increased from 5.4% to 7.1% (target: 9.0%); and enrollment of marginalized Janajatis increased from 14.7% to 17.7% (target: 19%). 252 250 Ibid.: 41. 251 Ibid.: 43. 252 World Bank 2017b, EVENT I ICR: 6. 96 • The project achieved its outcome targets, in several cases surpassing the output targets within a lower project financing envelope. According to the ICR, this was in part due to greater than expected competition among short-term training providers, which led to lower unit costs (US$ 511/trainee estimated at appraisal, US$ 386/trainee actual). 253 Change in indicators At the time of the first project restructuring, the wording of key outcome indicators was revised for clarity. The first indicator - on employment rate of training graduates - was revised to include the focus on "gainful" employment. The threshold for wages signifying "gainful" employment was raised. The target for the first indicator was also revised downward from 80% to 70%; the ICR reports that this downward revision was based on actual results from the first round of training, which indicated lower than expected outcomes. At the time of the second project restructuring, several output targets were revised upward due to a higher-than-expected level of implementation.254 Employment rates 70.3% of youth who received the short-term training were gainfully employed after six months, falling short of the original target of 80.0%. (Note: the baseline figure was not relevant due to the revised wording of the indicator.) According to the project's tracer study, which tracked graduates 3 to 18 months after completion of training, 87% of the graduates were employed, compared to 67% of youth who did not receive the training. The average monthly earnings increased from 2000 rupees to 6527 rupees (original target: 2500; revised target: 5000). The project's impact evaluation also reported that the probability of employment increased by 20.1 percentage points for graduates of the training compared to the control group. The increase in probability was higher for females (21.7 percentage points) compared to males (19.7 percentage points). 255 Rating Under the revised targets, “Efficacy” was rated “Substantial” due to meeting of the revised target for employability of graduates of the training programs. There was also evidence of increased access to training programs (enrollment in and completion of the training), especially when compared to youth not participating in training programs. Equity of access also increased, particularly among disadvantaged groups. 256 Cost-benefit analysis The ICR provides an updated cost-benefit analysis of Component 3 (short-term training), which comprises approximately US$ 23.0 million (63%) of IDA financing. Costs are calculated as the actual project cost for Component 3 activities and opportunity costs to enrollees, namely foregone income during training (which the ICR estimates at NPR 17,346 per trainee for a three-month session). The benefits from the short-term training activities are calculated according to data on higher employment rate of graduates (compared to youth who did not participate in the training), although 253 Ibid.: 8, 16. 254 Ibid.: 11. 255 Ibid.: 18. 256 Ibid. 97 there is no statistically significant difference in earnings. The cost-benefit analysis for the short-term training activities results in an internal rate of return of 23.1% and a net present value of US$ 60.5 million.257 Monitoring and verification A system that pays for results has to have a credible and reliable monitoring and verification mechanism in place that is able to measure and account for the results. To do so, the project Secretariat established mechanisms for monitoring of training in the field and verification of completion of training and placement of trainees. Orientation of training providers was conducted before the start of the training cycle on the monitoring and reporting requirements of the project. These included submission of complete documentation by the training providers from the start to the end of training (including skills tests taken by the trainees), and unannounced visits during training and interviews with passed-out trainees on a sample basis. Independent verification of placement of trainees will be done on a sample basis (at least 15% of trainees will be randomly selected for verification as mentioned above) after training providers have submitted their placement reports three months and six months after trainees have completed their training. Based on experience during the early phase of implementation, the strategy for field-monitoring of training and verification of outcomes was to be suitably adapted if necessary, to ensure greater fidelity of measurement and findings. Selected lessons that are applicable to the short-term training component of EVENT I • Strong linkage between training and labor market: The most relevant and effective training programs pay close attention to market needs. This requires that identification of the most suitable trades for training be decentralized, and the capacity of the training providers to conduct timely and robust market assessments be strengthened. • Results-based financing of training provision: Results based financing can align incentives and expected outcomes from training. Basing reimbursement to training providers at least partially on training and placement outcomes is one mechanism to ensure that the training provided is market needs oriented. There already exists in Nepal a good example of this mechanism being led by the SDC through their support to Helvetas and the Employment Fund. The program adopts an output/outcome-based approach to financing and the proposed project too will incorporate lessons learned from this experience. 258 • Incentive schemes for targeting disadvantaged groups: One approach to increasing the participation of disadvantaged population groups in training programs is by providing results - based financial incentives to training providers for training candidates from disadvantaged groups. There is successful experience of using this mechanism to incentivize training and placement of women, Dalits and youth from disadvantaged socio-economic groups. 259 Comments on M&E 257 Ibid.: 21. 258 Ibid.: 15. 259 Ibid.: 16. 98 The ICR recognizes the joint team’s commendable effort in instituting a strong M&E system, including its close collaboration with the Employment Fund (NEF) for the development of the M&E system for the short-term training subcomponent. The project successfully developed a decentralized web-based MIS with modules for all components and subcomponents of the project allowing the project secretariat to collect data necessary to guide the project implementation and generate comprehensive quarterly implementation progress report on time. A decentralized MIS allowed the project secretariat, through training providers, to collect data on short-term training as well as on labor market outcomes of all trainees, three and six months after training completion as part of the regular M&E. The independent monitoring arrangement for the short-term training, as envisioned during project design, evolved from having an independent third party to having a joint monitoring team (JMT), consisting of representatives from the District Education Office (DEO), Central TVET organization (CTEVT), and NEF, to verify the results. The JMT carried out field-based verification of the short-term training at different stages to verify the data provided by the training providers. Given the scale and intensity of short-term training activities, the JMT’s ability to carry out monitoring of the short-term training, maintaining a certain level of quality, is an impressive feat. 260 The JMT made four monitoring visits during short-term training and two visits, on a sample basis, post training to verify the labor market outcomes of trainees, three and six months after completion of the training. Lessons at completion • Monetary incentive scheme to training providers, performance grants to institutions, and a targeted approach, such as a special women’s window and pro-poor targeting, can be effective in improving equitable access and participation of women and youth from marginalized and disadvantaged groups. • Transfers based on outputs, directly to short-term training providers can generate greater accountability with respect to objectives, especially enrollment, completion, and employment. The same performance-based funding modality will be continued in the follow-on project.261 • The joint monitoring mechanism was one of the key factors contributing to the success of the short-term training activities under the project. The tripartite arrangement provided credible checks and balances and fortified the level of assurance in the results. The project show ed that with appropriate coordination, human capital, and capacity rigorous monitoring is possible even at the large scale. The same monitoring mechanism will be followed in the follow -on project moving beyond the headcount to a more enhanced focus on quality and market relevance of the trainings. 262 A.5. Liberia- Economic Empowerment of Adolescent Girls and Young Women (EPAG) The Economic Empowerment of Adolescent Girls and Young Women (EPAG) pilot project in Liberia (2010-12) aimed at removing barriers to employment and raising the incomes of young, 260 Ibid.: 27. 261 Ibid.: 32. 262 Ibid.: 33. 99 out of school literate women. 263 Young women and adolescent girls were provided a package that combined: 1) six months of classroom-based technical and life skills training, with a focus on skills with high market demand; and 2) a six-month placement and support phase in which trainees transitioned to wage employment or started a business. The program channeled one third of participants into a wage job track while two thirds underwent training in business development. The program was implemented by local Non-Governmental Organizations (NGOs) selected competitively. Roughly 2,500 young women aged 16-27 were recruited in the first round of EPAG. The program featured several innovations: attention to girls’ needs through flexible morning and afternoon sessions, nearby training locations and free childcare; intensive support through coaches and peer groups, incentives for participation as well as competitions and prizes, independent monitoring and performance bonuses for providers. Background264 The EPAG project was the first pilot under the Adolescent Girls Initiative (AGI). The project was part of a larger Adolescent Girls Initiative (AGI) by the World Bank with support from the Nike Foundation and several governments (Australia, Denmark, Norway, Sweden and the U.K.). Objective The Project’s objective was to increase employment and earnings among young women. Target group EPAG originally targeted young women (i) between the age of 16 and 24; (ii) who possessed basic literacy and numeracy skills; (iii) who were not in school (although night school was acceptable); and (iv) resided in one of nine targeted communities in and around Monrovia and Kakata City. The project sought to reach young women at an early enough age to improve significantly the trajectory of their working years, to ensure the basic literacy and numeracy skills needed to succeed in the labor market and to avoid enticing applicants to drop out of school. The application process had to rely primarily on self-reported data in the absence of other forms of identification and documentation. To counter false information, eligibility criteria were not made public and the outreach campaigns did not specify age or education requirements. Program managers did not choose whom to train. Candidates were assigned by lottery to one of two rounds of the program. Those assigned to the second round served as a control group for the evaluation. Initial trainee recruitment took place from December 2009 through January 2010 and involved community mobilization campaigns. EPAG originally recruited 2016 trainees. Components • Job skills training for wage employment, combined with job placement assistance . • Business development skills training and links to microcredit for young women entrepreneurs . • Project monitoring and rigorous impact evaluation. • Institutional strengthening and capacity building. 263 Economic Empowerment of Adolescent Girls and Young Women (EPAG) 264 AGI and Liberia Ministry of Gender and Development 2015; Adoho et. al. 2014; World Bank 2010. 100 Training Model Participants were given a package of skills training and complementary services to facilitate their successful transition to employment. The model included six months of classroom training followed by six months of placement and support for the transition to wage or self-employment (micro-enterprise advisory services, internship and job placement assistance). Training Content Upon recruitment, participants were assigned to a job skills (JS) track, or a business development services (BDS) track. • Job skills track- was limited to 35% of total training places owing to the expectation that few wage jobs would be available in the Liberian job market. The JS track provided training in one of six different skill areas: hospitality; professional cleaning/waste management; office/computer skills; professional house/office painting; security guard services; and professional driving. The focus was on practical, demand-driven skills identified through independent labor market assessments, review of market data and extensive private sector consultations. All JS trainees received training in entrepreneurship skills as well. • Business Development Services training- included 65% of the trainees. Training content included how to identify micro-enterprise opportunities based on market needs assessment, how to grow and manage any business they already had. The curriculum included entrepreneurship principles, market analysis, business management, customer service, money management and record keeping. Innovations • Attention to girls’ needs. Service providers hold morning and afternoon sessions to accommodate the schedules of participants. Training is held in the communities where the girls reside, reducing the time and cost of transportation. Every site offered free childcare. • Incentives for participation. Participants signed “trainee commitment forms” and were paid small stipends and a completion bonus contingent on attendance. Participation was also incentivized through attendance prizes, contests, business plan competitions, etc. They were assisted to open a savings account at a local bank in which to save their stipends. • Support. Participants received support throughout the program through two means: (a) coaches who attended classroom training and provided follow-up with trainees in their communities during placement; and (b) trainees were organized into peer teams for social and learning support. • Independent monitoring. The M&E plan included frequent and unannounced visits to ensure that service providers maintained high quality learning environments. • Performance bonuses. The project uses a “Withheld Incentive Payment” (WIP), meaning that 10 percent of the total contract value is withheld until six months after the classroom training ends. The training provider receives a share of the withheld amount that is proportional to the number of graduates employed, relative to a preset target (for example, if 90 percent of their 101 target is met, the provider receives 90 percent of the withheld amount). 265 These were paid about 12 months after the start of training. Service providers EPAG was implemented by four NGOs that were selected by the Ministry of Gender and Development through competitive bidding. Two of the NGOs further sub-contracted to four Liberian NGOs. The service providers were responsible for developing training curricula, selecting training venues (that met criteria on safety, accessibility and conducive learning atmosphere), arranging childcare, assisting with community mobilization, recruitment of participants and provision of follow-up support. Results Changes during implementation - Recruitment – particularly younger girls who were both not in school but literate – proved difficult. The age limit consequently was raised to 27. The second round of EPAG included a preliminary basic literacy program to help otherwise qualified young women to improve their skills before entering the program. Payment scheme - In Liberia the final withheld incentive payment ensures that service providers make their best effort to transition trainees into jobs and businesses. In addition, the service provider contracts include payments tied to key deliverables to ensure strong oversight in contract administration throughout the project cycle. The Liberia AGI monitoring system, which generates monthly quality ratings, also stimulates healthy competition among the service providers.266 Participation - EPAG achieved remarkable participation. The project achieved a 95% retention rate (higher than similar projects in Liberia and elsewhere) and an average attendance rate of 90%. Employment - At baseline 38% of the young women had at least one income-generating activity. At mid- line the number of trainees engaged in income earning activities increased by 18 percentage points relative to the control group. This translated into a 47% increase in employment from the baseline of 38%. This was a significantly greater improvement than had been seen in many other similar programs around the world. Moreover, this economic impact seems to be long-lasting. Comparative rates - Positive employment outcomes were stronger among the business skills trainees. By the end-line 70% of business skills graduates were engaged in at least one income-generating activity (wage or self-employment) compared to just under 60% of job skills graduates. Among those with any employment, most were self-employed regardless of track in which they were trained. Types of employment - Most self-employed EPAG graduates were engaged in petty trade. Most wage employed graduates worked in a shop, restaurant or hotel. There was little movement into common female professions in surplus, such as beautician, tailor and soap maker. 265 AGIa: 1 266 Ibid.: 4. 102 Income - EPAG increased average weekly incomes by 80 percent relative to the control group, i.e. starting from a small base this amounted to an additional $8 per week. The project improved earnings through two channels: (i) bringing more people into employment, and (ii) increasing the productivity of those who were already engaged in income-generating activities. Savings - The savings of the young women participants increased significantly. Participants were 50 percentage points more likely to have savings at midpoint in the evaluation and their savings averaged nearly $35, significantly more than the project completion bonus ($20). The amount of savings continued to increase to the end point. Results by participant profile - The project was most effective for young women in the middle of the wealth distribution (i.e. more effective among those in the middle wealth quintiles than among the poorest or wealthiest.) There was no discernible impact among those with no education. The greatest impact was found among those with complete high school. Costs - Per participant costs (including all categories of project expenses) were $1350 for business skills and $1650 for job skills training. The costs are probably an upper bound, given start-up costs, and could decrease in future rounds of training. Compared with average monthly income increases means that job skills participants would need 12 years to cover costs, and business participants would need just three years. Other findings (compared with the control group): • EPAG graduates were more likely to have accessed credit. • Only small but significant improvements were found in participants’ control over their own resources, in part because baseline levels were already high. • EPAG graduates reported that they worried less. • Two measures of non-cognitive skills – self-regulation and a task-oriented measure of self- efficacy – showed positive change. At mid-point EPAG graduates were more confident. • EPAG significantly improved household food security and dietary composition. Other findings were neutral or negative: • EPAG did not appear to have an effect on women’s experiences of violence. • EPAG did not affect young women’s sexual behavior. • The project did not lead to any changes in desired or actual fertility. • There was no consistent or strong evidence that the project experiences led to changes in household size or significant asset purchases or sales by households. Lessons • Results-based payments conditional on placement can improve outcomes. The Liberia pilot used an innovative performance-based incentive scheme to improve the employment of graduates.267 • The EPAG evaluation provides strong evidence that skills training can be an effective policy option for increasing employment among young women in Liberia. 267 Ibid.: 1. 103 • Factors contributing to high participation rates- In qualitative focus groups, trainees credited the stipend and childcare as features that facilitated their full participation. • EPAG findings validate the importance of delivering business skills training along with job and life skills training in contexts where wage employment is limited. • The results underscore the importance of doing comprehensive analysis at the start of local economic and social conditions and especially of young women’s conditions and needs. Sources: AGI and Liberia Ministry of Gender and Development 2015; Adoho et. al. 2014; World Bank 2010. A.6. Chile- Lifelong Learning and Training Project (Chile Califica)268 Date - 2002-2009 IBRD Loan Amount - US$75.8 million. PDO - (1) Reduce the social deficit in the provision of lifelong education and training opportunities for young adults and adults, especially those who are unemployed or living in conditions of poverty, and those who seek a second chance to improve their employability and quality of life. (2) Upgrade the skill level of the Chilean workforce to improve its productivity and competitiveness. Project component 1 - A flexible adult education modality targeting adults seeking to complete their basic and/or secondary education has been successfully implemented since 2003. Operating together with an independent learning assessment and certification system, this demand-driven, results-oriented model has been effective in ensuring that a wide universe of education providers adequately responds to the needs of adult students while producing the desired educational outcomes with minimum supervision. Flexible adult education component - Providing new opportunities for lifelong learning and training (US$59.2 million, incl. US$41.1 million of IBRD financing) This component was organized into three sub-components, with a focus on: (a) Promoting and increasing private and public supply of basic and secondary education opportunities for adults; (b) Establishing a learning assessment and certification system for adults seeking to complete their basic and secondary education; and (c) Developing new modalities that harmonize adult basic and secondary education with training. Primary target groups - (1) Adults with incomplete basic and/or secondary education, particularly those in the three lowest income deciles, including indigenous groups; (2) Workers in need of training, particularly those in SMEs, and those choosing to certify and upgrade their work competencies; and (3) Students in technical-professional educational paths. Intermediate indicator - To offer the opportunity to complete basic and secondary education to about 71,500 and 48,500 adults, respectively, ensuring that about 120,000 adults complete/certify basic and/or secondary education.269 RBC Modality - Chile Califica included a form of incentive-based payments. Providers were paid based on results (i.e., number of students who successfully pass the evaluation and cer tification tests). 268 The following synopsis refers only to the flexible adult education component. 269 World Bank 2002, Chile Califica PAD. 104 The project paid training providers 25% per person as an advanced payment based on the enrollment in a training module and 75% based on successful completion of a learning module (instead of merely attendance). An independently administered examination and certification system was used to determine whether students passed and whether the provider received payment. This approach made trainers ensure that students achieved the minimum level of competency required to pass the exam. Results Targets Achieved - The overall number of adults completing their basic or secondary education under the flexible program has exceeded the original target by over 20%: 38,800 adults have completed their basic education under the flexible modality and 109,500 adults have completed their secondary education under the flexible modality, a total of 148,350 adults. The project thus succeeded in expanding the provision of relevant lifelong education and training opportunities among those population groups lagging behind in human capital development. The flexible adult education program has substantially expanded enrollment in adult education, particularly women, the unemployed, and the poor. Efficiency outcome - The stream of benefits is estimated at roughly US$380 per year per adult that successfully completed his/her secondary education until retirement. In the case that the education and assessment process take one full-time month or less, the net present value (NPV) per participant is positive after three years of subsequent employment. If the time invested increases to two, four, or six full-time months, the NPV becomes positive only after four, seven and twelve years, respectively. Given that the median age of participants is 35-years old, the adult education program is considered to be economically efficient, as participants are expected to remain economically active for more than twelve years. Impact evaluation - The expanded learning and training opportunities provided under Chile Califica have resulted in measurable impacts in terms of income among beneficiaries. There was a larger increase in income among adults that successfully completed their secondary education relative to those in the control groups, with those that participated having between 10-14 percent increase in their income. Conclusions on RBC - The results report concluded that a sound system of incentives is crucial to the successful implementation of education programs. A sound system of incentives is crucial to the successful implementation of education programs. Chile Califica’s flexible adult education program was conceived as a demand-driven, results-oriented program. These characteristics were crucial determinants of the program’s success. Specifically: • Demand-driven nature (i.e., students signed up at individual education providers of their choice): It served, among others, to: i) maximize the enrollment of motivated students; ii) ensure a good match between educational providers and the student in terms of location, schedules, etc.; iii) ensure the students’ satisfaction with the services provided, as they could ‘vote with the feet’ and change providers for subsequent courses; and iv) promote competition among education providers. • Results-oriented nature (i.e., an independently administered examination and certification system was used to determine: a) whether students passed and, b) whether the educational provider received payment—i.e., payment was contingent upon individual students passing 105 the exam): It served to, among others, to: i) ensure a minimum level of competency among all students that successfully passed the corresponding examinations; and ii) motivate providers to ensure that students achieve the minimum level of competency required to pass the exam. In this way, providers devised multiple strategies to minimize absenteeism and dropout rates and to maximize learning. In some cases, monetary incentives were also provided to teachers who achieved a certain success rate.270 A.7. Colombia “Jovenes en Accion” (Youth in Action) The program was introduced between 2001 and 2005 and provided three months of in-classroom training and three months of on-the-job training to young people between the ages of 18 and 25 in the two lowest socioeconomic strata of the population (Attanasio, et al. 2011:189). In 2005, the government agreed to use lotteries to assign oversubscribed training courses among interested applicants. Training institutions were paid according to market prices and were paid conditional on completion of training by the participants of the program.271 Classroom training was provided by training institutions which had to be legally registered, show financial solvency and compete in a bidding process in order to be able to participate. The training institutions also had to guarantee apprenticeship in legally registered (formal sector) companies in order to be paid by the government for the provision of training. Since training institutions were paid based on graduation of participants and their ability to place them in apprenticeships, they were inclined to select participants on the basis of who was likely to succeed. Private sector institutions—some for profit and some nonprofit—offered the classroom training and chose, designed, and marketed the courses to the firms providing the internships. Training institutions, thus, had to offer courses that provided skills for which there was demand in the labor market.272 The program reached 80,000 young people (or approximately 50 percent of the target population) and was given to various cohorts over a period of four years. The first cohort received training in 2002 and the last one received training in 2005. In 2005, there were a total of 114 training institutions offering 441 detailed types of courses to 989 classes with a total of 26,615 slots for trainees (Attanasio et al. 2011: 191). The greatest number of courses was offered in administrative occupations (such as sales, secretarial work, marketing, warehouse and inventory work, and archival work). However, there were also a large number of courses in manual occupations (such as seamstresses, electricians, and cooking assistants), as well as courses in fairly skilled occupations including (IT specialists, data entry, surveyors, and accountant assistants). Results • Some 212 training institutions were strengthened, and a network was established of over 330 certified training institutions through a national register of training institutions. 270 World Bank 2010b, Chile Califica ICR 271 “The payment system consisted of an advance and payments during the training which were settled according to the progress of the participants throughout the course. This system acted as an incentive for training institutions to retain young people in the course and specially to ensure the realization of labor practice (internships)” (Government of Colombia 2008: 20). 272 Attanasio, et. al. 2011: 211-12. 106 • More than 3,000 companies participated in the program enabling young people to undertake labor internships. • 97% of young people completed the regular classes and 92% completed the practical phase. • 47,000 young people were trained and with labor experience per year (141,000 in total).273 Evaluation of short-term effects found that male formal wages increase by 23 percent, while the formal wages of women increase by a staggering 33 percent. A well-designed randomized vocational training program in Colombia, which matched the skills taught to the skills required in the labor market, increases earnings for women by close to 20 percent and increases formal employment for both men and women by up to 7 percentage points.274 One year after YIA training, randomly selected female participants are 7 percentage points more likely to be in formal sector employment and earn 20 percent more than non- participants. Among males, YIA increased formal employment by 6 percentage points but had no effect on earnings. Evaluation of long-term effects finds that early labor effects and earnings increases of the Colombian training program persist in the medium term. Up to 11 years after randomization, participants are more likely to enter and remain in formal employment and have formal sector earnings that are at least 11 percent higher than those of non-participants. The impacts of job training on both the extensive and intensive margin of formal employment are long-lasting—particularly among women. An evaluation after 11 years, using a combination of te randomization design and administrative data, finds that the implied internal rates of return range between 22.2% and 23.5% percent for females and between 10.2% and 20.5% males. The employment and earnings impacts are among the largest in the job training literature that have been documented using a credible research design.275 Comments on RBC “This pay-for-performance element of the program created incentives for training institutions to offer participants courses and skills that were relevant and in-demand by employers.” and “Taken together, the results suggest that a job training program such as Colombia’s YIA that combines classroom and apprenticeship time and has pay-for-performance incentives built in is an attractive social investment and a potential avenue for social mobility”276 A.8. Bangladesh SEIP I Design The ADB-financed Bangladesh Skills for Employment Investment Project (SEIP, Tranche I, 2014-2019) also used results-based contracting to deliver training to 260,000 people. Training programs are delivered through performance-based contracts with training providers aligned with industry associations. (Providers were not selected through competitive bidding). In the contract, milestone payments are tied to training performance and results: (i) enrollment; (ii) certification; and (iii) employment (at least 60%) and job retention for three months. A customized performance-based 273 ILO CINTERFOR. 274 Attanasio, et. al. 2011. 275 Kugler et al. 2019. 276 Kugler et al. 2019: 6, 28. 107 contract was developed for the Government to partner with industry associations. This unique contract incorporates a detailed business plan on training programs, which are prepared by each industry association (or nongovernment organization) based on skills needs specific to each industr y. Payments are made as follows: 40% on enrollment; 40% on certification; and 20% on job placement and retention after three months. Another important mechanism was an online trainee management system that tracks, and monitors trainees’ real-time progress based on inputs from training providers. 277 Results Table A.9: SEIP Tranche I achievements in relation to targets Indicator Target Results Difference % Target Achieved Number trained 253,700 227,600 -26,100 90% Number of females trained 76,200 72,700 -3,500 95% Number of trainees employed 152,500 152,000 -500 100% Source: ADB 2018 SEIP Tranche I Joint Review Mission, October. In addition, the skills of 80% of those trained were subsequently certified. This nationwide program delivered training for over 200,000 under the Tranche I loan, which was one of the largest national initiatives. One major achievement has been that training was directly linked to job placement, given that job placement was a key performance requirement in the contract. While this performance has varied by industry associations, most industry associations achieved the 60% job placement target. (The target applied to industry associations, rather than individual training provider. Industry associations selected training providers and organized training courses). The performance - based contracts provided strong incentives to achieve the targets of certification and job placement. Not all industry associations proved to be experts in understanding the skills needs of their sector and in organizing training courses. Some industry associations achieved nearly a 90% job placement rate because they recruited the number of trainees they planned to absorb after training. These industry associations are in textiles, leather goods and agro-processing. These mostly were those that offered training courses in their business premises. The industry associations whose initial numbers of target trainees were low have also performed far better in terms of job placement. Their target was based on realistic demand in the industry and trained the number which they could absorb with jobs. On the other hand, some associations did not perform well because they did not adequately consider absorptive capacity in their industries, offered training courses that were not up to date, planned for target numbers of trainees that far exceeded what employers could absorb and therefore struggled to place the graduates. (The performance risk for non-placement, after all, was only 20%.) One finding from the SEIP I experience is that employer associations may not always have accurate information about the potential scope of specific skill demands or absorptive capacity for trainees in their respective industries. The associations may be aware of the kinds of skills needed but may not accurately estimate the number of graduates that the labor market can absorb. Alternatively, estimates of absorptive capacity may turn out to be incorrect because of changes in the 277 ADB 2016 SEIP Periodic Financing Request Report: 2-3. 108 economy. One drawback of performance-based contracts is slow disbursement. For training of three months, it is necessary to wait for another three or six months (job retention period) to make full disbursement. A.9. Grenada- Skills for Inclusive Growth The World Bank-financed project Skills for Inclusive Growth in Grenada (2009-2013) also used an output-based financing scheme for training contractors. 278 Payments to each training provider were to be made in three installments: 40 percent advance upon the initiation of the training course, 30 percent upon completing delivery of the four-month life skills and technical training, and the final 30 percent payment upon the successful graduation and certification of the trainees. This payment schedule was expected to provide a strong incentive for training providers to offer quality training that ensures their trainees learn and become certified. 279 Results Table A.10: Achievements compared with targets Indicator Target Actual Diff. % Achievement Number of entrants 500 886 386 177% Number of enterprises offering internships 90 240 150 166% Source: World Bank 2014, Grenada ICR. The completion reports that the number of employers involved substantially exceeded targets. Some 240 private companies provided internships to trainees amply exceeding the final target of 90 companies, including 55% non-tourism companies compared with a target of 48 percent. 280 It also stated that “by requiring training providers to demonstrate employers’ commitment to provide on-the- job internships as part of their proposal, the Project ensured that employers played a role in determining the allocation of training resources among the various occupations and economic sectors, thus ensuring the relevance of the technical skills developed under the Project”.281 278 World Bank 2014, Grenada ICR: 4. 279 World Bank 2008, Grenada PAD: 12. 280 World Bank 2014, Grenada ICR: 11. 281 Ibid.: 12. 109 Annex B: Overview of Results-Based Financing (RBF) – with application to Skills Training This annex provides an overview of results-based financing in general. It seeks to answer the following questions: What is results-based financing and contracting? What are the main advantages and limitations/risks? What are the main findings on results? What are essential requirements for success? B.1. Introduction: Features of results-based financing B.1.1. Results-based financing includes a bewildering array of terms. These terms include but are not limited to payment-for performance, performance-based funding, output-based aid, results-based aid, payment by results, performance-based contracting, incentive- based financing. The various permutations can be arranged in the diagram below, where the first column represents the objective and the last column the action or means: Diagram A.1: Terms applied to results-based financing Financing Results Aid Output -based Payments Performance Contracting Incentives Source: Adapted from Musgrove 2011:7-8 B.1.2 There appears to be no common definition of terms; however, all types include some common features. All terms refer to payments of an incentive (in money or kind) by a funding agency (the principal) to a recipient (agent) conditional on achieving pre-determined targets. In other words, results-based financing is payment of an incentive conditional on achievement of pre-defined results. As explained later, RBF involves three main elements: incentive, payment for accomplishment, and pre -defined targets. Payments are not disbursed against individual expenditures or inputs but on results that are largely within the control of the recipient. 282 282 Perrin 2013: i; DFID 2014: 4; Oxman and Fretheim: 70-71; GPOBA 2016: 1; Pearson, et al. 2010: 10; UNESCO 2018: 1; USAID: 3; Gittner 2013: 1; AGI e 2015: 3. 110 This report uses the term “results-based financing” (RBF) as an umbrella term. It encompasses any program that rewards delivery of verified pre-defined results (outputs, outcomes or impact) with a financial incentive. The reward recipients may be governments (this is often called results -based aid), service providers (results-based contracting) or beneficiaries (e.g. CCTs).283 “Results-based contracting” (RBC) is a sub-set of RBF. It means a formal agreement (contract) is made between a funder and an implementing entity (agent, such as a non-government service provider) for delivery of, and payment for, pre-agree results (which, in turn, may be outputs, outcomes or other performance measure). In effect it means that service provision is contracted to a third party.284, 285 B.1.3 At its core RBF emphasizes achievement of results. RBF is characterized by a result focus. “Results” properly encompass outputs, outcomes (sometimes divided into intermediate and final) and impact. They exclude process indicators (inputs and activities.). In many cases output measures are used a proxy indicator of outcomes. 286 As O’Brien and Kanbur pointed out it depends on what is counted as results, since almost anything can be labeled results-based financing. The discussion of RBF has to be framed in the context of a chain of causality from inputs to outputs to outcomes. These are commonly presented in a “results chain.” In particulat, RBF rewards the production of desired outputs and outcomes. 287 It focuses on outcomes (or at least outputs further along the results chain) than inputs or activities. 288 These results are agreed in advance at the start (pre-defined). This contrasts with traditional financing, where payments are made on the basis of inputs, activities and processes. The underlying assumption is that the funder is likely to get more of what they pay for, so the funder should pay for what really matters—outcomes and impacts.289 Diagram A.2: Results Chain 283 UNESCO 2018: 1; IDA/GPOBA 2009:i; SIDA. 2015: 10; Pearson, et al. 2010: 10; Klingebiel 2012: 6. 284 Funders are treated as principals who are trying to achieve an objective by contracting with a recipient who is conceived as an agent. The underlying assumption is that differences exist in the objectives of the principals and agents which principals seek solve by offering a contract that aligns agent’s incentives with their own (Perakis and Savedoff 2015 : 8; Clist and Verschoor 2014). 285 Pearson, et al. 2010: 10; Musgrove 2011: 5; Loevinsohn 2008: 1; 8: 5. 286 SIDA. 2015: 10; 8: 3; WB 2017: 11; Musgrove 2011: 3 287 Macharia 2015: 1; Perrin 2013: 1. 288 O’Brien and Kanbur 2013: 21. 289 Perakis and Savedoff 2015: 5; SIDA. 2015: 9. 111 Source: Author’s compilation based on Klingebiel 2012:9 and OBA Approaches. The capacity of an intervention to cause achievement of results is inversely proportional to the stage on the result chain. That is, as things progress from outputs to outcomes and then to impact it becomes more difficult to link results to the intervention. 290 B.1.4. Results-based financing encompasses a wide array of mechanisms. These include cash-on-delivery, Payment for Results (PforR), results-based contracting, conditional cash transfers, etc. Various typologies have been advanced to organize these concepts. Schemes can be defined according to their funding sources, contracting parties, implementing agency, types of results desired and links between results and payments. Another typology organizes schemes according to two dimensions: (i) the extent to which financing is paid up-front before the results have been achieved, or ex-post on delivery of the results; and (ii) the stage in the results chain in which financing is delivered. 291, 292 B.1.5. The clearest distinction among schemes is by who receives the payment. The various schemes fall into three categories according to this typology: Table A.11: Classification of schemes by recipient of the funds Government Implementer/Provider Beneficiary Cash on delivery Results-based contracting Conditional cash transfers (CCTs) Program for Results (PforR) Fee for service Vouchers Performance-linked budget support Disbursement-linked to indicators (DLIs) Source: Author’s adaptation from Musgrove. 290 Musgrove 2011: 3; UNESCO 2018: 3. 291 See diagrams on results-based schemes in Klingebiel 2012: 7,Mumssen, et al. 2010: 4. 292 Pearson, et al. 2010: 13; SIDA. 2015: 9; and also DFID 2014: 1. 112 B.1.6. Performance risk is transferred from the funder to the implementer. Much of the performance risk is shifted from funders to service providers whose payment is now contingent upon delivery of results. 293 Performance risk includes three specific types of risks: • financial risk (i.e. the need to pre-finance inputs without assurance of final payment, and the risk of cost overruns). • operational risk (i.e. interruptions to delivery such as staffing problems, or beneficiary dropout before completion). • demand risk (i.e. failure of uptake, such as failure of the economy to generate new jobs for graduates, or failure of trainees to seek employment after skills training) .294 B.1.7. Payments are conditional. The defining element of RBF is that the funder only pays when the agreed results have been achieved. Providers are compensated for the results regardless of actual inputs, i.e. payments are ex -post. This is somewhat akin to paying for a service only when the pre-agreed service has been provided. 295 B.1.8. Payment involves incentives.296 Incentives are implicit in all forms of results-based financing. An incentive is the difference between (1) the cost of service provision plus normal profit margins, and (2) the actual payment. The purposes of the incentive are to compensate for performance risk and to motivate those taking action. The concept rests on the assumption that the agents are motivated by financial gain and will seek ways to maximize income. Ideally the incentive payment would be made only on realizing the outcome, but compromises are often made in practice. RBF can also create incentives for improved performance by rewarding progress in incremental steps. Incentives may be directed at either the supply (provision) side or the demand (beneficiary, in this use of the term) side. Rewards can be monetary or non - monetary. Prizes and competitions are alternatives to incentives that work much the same. 297 B.1.9. Achievements must be verified. Since payment depends on achievement under RBF, verification of achievements assumes overriding importance. Verification depends on measurable indicators of achievement that can be substantiated. It also requires objective evaluation of results and avoidance of evaluation bias. 298 293 SIDA. 2015: 10. 294 IDA/GPOBA 2009: 13-15. 295 Employment Fund Secretariat-a: 2; Perakis and Savedoff 2015: 2-3; Klingebiel 2012: 12; SIDA. 2015: 7, 10; IDA/GPOBA 2009: 14) 296 Results-based financing facilitates the alignment of interests among funders and implementers. Incentives can be seen as a way to resolve the “principal-agent problem”, i.e. how best to design a contract that will motivate the agent to behave in the way desired by the principal even when it is not possible to fully monitor what the agent is doing. With RBF the contract aligns the objectives of the principal and the agent. (USAID: 2; Eldridge 2009: 161). 297 Perakis and Savedoff 2015; Savedoff 2016: 4; Eldridge 2009: 161; Results for Development Institute 2016: 6; Musgrove 2011: 4; SIDA 2015: 10; AGI e 2015: 4, 9: UNESCO 2018 23: 2. 298OBA Approaches: 1-3; Klingebiel 2012: 12; SIDA 2015: 10; AGI e 2015: 5; WB 2017: 11. 113 B.2. Advantages of results-based financing This section presents some of the theoretical arguments in favor of RBF, i.e. the promise of greater benefits compared with traditional financing. B.2.1. RBF can lead to greater effectiveness. First, RBF provides motivation for results. The attraction of performance bonuses and the credible threat of termination of the contract help motivate contractors to achieve better results. By being paid on results, partners are strongly encouraged to examine what is not working, a first step in making adjustments that raise achievements. 299 Second, greater effectiveness manifests itself in better decisions, improved management and implementation. Service providers have every reason to plan and implement schemes well to ensure that they are fully compensated. Careful planning enhances sustainability. 300 Third, results-based financing tends to engage the capital and expertise of non-government service providers who often are more effective than public providers. Non-state providers are typically more inventive at overcoming absorptive capacity constraints that often plague public providers. 301 Fourth, output verification helps to ensure that the benefits of a program reach the intended beneficiaries. The incentives encourage non-state providers to serve beneficiaries (usually the poor) they might otherwise disregard, thereby achieving better targeting.302 B.2.2. Provider autonomy increases innovation. RBF can be used as a mechanism to promote innovative technical solutions. Moving from processes towards impacts tends to increase the scope for inventiveness on the part of the implementer. Implementers are given latitude, i.e. more discretion and autonomy, on how to accomplish the intended results. This greater flexibility can lead to creative responses to problems. 303 B.2.3. RBF brings clarity and transparency. Funding tied directly to outputs can make the objectives and their achievement more transparent. The agreement on end results makes clear what is being targeted. The heightened focus on measurement shows whether it is actually accomplished. 304 B.2.4. RBF enhances accountability. Transparent results also make service providers more accountable. 305 Shifting the performance risk to service providers increases accountability for results (i.e. they are only paid after they have delivered 299 Savedoff 2016: 4; Loevinsohn 2008: 7, DFID 2014: 10; 82. 300 IDA/GPOBA. November 2009: 23. 301 Loevinsohn 2008: 83. 302 IDA/GPOBA: 12-13; Mumssen, et al. 2010: 7-8, 99. 303 USAID: 2,5; Savedoff 2016: 4; SIDA. 2015: 11; WB 2017: 12, 14; Loevinsohn 2008: 82. IDA/GPOBA 2009: 21; Musgrove 2011: 3; Carnavan, et al. 2008: 21. 304 Felstead 1998: 17; DFID 2014: 10; WB 2017: 28. 305 Felstead 1998: 17. 114 an agreed output).306 More accountability should reduce cost overruns and benefit shortfalls. 307 B.2.5. RBF in theory lowers costs and raises efficiency. Payment for results induces providers to develop the most efficient ways of delivering services. Since funding is based on the costs of delivering outputs, waste or inappropriate allocation of such funding should be minimized. In theory corruption that diverts spending will reduce results and the amounts paid.308 As an example of efficiency, in vocational education and training results -based approaches gives providers an incentive to reduce training time to the minimum required for trainees to achieve the output .309, 310 B.3. Limitations and risks in results-based financing Results-based financing schemes may entail risks that daunt providers, increase costs and create unintended consequences. These risks are explained below. B.3.1. Performance risks may limit the number of providers and increase costs. The transfer of risk from funders to service providers is a core element of the RBF approach. Agreed upon results may not be reached because of factors outside the control of the provi der, thereby placing the provider at a disproportionate level of risk. The less control the provider has over the output, e.g. employment, the higher the risk to participation in the program. Employment is a function of many variables, macro and micro, and is not always under the control of the trainer, particularly in highly distorted economies that are not creating wage employment. Moreover, under PBF, activities have to be pre-financed regardless of results. The uncertainty of contingent disbursement may deter those more averse to risk and loss. Many service providers, particularly smaller ones, may be unable to shoulder the burden of pre-financing. An additional risk to providers is possible delays in payments. 311 Excessive risk can thus discourage providers and limit their number. Smaller actors may be particularly affected.312 Experiences in DFID’s Girls’ Education Challenge found that smaller NGOs were less likely to bid for contracts.313Alternatively, providers may require higher rewards to compensate for the extra risk. Recipients must have a large degree of control over outcomes. Otherwise risks may inflate prices.314 B.3.2. Verification requirements can increase costs. RBF entails considerable efforts in initial design and additional costs for monitoring and verification. An extra layer of bureaucracy sometimes has to be created to ensure adequate monitoring and evaluation of service delivery. Third parties are often used to conduct assessments. Transaction costs can be extremely 306 Mumssen, et al. 2010: 7-9, 99. 307 IDA/GPOBA 2009: 16,17. 308 Savedoff 2016: 12. 309 On the other hand, it may prove difficult to innovate when low costs are a primary concern. A focus on keeping costs low may also jeopardize quality at the altar of efficiency. (Results for Development Institute 2015: 39). 310 Loevinsohn 2008: 82; Mumssen, et al. 2010: 10; Felstead 1998: 16, 60; also Messen, et al. 2010: 3-4. 311 IDA/GPOBA 2009: 18. 312 IDA/GPOBA 2009: 18; WB 2007: 27; USAID: 2. 313 Holden and Patch, 2017 in UNESCO 2018: 7; WB 2017: 27; Results for Development Institute 2015: 36; Clist 2016, Holzapfel and Janus 2015 in WB 2017: 27. 314Clist and Verschoor 2014: 2. 115 high. As an example, in a Rwanda health program a team of five district health professionals used an average of five hours each per month just to validate in one health center.315 Parallel M&E system increase costs substantially often without building country capacity.316 The question must be asked whether the additional administrative costs of RBF are justified by additional benefits. B.3.3. Incentives may lead to unintended consequences. Results-based incentives might distort and undermine program goals. Gittner laid out seven types of unintended consequences, as follows: 317 • Unrewarded activities may be ignored. Incentives placed on one factor may skew the focus of the provider and lead to neglect of other important factors. For example, over-emphasis on reducing dropouts to improve learning achievements may divert attention from the need to raise teacher competencies. • Providers can inflate services to maximize payments. Providers may oversupply services to increase their performance payments. For example, payment on a per-trainee basis led employer associations in Bangladesh to expand training beyond apparent market needs. 318 • A short-term focus that may undercut institutional development and sustainability. Too much attention to short term goals diverts attention from long goals that are not rewarded immediately. For example, third-party verification may be introduced at the expense of long- term improvements in institutional capacity. • Incentives for quantity may overwhelm delivery quality. A DFID review found that basing payments largely on quantitative targets created negative consequences in terms of quality of service provision. • RBF offers opportunities for fraud because incentives can be “gamed”. Service providers might manipulate measurements or cheat on reporting to improve recorded performance instead of improving the actual delivery of services. • “Cream skimming” or “cherry-picking” may occur. 319 If a service is rewarded on a fixed price per recipient, providers could maximize output and remuneration by selecting those most likely to succeed or who require less effort. This can filter out high priority recipients, such as the disadvantaged. • Incentives may dilute intrinsic professional motivation. NGOs tend to have intrinsic motivation to do well in servicing disadvantaged groups. Extrinsic motivation through financial incentives may replace intrinsic motivation. B.4. Findings B.4.1. Overall, sources point to promising results, but note the lack of extensive evidence. 315 Pearson, et al. 2010: 39. 316 USAID: 2; UNESCO 2018: 7; Eldridge: 165; Pearson, et al. 2010: 38. 317 Gittner 2013: 19-21. See also Klingebiel 2012; Clist and Dercon 2014: 1; Pearson, et al. 2010: 4,33,36; SIDA 201511,13; Eldridge 2009: 164; Oxman and Fretheim: 79; Carnavan, et al. 2008: 20; USAID: 2. 318 SEIP. 319 These terms can be defined as selection and concentration on those most likely to succeed rather than those most in need (Holzapfel and Janus 2015: 14). 116 Reviews have found that the evidence base on RBF is weak.320 Moreover, it is difficult to single out the impact of the results-financing measure since it is typically just one component in a package of measures.321 Perrin indicated that most evaluations have been in the health sector but could not attribute changes observed to payment by results. As a result, the value of RBF is unproved so there is need for hea lthy skepticism. Others question whether better results are worth the unintended consequences and additional costs, or whether such schemes provide long-term, sustainable solutions. 322 Several important reviews have been more positive. The importance of a results orientation is largely uncontested. A World Bank review on output-based aid in 2009 and 2010 reported that some evidence indicates that incentive schemes have led to improvements in operational efficiency and the delivery of innovative, often pro-poor, service solutions. In addition, the output-based approach has demonstrated efficiency gains through competition in the selection of service providers (although competitive tendering can take time). A review by DFID also concluded that RBF schemes generally deliver results. Referring to health, one source claimed remarkable improvements achieved in health indicators under diverse settings. Other reviewers point to an increased supply response by the private sector, increased effectiveness through accountability, lower cost overruns and benefit shortfalls and reduced corruption as well as improved management and implementation. 323 One claimed that incentives for providers are associated with higher chances of success and larger size effects. 324 B.4.2. In practice there are few if any “pure” RBF schemes. RBF programs use a wide range of points on the results chain. Operations can be interpreted as more or less RBF depending on their choice of disbursement strategies. Many agreements pay for achievements at different points of the results chain. Indicators range from processes (protocols and procedures), to intermediate outputs (e.g. teacher recruitment) to outcomes (e.g. test scores). For example, most health projects used output indicators.325 Most projects do not elect for purely payment on delivery; instead they select a gradual approach as outputs are built.326 B.4.3. Results-based assistance depends first and foremost on the definition of ends being targeted. In identifying results, it is important to understand the complexities of the results chain and the location within it of the results sought. Definition, selection and measurement of performance indicators are critical. The process of contracting should start with a clear defini tion of objectives and 320 Pearson, et al. 2010: 36-38. 321 Felstead 1998: 15; 62. 322 Perrin 2013: i,iii; UNESCO 2018: 7; Pearson, et al. 2010: 1-2; Mumssen, et al. 2010: 10. 323 The 2009 IDA review compared 13 completed output-based aid projects completed in 2007 in water, energy and health with the sample of all 37 available ICRs for projects in those sectors that were closed in FY 2007. It found that 85 percent of output- based aid projects achieved or over-achieved the quantifiable PDOs, compared to 49 percent of traditional projects. Moreover, nearly 70 percent of the output-based projects were completed under budget, compare with about half the traditional projects. Some of the traditional projects recorded cost overruns, but none of the output-based projects did. (IDA and GOBPA 2009: 16). 324 Mumssen, et al. 2010: 9; Pearson, et al. 2010: 34-36; Carnavan, et al. 2010: 20; IDA-GPOBA 2009:ii; Loevinsohn 2008: 82; Kluve, et al. 2016: 38. 325 Perakis and Savedov 2014: 4 provide an instructive table showing examples of inputs, outputs, outcomes and likely indicators for six sectors: education, health, water, transportation, deforestation and energy. 326 O’Brien and Kanbur: 8,22; SIDA. 2015: 7; AGI e 2015: 4-5; O’Brien and Kanbur 2013: 22; Carnavan, et al. 2008: 20; O’Brien and Kanbur 2013: 17,22; Perakis and Savedoff 2015: 5; SIDA. 2015: 7; AGI e 2015: 4-5. 117 indicators. Results indicators should be precise, measurable, hard to dispute (given their importance for disbursement) and few in number. 327 B.4.4. The price of incentives must be calibrated high enough to motivate recipients but also low enough to avoid windfalls. Key design tasks are to set the size of the incentive and allocate it along steps in the results chain (assuming there are multiple stages for payments). The essential question is what level of incentive is required to reap optimum benefits. The amount of the incentive should strike a balance between the financial needs of providers while still garnering interest and motivating them. As an example, the small size of the bonus incentive offered in a Uganda transport progr am (between 5-7 percent of operating costs) was one of the main reasons why RBF had no effect on the use of services. 328 B.4.5. Adequate organizational and institutional capacity is an antecedent for success and must be developed further. A minimum threshold of administrative/delivery capacity has to be assured in both the managing agency and the service provider. In addition, various forms of assistance should be provided to develop capacity further. Capacity requirements limit the applicability of results-based financing schemes. For RBF to work implementing agencies must have sufficient prior capacity to carry out basic administrative functions. These include setting up incentive mechanisms, organizing bidding processes, managing contracts, monitoring and evaluation including verifying results, data collection and analysis, and financial management to process budgets and payments. RBF heightens the need for adequate systems for monitoring and evaluation of service delivery. Precise verification is vital for contract payments. Competitive bidding processes take time to develop and can require extensive capacity building. A key question is also the extent to which service providers can manage the task of rigorous monitoring, particularly on a large scale or outside urban areas Provider capacity must include ability to recruit and deploy the necessary physical, financial and human resources. Non-state actors may not have the capacity to scale up their efforts at service delivery. For example, p roviders in the DFID- financed Girls’ Education Challenge faced challenges in monitoring and evaluation reflecting weak capacity.329 Apart from minimum capacity requirements, assistance should also be provided to develop that capacity further. PBF needs to be implemented as a package with other kinds of support for building institutional capacity. Concomitant technical assistance is often needed to build institutional capacity. Provision of training to providers can reduce performance risks and enhance implementation.330 B.4.6. Provider participation depends importantly on mitigating their financial risk. 327 O’Brien and Kanbur 2013: 12; O’Brien and Kanbur 2013: 11; Pearson, et al. 2010: 45; Barder, et al. 2014: 1; USAID: 5; Loevinsohn 2008: 3,4; Clist and Dercon 2014: 2; 23: 1. 328 Results for Development Institute 2016: 6; Pearson, et al. 2010: 46. 329 Pearson, et al. 2010: 45; DFID 2014: 5,8; Carnavan, et al. 2008: 19; Macharia 2015: 6; Eldridge: 165; Oxman and Fretheim: 78; IDA/GPOBA 2009: 20; Coffey 2016 in UNESCO 2018: 5. 330 Perrin 2013:ii; Carnavan, et al. 2008: 19; Klingebiel 2012: 17; Results for Development Institute 2015: 37-38. 118 Three factors increase the costs faced by agents (contractors): (1) extent to which costs are under their control; (2) size of upfront investment required; and (3) time between investment and payout. The degree of performance risk depends importantly on the ability of the provider to pre-finance. Provider risks can be mitigated through several measures, including: • providing adequate information so that providers will be able to gauge risks correctly.331 • partial pre-funding- provide a portion of the payment upfront, such as for mobilization.332 • partial payment for partial results-- pro rata payments by degree of outcome.333 • training and capacity building for contractors, community engagement, knowledge sharing and peer learning. 334 B.4.7. Distortions may not be completely avoidable, but they can be moderated. One review concluded that despite careful design RBF schemes still face fraudulent reporting. Narrow high-stakes schemes were more strongly affected. It concluded that some problems may be inherent and cannot be eradicated fully. Still, potential unintended effects should be anticipated and monitored. In contracting health services, the major corruption issues tended to arise during selection of contractors and processing of payments. Corruption can be minimized by: • using competitive selection. • having large contract lots that attract more competition and larger providers. • making bid evaluation transparent by clear selection criteria and multiple stakeholder involvement in evaluation. • setting explicit conditions for payment. 335 B.4.8. Verification of results is vital for effective results-based financing. Indicators must be verified openly, transparently and independently, i.e., not by the service provider. Verification is the most tangible cost of results-based financing and can be time and resource intensive. 336 This suggests a need to strike a balance between thoroughness of verification and the need for timely reporting, i.e. a tradeoff between precision and cost. 337 Sufficient time should be devoted in advance to determine how data will be collected and the costs of verifying results. Verification can take many forms, including: (i) existing monitoring procedures if there is sufficient confidence, (ii) building in-country capacity and management information, (iii) impact evaluations, (iv) sample surveys, with results from the sample extrapolated to the whole population, and (v) independent, third part corroboration. Sample surveys are workable but may 331 WB 2017: 44. 332 Mumssen, et al. 2010: 9; IDA/GPOBA 2009: 14; Klingebiel 2012: 17; Results for Development Institute 2015: 36,38; WB 2017: 27. 333 Results for Development Institute 2015: 6, 37-8; WB 2017: 27. 334 Fritsche et al. 2014: 6. 335 Pearson, et al. 2010: 33; Perrin 2013: iii; Loevinsohn 2008: 90; see also checklist in Loevinsohn 2008: 67-8. 336 However, one source claimed may be partly offset by reductions in transaction costs and management time in implementation: Clist and Dercon 2014: 2 337 O’Brien and Kanbur 2013: 13,14; O’Brien and Kanbur 2013 : 13,14,22; Loevinsohn 2008: 4; 22; AGI e 2015: 4; Gittner 2013: 1; IDA/GPOBA 2009: 22. 119 introduce error if the sample is unrepresentative. Independent, third party verification is costly, often impractical and does not resolve capacity constraints in the long term. Some projects reportedly have assumed that since data are needed to demonstrate results, monitoring and verification systems inevitably will be strengthened. However, evidence suggests this is not guaranteed.338 B.5. Elements of good practice Several elements of good practice can be added, as follows: B.5.1. RBF should be introduced gradually. Experience suggests the wisdom of using a gradual approach. RBF schemes should be piloted with systematic evaluation before scaling up. 339 B.5.2. Examine carefully whether existing systems and capacity are sufficient. Appraisal has to examine carefully whether existing systems and capacity are sufficient. T his applies to both institutional and administrative capacity as well as financial capacity, i.e. the extent to which service providers can manage the additional risks of RBF compared with traditional input financing. 340 B.5.3. Competitive bidding helps lower costs but may raise entry barriers and delay implementation. The way contracts are awarded is a key step in achieving cost -effectiveness. Evidence demonstrates that competition is important in procurement and has resulted in lowering some delivery costs. However, competitive selection of providers may raise entry barriers to new training providers, especially in areas of high training costs and with smaller providers that cannot shoulder the financial risk. Principals and agents are often unfamiliar with competitive procurement processes which have resulted in underbidding and delays.341 B.5.4. To the extent possible risks should be largely within the control of the contractor. To what extent does the implementing organization have control over the results. Therefore, it is important in design to consider what factors the recipient can actually affect to achieve results. Anchoring payment to results outside the implementer’s control is both unfair and demoralizing. 342 B.5.5. Allow room in contracts for service providers to innovate. Contractor discretion can help lower costs and stimulate innovation. Consequently, it is important ask whether the contractor has flexibility in delivering intended results. An overly prescriptive approach stifles innovation. Contractor discretion should include use of personnel, procurement of needed 338 OBA Approaches: 1-3; Klingebiel 2012: 12; SIDA. 2015: 10; 8: 5; WB 2017: 11, 26; DFID 2014: 8; O’Brien and Kanbur 2013: 14, 22; Loevinsohn 2008: 4; Pearson, et al. 2010: 47; UNESCO 2018: 7. 339 Pearson, et al. 2010: 45. 340 DFID 2014: 8; Carnavan, et al. 2008: 19. 341 Pearson, et al. 2010: 46; Felstead 1998: 61. 342 DFID 2014: 5; WB 2017: 27; Results for Development Institute 2015: 36; Clist 2016, Holzapfel and Janus 2015. 120 goods, etc. 343 B.5.6. Contracts should balance rewards with sanctions. The possibility of non-payment should be explicit for results that fall below targets. Non-payment has to be credible and funders should be serious about withholding payments. Contracts should include mechanisms to resolve disputes; enforcement requires termination processes. Penalties should be considered for non-performance. 344 B.5.7. Provide ample consultation. Finally, reviewers appealed for consultation among the parties. Stakeholders should be involved in the design of indicators as well as the design of contracting systems. Mutual understanding and agreement in advance pay dividends in more effective execution and fewer disputes. In particular, close consultation with potential contractors about rules and expectations enables them to assess performance risks. The idea of nonpayment is so alien from normal contract parameters that it must be discussed and agreed upfront. Contracts for education providers under the Girls Education Challenge failed to specify how this would be applied in practice. This led to confusion and frustration among implementing organizations and delayed implementation.345 B.6. Funding mechanisms for vocational training Table A.12 shows various funding mechanisms for institutional training. Table A.12: Overview of funding mechanisms in vocational training Type of Description mechanism Direct Ad hoc Institutional funding based on Allocation funding previous year allocations, political influence, interest group pressure Normative Institutional inputs Typically based on size of trainee formula enrollment multiplied by a funding parameter of unit costs Performance related- training Based on numbers completing outputs training Performance-related- training Based on success in certification quality examinations; on-site inspection Performance-related- training Based on placement in outcomes employment: meeting labor market needs; on-site inspection Composite formulas Combinations of the above 343 IDA/GPOBA 2009: 21; Loevinsohn 2008: 5; Clist 2016 in WB 2017: 27. 344 WB 2017: 26, 44; Clist and Dercon 2014: 2; Loevinsohn 2008: 6; Results for Development Institute 2015: 6; Pearson, et al. 2010: 46. 345 Results for Development Institute 2015: 6; Loevinsohn 2008: 2; Pearson, et al. 2010: 45-7; 22; 54; Coffey 2016 in WB 2017: 26; Brenzel 2009: 32-35; Perrin 2013: iii. 121 Contract Institutions contracted to provide funding for enrollment of special target groups Competitiv Unifying training markets: e tendering tendering open to private and public providers Indirect Trainee- Vouchers/entitlements, allocation based particularly for continuing funding training and lifelong learning Source: Ziderman 2016: 72 with minor adaptation in order, and italics added. 122 Annex C - RBF and the Use of Disbursement-Linked Indicators (DLIs) in Skills Training A DLI is a measure used to trigger payment of funds from a loan or credit. Progress towards achievement of the indicator becomes the focus of implementation. Key design elements i nclude determining the level of result on the chain of causality, pricing the result, distributing the amount of incentive among the DLIs, choosing scalable or target conditions and how to verify results. Five DLI-type operations were reviewed in skills training plus two outside the sector. DLIs were approximately balanced between quantity and quality objectives. The majority of DLIs pertained to inputs and activities, relatively few to outputs and outcomes. The weighting of DLIs ranged from equal amounts allocated to each DLI in three operations to heavily skewed in two cases. Almost twice as many DLIs used target thresholds as incremental progress. About half the DLIs used percentages as the unit of measure. Complexity increases with the number of DLIs and compound content of the indicator. Literature sources recommended a reliance on outcomes for payments, but only a few of the skills related DLIs did so. Heavily weighting DLIs risks distorted attention. In general, scalable DLIs have advantages over threshold DLIs because of difficulties in setting realistic threshold targets. Percentage -based indicators may be susceptible to manipulation. Development agencies should refrain from setting up parallel systems for verification. Instead, they should rely on country-generated data with independent verification. Unintended consequences-- including distortions, data manipulation and cherry-picking – can be mitigated through precise payment conditions, careful verification and sanctions for abuse. DLIs help focus on results and can enhance donor coordination. The incentives can promote diversity and quality in training. In addition, they can support the pursuit of system reforms. However, results-based approaches should only be implemented where direct measures exist of intended results. Precision is necessary in formulating DLIs. A robust monitoring and verification system are critical for success. C.1. Overview of results-based schemes using DLIs 1. DLI mechanisms The DLI mechanism is results-based financing since payments are made for achievements of pre-agreed results. These results are defined in the form of indicators. Progress on the indicators is measured and accomplishment triggers release of funds. 346 DLIs become the focus of implementation. The choice of indicators, consequently, is critical to success of RBF. The DLI mechanism can be used with different lending modalities, including investment project financing and PforR. 2. Importance of indicators Indicators summarize information required, but do not explain why something has occurred 346In a sense disbursement-linked conditions are not unique to results-based instruments. Every financing operation has some sort of conditions to be met before payments are made, such as submission of evidence of approved expenditures (Gelb and Hashmi 2014: 2,7). 123 and may not strictly demonstrate causal chains. 347 Indicators can: • be quantitative (e.g. number, percentage or ratios), or qualitative (existence - yes or no) or category (high, medium, low). • be direct (e.g. competency obtained or test scores, employment), or indirect (i.e. measuring something similar to, but different from, the result; for example: % of trained instructors as a measure of quality). • measure results of an intervention (outputs, outcomes and impacts) or processes (inputs and actions taken/work performance- e.g. funding, number of training programs developed). • be threshold (met or not met), or scalable (open-ended payment proportional to degree of achievement up to a ceiling). 348 Selecting the right set of DLIs not only requires good sector knowledge and but also considerable skill on the part of the program designers. It may also entail long periods of negotiation with government over the key results and the annual DLI targets. 349 3. Choices in the design of DLI-based operations 350 The development of DLIs involves a series of important choices at the design stage. These are explained below: What level of results? The first choice is payment for what results, i.e. which stage on the results chain. Are the results outcomes, outputs or processes (inputs and activities)? What is the price of the result? Financial incentives are the bedrock of results -based approaches. The result and incentive have to be priced. No clear indicators exist for setting the overall price of results, but the literature suggests three important criteria: • Value for money, i.e. whether the financing amount is reasonable for the estimated cost of producing the result. • Leverage, i.e. whether the amount is sufficient to capture the interest of key parties (policy makers, implementers). • Level of risk- whether the amount compensates implementers sufficiently for bearing the performance risk. 351 How is the price distributed by DLI? Programs invariably use multiple DLIs. This makes it necessary to allocate a share of the total price to each DLI. 352 What type of indicator? Another design decision relates to the type of condition, that is, whether the indicator is scaled in proportion to performance, or conditional on achieving a target 347 Holzapfel and Janus 2015: 3. 348 Ibid. 349 O’Brien and Kanbur 2013: 9; ADB 2017a: 4. 350 The following section is drawn largely from Holzapfel and Janus 2015: 7-14. 351 O’Brien and Kanbur 2013: 16; and Holzapfel and Janus 2015: 9 -10. 352 O’Brien and Kanbur 2013: 16. 124 threshold.353 If it is incremental, another consideration is whether the increment is a fixed total or a total above a threshold (which requires a baseline). 354 4. How to verify results Design of DLI-based operations also faces the questions of how to verify results . To ensure credibility the measurement of DLIs must be valid (i.e. measure the result that is intended) and reliable (consistent over time). Data have to be independently verifiable to convince actors that the measurement and compensation of their efforts are fair and transparent. The ease of data verification varies by level of result. Inputs, activities and outputs are reported to be relatively easy to measure, although there may be room for interpretation (e.g. on whether a reform was implemented as planned.) Attributing changes to specific interventions is fairly straightforward up to the level of outputs. Measuring outcomes is more challenging because of the likelihood of intervening variables. These external factors make attribution difficult. 355 Options for measuring results include: (1) to rely on the country’s monitoring and evaluation systems for measuring progress, or (2) to use household survey data. However, survey data may not be timely, and are often unreliable and suffer from sampling errors. (A compensatory measure would be randomly checking administrative data). Other options, often costly, are to (3) create a parallel monitoring and evaluation system, or (4) commission independent (sample) surveys. 356 C.2. DLI examples in skills development programs 1. Project examples The use of DLIs seems to be relatively recent for skills training, but has been done more extensively in other sectors, e.g. education. This report includes a cursory review five skills training projects that use the DLI mechanism. All are recent and still under implementation .357 The review also includes an employment support operation in Bosnia and Herzegovina. The five skills training projects generally seek four things: strengthening of institutional systems, increased access, improved quality, and better relevance of training to the labor market. The Pakistan-Punjab Skills Development Project (PSDP- 2015-2020; World Bank US$ 50 million): aims at improving the quality, labor market relevance of, and access to, skills training programs in priority sectors in the Punjab. The main components include support for effective registration and certification of the skills training sector, strengthening the provincial training agency as a service provider, expansion of competency-based training and assessment, preparation of partnership agreements with industry and expanding the supply of skilled workers in market relevant trades. 353 Gelb and Hashmi 2014. 354 Holzapfel and Janus 2015: 10. 355 Holzapfel and Janus: 9. 356 Holzapfel and Janus 2015: 12. 357 The ADB-financed SL/SSEP completed its first phase in 2017 and a second phase was approved in 2018 with virtually the same content. 125 The India- Skills Strengthening for Industrial Value Enhancement Operation (STRIVE, 2017- 2022, World Bank- $125 million): focuses on building the capacity of Industrial Training Institutes and state management of skills training, strengthening employer participation in apprent iceship training as well as pre-employment training, and upgrading of instructors via distance training. The Skill India Mission Operation (SIMO- 2017- 2023; World Bank- US$250 million) features (i) institutional strengthening at the national and state levels for planning, delivering, and monitoring relevant training; (ii) improved quality and relevance of skills programs through better standards, curricula and learning materials, raised skills acquisition by female trainees and other disadvantaged groups; and (iv) expanded skills training through links with employers. The two Sri Lanka projects – SDP and SSEP - support implementation of the country’s Skills Sector Development Program (SSDP) and seek to: (i) improve the quality of training provision by strengthening registration and accreditation of providers, revision of qualification standards, and better policies to recruit, upgrade and retain qualified instructors; (ii) enhance industr y partnerships through skills gaps analyses, establishment of industry advisory groups and public -private partnerships; (iii) increase enrollment in areas of skill shortages and increase participation of women and other disadvantaged groups; and (iv) improve sector management, including better sub-sector coordination and ensure adequate budget allocations and utilization. 358 2. Characteristics of DLIs DLIs by subject. Table A.13 presents an overview of the content of the five training-related programs, showing a wide range of subjects. About three fourths (27 of 37) of the DLIs relate to institutional development and training inputs. Those referring to employer links, graduation and employment of graduates account for the remaining one fourth of the DLIs. In terms of frequency, the largest category of DLIs (11) referred to establishment or improvement of institutions, administrative systems and plans/agreements. Four DLIs explicitly made payments conditional on better linkages between training supply and employers. Surprisingly few (i.e. 3) pertained to increasing the number of qualified graduates (Pakistan and the two India projects.) Just three DLIs focused on outcomes (employment rate of graduates six months after completion of training. Table A.13: Number of DLIs in five skills training programs by subject Pakistan India India Sri Sri Total Subject PSDP STRIVE SIMO Lanka Lanka SDP SSEP Institutions, systems, plans/agreements 2 2 3 4 11 Training inputs- sub-total, of which: 1 4 4 4 3 16 Finance 1 1 1 3 Teachers 1 1 2 1 5 Curricula/ learning programs 1 1 1 3 Information 1 1 2 Enrollments 1 1 1 3 358Sri Lanka SDP 2014-2019, World Bank USD 101.5 million; and Sri Lanka Skills Sector Enhancement Program (SSEP) 2014- 2017, ADB USD100 million. 126 Employer linkages 1 2 1 4 Graduates 1 1 1 3 Employment 1 1 1 3 Total 5 6 8 9 9 37 Source: Author’s compilation. Given the limited number in the skills training sector, two additional operations are added to the examination. DLIs by stage. This small sample shows that multiple indicators are used at different stages of the results chain (see Table A.14, where the Bosnia and Herzegovina Employment Support Project (ESP) has been added as an employment program, although not focused on skills training). The DLIs show a wide distribution among subjects. DLIs in the six projects were roughly balanced between quality and quantity. In terms of stage in the results chain, 32 out 45 DLIs dealt with processes (inputs, such as the provision of financing and a reduction in the number of teacher vacancies , or activities, such as establishment of industry councils). A total of 8 out of 45 dealt with outputs (examples of outputs included completion of training and certification of graduates.); and only 5 out of 45 pertained to outcomes, such as employment rate of graduates six months after training. Table A.14: Summary Table- Number of DLIs by project and category Program or Objective No. of DLIs Number of DLIs by stage on Project results chain Quanti Qua Input Activi Output Outco ty lity ty me Pakistan Improve the quality, labor market 1 4 0 4 1 0 PSDP relevance of, and access to, skills training programs in priority sectors. India Improve access to quality and 5 1 1 3 1 1 STRIVE market-driven vocational training provided in ITIs and apprenticeships. India SIMO Enhance institutional mechanisms 5 3 1 4 2 1 for skill development and increase access to quality and market- relevant training for the workforce. Sri Lanka Expand the supply of skilled and 3 6 2 7 0 0 SDP employable workers by increasing access to quality and labor market relevant training programs. Sri Lanka Create a market-responsive and 3 6 2 6 0 1 SSEP inclusive TVET system that increases the employability of the Sri Lankan workforce, especially youth (imputed). Bosnia and Increase formal private sector 4 4 0 2 4 2 Herzegovina employment among targeted ESP groups of registered jobseekers. Note: the classification of DLIs by results stage can vary considerably according to perspective (see footnote 127 14, Gelb and Hashmi 2014: 8). The above classification uses the results-chain table at the beginning of the section. The two projects in Sri Lanka, one financed by the World Bank and one by the Asian Development Bank (ADB), have overlap in five of the nine DLIs. Source: Author’s compilation. The DLIs tend to be weighted differently. In the Punjab and Sri Lanka operations financed by the World Bank, all components were weighted equally in amounts, although in Punjab an extra year of disbursement on some DLIs meant greater total financing. Three of the projects had substantial variation. SIMO allocated 42% of the credit proceeds to DLI 6, in part because it is broadly defined to include three subjects: institutional development, relevance, and access. The Bosnia and Herzegovina ESP was also heavily weighted, with amounts ranging from about 9% (DLI 2) to 63% (DLI 1).359, 360 Disbursement types vary. The incentives in a substantial number of DLIs were scalable in proportion to performance. A good example of incremental payments is the Sri Lanka SDP. Each dollar raised from private enterprises earns another dollar in incentive payment. (DLI 9- “Upon achievement of target, US$1 for every US$1 of private funds credited to the CSR Skills Fund”). The two India operations also relied on incremental performance incentives. In fact, the two India cases combined both types in a single condition, i.e. scalable after first meeting the threshold. As an example, in STRIVE DLI 6, “USD 2,250,000 will be paid upon achievement of threshold of the second DLR; plus USD 450,000 for each additional industry cluster reaching a female enrollment of 20% thereafter ”.361 DLI operations use three types of triggers data on which to base disbursements: status descriptions (e.g. partnership agreement concluded), absolute numbers (e.g. number of trainees enrolled) and percentages (e.g. vacant posts reduced by 20%). Several operations relied heavily on percentages: four of the six DLIs in STRIVE use percentages; 4 of 8 in SIMO; and 5 of 9 in SSEP. Table A.15: Characteristics of the DLIs in Select Projects Project No. Weighting –range (%of total DLIs Disbursement type financing per DLI) Scalable Both Binary Pakistan- PSDP 5 0 0 5 18-22 India STRIVE 6 3 2 1 15-18.8 India SIMO 8 3 3 2 3.7-42.2 Sri Lanka SDP 9 0 0 9 All same Sri Lanka SSEP 9 0 0 9 5-17 Bosnia and Herzegovina ESP 8 2 0 6 8.6-63.1 TOTAL 45 8 5 34 359 Improved performance of states on institutional strengthening, market relevance of SD programs, and access to and completion of training by marginalized population. 360 Increasing job placement through active labor market programs. 361 Second DLR under DLI 6 = A minimum of 5 Industry Clusters receiving Industry Apprenticeship Initiative grants have reached a female enrolment of 20% in their respective apprenticeship programs. 128 Note: “scalable” (or incremental) means disbursements are pro-rated according to performance; “binary” means the condition/target or threshold either has or as not been met (yes or no); “both” means that some parts of the DLI are binary while others are incremental. Source: Author’s compilation. C.3. Findings and recommendations in the literature362 1. Level of results Holzapfel and Janus recommend that funders, to the extent possible, rely on outcome indicators for disbursements. The use of outcome measures has several advantages, but there are conditions when it may be appropriate to use DLIs lower on the results chain, such as when substantial pre- financing is required. The following table shows some of the advantages and disadvantages of using outcome indicators compared with process indicators. Table A.16: Advantages and disadvantages of DLIs according to scale on the results chain Payments based on: Process indicators Outcome indicators Advantages • Allows for more gradual and • Easier to reach consensus, earlier disbursements. • Leaves discretion to recipient to • Allows funding of steps needed choose inputs and activities and to strengthen recipient outputs, thereby increasing ownership capacity, and possibly innovation. • May be suitable if substantial pre-financing is required. Disadvantages Limits flexibility and discretion of Requires capacity of actors to pre-finance recipient implementation. and implement activities. Source: Adapted from Holzapfel and Janus 2015:8, 47. In contrast with the recommendation to use outcome indicators, a strong majority (30 of 37) of DLIs examined in skills training fell on the process side. Only 3 of 37 DLIs referred to training outcomes, i.e. employment. However, as stated, a focus on process indicators may be justified given the newness of the approach and extensive strengthening required in institu tions. 2. Pricing the results Among the three criteria for pricing results mentioned earlier, greater emphasis is recommended on potential leverage effects and compensation of risks as a method for pricing results, rather than value for money. Use of the value for money approach can reduce the incentive significantly. An efficient program (low input for outcome) in terms of value for money may not be effective in achieving desired objectives. 363 3. Allocating the amount by DLI Equal allocations to all DLIs should focus attention on the complete set of results, but it may 362 Holzapfel and Janus: 7-14 and 47-48. 363 Ibid. 129 lead to disproportionate attention on the results that are easiest to achieve. It may be better to allocate a higher share of funding to the most difficult indicators or those that have greater importance. However, too much value placed on one DLI risks distortion. 364 Two of the reviewed operations placed disproportionate emphasis on a single DLI- India SIMO DLI 6 (42% of total financing) and ESP DLI 2 (63% of total financing). In contrast, Sri Lanka SDP allocated an equal amount to all DLIs. Disproportionate financing on one indicator risks disproportionate attention and the neglect of other worthy targets. 4. Type of indicator Several reasons suggest the use of incremental DLIs (scalable) rather than threshold targets. Scalable indicators tend to be relatively stable incentives. The incentive for threshold targets can be substantial, but it is difficult to calibrate properly. The following table compares the advantages and disadvantages of scaled versus target indicators. Table A.17: Advantages and disadvantages of scaled and target indicators Type of Indicator Scaled (incremental) Target (threshold) Advantages Relatively stable incentives High incentive for performance at the margin Disadvantages Danger of performance • Low incentives if target is set too high (i.e. plateaus (more difficult to achievement is unlikely) or too low delivery results once certain (threshold can be achieved with minimum progress has been made). effort), • Difficult to make realistic assessment of what can be achieved with reasonable efforts and available resources within a given time. Source: Adapted from Holzapfel and Janus 2015:10, 47. Scalable DLIs tend to reward better performance and penalizes shortfalls. In addition, “…the evidence suggests that the performance risk inherent in results-based operations can be moderated by using several scalable DLIs that provide a graduated basis for disbursement rather than threshold-type on-off conditions. In addition, scaling disbursements provide relatively stable incentives compared with threshold-type on-off programs that might not disburse even if there is progress. Threshold-based DLIs provide a high-powered incentive at the margin, but also run the risk of discouraging effort when full target attainment seems unlikely. It may also increase pressure to modify the DLI so that disbursement can take place. 365 Only about one third of the operations reviewed above had scalable DLIs (16 scalable vs. 35 thresholds). 366 To the extent that scalable DLIs provide better incentives and moderate performance risk, their greater use may be justified in skills development projects. 5. Total increments, or above a baseline If incremental DLIs are used, a decision is needed on whether total achievements will be rewarded, or achievements above a baseline. Paying for additional improvements may be preferable 364 O’Brien and Kanbur 2013: 16; Holzapfel and Janus 2015: 10. 365 Gelb and Hashmi 2014: 15,18. 366 Adding the five ‘mixed’ to each side. 130 because it creates a higher incentive for the recipient to improve current conditions. Paying for additional improvement is only possible if an adequate baseline exists. 367 6. Ceiling on incremental progress Another question is whether a ceiling should be set on incremental progress, either in terms of performance or financing. Absent a ceiling, a program the budget could be exceeded. However, ceilings serve as implicit targets and may need to be avoided. 368 7. Complexity Complexity increases with the number and expanded content of DLIs. First, compound DLIs pack multiple subjects into one indicator. The India SIMO DLI No. 6 exemplifies this complexity. It encompasses improved performance of states on institutional strengthening, market relevance of training programs and access to, and completion of training by, marginalized populations. Second, even if the criteria for the indicator are transparent and monitorable, their number increases the complexity of the operation.369, 370 The Sri Lanka SDP program has 18 DLIs and sub-DLIs that almost inevitably increase implementation risks. 8. Consistency between agents and beneficiaries An important question for incentives is whether the implementing agent receives the financial reward. For example, the links may be diffuse if one level of government implements, but another level (or unit of government) reaps the financial reward. A disconnect between the two may weaken incentives. 9. Verification of results Holzapfel and Janus recommend relying on country-generated administrative data with independent verification to ensure fairness in determining disbursements. Funders should refrain from setting up duplicate parallel monitoring and evaluations systems as this weakens country ownership and increases costs. 10. Causation and attribution A results orientation should take precedence over concerns with attribution. The implementer/provider should have plausible control overachieving the activities and results that are measured by DLIs. “Outcome indicators should primarily be used to guarantee strong results orientation, while making sure that they can be influenced by the incentivized actor within the program time frame.”371 11. Minimizing unintended consequences 367 Holzapfel and Janus 2015: 10. 368 Ibid.: 11, 47. 369 Gelb and Hashmi 2014: 16. However, this assertion seems to contradict anot her, namely: “performance risk is somewhat moderated by having disbursement depend on a number of DLIs rather than a single indicator” (Gelb and Hashmi : 15). 370 Gelb and Hashmi 2014: 16. 371 Holzapfel and Janus 2015: 47. 131 DLIs are subject to the same risks as other types of results-based financing for unintended consequences. • Results-based approaches should only be implemented if good measures exist, i.e. if indicators are direct measures of the intended result. Proxy measures should only be used if there is a clear overlap between actions needed to raise the value of the proxy and actions needed to achieve the underlying objective. 372 • Incentives may divert attention from the underlying objective to the measure itself. Distortions may also include neglect of other important goals, e.g. targeting the enrollment rate but neglecting the quality of education. This can be partly mitigated by using several DLIs that encompass all important objectives.373 The agent should require genuine effort to influence results.374 • Data manipulation is always a danger with self-reporting. This risk can be mitigated by using several related DLIs and by strengthening public accountability mechanisms. Independent verification of results is crucial. Transparency and public peer pressure can be important complements. 375 • “Cherry-picking” can be reduced by disaggregating indicators so that they identify priority types of recipients, and by increasing the rewards paid for disadvantaged groups or remote regions.376 C.4. Findings in the specific operations reviewed The DLI-based operations in skills training, as reviewed above, are too recent to have findings from experience. However, some general lessons are cited in the documentation. The DLI mechanism appears well suited to broad reforms in skills development as well as quality upgrading and incremental improvements of system components. 1. DLIs help focus on results The use of DLIs with predefined implementation progress and performance targets in the results-based design of the Punjab Education Sector Program helped the sector dialogue to focus on the most critical factors necessary for the achievement of project development outcomes.377 Clear articulation of linkages in formulating indicators, DLI targets and protocols is important to focus efforts on factors that are critical to the achievement of the project objectives.378 2. DLIs linked to government program enhance donor coordination The results-based investment operation improved donor coordination by linking implementation 372 Clist and Verschoor 2014: 2; Holzapfel and Janus 2015: 11, 48. 373 However, a complex set of indicators may diffuse attention and reduce the effect of the incentive. 374 Pearson, et al. 2010: 28; Holzapfel and Janus 2015: 13,48. 375 Holzapfel and Janus 2015: 13,48. 376 Ibid.: 14. 377 World Bank 2012b: 25. 378 World Bank 2015b. 132 support to government strategy under a common results-framework (DLI matrix) and reducing the Government’s reporting burden.379 3. Imprecisely formulated DLIs hamper early programs Stronger analytical work is desired to select the set of PDO indicators, with greater rigor towards establishing, measuring and forecasting the extent of change expected on these indicators as result of the successful achievement of the DLIs. In addition, the DLI targets need to be formulated with precision to avoid any ambiguity and to ensure that the agreed protocols are also adhered to. 380 4. Ratio-based indicators are easily manipulated Ratios should not be included as PDOs because of sensitivity of these indicators to slight changes in numerator and denominator resulting in volatility and noise .381 5. A robust M&E system is critical to achieving results As demonstrated by RBF projects globally, a robust M&E system is critical for measuring performance and refining interventions through better-informed decision making. 382 6. Incentives can enhance diversity and quality in training Policy interventions to incentivize training providers are essential for enhancing diversity and excellence in training and improving accountability for training results. 383 379 World Bank 2012b, PESP ICR: 25. 380 Ibid. 381 Ibid. 382 World Bank 2015b. 383 World Bank 2014b. 133 Annex D: “How to” guidelines on RBC D.1. How to apply a results-based contracting system Table A.18 provides additional guidance and steps to be considered , borrowing from the health sector in which RBC has grown a lot over recent year. Table A.18: Checklist for Performance-based Contracting (in the Health Sector) Task Step 1: Conduct 1. Establish a consultative process with stakeholders Dialogue with • Hold a few discussions with each set of stakeholders Stakeholders • Get back to stakeholders with draft proposals and contracts 2. Identify champions 3. Address the legitimate concerns of stakeholders: • Government health officials, • Local politicians and government officials, • Government health workers, • Private sector, including NGOs, • The community, • Development partners and donors. Step 2: Define the 4. Define the objectives of the contract: Services • Limit the number of indicators, (Develop • Design bias toward outcome/output indicators, TORs) • Ensure that indicators are independently measurable, • Define indicators as precisely as possible, • Set targets broadly. 5. Include objectives related to equity and quality 6. Consider pay for performance 7. Ensure that purchasers and contractors focus on objectives 8. Define the size and location of each contract “lot” 9. Define the scope of services—focus on “what” not “how” Step 3: Design the 10. Decide how data will be collected: Monitoring • Health management information system, and • Household surveys, Evaluation • Health facility assessments, • Supervisory checklists. 11. Collect baseline data 12. Devise a clear schedule for data collection 13. Look for comparison/control groups 14. Assign responsibility for collection, analysis, and dissemination of data 15. Budget sufficient funds for monitoring and evaluation Step 4: Decide How 16. Use a competitive selection process to Select 17. Develop clear selection criteria Contractors 18. Establish a transparent and independent evaluation process and Establish 19. Maximize interest of possible contractors 134 the Price 20. Select a contractor Step 5: Arrange for 21. Define responsibility and clarify the contract management structure Contract 22. Ensure that proper staffing of the contract management unit Management 23. Allow sufficient budget for contract management 24. Consider computerization of contract management 25. Develop a written contracting plan (or contract manual) Step 6: Draft the Contract 26. Maximize managerial autonomy: and Bidding • Clarify authority of both parties, Documents • Focus on “what” not “how”, • Discover other ways of increasing autonomy. 27. Ensure that contractors can manage personnel effectively 28. Use lump-sum contracts 29. Leave procurement of supplies, equipment and services to contractors 30. Ensure that duration of the contract is sufficiently long 31. Have clear procedures for making payments: • Mobilization payments, • Regular payments. 32. Establish a clear process for termination and imposing other sanctions 33. Establish dispute resolution mechanisms 34. Define reporting requirements of the contractor 35. Have an explicit policy on user charges 36. Ensure that contractors use independent, private sector auditors 37. Ensure that contractors build the capacity of health workers 38. Address the capacity needs of contractors 39. Clarify responsibilities for physical infrastructure 40. Formulate the bidding documents Step 7: Carry 41. Track the schedule of the bidding process out the 42. Conduct regular monitoring visits Bidding 43. Meet with stakeholders frequently Process and 44. Review the contracting plan and the contract Manage the Contract Source: Loevinsohn, 2008: 67-8. Note: Tasks in italics should receive particular attention. D.2 Lessons from NEF on how to apply RBC to an employment and training program Table A.19 distills the lessons from the NEF on how to apply RBC in an employment and training program. The step-by-step guide indicates the salient stages and highlights the lessons learned from the NEF. Annex D points to additional guidelines, instruments, and references, in the application of RBC for short-term training. Table A.19: How to apply a results-based financing system: a step-by-step guide from the NEF Step Activities Lessons Learned, Risks, Precaution 1. Define Discuss and agree with key • Since the outputs and outcomes will be project outcomes stakeholders (including training linked to payments, they need to be and outputs and employment providers) the carefully formulated and defined (see 135 outputs and outcomes of the indicators below). intervention planned. • Be careful to define achievable outcomes: Attainment of these is not Outline the activities that will only dependent on the performance of lead to the achievement of collaborating training providers, but intermediate results (outputs) also a range of externalities. and ultimate results (outcomes). • Achieving a project objective of improving labor market outcomes Harmonize project’s outputs and such as employment or earnings is outcomes with overall (country) more appropriate and measurable program’s outputs and outcome. rather than attempting to achieve better “employability” or measure the training completion rate only. 2. Make Define the indicators based on • Define SMART indicators (specific, smart result which the results achievement measurable, attributable, realistic, indicators will be measured. time-related). • Timing: consider pre-, during- and Define and agree on the result post-training phases when setting measurement process with the indicators. Training Providers. • Think well about how monitoring will be conducted logistically, since it is time-intensive and requires sufficient dedicated human resources. • Think well about the results verification method. Will it be done on a population or on a sample basis? 3. Agree on Define competitive prices for • Information with regards to market the price that will each of the outputs and price needs to be collected by the be paid upon the outcomes. project to avoid excessive quote. achievement of the • The price ceiling needs to be given to results (output and Request training providers to the training providers before asking outcome price). submit their outcome price quote for the quote. based on market prices. • The detail cost items needs to be defined based on which training Negotiate prices based on providers can submit their quote. market rates (where available) or based on the estimated cost including a profit margin. 4. Discuss Conduct information workshops • Most training providers will not have the concept with on the payment approach. worked under (the RBC) approach. the partners Their concerns revolve mostly around Allow enough time for the high financial risk they bear, since questions, have examples ready, they will lose their investment if the be prepared for skepticism. trainees are not gainfully employed. 136 • Meet partners, discuss their plans individually. • Provide calculated examples in which partners have made a profit upon the achievement of results. 5. Agree on Take time to ensure that the • It is crucial to have the collaborating the result training providers understand training providers “on board” the indicators with the result indicators, and when project. They need to be fully aware of partners as well has how they are the opportunities and risks of this measured. financial model. • Developing relationships at “eye” Respond to (critical) questions level with the partners because they and queries in order to ensure might feel that the project is training providers understand “controlling” them too much because them, and that the project can of performance-based indicators and count on their commitment. the related rigorous monitoring system. 6. Incorporate Incorporate result indicators and • Orient partners on the conditions of indicators in the their measurement system the contract, so that all parties have contracts, and link clearly in the contract with the same level of understanding on the them to progress partners. contractual provisions. and payment • Pay for the partners’ services in installments Link the progress with payment installments. Link the larger part of the installments for the partners payment with the achievement of the against these defined and agreed outcome, and the smaller part to indicators. outputs. • There is no payment for activities. Ensure the contract is fair for Do not pre-finance: the first payment both parties Clearly indicate is released only after the achievement deliverable results. of the outputs. Clearly specify the payment mechanism linked to the results. Reward better results through higher payment (incentive) and deduct payment for poor result (dis-incentive). 7. Clarify the Define and agree on • Even though it is a results-based process of implementation steps. payment system, the processes and the implementation quality of their services need to be Define and agree on quality discussed and agreed with the indicators that need to be partners. Neglecting this part endanger followed during the achievement of the desired result. implementation. 137 8. Monitoring and Set up system that allows project • Employment Fund used an online data storage staff and training providers to database accessible to both partners follow and see the progress, and project staff. In that way, the data intermediate achievements and can be accessed anytime, even from results transparently. remote locations (provided there is internet connection). • Graduates can be tracked and their progress followed through this online system. The data can also be shared with other projects in order to avoid duplication. 9. Periodically Have regular review meetings • Bi-annual meetings were held with review and with training providers. training providers. Regular feedback improve was obtained from them between these implementation Hold reflection meetings with meetings through the monitoring staff, project/ monitoring staff. and in bilateral meetings. • While piloting innovations, regular feedback workshops were held to see how the pilots were going, and in which areas improvement was needed 10. Assess The training providers are • Poor performing training providers are not performance of assessed based on their eligible to work with Employment Fund in training providers performance, collaboration and the future; in some cases, Employment the success in achieving the Fund also terminated contracts mid-way defined goals. due to non-compliance. • The best training providers are rewarded, but even successful partners have to pass the procurement process annually to be able to continue their work with Employment Fund. Source: Nepal Employment Fund Secretariat 2015. D.3. Step-by-step guide to monitoring of RBC Table A.20 explains the stages, activities and lessons on monitoring based on the NEF program. For additional guidance, the Adolescent Girls Initiative (AGI) tools on monitoring are also a useful resource.384 384 https://www.s4ye.org/agi/html/Monitoring_and_Evaluation_Verifying_Employment.html 138 Table A.20: How to conduct monitoring in a results-based contract: Step-by-step guide from the NEF Stage Purpose of Activities Application in Lessons learned, Risks, monitoring Employment Precautions Fund 1. Pre-training Compliance • Monitoring of 60-75% training • For adequate monitoring in a monitoring (e.g. trainee training events are countrywide project, employment selection) announcement monitored fund has 5-6 regional offices, • Monitoring of headed by regional Monitoring trainee officers and supported by field selection officers and field Monitors. • Training venue • Around 80% of the employment monitoring fund staff is almost exclusively 2. Monitoring Compliance • Monitoring of All the training dedicated to monitoring. during training (e.g. the quality of events are • Monitoring of process as well whether the training monitored up to result (especially the compliance training (physical three times during with agreed quality standards for standards facilities, their conduction. training) is a key contribution to are met) curricula, tools the achievement of the anticipated and result. equipment, consumables, training delivery, qualification of trainers etc.) • On the job training (OJT) monitoring Skills test process monitoring • Monitoring of employment placement preparation 3. Post- Verification • Employment • 10-15% • The monitoring workload is training of Results verification/ stratified highest towards the post-training monitoring (i.e. wage establishment random phase. Employment Fund hired employmen of enterprises sample during additional staff (field monitors) to t & income; 3 months post three months be able to physically monitor 30- self- training employment 40% of all trainees. employmen • Employment verification • Due to the project cycle, the t and job and • 30-40% monitoring needs to be completed creation) income/job stratified at the end of the year. Allow creation random enough time for planning the verification sample during logistics and for the monitoring to after 6 months 6 months for be able to complete the task in post training physical time. employment • One field monitoring staff member and income visits approximately 1,500 verification trainees, i.e. 75 training classes 139 • Extrapolation annually. of findings to • Statistically sound system of the whole generating stratified random population sample with an online database software (Dot Net). • Samples are drawn randomly based on five strata: gender, training events, categories A-D, districts, and trades. • Send monitoring staff to different regions for the final verification of the graduates. This enhances independence and fosters cross learning • In contexts that disable physical verification, e.g. graduate working in very remote area, or migrated overseas, employment fund gathers information from close family members and/or employers. as a last option, it conducts telephonic verification Source: Employment Fund Secretariat, 2015. D.4. Model contract for short-term training using RBC This section presents specific guidance on drafting contracts with training providers. It provides a model of the essential subjects and clauses to include in a results-based contract for short-term skills training. Agreement between Training & Employment Provider (T&E) and Nepal Employment Fund Contract between Employment Fund Secretariat (EFS) and……………………………… Whereas, Employment Fund, (hereafter called NEF), a fund jointly established by SDC, DFID and Helvetas Nepal and funded by SDC, DFID and The World Bank and managed and administered by Employment Fund Secretariat (EFS)/Helvetas Nepal (HN); and ………………………….. (hereafter called T&E) entered into agreement to provide Training and Employment (T&E) services to economically poor and socially discriminated youths in Nepal using outcome-based payment modalities as elaborated in this contract and the subsequent annexes: 1. Parties of the contract T&E, represented by Chief Executive Officer duly authorized by the board of directors, and EFS, represented by the Team Leader of Employment Fund Secretariat 140 2. Contracted Services Under the terms and conditions signed in this contract, EFS agrees to purchase services from the T&E which will enhance the skills, provide gainful employment and income to ……. young persons who are economically poor and socially discriminated. The details of services to be procured are presented in the service proposal by T&E which has been evaluated by Employment Fund Secretariat. The final agreed summary of trade and location wise training events is presented in Annex 1 and the detail action plan for implementation is presented in Annex 2. 3. Duration of this Contract and Milestones Start of Contract:…..,, 2011 End of Contract: …….., 2012 Training Completion: ……..,, 2011 Final Reporting: ……..,, 2012 4. Funding Envelopes and Expected Outcomes Outcome under Envelope A (Training and Employment) Envelope A covers training and employment to ……., youths (including the costs of monitoring income verification by the T&E). The T&E has to ensure the following outcomes: • ……. youth skill trained • At least 90% of the trainees participate in the compulsory NSTB skills test (if the respective occupational skills standard is available) and at least 80 % of the trainees pass the test. • Out of which at least 80 % are in gainful employment, for at least six months within an eight months period immediately after the training. Gainful employment is defined as a minimum income of NPR 3,000 per month for elementary and NPR 4,600 for level 1 training graduates within Nepal, NPR 7,000 per month for India and NPR. 15,000 per month for abroad after the training for six months of employment within eight months of the completion of the training. • At least …….., % of the trainees are women. The occupation-wise details are described in Annex 1. Outcome under Envelope B (Market Research and Product Development) The outcome of Envelope B is the Rapid Market Appraisals conducted during the procurement process. Outcome under Envelope C (Capacity Building) The outcome of Envelope C is the Capacity Building Activities conducted during the procurement process. 5. Outcome Budget, Installments and Invoicing The total agreed outcome budget under this contract is NPR ……., (Rupees ……..). The breakdown of the outcome budget as per funding envelopes and their installment release plan are presented in the 141 following table. Table A.21: Outcome budget and installment plan Funding Envelopes Total Outcome Installments Budget (NPR) and their release plan First Second Third A. Training & …….., …….., …….., …….., Employment (…….., persons) B. Market Research …….., …….., …….., …….., and Product Development C. Capacity Building …….., …….., …….., …….., Total (NPR) Planned date for ……..,2011 ……..,2012 ……..,2013 installment release by EFS Further breakdown of direct training cost is presented in Annex 3 and summary of total outcome budget is presented in Annex 4. Cost variations T&E needs prior written approval from EFS management if the direct training cost exceeds the agreed amount. EFS may approve such additional cost, to a limit of 10% of the total amount for direct training costs on a case to case basis subject to valid reasons for the increment. However, this increment will have no effect on calculation of incentive amount. Taxes VAT bill is mandatory, and VAT will be added extra at applicable rate. Taxes on payments to the T&E will be deducted as per prevailing income tax act. The T&E needs to submit annual tax clearance certificate to EFS by December end 2011. 6. Disbursement Procedure The agreed outcome payment amount shall be disbursed in three installments as follows: • After successful completion of the training and NSTB skill test (if applicable) - 40% of the outcome payment. • After three months employment and income verification - 25% of the outcome payment. • After six months employment and income verification – 35% of the outcome payment. The details of the disbursement are described in the General Conditions of Contract (Annex 8) 7. Quality of training & employment service delivery The T&E must ensure that all the training events are conducted in a good quality learning environment 142 and have appropriate physical facilities equipped with adequate tools, machinery and consumables as described in the General Conditions of Contract (Annex 8). The T&E must validate the trades previously identified during the Rapid Market Assessment and confirm the employment market before announcing the training events. The T&E is required to establish an adequate Counseling and Placement Unit which provide s the trainees and graduates with good quality coaching and advice on opportunities for gainful wage and self-employment as well as linkage to employers or credit/ business development services. 8. Occupations and area of implementation The agreed geographical locations (districts) and occupations for training and employment are listed in Annex 1. Changes on the agreed geographical location (district) for program implementation and changes on agreed occupation require prior approval from EFS management. 9. Monitoring and Reporting Monitoring and reporting by T&E : The T&E must monitor the performance and outcome of the agreed activities regularly and in line with the requirements of the EFS monitoring guideline and database manual. The T&E must enter the monitoring data into the EFS online monitoring system in compliance with and within the time periods specified in the EFS database manual. Based on the monitoring, the T&E must submit periodic financial and progress reports together with the invoices in the format presented in Annex 5 and Annex 6. Additionally, an analytical annual report focusing on outcomes and lessons learned together with the summarized financial report needs to be submitted before the end of this contract. In order to assess progress and share lessons learnt with EFS, the T&E has to organize and conduct bi-annual planning & review meetings (mid-term and end of year) during which key staff members and T&E management involved in the training & employment services as well as at least one EFS representative need to be present Monitoring and income verification by EFS : EFS reserves the right to deploy its own- or third-party field monitors at any time during the contracting period to verify the inputs, outputs and outcomes reported by the T&E. The monitoring and income verification may encompass the whole contracted target or be based on random sampling. In the case of random sampling during income verification, EFS will use extrapolation as described in the General Conditions of Contract (Annex 8) for defining the number of gainfully employed and the total outcome payment. 10. Contract amendments, termination and resolution of disputes 143 Amendments This contract can be amended through mutual understanding with a written note signed by both parties. Termination This contract automatically expires with reaching the contracting end date specified at the end of the contract. For any subsequent contract, the T&E has to undergo the complete procurement process of EFS. This contract can be terminated during the contracting period by either party with one-month advance notice in writing with specification of the reasons to the other contracting party. In case of serious violations of this contract including its annexes, EFS reserves the right to immediately terminate the contract by giving one-week written notice in advance specifying the reasons. In any of the two cases of early termination of the contract, EFS pays only for already successfully completed training activities and achieved outcomes the actual cost incurred by the time of termination up to the maximum agreed amount. Resolution of disputes This contract is governed by the law applicable in Nepal. Any disputes between EFS and T&E will first be taken to the NEF Steering Committee for mutual understanding and settlement. If either party does not agree to the decision of the NEF Steering Committee, the matter may be put to arbitration in accordance with the Nepal Arbitration Act. Decision so reached will be final and binding to both parties. Annexes to this contract : The following annexes form an integral part of this contract: • Summary list of Trades, location and per capita outcome – Annex 1 • Detail Action plan – Annex 2 • Direct training cost calculation – Annex 3 • Summary of Total Outcome budget – Annex 4 • Financial reporting format – Annex 5 • Progress reporting format – Annex 6 • Code of conduct – Annex 7 • General conditions of contract – Annex 8 144 Annex E: List of additional relevant project-related sources Chile • World Bank. 2012. “An IEG Comparative Review for Higher Education Reform in Latin America: Based on Project Performance Assessments of Three Projects: Chile Lifelong Learning and Training Project (L71060), Science for the Knowledge Economy Project (L71720), and Colombia Higher Education Improving Access Project.” Independent Evaluation Group. Report No.: 68251. http://documents.worldbank.org/curated/en/921661468240325090/pdf/682510PPAR0P060C0dis closed060250120.pdf. India • World Bank. 2015. “Labor Market Impacts and Effectiveness of Skills Development Programs in 5 States in India: Assam, Andhra Pradesh, Madhya Pradesh, Odisha and Rajasthan. ” Report no. 94682-IN. http://documents.worldbank.org/curated/en/798601468034846280/pdf/94682-Change-in- Title-Final-Revised-Output-Skill-Report-5-States.pdf. • World Bank. 2017. “Program Appraisal Document: Skill India Mission Operation,” (SIMO).Report No. 108103-IN. http://documents.worldbank.org/curated/en/979401503494345541/pdf/INDIA-SKILL-PAD- 06062017.pdf. • World Bank. 2017. “Program Appraisal Document: Skills Strengthening for Industrial Value Enhancement Operation,” (STRIVE) Report No. 108103-IN. http://documents.worldbank.org/curated/en/849071488682856000/pdf/India-Skill- Strengthening-PAD-02102017.pdf. Kenya • World Bank. 2016. Youth Employment and Opportunities Project (KYEOP). Project Appraisal Document. http://documents.worldbank.org/curated/en/867311468197383244/pdf/PAD1654- PAD-P151831-IDA-R2016-0077-1-Box394888B-OUO-9.pdf. Nepal • World Bank. 2017c. Nepal - Enhanced Vocational Education and Training Project II (EVENT II) Project Appraisal Document (PAD). Report: Report No: PAD2373. September 18, 2017 http://documents.worldbank.org/curated/en/387251506909647717/pdf/NEPAL-EVENT-PAD- 09122017.pdf. Sri Lanka • Asian Development Bank. 2014. “Report and Recommendation of the President to the Board of Directors Proposed Results-Based Loans Democratic Socialist Republic of Sri Lanka: Skills Sector Enhancement Program.” Project Number: 42251. https://www.adb.org/sites/default/files/project-document/80480/42251-018-rrp.pdf. • Asian Development Bank. 2018. “Proposed Results-Based Loans and Administration of Grant for Additional Financing Democratic Socialist Republic of Sri Lanka: Skills Sector Enhancement Program.” Project Number: 42251-019 Loan Numbers: 3119-SRI and 3120-SRI. https://www.adb.org/sites/default/files/project-documents/42251/42251-019-rrp-en.pdf. 145