33398 H N P D I S C U S S I O N P A P E R A Practical Guide to Economic Analysis of Youth Projects James C. Knowles and Jere R. Behrman November 2004 A PRACTICAL GUIDE TO ECONOMIC ANALYSIS OF YOUTH PROJECTS James C. Knowles and Jere R. Behrman November 2004 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Family (HNP) of the World Bank's Human Development Network. The papers in this series aim to provide a vehicle for publishing preliminary and unpolished results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author(s) whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Managing Editor, Rama Lakshminarayanan (rlakshminarayana@worldbank.org). Submissions should have been previously reviewed and cleared by the sponsoring department, which will bear the cost of publication. No additional reviews will be undertaken after submission. The sponsoring department and author(s) bear full responsibility for the quality of the technical contents and presentation of material in the series. Since the material will be published as presented, authors should submit an electronic copy in a predefined format (available at www.worldbank.org/hnppublications on the Guide for Authors page). Drafts that do not meet minimum presentational standards may be returned to authors for more work before being accepted. For information regarding this and other World Bank publications, please contact the HNP Advisory Services at healthpop@worldbank.org (email), 202-473-2256 (telephone), or 202-522-3234 (fax). © 2005 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper A Practical Guide to Economic Analysis of Youth Projects James C. Knowlesa Jere R. Behrmanb aConsultant, Health, Nutrition, and Population, Human Development Network, World Bank, Washington, DC USA bConsultant, Health, Nutrition, and Population, Human Development Network, World Bank, Washington, DC USA Paper prepared with funding from the Swedish International Development Cooperation Agency Abstract: Investments in youth are particularly important in light of the unprecedented numbers of young people now entering their reproductive and early adult productive years. Of the more than one billion youth aged 15-24, 86% live in developing countries. This guide is for those involved in appraising the economic merits of youth projects or projects with an important youth component. It builds on two previous publications in the HNP Discussion Paper series, Assessing the Economic Returns to Investing in Youth in Developing Countries (Knowles and Behrman 2003) and The Economic Returns to Investing in Youth in Developing Countries: A Review of the Literature (Knowles and Behrman 2005). The guide is designed to explain and illustrate basic economic principles relevant to ex ante project appraisal, and their application to the task of making choices related to youth projects. It is also intended to familiarize those who do not specialize in youth projects with some of the issues particular to youth projects and to provide some basic understanding of the concepts, relationships and data relevant to undertaking economic analysis of youth projects. The guide comprises seven sections. Following a brief introductory Section 1, Section 2 sketches out a framework for economic analysis of youth projects and the basic economic motives for policies and policy options. Section 3 turns to estimating costs, effectiveness and benefits. Section 4 considers poverty and gender analysis. Section 5 includes seven illustrative case studies covering a variety of youth investments in several different countries. Section 6 presents model terms of reference for the economic analysis of a hypothetical continuing education project in Serbia-Montenegro corresponding to one of the case studies presented in Section 5. Section 7 provides a list of references. Keywords: youth; adolescents; economics; poverty; gender; reproductive health Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: James C. Knowles is an independent consultant and can be reached at: N&N Mansion, Apt. 7C, 78 Soi Chulin, Sukhumvit 21 (Asoke), Bangkok 10110, Thailand. telephone 662-260-3432, JKinUSA@aol.com; Jere R. Behrman is the Director of Population Studies Center and W.R. Kenan Jr. Professor of Economics, Economics Department, McNeil 160, University of Pennsylvania, 3718 Locust Walk, Philadelphia, PA 19104-6297, USA; telephone 1 215 898 7704; fax 1 215 898 2124, 1 215 573 2057; e-mail jbehrman@econ.sas.upenn.edu. iii iv Table of Contents FOREWORD...................................................................................................................IX ACKNOWLEDGEMENTS ...........................................................................................XI EXECUTIVE SUMMARY..........................................................................................XIII SECTION 1. INTRODUCTION...................................................................................... 1 1.1 PURPOSE OF THIS GUIDE ............................................................................................. 1 1.2 NEED FOR THIS GUIDE................................................................................................ 1 1.3 SCOPE OF THE GUIDE.................................................................................................. 6 1.4 ORGANIZATION OF THE GUIDE ................................................................................... 7 1.5 RELATED WORK.......................................................................................................... 7 SECTION 2. BASICS OF ECONOMIC ANALYSIS ................................................... 9 2.1 PURPOSE(S) AND CONTEXT OF ECONOMIC ANALYSES ................................................. 9 2.2 METHODOLOGY........................................................................................................ 10 2.3 CONSIDERATION OF ALTERNATIVES ......................................................................... 12 2.4 RATIONALE FOR PUBLIC SECTOR INVOLVEMENT....................................................... 14 2.5 SUSTAINABILITY....................................................................................................... 15 2.6 RISK AND SENSITIVITY ANALYSIS............................................................................. 16 2.7 DISTRIBUTIONAL ANALYSIS...................................................................................... 17 SECTION 3. ESTIMATING COSTS, EFFECTIVENESS AND BENEFITS .......... 18 3.1 ESTIMATING COSTS ................................................................................................. 19 3.1.1 The identification of project inputs .................................................................. 19 3.1.2 Costing the inputs............................................................................................. 21 3.1.3 Other project costs ........................................................................................... 23 3.1.4 Treatment of some specific cost items .............................................................. 24 3.2 ESTIMATING EFFECTIVENESS.................................................................................... 25 3.2.1 Identifying outcomes......................................................................................... 26 3.2.2 Selecting indicators .......................................................................................... 27 3.2.3 Impact Estimation............................................................................................. 28 3.2.3.1. Data .............................................................................................................. 29 3.2.3.2 Estimation...................................................................................................... 32 3.3 ESTIMATING BENEFITS............................................................................................. 38 SECTION 4. POVERTY AND GENDER ANALYSIS............................................... 40 4.1 POVERTY ANALYSIS.................................................................................................. 40 4.1.1 Variations in project costs, effectiveness and benefits between the poor and non-poor .................................................................................................................... 41 4.1.2 Distribution of project benefits between the poor and non-poor ..................... 41 4.1.3 Poverty impact.................................................................................................. 41 4.2 GENDER ANALYSIS ................................................................................................... 42 4.2.1 Gender variation in costs, effectiveness and benefits....................................... 43 v 4.2.2 Distribution of project benefits between males and females............................ 44 4.2.3 Gender considerations in project implementation .......................................... 44 4.2.4 Gender impact .................................................................................................. 44 4.3 TARGETING INVESTMENTS IN YOUTH........................................................................ 45 SECTION 5. CASE STUDIES....................................................................................... 47 5.1 CASE STUDY #1: ANALYSIS OF DE-WORMING INTERVENTION IN KENYAN RURAL PRIMARY SCHOOLS ......................................................................................................... 47 5.1.1 Problem and identification of alternatives....................................................... 48 5.1.2 Measurement of effectiveness........................................................................... 48 5.1.3. Measurement of costs ...................................................................................... 49 5.1.4 Results and conclusions.................................................................................... 50 5.2 CASE STUDY #2: SCHOLARSHIP PROGRAM FOR POOR SECONDARY SCHOOL STUDENTS IN URBAN COLOMBIA ..................................................................................................... 53 5.2.1 Problem and identification of alternatives....................................................... 53 5.2.2 Measurement of effectiveness........................................................................... 53 5.2.3 Measurement of costs ....................................................................................... 55 5.2.4 Results and conclusions.................................................................................... 56 5.3 CASE STUDY #3: HIV, STI AND PREGNANCY PREVENTION IN TANZANIA................ 58 5.3.1 Problem and identification of alternatives....................................................... 59 5.3.2 Measurement of effectiveness........................................................................... 59 5.3.3 Results and conclusions.................................................................................... 60 5.4 CASE STUDY #4: MEXICO'S PROGRESA/OPORTUNIDADES PROGRAM SCHOLARSHIPS' IMPACT ON SCHOOLING......................................................................... 61 5.4.1 Background....................................................................................................... 62 5.4.2 Importance of evaluation and initial large-scale experiment .......................... 62 5.4.3 Impact on schooling in rural communities ­ enrollment, attainment, drop-outs, re-entry, grade progression, spillovers within families and within communities ..... 63 5.4.4 Impact on child and adult labor ....................................................................... 65 5.4.5 Counterfactual evaluation of experimental variants of the PROGRESA program ..................................................................................................................... 66 5.4.6 Ongoing data collection and evaluation .......................................................... 68 5.5 CASE STUDY #5: CONTINUING EDUCATION IN SERBIA AND MONTENEGRO .............. 70 5.5.1 Problem and identification of alternatives....................................................... 70 5.5.2 Measurement of effectiveness........................................................................... 71 5.5.3 Measurement of costs ....................................................................................... 72 5.5.4 Results and conclusions.................................................................................... 72 5.6 CASE STUDY #6: INVESTMENTS IN BASIC EDUCATION IN PAKISTAN......................... 74 5.6.1 Problem and identification of alternatives....................................................... 75 5.6.2 Estimation of effectiveness ............................................................................... 75 5.6.3 Results and conclusions.................................................................................... 76 5.6.4 Extensions to include additional benefits......................................................... 76 5.7 CASE STUDY #7: YOUTH TRAINING PROGRAM (PROGRAMA JOVEN) IN ARGENTINA. 77 5.7.1 Problem and identification of alternatives....................................................... 78 5.7.2 Measurement of effectiveness........................................................................... 79 5.7.3 Measurement of costs ....................................................................................... 80 vi 5.7.4 Results and conclusions.................................................................................... 80 SECTION 6. MODEL TERMS OF REFERENCE FOR AN ECONOMIC ANALYSIS OF A CONTINUING EDUCATION PROJECT IN SERBIA- MONTENEGRO............................................................................................................. 81 6.1 PROJECT BACKGROUND AND OBJECTIVES................................................................. 81 6.1.1 Country background......................................................................................... 81 6.1.2 Project description ........................................................................................... 82 6.2 PURPOSE OF THE ECONOMIC ANALYSIS..................................................................... 82 6.3 PROFILE OF SKILLS REQUIRED................................................................................... 83 6.4 RESOURCES .............................................................................................................. 83 6.4.1 Key reference documents.................................................................................. 83 6.4.2 Description of key data sets.............................................................................. 84 6.4.3 Key contacts...................................................................................................... 84 6.5 SCOPE OF WORK ....................................................................................................... 84 6.6 SUGGESTED OUTLINE OF THE ECONOMIC ANALYSIS.................................................. 85 6.7 LEVEL OF EFFORT AND BUDGET................................................................................ 85 REFERENCES................................................................................................................ 87 FIGURES Figure 1: School enrollment and labor force participation of children in PROGRESA communities prior to program implementation ........................................................ 64 Figure 2: Positive enrollment effect for children at primary and secondary school levels ................................................................................................................................... 64 TABLES Table 1: Alternative Investments in Youth......................................................................... 5 Table 2: Present discounted value (PDV) of $1000 gained at different years in the future with different discount rates ..................................................................................... 11 Table 3: Frequently used inputs in youth projects............................................................ 20 Table 4: Common outcomes of youth interventions......................................................... 26 Table 5: Three-year costs per lottery winner in the Colombia PACES program (US$) .. 55 Table 6: Calculations of discounted benefits and costs (US$) per lottery winner for the Colombia PACES voucher scheme .......................................................................... 56 Table 7: Estimates of the annual earnings benefits from completing high school in Serbia, based on private sector earnings differentials by level of work experience and sex, 2000 (October).......................................................................................................... 72 Table 8: Cost per secondary graduate in selected nonformal schools, Hungary 1998-99 72 Table 9: Estimates of the annual earnings benefits (including those from reduced unemployment) from completing high school in Serbia, based on private sector earnings differentials by level of work experience and sex, 2000 (October) ........... 74 Table 10: Non-earnings benefits from educating 1,000 girls in Pakistan, 1992............... 77 vii viii FOREWORD The authors of this Discussion Paper make a valuable contribution to our understanding of the role of health, nutrition and population as determinants of economic growth and development. Beyond basic rights arguments, the investment in children and youth has important social returns such as future productivity of the workforce and poverty alleviation. Factors contributing to economic growth and prosperity include policies and institutions that foster good governance, private sector investment, trade liberalization, natural resource conservation, basic education, and health. Good health, nutrition and population policies are now, also, recognized as important contributors to economic growth and development. Good health improves growth and development in the following ways: nutrition positively affects labor productivity and growth; fertility and population dynamics affect growth; child health and youth health affect growth. Poor health and unhealthy habits on the other hand, reduces economic growth and development in the following ways (Hammoudi 1999): HIV/AIDS, malaria, tuberculosis (TB) lower labor productivity, growth, and household incomes; tobacco use adds an economic burden on households; disability in most cases, contributes to earnings loss and unemployment; treating diseases and the needed health care systems are expensive. More than a quarter of the world's population--1.7 billion people--is between the ages of 10 to 24, and the numbers are growing. The vast majority of these young people--86 percent--live in developing countries where, in many places, they represent 30 percent of the population. To a large extent, the choices young people make--with regard to sex and reproduction as well as to other critical aspects of their lives--will determine the size, health and prosperity of the world's future population. Youth have many pressing needs. Nonetheless, a stronger and more coherent economic argument for youth investment is needed to spur appropriate action by governments and donors. The timing of this guide is particularly opportune. Development institutions such as the World Bank and its government partners are increasingly focusing on investments in the 10 to 24 age group as a way to reduce poverty, break the cycle of poverty, build human capital, and stimulate economic growth. At the same time, the work complements World Bank-sponsored analyses of investments in the youngest citizens--infants and children. We hope this Discussion Paper will galvanize even greater attention to the problems and promise of youth in developing countries. With a greater understanding of appropriate policies and of the analytic techniques needed to analyze such policies, countries are more likely to give young people the best possible chance to stay healthy, learn, obtain a job or livelihood, and to participate fully in society. Elizabeth Lule Adviser for Population, Maternal and Child Health Health, Nutrition and Population World Bank ix x ACKNOWLEDGEMENTS The authors would like to thank Linda McGinnis, Wendy Cunningham and Alexandre Kolev for their valuable input as peer reviewers for this paper. Further, the authors are grateful to James E. Rosen and Elizabeth Lule for their close collaboration on the development of this guide. The authors alone, and not the World Bank, are responsible for any errors in this guide. The authors are grateful to the Health, Nutrition and Population and Children and Youth Units of the World Bank for publishing this report as an HNP Discussion Paper. xi xii EXECUTIVE SUMMARY The value of conducting systematic ex ante economic analysis in deciding whether or not, or how better, to implement a project is well-documented, as are the techniques themselves. Applying them to a particular project is not always easy. This Guide is for those involved in appraising the economic merits of youth projects or projects with an important youth component. This Guide is designed to explain and illustrate basic economic principles relevant to ex ante project appraisal, and their application to the task of making choices related to youth projects. It is also intended to familiarize those who do not specialize in youth projects with some of the issues particular to youth projects and to provide some basic understanding of the concepts, relationships and data relevant to undertaking economic analysis of youth projects. Investments in youth are particularly important in light of the unprecedented numbers of young people now entering their reproductive and early adult productive years. Of the more than one billion youth aged 15-24, 86% live in developing countries. Investing in the human and social capital of these young people is key to ending the cycle of poverty. And there are important aspects of youth projects that distinguish them in degree, if not in kind, from other investments: · The changing loci in decision-making for youth from their parental family to the youth themselves · The engagement of youth in a number of important transitions, often including completing formal schooling, initiating work (increasingly outside of family enterprises), initiating sexual relations, establishing residence outside of their parental household, migrating, becoming married and having their own children · The important longer term impacts of the decisions made by youth during these transitions, requiring a life-cycle approach in economic analysis · The limited availability of evidence on the cost effectiveness of many youth investments, requiring particularly careful evaluation of this limited evidence · The tendency of youth ­ perhaps in part because of the multiple transitions in which they are immersed or in part because of rapid changes in knowledge or maturity ­ to discount relatively heavily longer term effects of the decisions that they are making (more heavily than they are likely to do after they become adults) · The engagement of many youth in risky behavior that is likely to affect the welfare of others in society in addition to the effects on themselves · The substantial opportunity afforded by youth in some countries to achieve improved social cohesion because they may not bear the same scars from recent ethnic or religious conflict as do adults · The necessity of youth investments in some cases to compensate for important investments not made during childhood xiii · The susceptibility of youth to engage in activities disruptive and costly to society, partly because youth are a large, politically visible and sensitive group that is usually without any formal voice · The tendency for youth investments to be components of larger projects and to have important synergies with other investments and projects For these and other reasons it is useful to have a practical guide focused on the economic analysis of youth projects rather than only to refer to the general procedures for such analysis. The goal of this document is to provide such guidelines. Following a brief introductory Section 1, Section 2 sketches out a framework for economic analysis of youth projects and the basic economic motives for policies and policy options. This framework is essential for organizing the applied economic analysis in what follows. This framework discusses the purpose(s) and context of economic analyses, methodology for such analysis, the centrality of the consideration of alternatives, the rationale for public sector involvements, project sustainability, risk and sensitivity analysis, externalities, and distributional analysis. Section 3 turns to estimating costs, effectiveness and benefits. To estimate costs a systematic process needs to be undertaken to identify project inputs, cost those inputs, and incorporate other project costs. Critical to estimating such costs is to identify all true resource costs (not financial or budget costs) incurred by the government or by other entities because of the project. In this process it is important to distinguish between transfers and true resource costs. It also is important to include any distortionary costs due to the project, including those incurred through raising revenues to finance the project. To estimate effectiveness it is necessary to identify project intermediate outcomes and impacts, select appropriate indicators and estimate the project effect and impact on those indicators. The latter estimation has two critical intertwined components ­ data and estimation approaches ­ that together can help make the desired hypothetical comparisons between behaviors of an individual affected by the project with behaviors of the same individual were s/he not affected by the project. Controlled experiments with random assignment of project treatment and controls provide an attractive conceptual framework that at times (but not with sufficient frequency) is implemented to make such comparisons. Non-experimental data with special features ­ "natural" experiments, control groups based on matching, program discontinuity ­ together with appropriate estimation techniques to control for unobserved factors and behavioral choices may permit good comparisons. Well-specified and well- estimated structural models permit undertaking counterfactual policy experiments to inform the analysis of policies that were not undertaken or not undertaken in the particular context of relevance. While ex ante project analysis usually does not involve data collection and analysis directly, it is essential that the analyst be sensitive to data and analysis issues for two reasons. Firstly, the collection of relevant information and the analysis of it is a key component of good project design. Secondly, the analyst typically must use estimates xiv from other studies as part of the ex ante economic analysis. To do so well requires understanding of and sensitivity to the basic data and analysis issues in undertaking such studies. Thirdly, because economic evaluations of youth investments are still in very limited supply and because good analyses often are context dependent due to differences in markets and other factors, most youth projects should include an impact evaluation in their monitoring and evaluation (M&E) framework. To estimate benefits requires weighting the impacts on all the important indicators affected by the project. That the benefits in principle include all the impacts is their major strength in comparison with single indicators of effectiveness as are typically used in cost-effectiveness analysis. But finding appropriate "prices" to weight all the impacts in order to construct the benefits also poses a challenge. For some impacts (e.g., labor productivity) market prices may exist that provide the appropriate weights (perhaps with some adjustments so that they reflect true marginal resource costs or "shadow prices"). For other impacts (e.g., mortality averted), however, some other methods probably have to be used, such as the resource cost of the most cost-effective alternative means of attaining the same objective or measures of willingness to pay to attain the same objective. The estimates may be quite sensitive to which of these alternatives is used. Section 4 considers poverty and gender analysis. The discussion of poverty analysis considers variations in project costs, effectiveness and benefits between the poor and non-poor; the distribution of project benefits between the poor and non-poor, and the poverty impact of projects. The discussion of gender analysis considers gender variation in costs, effectiveness and benefits; the distribution of project benefits between males and females; and the gender impact of projects. This section concludes, finally, by considering issues in targeting investments to youth. Section 5 includes seven illustrative case studies covering a variety of youth investments in several different countries. Each of these case studies discusses the problem(s) that is being addressed by the project, identification of alternatives, measurement of effectiveness, measurement of costs and the results and conclusions of the analysis. In the process many of the ideas that are presented in Sections 2, 3 and 4 are illustrated. These include how to estimate public and private resource costs and multiple impacts, how multiple effects can be valued, how spillovers might be addressed, what types of information might be used and which methods may be used to exploit the available information most effectively. Section 6 presents model terms of reference for the economic analysis of a hypothetical continuing education project in Serbia-Montenegro corresponding to one of the case studies presented in Section 5. These model terms of reference discuss in turn the project background and objectives, the purpose of the economic analysis, the profile of skills required, various resources (e.g., key reference documents, key data sets, key contacts), the scope of work, a suggested outline for the economic analysis and the level of effort and budget. Section 7 provides a list of references. xv xvi SECTION 1. INTRODUCTION 1.1 PURPOSE OF THIS GUIDE The importance of conducting systematic ex ante economic analysis in deciding whether or not, or how better, to implement a project is well-documented, as are the techniques themselves. Applying them to a particular project is not always easy. This set of guidelines is for those involved in appraising the economic merits of youth projects or projects with an important youth component.1 For purposes of this Guide, youth are defined as persons aged about 15-24 years. This Guide is designed to explain and illustrate basic economic principles relevant to project appraisal, and their application to the task of making choices related to youth projects. It is also intended to familiarize those who do not specialize in youth projects with some of the issues particular to youth projects and to provide some basic understanding of the concepts, relationships and data relevant to undertaking economic analysis of youth projects. This Guide is targeted mainly to economists who will be preparing economic analyses of youth projects. However, it is intended to be useful as well to those responsible for designing youth projects, including Task Team Leaders (TTL) and those responsible for designing a project's monitoring and evaluation (M&E) framework. 1.2 NEED FOR THIS GUIDE Investments in youth are particularly important in light of the unprecedented numbers of young people now entering their reproductive and early-adulthood productive years (World Bank 2004). Of the more than one billion youth aged 15-24 (nearly one-fifth of the world's population), 86% live in developing countries where they currently account for 29% of the total population (Sundaram, et al. 2004, World Bank 2004). Investing in the human and social capital of these young people is key to ending the cycle of poverty. Success in some areas (for example, universal primary education) has increased the demand among youth for secondary schooling and for non-agricultural employment. In some countries, investments in youth are required to compensate for previous underinvestment in children. Estimates of the cost of not investing in youth have been impressively large in some cases (World Bank 2003). Investments in youth are likely to be central to the successful pursuit of the Millennium Development Goals (MDG) that are accepted widely as shaping the development agenda in the international community (World Bank 2004). The successful achievement of many of these goals will require that policy makers center much of their attention on youth. Youth currently growing up in poverty face much greater health risks in both the short 1 There are a number of previous World Bank documents directly related to this topic. Examples include Barnum (1995), McGuire (1996), Phillips and Sanghvi (1996), Preker (1996), Tan (1995), Devarajan, Squire and Suthiwart-Narueput (1997), Harberger (1997), Belli, et al. (1998), and Knowles and Behrman (2003a, 2003b). 1 and longer term and are much less likely to attend schools of adequate quality, to complete primary school, to attend secondary school (particularly girls), to avoid HIV infection, to find secure and productive employment, to have opportunities for community participation, to marry well, or to be able to provide good care and support to their children. To meet the MDG goals, most, if not all, of these issues related to youth must be addressed. At a sufficiently general level, the systematic analysis of youth projects is like the systematic analysis of projects that involve investment in people of any age or, for that matter, investments in physical capital. But there are important aspects of youth projects that distinguish them in degree, if not in kind, from other investments in people. These include: · The changing loci in decision-making for youth from their parental family to the youth themselves. Youth are in the midst of transitions from childhood to adulthood. Associated with these transitions are shifts of decision-making from their parents and other adults to themselves. In some respects these shifts in the loci of decision-making power, in combination with the multiple transitions noted next, make effective project development very challenging in part because the target is rapidly moving (rapidly changing youth instead of their relatively stable ­ in respects related to life cycle ­parents). While this dimension of youth is widely recognized, project evaluations to date do not incorporate well the changing loci in decision-making that may be germane to a youth project's success or that may be affected directly by the project.2 · Youth are engaged in a number of important transitions, often including completing formal schooling, initiating work (increasingly outside of family enterprises), initiating sexual relations, establishing residence outside of their parental household, migrating, becoming married and having their own children. The combination of multiple transitions within a fairly short period of the life cycle means that youth projects can involve investments in many areas (Table 1.1). It also means that although youth projects may be directed towards some particular aspect of youth (e.g., employment, education, health), they are also likely to have ramifications on many other dimensions of their lives, whether intended or not. During such transitions, moreover, the potential is substantial for longer-run impact (including intergenerational effects) because affecting these transitions often has persistent impacts for the youth when they become adults. This means that the evaluation of project impact is more complicated for investments in youth both because of the broad range of interventions and effects involved and because the effects are likely to be distributed over an extended 2Instead, typically, the analysis implicitly assumes that the locus of decision-making on the demand side is whatever residential unit in which the youth live. ­ in part because that is what standard data permit. But such approaches, as noted, may miss the key shifting of the locus of decision-making that occurs as youth mature and that projects may affect the relative importance of youth versus their parental family and kin in making decisions. 2 period of time.3 This further raises the question of how they should be discounted to compare them with effects of other projects. · The longer term impacts of the decisions made by youth during these transitions requires a life-cycle approach in the economic analysis of youth projects. For example, terminating schooling prematurely may result in substantially lower earnings over a person's working life. Engaging in unprotected sex may result in a youth contracting HIV. Engaging in some other types of risky behavior may also have longer-run adverse health impacts (for example, becoming a smoker or abuser of alcohol or drugs). · The available evidence on the cost effectiveness of many youth investments is very limited. In part because many of the expected effects of youth projects are longer- term, the available evidence on the cost effectiveness of youth projects is quite limited (Knowles and Behrman 2003b, World Bank 2004).4 Because suitable data will typically be unavailable in a given country, it is often necessary for those evaluating youth projects to draw on the findings of available research from other countries. · Youth ­ perhaps in part because of the multiple transitions in which they are immersed or in part because of rapid changes in knowledge or maturity ­ may discount relatively heavily longer-run effects of the decisions that they are making (more heavily than they are likely to do after they become adults). Youth are often characterized as being relatively impatient either because they are still maturing or because they are not well informed. If the latter is the case, then there may be a high return to improving the information on which they are making decisions. If the former is the case, then there is a problem regarding what discount rate is appropriate to use for the analysis. The existing project evaluation literature (discussed in Section 1.5 below) does not address this issue nor do most previous analyses explore the implications of using alternative discount rates for this reason. · Youth are more often engaged in behaviors that are likely to affect the welfare of others in society, in addition to effects on themselves, other than through markets. These include investing in education if education has spillover effects as is often claimed. They also include various risky behaviors, such as unprotected sex with multiple partners, exploring drugs, driving vehicles with some abandon and 3Projects that have impact earlier in the life cycle on, for example, early childhood development, also may have multiple effects over a long time horizon. But their impact is not complicated in the same way as are youth projects because of the shift in the locus of decision-making and multiple major transitions noted above. Also they are likely to have longer gestation periods before effects related to productivities (e.g., in work or in parenting) are realized. 4In some respects, youth projects share this feature with early childhood development (ECD) projects. However, many of the important effects of ECD projects are experienced earlier (for example, in the health and survival of children and their mothers), and there is more information available on some of the longer- term effects of ECD projects, such as their impact on primary schooling. 3 engaging in illegal activities.5 This suggests that at least some types of investments in youth may yield substantial social benefits, due to externalities, in addition to private benefits. The possible presence of greater social benefits may justify more public involvement in the youth area than in some other areas. · Youth are a large, politically visible and sensitive group, usually without any formal voice, and therefore susceptible to disruptive activities that are costly to society (World Bank 2004). Accordingly, some youth projects may have a political or participatory focus. In some countries (for example, in South-Eastern Europe), youth may offer a substantial opportunity to achieve improved social cohesion because youth do not bear the same scars from recent ethnic or religious conflicts as do adults. In other contexts such as in a number of areas in Sub- Saharan Africa, in contrast, youth have been immersed in such conflicts (e.g., as child soldiers, or targeted victims of social unrest), which means that they may represent less of an opportunity in this sense than a necessary focus for rehabilitation. The desired political and social outcomes of some youth projects may be as important as their more standard outcomes, such as enhanced productivity or improved health. · Youth investments are often components of larger projects, and youth investments may have important synergies with other investment projects (World Bank 2004). For example, water and sanitation projects may free up girls' time from having to fetch water, enabling more girls to attend school. Similarly, employment projects may have synergies with continuing education projects. The methods used to evaluate youth projects must be sufficiently flexible to accommodate these synergies. For these and perhaps other reasons it is useful to have a practical guide focused on the economic analysis of youth projects rather than only to refer to the general procedures for such analysis. The provision of such a guide is the intent of this document. 5According to the World Bank (2004), about one-half of new HIV infections are in youth (an estimated 6,000 youth per day become infected with HIV). Each year 13 million teenage girls (15-19) give birth, while another 4 million seek abortions (many of which are unsafe). Teen mothers are twice as likely to die of pregnancy-related causes, and their children are at higher risk of illness and death. Youth also account for a disproportionately large share of criminal activity, drug and alcohol abuse and of motor vehicle accident victims. 4 Table 1: Alternative Investments in Youth Investments Schooling School quality-improving investments (e.g., strengthening inputs, administration decentralization, school autonomy) Scholarship programs (including vouchers and conditional cash grants) Enactment and enforcement of compulsory attendance laws Training Vocational and technical training Adult basic education and literacy (ABEL) training Continuing education Military training Work Short-term employment programs (jobfare) targeted to youth Enactment and enforcement of child labor regulations Enactment and enforcement of other employment regulations (e.g., safety regulations, hours of work, minimum wage, restrictions on layoffs) Reproductive health School-based reproductive health education Social marketing of reproductive health services targeted to youth Youth-friendly reproductive health services Linked services Peer counseling programs Mass media programs Workplace/community outreach services targeted to youth Investments designed to delay age at marriage Reproductive health policy development Anti-retroviral treatment of HIV-infected youth School-based health investments School health policies School-based health education (apart from RH education) School lunch/feeding Micro-nutrient supplements administered to school children Mass de-worming of school children Water and sanitation facilities in schools Presumptive malaria treatment of school children Periodic physical examinations of school children School health insurance Other health investments Increasing the tax on tobacco products Ban on tobacco advertising and promotion Anti-alcohol abuse investments Anti-drug abuse investments Mass media investments (apart from reproductive health) 5 Investments Food supplements for pregnant and lactating women Food fortification Road accident prevention Investments to improve the mental health of youth Community and other investments Youth centers Youth development programs Micro-credit and entrepreneurship training programs targeted to youth Youth rehabilitation programs Sports and recreation programs Source: Knowles and Behrman (2003b). 1.3 SCOPE OF THE GUIDE Before turning to how this Guide is organized, it is useful to clarify that there are several things it does not attempt to do. Firstly, it does not provide a critical assessment of the "state-of-the-art" in the economic analysis of youth programs.6 Nor does it attempt to give detailed descriptions of the procedures involved, exact data sources or calculations for either cost or impact estimates. However, this Guide does include some, though not detailed, discussion of issues related to data and to analysis. This is for two reasons (as discussed above): (1) to be able to choose better estimates from the existing literature, and (2) to be able to design effectively the important evaluation component of the project under consideration. Further, this Guide focuses specifically on the techniques of cost-effectiveness and cost- benefit analysis from a micro perspective. It does not deal in any detail with the broader context in which such analysis takes place. For some kinds of projects (for example, those involving education and labor market institution-building, decentralization, financial reforms and price restructuring), macro considerations may be crucial. For many others they are of more marginal relevance and the framework provided by cost- effectiveness analysis and cost-benefit analysis should be the central focus of the process. In either case, it is important to be aware of the broader context. This Guide also does not discuss in detail the role that qualitative data and their analysis might play in the economic analysis of youth projects. Although there may be considerable scope for using qualitative data in the economic analysis of youth projects, it is not clear yet what that role should be and whether it should be confined to ex post analysis. Despite the fact that this Guide does not discuss in detail the role of qualitative analysis in economic analysis, it makes some reference to the use of qualitative analysis as a valuable complement to quantitative analysis (for example, in the case studies discussed in Section 5). It is also our belief that much of the material in this Guide would be useful as well to those preparing a qualitative analysis of a youth project. 6For such an assessment, the reader is referred to Knowles and Behrman (2003b). 6 Finally, this Guide focuses on ex ante appraisals for the purpose of decision-making in advance of project implementation. While much of what is described in the Guide can be applied ex post to project evaluation or to project monitoring during implementation, there are some differences that should be borne in mind. One key difference relates to the information used for the analysis. Ex ante economic analysis must often use information from other studies to attempt to represent critical empirical magnitudes, whereas ex post evaluation more often uses information that is generated by the project. To undertake good ex post evaluation requires thorough knowledge of the data available and the use of appropriate techniques to analyze the data. Ex ante analysis does not require the same active immersion in data and measurement issues. And yet the difference may not be as great as it first appears to be because, as noted above, the ex ante analyst needs to be able to make good judgments about data and methods that others have used to assess their usefulness for the ex ante analysis being undertaken. 1.4 ORGANIZATION OF THE GUIDE This Guide begins by sketching out in Section 2 a framework for the economic analysis of youth projects and the basic economic motives for policies and policy options. This framework is essential for organizing the applied economic analysis in what follows. Section 3 turns to estimating costs, effectiveness and benefits. Section 4 considers poverty and gender analysis as well as targeting. Section 5 provides some illustrative case studies and Section 6 presents model terms of reference for the economic analysis of a youth project. Section 7 provides a list of references. 1.5 RELATED WORK The World Bank has published several manuals and methodological guides on economic analysis of projects during the past ten years. The main reference is the Handbook on the Economic Analysis of Investment Operations (Belli, et al. 1998) that is intended to provide general guidelines on appropriate methodologies for all sectors. In addition to providing general methodological guidance, the Handbook provides in depth case studies in the education, health and transport sectors. In addition to the Handbook, several important sector-specific methodological guides on economic analysis have been produced. For example, The Economic Analysis of Nutrition Projects (Phillips and Sanghvi 1996) provides an introduction to the methods used in the economic analysis of nutrition projects as well as several case studies of the economic evaluation of nutrition projects as well as one sector-level evaluation of nutrition investments. The Economic Analysis of Health Projects (Hammer 1996) applies the framework presented in Devarajan, Squire and Suthiwart-Narueput (1997) to the economic evaluation of health projects. Topics discussed include: the justification for public involvement, impact estimation, substitution of public services for private services, the fiscal impact of projects (including the role of cost recovery), and the fungibility of project resources. In addition, to these World Bank resource documents, the Asian Development Bank has prepared the Economic Analysis of Health Sector Projects 7 (Adhikari, Gertler and Lagman 1999) to help project designers assess the complexities of health sector projects. In the youth area, the World Bank has sponsored two studies by Knowles and Behrman that focus on the economic evaluation of investments in youth (2003a, 2003b). The first study presents a methodology to estimate the benefits and costs of investing in youth, including the presentation of several case studies, while the second study reviews the existing literature on the effectiveness and cost-effectiveness of a wide range of investments in youth. Knowles and Behrman's main finding is that there is currently very little reliable information on the cost and effectiveness of many types of investments in youth.7 Together, these various reference materials on the economic analysis of projects provide much useful guidance to those preparing economic analyses of youth projects or projects with an important youth component. The present Guide does not attempt to duplicate or supplant these useful reference materials. Instead, it attempts to complement them by indicating to the reader when appropriate where he/she can find a more detailed treatment of a given topic as well as which of the various topics discussed in these reference materials are likely to be most relevant to youth projects. In addition, this Guide provides additional information on some topics that are not treated thoroughly in the existing reference materials on the economic analysis of projects. These include: · How to identify and measure the economic costs of a project (including importantly the distinction between real costs and transfers) · What kinds of data and estimation techniques can be used to obtain good estimates of project effectiveness · How to assess the reliability of available estimates of the effectiveness or benefits of project interventions · How to attach monetary values to a wide range of project effects, some of which may difficult to "monetize" · How to use a life-cycle approach to evaluate the benefits and costs of a youth project · How to incorporate multiple and varied benefits into a cost-benefit analysis · How to incorporate an effective monitoring and evaluation strategy during the implementation of a project · How to choose an appropriate discount rate In addition to the various reference materials on the economic analysis of projects, there has been some interesting and relevant work in recent years to assess the value of economic analysis itself in the design and implementation of projects. For example, a 1995 World Bank study investigated whether projects rated as good or excellent in terms of the quality of their economic analysis were more likely to receive high ratings in terms of their expected achievement of development objectives three years later (Belli and 7This conclusion is re-affirmed in the most recent draft of the World Bank's Children and Youth Framework (World Bank 2004 8 Pritchett 1995). The study found that the probability of a poor project rating after three years of implementation was seven times higher if the economic analysis was rated poor. A 1999 study investigated the relationship between the quality rating of the economic analysis of 104 education projects and their subsequent ratings during implementation (Vawda, et al. 1999). That study found no relationship between the rated quality of the economic analysis and the project's most recent rating in terms of the likelihood of achieving its developmental objectives. However, the study found a significant positive relationship between a project's implementation progress rating and the rating of its economic analysis. That study also found that all of the most highly rated economic analyses had been preceded by a significant amount of analytical work prior to the design of the project. It cites another World Bank study that concluded more generally that an increase of one staff week in the amount of time devoted to analytical work prior to project initiation was associated with an increase in the project's rate of return of between 0.02 to 0.04 percentage points (Deininger, Squire and Basu 1998). SECTION 2. BASICS OF ECONOMIC ANALYSIS8 2.1 PURPOSE(S) AND CONTEXT OF ECONOMIC ANALYSES A project's financial analysis demonstrates the impact of the project on the financial status of the entity implementing the project or of entities affected by the project. For example, in the case of a health project, the implementing agency will generally be the Ministry of Health and entities affected by the project will include households and private-sector health providers. In contrast, economic analysis estimates the impact of the project on the entire society in the country in which the project is implemented and compares that impact with the project's real resource costs. The focus is not on financial budgets, but instead on real resource costs and benefits. The purpose of economic analysis is to guide the design of the project in order to maximize its social benefits in relation to its social costs. Accordingly, economic analysis should be undertaken early in the development of a project. If it is undertaken after the project design has been completed and the design is immutable for some reason (e.g., already existing political commitments), the economic analysis will only be helpful in deciding whether or not to proceed with the project. The economic analysis should consider the sector and country context of the project and determine whether it is consistent with the general directions of policy reform required in the sector and country. An important element to consider in such an assessment is the incentives available to those who will be responsible for implementing a project and how this relates to the fungibility of project resources (Hammer 1996). Key documents to be 8Readers who are familiar with the methodology used to prepare an economic analysis of projects may wish to skip this section of the Guide. However, it contains a number of examples of the special features of youth projects in this context that even such readers might find useful. 9 examined for this purpose typically include the World Bank's Country Assistance Strategy, the country's own development plan and sector strategies, the country's poverty reduction strategy, and the country's most recent Public Expenditure Review. The economic analysis should also consider the overall quality of a country's public expenditure program, and particularly of the program within the sector (or sectors) in question. The reason is that money is fungible, and a project funded by the World Bank, even if highly beneficial to society, may make it possible for the government to spend on other activities that are less socially beneficial. If this is the case, the World Bank project has essentially financed the less desirable expenditure. 2.2 METHODOLOGY When the expected benefits of a project can be measured in monetary terms, the appropriate methodology is cost-benefit analysis (CBA). In this case, the sum of the discounted value of benefits net of costs (referred to below as "net present benefits") is the appropriate yardstick to apply in deciding whether the project is economically attractive. A project with negative net present benefits should not be undertaken. Equivalent ways of saying essentially same thing that we sometimes use below (particularly in the case studies in Section 5) and that are used elsewhere in the literature are that the benefit-cost ratio exceeds one or that the internal rate of return to the project is equal to or greater than the discount rate.9 In the context of youth projects, because expected benefits often occur many years later, conclusions regarding a project's net present benefits will often be very sensitive to the discount rate used to convert a project's future benefits and costs to present values. Typically some of the benefits of youth projects occur years or decades later in adulthood, or even carry over to the next generation (e.g., improved schooling for female youth is thought to improve the education, health and nutrition of their children). The choice of a discount rate, therefore, can be critical in determining the conclusions drawn from an economic analysis. The Handbook (Belli, et al. 1998) states that the World Bank generally uses a discount rate of 10-12% in its economic analyses.10 It explains that this rate is used as a "rationing device" and that it does not necessarily reflect the true opportunity cost of capital in borrowing countries. It invites analysts to use another rate if it can be justified.11 9 Criteria based on net present values, cost-benefit ratios and internal rates of return are not equivalent in all cases. In the case of mutually exclusive alternatives (e.g., which bridge to build across the gorge when there is the possibility of building only one bridge), for example, the advantage of net present value is that it is able to reflect the absolute size of the potential benefits (see Belli, et al. 1998 for further discussion). But for most (perhaps all) youth projects they are equivalent. 10 The Asian Development Bank (ADB) also recommends the use of a discount rate of 10-12% for the economic evaluation of its loan projects (Adhikari, Gertler and Lagman 1999). 11 Country Assistance Strategies may recommend a discount rate for use in the country. If this is the case, the analyst should determine what the country-specific rate reflects (for example, the distortionary effects of the country's tax system) in order to avoid double counting of costs. Even in such cases, moreover, it 10 However, the Handbook cautions that use of a rate lower than 10% might be difficult to justify in light of research suggesting that this is the low end of estimates of the opportunity cost of capital in developing countries. Some other World Bank references on economic evaluation claim that in most countries the opportunity cost of capital is correctly reflected by the real rate of interest for low-risk securities, which is generally lower (usually less than 5%) and that the real rate of interest on low-risk securities such as World Bank Bonds and United States' Treasuries has remained at about 3% for many decades (Barnum, 1995, Phillips and Sanghvi 1996).12 Table 2.1 provides some illustrations of how the present discounted values of benefits change with different lags and selected different discount rates. As is clear from this table, using the appropriate discount rate may be one of the more important components of a good economic analysis of youth projects that have some benefits with long lags. Accordingly, good practice in the economic analysis of youth projects is to use a range of alternative discount rates (for example, from 3-10%) in calculating net present benefits (see Section 2.6). Table 2: Present discounted value (PDV) of $1000 gained at different years in the future with different discount rates Years into the Annual discount rate future 3% 5% 10% 20% 5 $862.61 $783.53 $620.92 $401.88 10 $744.09 $613.91 $385.54 $161.51 20 $553.68 $376.89 $148.64 $26.08 30 $411.99 $231.38 $57.31 $4.21 40 $306.56 $142.05 $22.09 $0.68 50 $228.11 $87.20 $8.52 $0.11 60 $169.73 $53.54 $3.28 $0.02 Estimates of net present benefits should include the full range of relevant benefits and costs. There is often a tendency in youth projects to neglect possible benefits that may occur outside the sector in which the investment is focused (for example, neglecting possible health benefits in projects that involve educational or training investments, and vice versa). There is also often a tendency in youth projects to neglect benefits for which it is difficult to assign a monetary value (for example, health benefits as compared to gains in labor productivity). Instead of neglecting such benefits, it may be possible to employ fairly simple methods to value them (for example, using an estimate of the least- cost alternative used by society to obtain the same outcome).13 This and other approaches still is very valuable to illustrate how sensitive the results are to a range of plausible discount rates, not just the recommended one. 12 The World Bank and WHO have also used a 3% discount rate in calculating the number of disability- adjusted life years (DALY) gained from alternative health interventions or in Global Burden of Disease estimates. 13 In such cases, it may also be possible to use cost-effectiveness analysis or weighted cost-effectiveness analysis to obtain a quantitative (albeit partial) assessment of the project's efficiency in achieving clearly 11 for assigning monetary values to such difficult-to-monetize outcomes are discussed in Section 3.3. Additionally, it is important to include external benefits and costs that may be experienced by persons other than the project's intended (or main) beneficiaries. In cases where it is impractical to obtain a quantitative estimate of a project's benefits, a qualitative14 assessment of the project's likely net present benefits in relation to its costs may be acceptable (Belli et al. 1998). Such a problem may arise when a project supports a large number of interventions, each of which has a number of different expected outcomes (for example, a youth center providing many different services). In such cases, the economic analysis should provide a clear rationale for why the project's net present benefits are expected to exceed zero. 2.3 CONSIDERATION OF ALTERNATIVES A key element in economic analysis should be consideration of alternatives to the proposed project, including of course the alternative of not undertaking the project at all. At a general level, the use of net present benefits (or cost-benefit ratios or internal rates of return) builds in the consideration of alternatives through incorporating a project's true resource costs. But it is useful here to go beyond this general point by discussing some specific examples of what it means in practice to consider alternatives. Economic analysis, for one example, should consider the separability of the various components of a project and should assess the benefits and costs of each separate component as if it were the marginal component. In doing so, however, it is also important not to neglect possible synergies between various project components, as well as possible economies of scope.15 The economic analysis should not be confined to the components that are more economically attractive. Unsatisfactory (separable) components should be deleted from the project. When youth projects are only one component of a larger project, the youth component should be analyzed separately. The youth component should not be justified on the basis of the economic strength of the larger project of which it is a part. Economic analysis should also consider the economic implications of using alternative technical approaches in a project, for example, the implications of developing a given type of training capacity within a country versus sending students to other countries to receive comparable training, or the implications of having a given type of service provided by the private sector instead of the public sector. In the context of youth projects, another important alternative is often whether to finance projects through direct stated objectives, compared to similar projects in the same country or internationally. Cost-effectiveness analysis and weighted cost-effectiveness analysis are discussed in detail in Belli, et al. (1998). 14 The term "qualitative" as used in this context refers to an assessment of whether net present benefits are likely to be positive or negative based on the available quantitative information. 15 Synergies and economies of scope may not be limited to a single project. In some cases, the net present benefits of a given youth project may depend on whether another project is implemented. In such cases, it will be useful to calculate the project's net present benefits under two scenarios, one in which the other project is implemented and the other in which it is not implemented. 12 budget support to public providers of services (sometimes referred to as "supply-side financing") or through such demand-side financing methods as scholarships or other conditional cash grants, loans, or vouchers (World Bank 2004). Lastly, it is useful to consider whether a project's objectives, including distributional objectives, might be met more efficiently through policy reform or improved regulation. Cost-effectiveness or cost-benefit analysis may be very useful in conducting analyses of such alternatives. Because investments in youth have many potential effects, including important effects in some cases in sectors different from the one in which the investment is undertaken (for example, possible reproductive health outcomes of investments in continuing education), there are often going to be a variety of investments that can be used to achieve a given objective, or set of related objectives. Cost-benefit and cost-effectiveness analysis may be very helpful in narrowing the range of choices. However, it is also useful to consider other important features of projects in selecting among them. Some of these are discussed below (for example, the rationale for public sector involvement is discussed in Section 2.4 and the project's distributional impact is discussed in Sections 2.4, 2.7 and 4). In addition, it will also be useful to consider where a given project fits into the "efficiency policy hierarchy," as discussed below. An efficiency policy hierarchy can be defined in which alternative policies to attain the same objectives (including alternative investments in youth) are ranked according to their social marginal costs, including both direct and possible distortionary costs and the costs of administering and monitoring the policy's effects. There are tremendous information problems regarding exactly what effects policies have, particularly in a rapidly changing world. There is an argument in favor of policies that are as direct as possible, both because more direct policies are likely to have lower distortionary costs and because more direct policies are likely to be more transparent. For example with regard to the latter, information problems may provide an argument for price policies (taxes or subsidies, as compared to policies focused on quantities such as quotas or prohibitions) because if there are shifts in the underlying demand and supply relations they are likely to be more visible in a more timely fashion to policymakers if they have impact on the governmental budget than if they only change the distortions faced by private entities as tends to happen with quantitative policies.16 Thus, for efficiency/productivity reasons, particularly given that in the real world information is imperfect and changes are frequent, there is an argument generally for choosing policies that are focused as directly on the problem as possible. Note that this means that, for example, if there is a good efficiency reason for public support for particular youth projects, that does not mean that the best way to provide that support is through governmental provision of the relevant services. Higher in the efficiency policy hierarchy than direct governmental provision of such services, for 16 Nevertheless there are likely to be some cases, such as providing information regarding the quality of goods and services related to investments in youth, for which quantitative regulations may be higher in the efficiency policy hierarchy than price policies because of the nature of the information requirements. 13 example, may be subsidies or taxes that create incentives for the efficient provision of these services, whether the actual providers are public, private or some mixture. On the other hand, policies that discriminate against one type of provider -- for example, by making the availability of such subsidies dependent on whether the provider is public -- are generally likely to be lower in the efficiency policy hierarchy than policies that do not have such conditions. 2.4 RATIONALE FOR PUBLIC SECTOR INVOLVEMENT A key element of the economic analysis is to provide a rationale for public sector involvement. There is a fairly-widely held presumption that the private sector is more efficient in producing goods and services for which markets create pressures for efficient production than the public sector. Public sector involvement is often rationalized in terms of the presence of various types of market failure. Examples of market failures include: · Incomplete information and/or asymmetric information between buyers and sellers · Externalities (both producer and consumer and both positive and negative) · Public goods (non-excludable goods, non-rival goods) · Economies of scale that are large relative to the size of the market Market failures cause a divergence between private benefits and costs and social benefits and costs. In this case, an important rationale for public involvement is to ensure that a given service or product is produced up to the point where the last unit produced and consumed has the same value to society as the inputs used to produce it (i.e., to ensure that social marginal benefits are equal to social marginal costs). Externalities are an example of how market failures may be important for some types of youth projects. For example, investments in youth may reduce negative externalities in the form of crime and violence, communicable diseases (e.g., HIV/AIDS), and those related to alcohol and tobacco use (for example, accidents and effects of second-hand tobacco smoke). And they may increase positive externalities through education, which is thought to have positive spillover effects on others. The presence of such externalities provides a justification for public involvement in some types of investments. Because rates of emigration are high for youth in some countries (for example, from some Eastern European countries currently), the presence of externalities may also provide a justification for the participation of receiving countries in financing some investments in youth while still in their home countries (for example, investments in continuing education). The presence of one or more market failures in a given market does not, however, necessarily imply that the best approach is for the government to provide services directly, especially since policy failure may be a more important consideration in some contexts than market failure (i.e., some of the most important differences between market prices and opportunity costs or between financial and economic resource flows may be due to governmental policies rather than market failure). Highest in the efficiency 14 hierarchy may be lessening or eliminating policy failures. Governmental financing of privately-provided services, moreover, may be more appropriate than governmental provision of such services, even in the case of public goods. In addition to market failure, there are often strategic, political and equity considerations that suggest that public involvement (including in some cases the direct provision of goods and services) is more socially beneficial at a given point in time. For example, efficient provision by the private sector may require effective regulatory capacity on the part of the government that does not currently exist and that cannot be provided in the immediate future. Distributional considerations may also justify public involvement in some investments in youth. For example, those in charge of allocating household resources may fail to give adequate weight to the longer-term interests of youth generally (and of girls in particular) in making such allocations. Under these circumstances, public involvement in some form may be desirable to protect the longer-term interests of youth. Another possible justification for public involvement that may be particularly relevant to investments in youth is the merit good argument, i.e., that even well-informed people (and particularly youth) may not always act in their best interest (for example, mandatory use of helmets by motorbike riders or drivers).17 The point made earlier in Section 1.2 about the possibility that some youth may have higher discount rates as youth than as adults suggests that youth projects involve merit goods. However, the merit good argument is controversial. It is often argued that no one other than the person involved is in a position to know what is in fact in another person's best interests. 2.5 SUSTAINABILITY The economic analysis should also address the issue of whether the activities supported by the project are likely to be sustained after the project terminates (i.e., when project- provided financing and other inputs are no longer forthcoming). The dimensions of sustainability are financial, institutional and political. The analysis of financial sustainability should include a projection of the project's fiscal impact, including project-related taxes and subsidies and their incidence, both during and after the project. The analysis of fiscal impact should also assess the "fungiblity" of project resources, for example, the likelihood that project resources will substitute for resources that would otherwise have been provided to a sector or whether additional resources provided to the sector are used to support other, possibly less efficient and less equitable purposes (Hammer 1996). 17Yet another possible justification for public involvement in investments may be the large scale of the investment required. However, this is unlikely to be an important factor in the case of most youth projects. 15 In the case of some youth investments (for example, projects supporting the delivery of health or training services), there may be opportunities to recover costs from beneficiaries. From the point of view of the efficiency policy motive (though not necessarily from the point of view of distributional concerns), it is desirable that the project charge private beneficiaries a "user fee" equal to the social marginal cost of the project so that private individuals make their decisions in light of the true social marginal costs. Thus, quite aside from concerns about financial sustainability, there are important efficiency arguments for charging such user fees ­ that also may have a positive effect on financial sustainability.18 In some cases, policy reforms or additional project investments may be needed to enhance cost-recovery prospects (for example, the targeting of scholarships or development of credit markets to provide loans to finance continuing education or vocational/technical training; the improvement of the tax system to recover partial returns from publicly-subsidized human resource investments).19 The analysis of institutional sustainability involves an assessment of the capacity of institutions previously supported by the project to continue delivering the project- supported services after the project ends. In the case of governmental institutions, important considerations include whether staff have adequate incentives to continue performing the services they performed during the project In the case of NGOs that may have been supported by the project, institutional sustainability should include an assessment of the organization's own sustainability, including its ability to attract continuing financial support and to use that support effectively beyond the life of the project. The analysis of political sustainability involves an assessment of the degree of political support for the continued financing and provision of project-supported services after the project ends. Some youth-related investments may be politically controversial and may only be acceptable to governments as long as their costs are financed by donors, possibly as a condition of obtaining donor support for other, more popular investments. Another consideration is that the benefits of some youth investments may not be widely enough distributed among the population to garner political support. The degree of political support may be related, in part, to the degree to which benefits are targeted to the poor and other vulnerable groups, who often lack the power to influence political decisions. There may be a tradeoff between well-targeted programs and programs that have a large enough clientele to assure longer-run political support. 2.6 RISK AND SENSITIVITY ANALYSIS Risk and sensitivity analysis are two closely related approaches to handling uncertainty either about the future value of the prices of key inputs or outputs (risk analysis) or about key parameters used in the economic analysis (sensitivity analysis). The economic 18 But while having positive rather than zero user fees is likely to contribute to financial sustainability, it does not guarantee financial sustainability if there are public goods or scale economies or price distortions. 19 Because youth are likely to emigrate from some countries, consideration should be given in such cases to how loans can be repaid after youth emigrate. 16 analysis should identify those factors in a project about which there is significant uncertainty and suggest measures that can be used to manage the key risks.20 In the context of youth projects, for example, there may be uncertainty about the impact of the project-provided services on actual youth behavior (for example, does a better understanding of sexual risks translate into less risky sexual behavior?). In this case, the key parameter(s) should be varied by some arbitrary amount (usually 10-15% or one standard deviation) to see how such variations affect conclusions about the project's net present benefits or its cost-effectiveness.21 Sensitivity analysis is often applied in the case of key parameters in the economic analysis about which there is an unusual level of uncertainty. Another common use of sensitivity analysis for evaluation of youth projects may be to determine how, for example, a different choice of discount rate may affect both the project's net present benefits as well as those of individual project components or alternative technical approaches. The results of sensitivity analysis may indicate the areas where additional information would provide the greatest payoff in terms of obtaining a more accurate estimate of the project's net present benefits. However, even in this area, cost-benefit analysis is useful (at least conceptually) in determining whether the marginal benefits of additional information are worth the marginal cost. Risk and sensitivity analysis as usually practiced have the following limitations (Belli, et al. 1998): · They do not take into account the probability of the occurrence of the events whose impact they analyze · They do not take into account the possible correlations among the uncertain events or parameter values analyzed · The percentage variations in key parameters may not bear any relation to their observed (or most likely) degree of variation Various methods that can be used to address these limitations (for example, the use of Monte Carlo simulation techniques to analyze the impact of joint variations in key parameters) are discussed in Belli, et al. (1998). 2.7 DISTRIBUTIONAL ANALYSIS The economic analysis should consider not only whether the project and its various components yield the maximum possible net present benefits but also to whom the 20Governments (unlike individual investors) should not in general favor less risky investments with lower expected returns over riskier investments with higher expected returns because governments can better pool such risks (Belli, et al. 1998). However, there are exceptions to this rule (for example, very large projects). 21The concept of a "switching value" for a given parameter (i.e., the percentage change in the parameter required to reduce net present benefits to zero) is sometimes used with sensitivity analysis. 17 benefits accrue and who pays the costs, i.e., the project's main "winners" and "losers."22 Although cost-benefit analysis and cost-effectiveness analysis are designed mainly to assess the efficiency of projects, methods exist for weighting benefits or effectiveness estimates to reflect distributional preferences (for example, weighting more heavily benefits that accrue to the poor or to women).23 Benefit-incidence analysis may also be useful in identifying the winners and losers from a project (Adhikari, Gertler and Lagman 1999). The distributional analysis should also analyze the likely impact of any cost- recovery measures that may be introduced by the project or by others on the subsequent distribution of project benefits. SECTION 3. ESTIMATING COSTS, EFFECTIVENESS AND BENEFITS A key feature of economic analysis ­ in contrast to financial or budgetary analysis -- is that the costs and benefits are based on opportunity costs to society, not financial flows. For example, in evaluating the cost of delivering health services to youth in public health facilities, the opportunity cost of inputs such as labor (including volunteer labor), drugs and medical supplies, buildings (including land) and equipment are used, instead of the financial costs that may be reflected in governmental or project accounts. Taxes and subsidies should not be included in the resource costs and benefits (even though their values are likely to affect both the project's fiscal impact and its distributional impact).24 Economic analysis should, however, take fully into account "externalities," such as the opportunity cost of a project's "spillover" costs and benefits on others' labor productivity, health or learning. Economic analysis should also clearly identify the project's expected impact, i.e., the difference between what is expected to happen as the result of the project versus the counterfactual of what would be expected to occur in the absence of the project. A key aspect of measuring project impact is to assess the possible displacement of existing services (including privately-provided services) by project-provided services. Economic analysis requires that the project's expected impact be "monetized" (i.e., expressed in money terms) so that its attractiveness as an investment can be compared to other possible uses of the same resources. This is done by attaching a "price" to each of the project's impacts to obtain an estimate of the project's "benefits." The sum of the project's net benefits (or almost equivalently, its benefit-cost ratio or internal rate of return) can then be readily compared to those of other projects. Although many project impacts can be readily monetized (for example, increases in labor productivity), it is more difficult to attach a money value to some project impacts (for example, those that are not traded in markets such as "improved health"). 22Such distributional analysis is discussed in detail and illustrated in Belli et al. (1998). 23These techniques are discussed in Adhikari, Gertler and Lagman (1999). 24However, activities that are financed by tax revenue may need further adjustments to reflect the distortionary costs of collecting additional taxes or of governmental expenditures, as discussed below. 18 This section of the Guide elaborates on some of these issues with regard to the components of cost-effectiveness and benefit-cost estimates that are widely used to evaluate projects­ first with respect to costs, then effectiveness (impact), and finally benefits. The discussion in this section is meant to complement the corresponding discussions in existing reference documents on the economic analysis of projects and focuses on the special features of youth projects as they relate to these topics. 3.1 ESTIMATING COSTS This section of the Guide discusses issues related to the estimation of project costs in youth projects. The discussion does not cover much of the mechanics involved in preparing actual cost estimates. For discussions of these topics, the interested reader is referred to Over (1991). 3.1.1 The identification of project inputs Youth projects typically involve several distinct activities. The first step in estimating project costs is to identify the distinct activities that are included in the project and to identify their inputs, outputs and outcomes. The project's Results Framework is a good starting point for such an analysis. It is critically important to identify all relevant inputs, regardless of whether they are provided by the project and regardless of whether they involve any financial expenditure (inputs not provided by the project will often be missing from the project's Results Framework). Table 3.1 lists some of the inputs that are frequently used in youth projects. These inputs are grouped into broad categories: (1) capital inputs (i.e., inputs that are not completely consumed during a given year), and (2) recurrent inputs (all other inputs). It is necessary to separate capital inputs from recurrent costs not only because they should be handled differently in calculations but also because the distinction is an important one in analyzing the issue of sustainability that is largely a function of recurrent costs. Project inputs should be limited to the additional inputs required to perform various project activities. For example, in the case of personnel, project inputs should be limited to the additional time required by personnel to perform project-related tasks. If the project uses capital inputs (e.g., existing equipment or buildings), the project's inputs should be limited to the portion of the input's total use that is project-related, for example, the space occupied by the project in an existing building or the time during which an existing vehicle is used by the project. This principle should apply equally to both existing capital inputs and to capital inputs purchased by the project. Care must be taken to look closely at the implications of "piggybacking" additional activities onto already existing activities. For example, the implications of piggybacking treatment for schistosomiasis onto a de-worming intervention that already covers geohelminths are discussed in the Kenya case study in Section 5.1. If, in order to be involved in the proposed project, it is necessary to divert time or space from some other activity, the inputs so used should be included in the cost calculations. Only if those resources are otherwise idle or unoccupied and expected to remain so for the life of the project and beyond, should they not be considered inputs used by the project. 19 Table 3: Frequently used inputs in youth projects Inputs Capital inputs Buildings Land Vehicles Other equipment Renovations/major repairs Basic training (including training of trainers and the cost of developing training materials) Recurrent inputs Personnel Client time Client transportation and related costs Materials and supplies (e.g., Medicine/contraceptives, Food, Micronutrients, Books, Training materials) Utilities (water, electricity) Telephone/communications Petrol Maintenance and repair Services (for example, legal, accounting) Refresher training The principles of cost measurement are the same for integrated youth projects or youth project components included in broader (non-youth) projects, although their application may be more difficult the more complex is the project. It is important to consider for such joint programs the possibility that the costs are not simply the sum of the costs of the individual components. It is likely that there will be some economies from sharing costs, particularly capital costs and management overheads. In fact the perception that there are such economies of scope is one of the major reasons for having integrated programs or projects (the other being possible synergies in effectiveness/benefits). For integrated programs, it may be very difficult to estimate the cost in isolation of any one component. In such cases, economic analysis may have to focus on the program as a whole rather than on its separate components. Special attention should be given to identifying the inputs correctly of project activities that are scaled-up from initial "pilots." The cost of providing a good or service for youth is likely to change with the scale of operation. Often the cost per unit of output will initially fall, due to economies of scale or increased rates of utilization, as the level of output increases. On the other hand, costs may rise with considerable expansion because the conditions under which the initial "pilot" operation was conducted may have been more favorable than conditions that more generally prevail and the dedication and enthusiasm of innovators may be hard to replicate on larger scales. It is important to include as inputs the management and other inputs that pilots may have received from the organizations that supported them. There is often a tendency to focus only on the direct costs of services without considering the various management costs that may be involved. Costs also are likely to rise with extensive expansion if, for example, youth who are less 20 well situated (e.g., because of location, family background, innate characteristics) increasingly are served. Since one of the important decisions to make concerning project design may be the choice of scale, it is useful to explore how costs are likely to vary as output increases. The level of inputs required by a project may decline at least initially because there is important learning about how to procure, organize, distribute and manage the inputs and outcomes and how to produce the desired outcomes more effectively. These learning effects are conceptually distinct from scale effects, and it is useful to keep them distinct in undertaking analysis. Learning can occur, for example, even if there is no change in scale. Since learning may be important for many projects, it is useful to explore how the needed inputs are likely to vary as the result of such learning (especially if sensitivity analysis indicates that alternative assumptions in this regard have a significant effect on the conclusions). 3.1.2 Costing the inputs Once the project's inputs have been correctly identified, the next step is to assign an economic cost to each input. The economic cost of an input is sometimes referred to as its "opportunity cost," i.e., the value of the input in its next best use. As previously indicated, the opportunity cost of an input is not necessarily the same as its financial cost (for example, the public expenditure cost of the input, or what the project expects to pay for the input). There are at least two important ways in which opportunity costs differ from financial costs: (1) The prices actually paid for inputs may not correctly reflect their opportunity cost. An important example is inputs that are contributed by individuals or by the community (for example, volunteer labor). Although the project does not have to pay for these inputs (or may pay less than market prices for some of them), they do have an economic cost. In all such cases, the full market value of contributed inputs should be estimated and included in the project costs. Governmental personnel are often paid a salary that is considerably lower than the market wage, even when the market value of all allowances and benefits is included.25 In such cases, the economic cost of the personnel is not what they are actually paid, but is rather the full market value of the time they spend performing their jobs. The use of some project inputs may impose external costs, for example, the effect of project vehicles in generating additional air pollution and congestion. Ideally, the cost of project inputs should reflect any such externalities. In the case of imported inputs, the local price paid may not reflect the opportunity cost of the input (for example, if the exchange rate is significantly over or under-valued). However, this type of distortion is less important now than it was in past decades.26 Lastly, all taxes and subsidies should be removed from 25For this reason the case study of continuing education in Serbia-Montenegro in Section 5.5 estimates earnings gains from additional education on the basis of private earnings (i.e., excluding earnings of governmental workers or workers in state enterprises). 26 Almost a decade ago, for example, in this regard Devarajan (et al, 1997) stated: "Reforms of trade policy and exchange rate systems have reduced the distortions of most concern. In these circumstances, paying only modest attention to shadow prices may be a sensible allocation of the time of the economists." 21 the prices paid when costing project inputs (for example, subsidies for petrol or electricity). (2) Some project expenditures or outlays do not represent real resource costs of the project but are instead transfers that represent changes in the command over existing resources: The distinction between economic costs and transfers is one of the least-well understood aspects of economic analysis. It is particularly important in the context of youth projects because many youth projects include cash grants or stipends, as well as in- kind benefits. The case of cash transfers generally is easiest. If for example the government taxes adult workers 100 rupees and gives those rupees to unemployed youth, the cash transfer generally does not represent a project cost. The cash transfer represents only a change in the command over existing resources (e.g., from tax payers to unemployed youth). The actual costs related to such a cash transfer include the cost of administering the cash transfer program and related distortionary costs. The latter would include any reduction in resources available to the economy from any effect the cash transfer may have on work effort on the part of unemployed youth receiving the cash transfers (for example, by reducing their job search effort or by discouraging them from accepting a relatively low-paying job) and from the additional cost of collecting the government revenue needed to finance both the cash transfers and the program's administrative costs (this includes both the administrative costs of collecting the additional tax revenue and the distortionary costs that result from increases in the collection of most types of taxes). Conditional cash transfers (i.e., transfers given to an individual who fulfills one or more conditions, such as enrolling in and regularly attending school) are also generally primarily or exclusively transfers.27 In this case, project costs include (in addition to those discussed above for any cash grant) the additional cost of the activity that is encouraged by the condition attached to the transfer. For example, in the case of a conditional cash transfer provided to youth who enroll in and attend school, these costs include the cost of additional school inputs required to accommodate any increased enrollment (including any additional costs incurred to offset any negative impact on students previously attending school, i.e., the costs of "crowding"),28 increases in household out-of-pocket expenditure on school-related items (for example, uniforms, books, transportation--but not fees to schools, as that would be double-counting), and 27Some qualification is made here because conditional transfers, like in-kind transfers, may have real resource costs if, as summarized at the end of this section, (1) they lead to a significant input into the activity being supported and (2) the project results in a net increase in the utilization of this input. For example, consider scholarships (transfers conditional on attending school) for poor, malnourished youth. Empirical studies suggest that the increased monetary income provided by such scholarships will lead to increased nutrient consumption by the youth (among other effects) that will increase their learning in school (e.g., Behrman, et. al. 2003, Behrman, Alderman and Hoddinott 2004). To the extent that this occurs, part of the conditional transfer is not a pure transfer, but a resource cost. 28Even if additional resources have been used in an effort to offset any negative effects of an intervention on others (non-beneficiaries), the estimation of project impacts should still include the possibility that some negative (or possibly positive) effects have been experienced by others. It is conventional to include the monetary value of any negative external effects as part of a project's costs, while the monetary value of any positive external effects is included among the project's benefits. 22 the opportunity cost of the additional time that grant recipients spend in school-related activities, including travel and homework. The distinction between project costs and in-kind transfers is a bit more subtle. The first relevant question is whether the item or items transferred in connection with the project are a significant (i.e., quantitatively important) input into the project-supported activity. Consider, for example, the case where food is given to a family in lieu of a cash grant in the above example. If the food is not a significant input into the educational activity that the project is attempting to encourage (for example, if children are already adequately nourished), the value of the food should be regarded as a transfer, and benefits and costs are the same as the case of a conditional cash transfer. If the food is an important input into the education activity (for example, food provided to malnourished children whose malnourishment precludes them from learning to their potential in a school-feeding program), some or all of the food provided through the program might be regarded as a cost of the program. To be considered a cost of the project, the food provided through the program has to increase the total daily nutrient intake of the children participating in the program. If it simply substitutes for food previously provided to the same children by their households, it would not represent an increase in the level of this input into the education of school children and the value of the food provided should in this case be treated as a transfer. Even if the feeding program does not result in any increase in children's nutrient intake, the project might still yield benefits by encouraging children to attend school. In this case, however, benefits and costs would be similar to those with a conditional cash grant, as discussed above. Summarizing, the key questions in determining whether a given project expenditure is a project cost or an in-kind transfer are: (1) is the item a significant input into the activity supported by the project? (2) does the project result in a net increase in the utilization of this input in the project-supported activity, or does it simply affect how an unchanged level of the input is financed? 3.1.3 Other project costs In addition to the costs of direct project inputs, projects often entail costs borne by the private sector that are often not readily apparent. Several examples of such costs are provided in the case studies discussed in Section 5 below. For schooling and training programs (and to a lesser extent, in health programs) the most important private cost is likely to be the opportunity cost of time. Another important private cost is likely to be the distortionary cost of raising the necessary additional tax revenue to finance project- related governmental expenditure (including transfers) (Devarajan, Squire and Suthiwart- Narueput 1997). In addition to the cost of collecting additional revenue, all taxes other than a lump sum tax lead to efforts to avoid the payment of the tax that involve real resource costs. Society has less product because of such distortions. These distortionary costs may be considerable. For example, it has been estimated that the distortionary cost (often called the "deadweight loss") of raising a dollar of tax revenue in the United States ranges from $0.17 to $0.56, depending on the type of tax used (e.g., Ballard, Shoven and 23 Whalley 1985, Feldstein 1995). Estimates for some other countries range from $0.18 to $0.85, depending on the tax (van der Gaag and Tan 1997). Harberger (1997) suggests using a shadow price of $1.20-1.25 for all fiscal flows on a project. The existence of substantial distortionary costs related to the collection of taxes provides one important justification for careful consideration of cost recovery possibilities in projects (Hammer 1996).29 In addition to distortionary costs related to the collection of additional tax revenue, some project activities may be responsible for modifying other types of private behavior that result in real resource costs. For example, conditional cash grants provided to a family to encourage school enrollment may affect the work effort of some family members. 3.1.4 Treatment of some specific cost items To clarify the treatment of some key components of project costs, this sub-section briefly discusses some specific cost items: Start-up costs (including costs of technical assistance): Start-up costs involve real resource costs, so they should be included in cost calculations. They also by definition occur early in the project life, so they are not discounted to the same extent as recurrent costs and capital service costs that are distributed over time. Monitoring, evaluation and analysis costs: These are important activities for any youth project, particularly new projects. Good projects should include such activities. Of course they have resource costs in terms of personnel and other recurrent expenses (e.g., materials and supplies) and capital services. But within the range of resource uses for these activities that usually are considered, it would appear that the expected gains in terms of program modification or possible abandonment exceed the real resource costs. The Mexican PROGRESA/Oportunidades program that is summarized in Section 5.4 below is a noteworthy example, though all too rare, of a youth-related program that has had serious information collection and economic evaluation incorporated since its inception. Capital costs: Many capital costs will typically involve investments made by the project. In this case, it is important to consider whether the capital input is consumed completely during the period in which costs and benefits are analyzed. For example, a building financed by a project will probably have a useful life beyond, say, a 10-year period used to evaluate the project's benefits and costs, whereas project-purchased vehicles will probably not. The easiest way to handle the former case is to subtract the present value of the estimated residual value of the capital item at the end of the project from its initial cost. Some projects may use all or part of an existing capital item as a project input (for example, space in an existing building). In such cases, a capital cost should be imputed 29As noted above in Section 2.5, another important reason for using user fees that will result in some cost recovery is to encourage more efficient use of the goods and services provided. 24 for the input. Capital costs include changes in the market value of the item during the year, such as may occur from the item's depreciation due to use, and the opportunity cost of the capital invested in the item during the year (averaged over the year).30 Maintenance costs may reduce actual depreciation costs, so it is necessary to avoid double counting these two items. Although it is conventional to regard capital costs as a "fixed" cost, some capital costs increase with output levels (for example, the depreciation of vehicles used to provide outreach services). If estimates of building costs are included, it is important if possible to treat building costs differently from land costs. Buildings depreciate (unless maintained impeccably), but land does not. The appropriate capital cost for land is an estimate of its market rental value (with the estimated cost of the building removed). If such an estimate is unavailable, an estimate of rental cost can be obtained by multiplying the approximate market value of the land by the real interest rate (i.e., the opportunity cost of the capital tied up in the land). Training costs: Training of trainers is clearly a capital cost, as is the cost of developing training materials. The cost of periodic re-training is a recurrent cost. This distinction is most important in the context of sustainability analysis. 3.2 ESTIMATING EFFECTIVENESS The objective in estimating effectiveness is to estimate how identical individuals and entities would behave with and without the youth project intervention, all else equal. The difference in such behaviors (outcomes) measures the effectiveness or impact of the project. It is conventional in the literature to use the term "effectiveness" to refer to the impact of a project on one summary indicator and to use the term "benefits" (see Section 3.3) to refer to the price-weighted sum of multiple impacts of a project.31 This Guide follows that convention.32 There are three basic steps involved in estimating effectiveness of each of the interventions in a project: (1) identifying relevant outcomes, (2) choosing appropriate indicators for these outcomes, and (3) estimating the impact of the project interventions on these indicators. Once again, the project's Results Framework (though possibly incomplete) is a good starting place. 30 Capital cost estimates often include an estimate of depreciation but not of the opportunity cost of capital. 31 In the absence of prices, sometimes other values are used to weight multiple outcomes in "weighted cost-effectiveness analysis." See Belli, et al. (1998). Although weights other than prices may be preferred for comparisons to investments with similar outcomes, comparisons across sectors require the use of a price-weighted "benefits" measure. 32 In principle, the relevant individuals and entities in assessing such impacts (effectiveness, benefits) include all those affected by the intervention, not just the intended beneficiaries. For example, if the project provides a service, this may displace alternative providers of the services and discourage new providers of the service from entering this market (including privately-provided services). These other effects are the focus of distributional analysis. For examples, see Belli, et al. (1998). 25 3.2.1 Identifying outcomes If a set of possible youth projects all have just one measurable outcome (say, reducing STIs), then comparisons of project impacts on indicators of that single relevant outcome across the projects yield information on relative effectiveness. By combining this information with information on resource costs (see Section 3.1), project choices can be guided by selecting, for example, the project with the greatest effectiveness for given resource costs. This is an application of cost-effectiveness analysis (CEA). If it is desired to compare the project to other projects with a different outcome (either in the same sector or in a different sector), it will be necessary to assign a monetary value to the project's impact on this single outcome. Doing so makes it possible to estimate the project's net present benefits, i.e., to prepare a cost-benefit analysis (CBA). Many, perhaps most, youth projects, however, have a multiplicity of outcomes (Table 3.2).33 A youth project that affects education, for another example, may also have impacts on employment, wages, physical and mental health (including reproductive health), fertility, age of marriage, and on various types of risky behavior (for example, crime, unprotected sex, drug and/or alcohol abuse, and tobacco use). If one wants to compare alternative youth interventions, as above, the implications of multiple benefits for CEA requires weighting the various outcomes consistently, with the weights in principle related to the impact of a change in each outcome on welfare. This seems naturally to lead to considering "benefits" (Section 3.3), in which prices are used as weights. Using prices as weights also makes it possible to compare the economic returns of youth projects with those of projects in other sectors. Table 4: Common outcomes of youth interventions Labor market outcomes Labor productivity Labor supply Employment/unemployment Child labor Education outcomes School enrollment/attendance Cognitive achievement Educational attainment Age at which a given schooling grade is completed Health outcomes Mortality Morbidity Disability Nutritional status (e.g., anthropometry, micronutrients, obesity) Health care expenditure Diseases (HIV, STI, TB, worms) 33In assessing effectiveness or impact, only outcomes are relevant. Inputs and outputs are relevant in measuring costs, as discussed above. 26 Fertility Abortions Infertility Risk-pooling services (e.g., school health insurance) Tobacco use Drug/alcohol abuse Physical/sexual abuse Female genital cutting Other outcomes Crime Violence/civil conflict Tax revenue (e.g., tobacco tax) Age at marriage Social solidarity/social capital Self esteem Orphans Teen pregnancies Social exclusion Source: Knowles and Behrman (2003b). The first step in estimating project effectiveness is to identify all of the outcomes that are expected to be impacted by the project. Many past youth projects make the mistake of focusing on only one or two potential outcomes while neglecting the others.34 In some cases, this neglect significantly reduces the project's benefits.35 For example, by adding another activity to a youth center (e.g., reproductive health services), the project's benefits might increase significantly without adding much to its costs. Even if failure to identify all potential outcomes in project design does not adversely affect the project's content, it may limit the type of information that is monitored or for which information is collected in baseline and follow-up surveys. This may mean that although the project may have achieved impact in certain other important areas (as might be revealed, for example, in qualitative research), there is no information available to document this impact quantitatively. 3.2.2 Selecting indicators Once the project's relevant outcomes have been identified, it is necessary to select appropriate indicators for measuring the outcomes. Once again, a good starting is the project's Results Framework and relevant sector analyses. The choice of appropriate indicators is important. Sometimes it is difficult to predict exactly how a given outcome will be affected by a project intervention. For example, the Kenya case study (Section 5.1) found that de-worming affected both health and schooling outcomes. However, surprisingly in light of findings from other de-worming studies, neither cognitive achievement nor hemoglobin levels were affected (only infection rates and school 34One possible reason for this practice is the perception that it is difficult to assign monetary values to many of the outcomes of youth projects (for example, improved health). This perception may lead project designers and analysts to limit economic analysis to cost-effectiveness analysis (CEA). Because CEA necessarily involves a single effectiveness measure (although the single measure may be a weighted sum of several outcomes, as in weighted CEA), this narrower focus may result in ignoring several relevant outcomes. 35The case studies in Sections 5.5 and 5.6 illustrate this point. 27 enrollment and attendance). This study illustrates that it is better to collect data on multiple indicators, even for a single outcome such as improved health or improved schooling, rather than relying on only one or a few indicators. Another important consideration in selecting outcome indicators is accuracy. If improved health is the expected outcome, it is better to measure the expected health improvements as objectively and accurately as possible. Physical measurements or lab tests conducted by skilled professionals are obviously more reliable outcome measures than highly subjective responses to questions about one's state of health or recent morbidity. Similarly, scores from independently administered tests are likely to provide more accurate measures of cognitive achievement than self-reported test scores. Several of the case studies in Section 5 illustrate the use of such reliable outcome measures. A number of candidate indicators may have nonlinear effects, perhaps having strongly positive welfare implications at low levels of the indicator but with declining incremental effects as the value of the indicator improves, possibly even having negative welfare implications at higher levels. For instance historically the dominant concern about macro nutrition for youth in developing countries has been undernutrition, as reflected in low Body-Mass Indices (BMI). So an increase in BMIs due to a project has been regarded as a positive outcome historically. But increasingly there is concern about the rapidly spreading prevalence of overweightedness and obesity among youth in developing countries, for which an increase in BMI is a negative outcome. Thus indicators tend to be more useful in terms of capturing important effects the more specific they are with regard to details such as time horizon, target groups, age, gender, pregnancy status and the more that they recognize the importance of any nonlinear effects. 3.2.3 Impact Estimation Once an indicator(s) is selected, the third basic step is to obtain estimates of the project impact on the relevant indicator(s). It is also desirable if feasible to identify which impacts are private (i.e., received by those directly receiving project-provided services or other benefits) and which accrue to the rest of society, since this provides essential information for determining whether there is an efficiency rationale for public involvement (for example, in the Kenya and Mexico case studies in Sections 5.1 and 5.4, such external effects are estimated separately). For ex ante analysis, the options are likely to be limited. But for two reasons it is important to discuss here how such estimates can best be obtained. The first reason is that the analyst must display critical judgment in selecting among the estimates in the literature or in assessing the validity of estimates that he/she has prepared. Good impact estimates must control for endogenous choices of those affected by projects in the presence of important unobserved factors that also affect these same behaviors. The second reason is that the analyst should incorporate into the economic analysis of a project the design of an appropriate impact evaluation component so that the project's success in achieving the expected effects can be documented. Impact evaluation is also useful in determining whether an ongoing project warrants modification, expansion 28 (scaling-up) or abandonment. The same considerations are applicable in all of these cases. The basic objective in estimating project effects is to understand how a representative youth in the target population will behave or be affected (as reflected in the indicator(s) of interest) if exposed to the prospective youth project versus how the same individual will behave or be affected if not exposed to the program being analyzed. Since the same youth cannot be observed being simultaneously exposed to a youth project and not being exposed to the same project, good estimates of project impact must combine good data with appropriate estimation techniques. 3.2.3.1. Data A central quality of good data is knowledge of the population it represents and how that population relates to the target population for the youth project being analyzed. Unfortunately much available data relate to selected subpopulations of youth, without information that permits understanding the implications of estimates based on the data for the population of interest. For instance, much data is institutionally-based in contexts in which only selected youth choose to use certain institutions ­ secondary schools, prenatal clinics, health clinics, training services, military services and employment services are some important examples. Program estimates based on such data without controlling for the selected nature of which youth participate in the relevant institutions are likely to be contaminated in an unknown direction and thus may not be very informative.36 Desirable data characteristics: What data are most desirable? There are at least six critical characteristics. (1) As suggested in the previous paragraph, one central characteristic is that the data should include a known representation of the target population. This can be accomplished through a random sample of that population, perhaps stratified, but with known sampling probabilities for each stratum. (2) The data should include information from prior to the implementation of the program ("baseline" data). Without such information it is very difficult (or requires stronger assumptions) to learn what changes from the pre-baseline situation occurred, to assess what the properties of the data are (e.g., whether experimental assignment in fact was random), and to control for possibly systematic attrition from the sample. (3) The data should follow the individuals in the sample over the period of time that is of interest for evaluation of program effects with periodic information that permits the characterization of the dynamics of those effects (panel or longitudinal data). Because some of the impacts of interest may have long lags, this may require panel data over many years or the collection of indicators that are available in shorter time periods and that are "sufficient statistics" for the longer-run effects. For example, if there is interest 36The case studies in Sections 5.4 and 5.7 illustrate various approaches for dealing with such selectivity problems. 29 in the impact on adult health and adult productivity of de-worming youth or providing youth with school scholarships, either the sample members need to be followed for many years or some "intermediate" outcomes measured over shorter periods of time that arguably are sufficient statistics for the longer-run impacts (e.g., cognitive achievement, long-run anthropometric indicators).37 Even if there are such short-run "sufficient statistics," data are likely to be needed for several years to enable a confident assessment of the program impact on such indicators given the combination of start-up adjustments in programs and lagged responses to new programs by individuals, families and other entities. (4) The information in the data should include the indicators of program impact of interest (including importantly intermediate indicators as discussed in the previous point) and, generally at least, information on the relevant youth and the contexts in which they operate (e.g., household, community, service providing institutions such as schools and clinics). Only with such information will it be possible to assess, for example, whether program impacts differ by types of youth, characteristics of their parental households, the nature of their communities, and the nature of relevant service providers.38 (5) The data should include information on any broader population that is required to assess all dimensions of the program impact. For example, if a program is targeted at pregnant youth, it is likely to be important to include youth who are not pregnant in order to understand how the program may affect the probability of youth becoming pregnant. Samples that are limited to, say, youth in schools or youth who are eligible for a program limited to poor families do not permit the investigation of spillovers beyond those in the sample (e.g., those in school, those who are designated to be "poor") and do not permit assessment of the program impact on distribution (because not all youth are in school or in poor families).39 (6) The data should permit the investigation of program effects through comparing some youth (and others affected by the program) with another comparable group. Considerable emphasis has been placed on the sixth characteristic, about which we now elaborate. But because we discuss this characteristic more extensively here than the other five should not be misconstrued to mean that the other five data characteristics are relatively unimportant. 37 The case studies in Section 5 all use such intermediate indicators though in some cases the relatively long panels permit such more direct estimates of longer-run effects (e.g., the rural component of the Mexican program that is discussed in Section 5.4, for example, currently has longitudinal data for six years after program initiation). A few studies exist that have followed individuals for relatively longer periods. Behrman, et al. (2003) is an example in which the impact of experimental assignment of extra nutrients to children on adult cognitive achievement about 30 years later is assessed. Another alternative is to use structural models, as is discussed in Section 3.2.3.2. 38 Such information also is likely to be critical in assessing various sample properties (again, including whether experimental assignment was random) and sample attrition. 39 For example, the Kenya case study in Section 5.1 has the shortcomings of having a school-based sample rather than a population-based sample. 30 Controlled experimental data with random assignment of treatment: The "Gold Standard" for assuring the sixth characteristic is through a good controlled experiment in which the youth that are exposed to the program impact ("treatment" group) and those who are not ("control" group) are determined randomly. With a well designed and carefully implemented experiment (i.e, random assignment to treatment versus control, double-blind procedures so that neither the treated nor those administering the experiment know who is receiving treatment and who is receiving a placebo, no spillover effects between the treatment and control groups, no systematic attrition, baseline and longitudinal data, large enough samples so that any other effects are random, etc.) good estimates of the impact of a particular program in a particular context can be obtained. If an analyst of potential youth projects can find estimates of the impact on the indicator(s) of interest based on a well designed and carefully implemented experiment for an identical program in a similar context, then adapting those estimates is the dominant strategy. However, that experiments differ in their ability to provide relevant information is well illustrated in the Kenya, Tanzania and Mexico case studies in Sections 5.1, 5.3 and 5.4. All of these experiments, for example, have information with which to estimate critical dimensions of spillovers and have (or aspire to) panels of relatively long duration. These are substantial strengths.40 However it is not often that the ex ante project analyst can identify existing impact estimates that fit the criteria mentioned in the previous paragraph. Nevertheless, experiments with random assignment of treatment are very important for purposes of this Guide for at least two reasons. Firstly, the procedures followed in good experiments provide a benchmark with which to evaluate other procedures used to generate impact estimates. Analysts that have this benchmark explicitly in mind are better prepared to make judgments about the quality of non-experimental impact estimates that they are attempting to build upon in their economic analysis of prospective youth projects. Secondly, a good youth project will have an integrated monitoring and evaluation component in order to assess its impact, if it is implemented, how well it is performing over time, what changes might make it better, and whether it should be continued. In many contexts, the most informative approach to this important component of a project will be an experimental design. Often resources are not available to institute universal implementation of new programs immediately. This presents an opportunity for creating the basis for an experimental evaluation through selecting randomly those who receive the program immediately and phasing others in later (under the assumption that those who will be phased in later do not know that they will be phased in later). This type of experimental design is illustrated in the Kenya, Tanzania and Mexico case studies in Section 5. 40Despite these strengths, these data also have limitations. The Kenyan data, for example, are not based on a random sample of households but are school-based, so they do not permit satisfactory exploration of spillover effects outside of school, the impact of the project on school enrollment and the distributional effects of the program since not all youth are in school. Both of these and other experiments, moreover, do not permit confident evaluation of variants of the program not actually subject to experiments, though counterfactual policy experiments have been evaluated with the Mexican data. 31 Non-experimental data: Most existing prior estimates on which an ex ante analysis of youth projects must draw are based on non-experimental data. But that hardly means that all non-experimental data sets are the same. In fact, they may differ substantially with regard to the other five characteristics mentioned above. These differences, together with how the data are analyzed (Section 3.2.3.2 below) may affect substantially the quality of the impact estimates reported in these studies. Moreover, non-experimental data sets may differ in the extent to which they may be able to approximate an experimental design with random assignment even though they are not controlled experimental data. Some data sets, for example, include one or more of the following features that permit comparing youth affected by a program with similar youth not affected by the program: · "natural experiments" -- in which youth who appear to be similar in some cases are exposed to a program but in other cases not exposed for reasons that are claimed to be unrelated to their observed and unobserved characteristics. For example, the Colombia case study in Section 5.2 is based on a natural experiment.41 · "matching" ­ in which youth who did not participate in the program or have program access were selected to be as like as possible -- in terms of observed individual, household and community characteristics ­ youth who participated in the program. For example, the Mexico and Argentina case studies in Sections 5.4 and 5.7 use this data feature. · Program "discontinuity" ­ in which data are collected on youth who are eligible for the program according to some eligibility criteria and data on other youth who are not eligible but close to being eligible in terms of the same criteria. Use of this data set feature is discussed in the Mexico case study in Section 5.4. Finally, in most data sets the participation or non-participation of some youth is determined simply through a process of selection. In the absence of experimental data and of any one of the three special features in non-experimental data discussed above, most data sets are limited to data on youth who select to or not to participate in the program. That is, the program may be available to all youth in an area, but not all choose to participate. For instance a program may enhance the quality of upper secondary schools or provide contraception to those who go to clinics. But not every youth goes to upper secondary school or to clinics. Some select out for a variety of reasons that may or may not be observed in the data. That means that such individuals are not very good controls for evaluating the program unless the analysis can control for why they select not to participate. 3.2.3.2 Estimation It is useful for analysts of prospective youth projects to have some understanding of estimation issues for the same two basic reasons noted above with respect to data issues: 41 Another well-known example of a natural experiment is Duflo (2001). 32 (1) To be able to choose better estimates from the existing literature, and (2) to design the important evaluation component of the project they are considering. Different types of data ­ with the control samples generated alternatively by random selection in controlled experiments, in "natural " experiments, through matching, through program discontinuities, or through simple selection ­ have been used for different analyses of youth program impacts. At the analysis stage as at the data collection stage, whatever features non-experimental data have and however they were generated, statistical matching methods can be used to improve the comparisons between those youth observed in the data that are affected by the program and those not affected by the program.42 Indeed, matching may improve the comparisons even with experimental data if there is selective attrition from the original sample design, as plausibly often may occur if the program confers a perceived benefit so that the treatment group has stronger incentive to stay in the sample over time than the control group. This matching should be implemented using variables that are not affected by the program, which means that for this purpose (as well as some others noted below) pre-program baseline data are very important. Major estimation problems: The nature of possibly major estimation problems can be illuminated by considering the simplest and most common estimates for youth program effects. These are cross-sectional comparisons of indicator(s) between those youth exposed to a program and those who are not, based on non-experimental data in which the youth selected whether to participate in the program, perhaps with control for some observed characteristics of the youth, their families, local institutions and their communities. The estimation problems for such comparisons originate from: · Unobserved factors that may be fixed such as ability and motivation or that may vary such as changing prices or disease environments that are correlated with program participation and with program impact. · Endogenous behaviors among right-side variables that have been determined either currently or in the past by some of the same factors that determine the program indicators of interest. An example is the inclusion of schooling attainment in comparisons to assess program effects on reproductive health without controlling for the fact that schooling attainment itself is determined by observed and unobserved right-side variables.43 42 In general this is true even if the control sample is generated by matching because at the sample design stage there is likely to be much less information available on which to match (e.g., census data) and the matching is likely to occur at some aggregate level (e.g., census units). Once the data are collected, there is likely to be much richer information on individual youth, their families, their communities and local institutions that permit much better matching on observed characteristics. This point is illustrated in the Argentina case study in Section 5.7. 43 Two important implications of standard economic models of households are that endogeneity is likely to be pervasive (1) in part because at every point of time individuals and households make decisions about all their behaviors (not just ones that are directly targeted by particular programs) so programs may affect all behaviors (e.g,. a scholarship program may affect not only school enrollment but also time use of all 33 · Sample selection such as limiting the analysis to those who utilize certain institutions (e.g., cognitive tests or health programs only for those in school or in the military, clinical tests only for those who attend clinics or other formal medical facilities, earnings of wage employees) without controlling for those choices. · Measurement problems that may be random in right-side variables such as self- reports of program exposure, schooling, sexual experience, health and employment experience that tend to cause biases towards zero in estimated effects of these right-side variables or that may be systematic in similar variables that cause biases in either direction (e.g., boys overstating sexual experience, girls understating sexual experience; the poor understating and the better-off overstating disease experience). · Limited or no basis for assessing counterfactual policy experiments as opposed to the particular detailed program in the particular market and policy context that is observed. · Feedback through endogenous effects on markets or policies that may occur if programs are relatively large and markets are somewhat segmented, as markets in developing countries often are characterized to be. Increasing the schooling of a generation of youth in rural areas, for example, might be expected to reduce the wage skill premium for schooling in such areas, as well as induce outmigration. Strategies for dealing with these estimation problems: These are difficult problems, the best resolutions of which involve a combination of better data, more sophisticated estimation methods and explicit modeling of the underlying behaviors. The best analyses are sensitive to the synergies among data, estimation strategies and modeling ­ though they may blend these three components differently, with a tradeoff among better data, better estimation techniques and better modeling. Controlled experimental data with random assignment among the population (as opposed to some self-selected sample such as youth in school or in the military or utilizing clinics) are least likely to be subject to many of these problems, though even experimental data may have some of them such as measurement error in key variables or selection on unobserved characteristics if there is differential attrition over time between program (treatment) and control groups. Experiments on their own, moreover, do not permit direct estimation of the impact of counter factual policies or contexts for policies. household members, nutrition and health of all household members, fertility, migration, etc.) and (2) in part because endogeneity is not limited to concurrent outcomes since the constraints and resources that individuals and households currently have in some cases persist over time (e.g., innate health, ability, motivation) and in other cases were determined by past behaviors in part by such persistent factors (e.g., schooling attainment of youth, their current stock of health, schooling and health of their parents if there are intergenerational correlations in genetic and other endowments). 34 At the other end of the data spectrum in terms of the sixth characteristic mentioned above ­ which youth are affected by the program and which are not -- are data in which program participation is due purely to selection by youth. Simple cross-sectional comparisons with such data, which are the most common form of impact evaluation in the literature, are likely to be subject to all of these estimation problems with the result that simple impact estimates from such data are very difficult to interpret. If these are the only estimates available, the economic analysis should use sensitivity analysis to examine how alternative assumptions regarding an intervention's impact affect the conclusions. In between are the three special non-experimental approaches discussed above ­ natural experiments, matching and program discontinuity. If unobserved fixed or time-varying factors are not important, these types of data have as good controls as do random assignments, with the same substantial strengths and limitations (e.g., selective attrition, not providing much information with which to assess counterfactual policies). But it would appear from many micro studies that unobserved factors are often important and that the failure to control for them often affects impact estimates substantially. Finally, it probably is best to reiterate explicitly that the five other data characteristics noted above are important for all of these alternative data sources. In what follows we attempt to summarize the complicated issues relating to the nature of the data and various common estimation strategies and to what extent they are likely to deal with the estimation issues.44, 45 · Simple comparisons of indicators for youth program participants versus non- participants: For good experimental data or for the three non-experimental types of special data that may approximate well experiments, this comparison may lead to good estimates of program effects (though it is unlikely to do so for non- experimental data in which selective program participation determines the 44 In this discussion we presume that lessening any particular problem is likely to move the impact estimates closer to the true values. Because there may be partially offsetting biases so that eliminating one of the two can make the estimates worse, this assumption does not always hold. But in the absence of a strong prior on the existence of such offsetting biases, it is reasonable to assume that lessening any one bias is more likely than not to move the estimate towards the true effect. 45Examples of most of these procedures can be found in the ongoing evaluation of the Mexican PROGRESA/Oportunidades program that is summarized in part in Section 5.4. For example, Behrman, Sengupta and Todd (2005), Parker and Skoufias (2000, 2001), Rubalcava, Teruel and Thomas (2002) and Schultz (2004) make simple comparisons between eligible program participants and eligible non- participants using data from a controlled experiment and Behrman, Parker and Todd (2004) make such comparisons using matching to identify controls (with a comparison sample also identified by matching) with multivariate controls for observed characteristics; Angelucci, Attanasio and Shaw (2004), Behrman and Hoddinott (2005), Behrman, Parker and Todd (2004), Buddelmeyer and Skoufias (2003), Parker, Behrman and Todd (2004), and Todd, et al. (2004) difference over time for youth program participants and double-difference over time between youth program participants and eligible non-participants in the second case with controlled experimental data, in the fourth case with regression discontinuity estimates, and in the other cases with matching to identify controls (again, based on a comparison sample identified by matching) with and without control for pre-program observed characteristics; Attanasio, Meghir and Santiago (2002) and Todd and Wolpin (2003) develop and estimate a structural model and use it to explore counterfactual policy possibilities. 35 controls). But for all of these cases, including most real-world experiments with possibilities of selective attrition, fixed and time-varying factors that are not controlled in this procedure may be correlated with program participation and affect program outcomes. So the strong assumption needs to be made that these factors are not important or are not correlated with program participation in order to interpret the estimates as the true effects. · Comparisons of indicators for youth program participants versus non- participants with multivariate controls for observed characteristics: This is likely to improve the impact estimates particularly for those based on data with controls determined by selective participation of youth, but also somewhat less so for the other non-experimental data sources and even for real world experiments if the additional controls for observed characteristics are not endogenous. Nevertheless, there remain estimation problems due to unobserved fixed and time-varying factors. If the controls for observed characteristics that are used are endogenous, then whether the impact estimates are closer to the true effects or not than the comparisons without such controls is not clear. · Differencing over time of longitudinal data for youth program participants to control for fixed effects: This may improve the estimates, again most likely more so for the less satisfactory data, because this nets out the effects of unobserved fixed factors including ones that may be important in program participation (e.g., innate health, ability, motivation) and some aspects of measurement error (e.g., always over- or under-reporting by a fixed amount). Such procedures, however, exacerbate biases towards zero due to random measurement error and do not control for time-varying unobserved variables nor for time varying changes (e.g., the state of the macro economy, weather fluctuations, price fluctuations) that are common to treatment and control groups. · Double-differencing both over time and between youth program participants and non-participants of longitudinal data to control for fixed effects and common unobserved time-varying effects: This is likely to improve further the estimates, again most likely more so for the less satisfactory data, because this not only nets out the effects of unobserved fixed factors and some aspects of systematic measurement error but also controls for time-varying changes that are common to both program participants and non-participants. Such procedures still exacerbate biases towards zero due to random measurement error and do not control for time varying unobserved changes that may differ between participants and non- participants. · Comparisons of indicators for youth program participants versus non- participants with multivariate controls for observed characteristics and instrumental variables: In instrumental variable techniques, right-side endogenous variables are replaced by their estimated values conditional on a set of instruments that are not correlated with the disturbance term in the relation being estimated, that are not included in the relation being estimated and that 36 predict reasonably well the right-side variables being instrumented. Good instrumental variable estimates eliminate biases towards zero due to random measurement error and control for fixed and time-varying changes and endogenous choice variables. Again, they are likely to improve the estimates most for the weakest types of data. However finding good instruments is difficult because of the considerations about individual and household behaviors that are summarized above in the discussion of the estimation problem of endogeneity. Indeed, from the perspective of such modeling, it would appear that many variables used as instruments in the literature should be included directly in the relations being estimated or are correlated with the disturbance terms in those relations and therefore are not really good instruments. Program eligibility if assigned randomly, however, would seem to be a good instrument ­ pointing to another, not always recognized advantage of experimental data with random assignment of treatment.46 · Double- differencing both over time and between youth program participants and non-participants of longitudinal data to control for fixed effects and common unobserved time-varying effects with instrumental variables: This combines the advantages of controlling for fixed effects with the use of instrumental variables to control for endogenous behaviors, unobserved time-varying factors that differ between program participants and non-participants and measurement error. If there are good instruments, this moves the estimates towards the true effects, particularly for poorer data. But the caveat about good instruments again applies. On the other hand, with control for fixed factors, fewer instruments may be needed and program eligibility may be a good instrument even if there is not random assignment (e.g., due to program discontinuity). · Structural modeling of individual and household behaviors: If models of the relevant behaviors can be estimated (and in the process the other estimation problems discussed above controlled), such models can be used not only to evaluate programs that have been implemented but also to evaluate counterfactual programs or variants of actual programs that have not happened. This is an advantage even over estimates based on good experimental data because experimental estimates are not informative about how behavior changes if the program or the context changes unless such variants were part of the experimental design. Structural modeling, however, is conditional on the structural forms assumed and is challenging if dynamic forward-looking behavior with multiple states of the world is allowed.47 For these reasons there are very few such studies for the evaluation of youth programs, though there are some for PROGRESA (see Section 5.4). On a micro level this approach also assumes that prices and policies are not affected by the program. 46This estimation strategy is illustrated in the Colombia case study in Section 5.2. 47In economics the term "structural modeling" generally refers to modeling within such a dynamic framework, and not to static relations that ignore the complications of dynamics. 37 · Structural modeling of individual and household behaviors embedded in models of markets and policy determination so that there is feedback though endogenous prices and policies: This approach allows for the possibility that prices and policies are affected by the program in addition to having the other strengths and weaknesses of micro structural modeling discussed immediately above. 3.3 ESTIMATING BENEFITS Once the impacts of projects have been estimated, cost-benefit analysis requires that a monetary value be placed on the estimate of the change in each outcome resulting from the project. For a number of outcomes, market prices can be observed. For instance there are likely to be directly observed labor market returns to increases in labor productivity that arguably represent the expected gains that youth might expect to obtain from being more productive due to better education, health and nutrition. Market prices can usually be used to value several of the other outcomes that are listed in Table 3.2 above. Examples include: labor supply, employment, educational attainment, age at which a given schooling grade is completed, health care expenditure, increased tax revenue, and risk pooling services.48 But there also are a number of other outcomes for which market prices are not available. There is some controversy about how best to value such outcomes. One option, that we prefer, is to use the amount of real resources used by a given society to attain the same objective by the least-cost alternative means. This method was first used to our knowledge by Summers (1992, 1994) in a study of the costs and benefits of girls' schooling (see the Pakistan case study in Section 5.6). This approach has a certain attraction in terms of reflecting what resources society has actually demonstrated that it is willing to use to save a life. But it also has some limitations. One limitation is that it requires identifying an alternative investment that impacts only the outcome of interest. Unfortunately, many (if not most) interventions have multiple impacts, and it may be very difficult to find a suitable alternative to use for this purpose. A second limitation is that there may be some benefits for which even this approach will not work even crudely because there does not seem to be information available on the resource costs of alternative means of obtaining many important expected outcomes of youth projects (even if obtaining such information is at least conceptually possible). For example, there is currently very little if any information for developing countries on the cost-effectiveness of alternative interventions to avert such negative outcomes as crime, alcohol and drug abuse, physical or sexual abuse, or social exclusion (Knowles and Behrman 2003b). Other potentially important outcomes of youth projects may be inherently difficult even to measure (for example, self esteem or social solidarity). In such cases, it may be 48 See Knowles and Behrman (2003b) for details. However, as noted above, if youth expect that their increased skills will change the wage structure, then these expectations should be taken into account in principle, though we are not aware of such adjustments in practice. 38 possible to use willingness to pay or contingent valuation methodologies to attach a monetary value to these outcomes. Such methods have been widely used in the environmental field (Dixon and Pagiola 1998).49 Contingent valuation uses direct questioning of respondents to determine their willingness to pay for a carefully described good or service (even one that is not traded in markets). The actual valuation can be obtained in a number of ways, such as asking respondents to name an amount they would be willing to pay or asking them whether they would be willing to pay a specific amount of money (often randomly varied from one respondent to another) for the good or service. Contingent valuation can be used to attach a monetary value to almost any project outcome. However when benefits are estimated using this technique, it is best to use sensitivity analysis (Section 2.6) to investigate the effect of fairly substantial variations in the estimates (for example, 50-60%) on the results. Health outcomes (including but not limited to mortality) long have been recognized as difficult to "monetize." One approach that has been mostly rejected in recent years (Belli, et al. 1998) is to value health outcomes in terms of labor market outcomes such as the present discounted value of future labor earnings lost through death or illness. The main shortcoming of this method is that it considers individuals only as production machines. Moreover, most individuals consume a considerable share of what they earn, which raises the question, at least from the point of view of others, whether their consumption should be netted out from their labor earnings. Alternatively, the Commission on Macroeconomics and Health (2001) recommends valuing an additional disability- adjusted life year (DALY) at between 1-3 times annual earnings or annual income per capita.50 These estimates are all considerably higher than estimates based on the use of Summers' methodology, which might for example be based on the cost per DALY gained from immunization (typically under $20). There is no correct answer in this area, but current practice seems to be to value a DALY conservatively at about the level of annual earnings or income per capita. It may be tempting to include estimated savings in the form of reduced levels of government welfare payments (for example, unemployment benefits or assistance provided to young mothers) as a benefit of some youth projects. However, such government payments are transfers (i.e., they lead to a change in who consumes society's resources, not in the level of resources available to a society) and should not therefore be included as benefits of a project. Similarly, if youth projects lead to increased economic activity in some areas (for example, an increase in tourism as the result of a reduction in crime rates), these should not be included as benefits (as such activities also involve costs). On the other hand, if a project leads to a reduction in the resources society must devote to addressing a given problem (for example, the costs of treating HIV/AIDS patients or the cost of apprehending and incarcerating youthful criminals), the value of the resources saved should be considered a benefit. 49 Belli et al. (1998) provide an example of the use of contingent valuation to estimate the benefits from creating a new park in Madagascar. 50 This estimate is broadly consistent with recent estimates for the US that value one human life at $1-1.5 million (Ashenfelter and Greenstone 2004). 39 One last consideration about benefits is that it is often important to make alternative assumptions about how long the benefits are likely to last as well as about how they may possibly change over time. Such assumptions are often necessary because estimates of effectiveness may be based on too short a period to provide this type of information directly. The Kenya case study in Section 5.3 illustrates this problem with respect to the estimated impact of a program designed to prevent HIV and STI infections and adolescent pregnancies. The Argentina case study in Section 5.7 also illustrates this problem and how to address it through sensitivity analysis. That study assumes that the estimated short-term impact of a job-training program on earnings (the program's only assumed benefit) will remain constant during a time period that is alternatively assumed to be 1, 3, 6, 9, 12, 15 or an infinite number of years (an example of the use of sensitivity analysis).51 More generally, sensitivity analysis will often be appropriate to investigate the effect of alternative assumptions regarding the duration and pattern over time of a project's estimated benefits. SECTION 4. POVERTY AND GENDER ANALYSIS Cost-benefit analysis and cost-effectiveness analysis are designed to investigate whether project interventions are efficient. In principle, any gains to society that result from youth projects can be redistributed between "winners" and "losers" in such a way as to bring about whatever distributional outcomes policy makers desire. In practice, however, such re-distribution may be difficult to effect and is seldom done. It is therefore important to consider as well how a project's benefits and costs are distributed among vulnerable groups. This section of the Guide illustrates this approach for two important vulnerable groups: the poor and women. In some settings, it may be appropriate for an economic analysis to include similar analysis for other groups, such as ethnic minority groups, disadvantaged religious minorities or castes. 4.1 POVERTY ANALYSIS In addition to cost effectiveness and net present benefits, the economic analysis of youth projects should address the following questions: · Are there reasons to believe that costs, effectiveness or benefits of project interventions will vary between poor and non-poor?52 · To what extent are the poor likely to share in the benefits provided by the project? 51It is of course also possible that the effects of the training on earnings will either diminish or increase within each assumed time period. 52Although the discussion in this section focuses on differences between the poor and non-poor, however defined, there may also be important differences between those who are just below (and just above) the poverty line and those who are extremely poor or relatively well off. The degree of disaggregation that is most appropriate to use in analysis of this type will vary depending on the country context. 40 · What is the expected impact of the project on the incidence, depth and severity of poverty? 4.1.1 Variations in project costs, effectiveness and benefits between the poor and non-poor It is frequently (usually implicitly) assumed that project interventions affect the poor and non-poor equally. For example, the economic analysis of educational investments frequently assumes that the estimated benefits and costs are the same for the poor and the non-poor.53 However, such an assumption is inconsistent with the well-established finding that children of the poor do less well in school than those of the non-poor and with the conventional wisdom that the poor are more constrained by capital market imperfections than the non-poor. It is also reasonable to expect that poor youth, even with the same level of education, will earn less at least initially because they have less favorable access to labor markets (for example, they may need to migrate from rural to urban areas to take advantage of their education) or because of disadvantages in their initial job searches (for example, they are less likely to have family connections that facilitate entry into good jobs). 4.1.2 Distribution of project benefits between the poor and non-poor Some types of youth projects may affect the poor and non-poor differently. For example, projects that involve improving the quality of existing education or health services may primarily benefit those who are already utilizing these services. Such investments may yield high net present benefits because they do not increase the opportunity cost of the time involved in utilizing such services (The case study on basic education of the rural population in Pakistan in Section 5.6 illustrates this point). However, in most countries, the poor do not utilize existing education and health services as intensively as the non- poor, so the benefits of projects that focus on quality improvements will tend to be received mainly by the non-poor. Another example is projects that involve school-based health services, which mainly benefit youth who are attending school. However, there may be significant externalities for some types of school-based health services (as illustrated by the school-based de-worming intervention in the Kenya case study discussed in Section 5.1). Externalities would also certainly be expected in a school- based program to educate youth about HIV (as illustrated in the Tanzania case study in Section 5.3), while they might not be available in a school-based program providing iron supplements. 4.1.3 Poverty impact Economic analysis should estimate the impact of the project on the incidence, depth and severity of poverty. For this purpose, "poverty" should refer to a multidimensional concept that includes not only income poverty but also limited capabilities (for example, poor health status and low educational attainment), absence of security (including 53 One exception is Ravallion and Wodon (2000), which estimates that the benefit-cost ratio of the Food for Education program in Bangladesh is lower among the poor. Some of the studies reviewed in Section 5 are based on selected samples such as youth in school and therefore cannot address directly the impact of the program on distribution in general or on such poverty measures in particular. 41 heightened risk of income or asset losses, lack of personal safety, and high exposure to health risks) and powerlessness (lack of information, inability to participate or influence decision-making, lack of accountability, poor access to social capital). In general, a project's poverty impact will be closely related to the share of project benefits received by the poor. This is particularly the case when youth projects increase the capabilities of youth beneficiaries by improving their health and increasing their education or provide enhanced security by protecting them from HIV infection or other health risks. However, some youth projects may have a significant impact on poverty through other channels, for instance, by contributing significantly to a higher rate of economic growth. An example would be a youth project that reduces crime or ethnic tensions and that may therefore contribute to increased economic growth by encouraging tourism and foreign investment (World Bank 2003). Youth projects may also contribute to poverty reduction by addressing generational disadvantages of youth that would otherwise be likely to translate into higher future poverty rates. For example, in some South Eastern European countries, youth may not have the same opportunity to acquire assets, such as land and housing, that their parents had during more favorable economic times. The absence of asset ownership may prevent youth from obtaining loans to start businesses. In such an environment, a project that provided even currently non-poor youth with increased access to credit might contribute to poverty reduction over time. The economic analysis should include an analysis of the expected impact of a youth project on poverty and the channels through which the expected poverty impact is expected to occur.54 4.2 GENDER ANALYSIS The economic analysis of youth projects also needs to address a parallel set of questions with respect to gender: · Are there reasons to believe that costs, effectiveness or benefits of project interventions will vary between males and females? · To what extent are women and men likely to share in the benefits provided by the project? · Does it matter whether the project is implemented by women or men? · Does it make a difference whether the project benefits are provided through men or women (e.g., if they are channeled through the parents of the youth, as are the scholarships in the Mexican PROGRESA/Oportunidades program discussed in Section 5.4, does it matter that they are channeled through the mothers)? · What is the expected impact of the project on the status of women and men? 54In the Mexican PROGRESA/Oportunidades program discussed in Section 5.4 there has been extensive analysis of the impact of the program on the incidence, depth and severity of poverty (see Behrman and Skoufias 2004). 42 Gender equality may be an issue of development effectiveness, not only a matter of distributional fairness. The available evidence suggests strongly that when women and men are relatively equal, economies tend to grow more rapidly, the poor move more quickly out of poverty, and the well-being of men, women and children is enhanced (World Bank 2001c, 2002e).55 4.2.1 Gender variation in costs, effectiveness and benefits The costs, effectiveness and/or benefits of some investments in youth may vary importantly between males and females. For example, investments in girls' schooling may yield higher benefits than similar investments in boys' schooling for any of the following reasons: · Because girls' enrollment rates are initially lower, a given intervention may have a larger effect in raising girls' enrollment rates (this point is illustrated in the Pakistan case study in Section 5.656) · Because girls may have much lower enrollment rates, girls not currently enrolled in school may have higher levels of innate ability than boys not currently enrolled in school · Empirical studies suggest that the labor market rates of return to more schooling above the basic levels are if anything higher for girls than for boys (e.g., Behrman and Deolalikar 1995, King and Mason 2000, Schultz 1988, 1993) · Increases in girls' schooling may provide additional health and nutrition benefits, both for themselves and their children, in addition to labor market benefits (again, see Section 5.6)57 There may also be differences in the cost of interventions by gender. For example, providing women with access to some types of health services may require home visits if there are social constraints on women's mobility. An interesting feature of many investments in youth that is emphasized in the second bullet in Section 1.2, although shared with investments in some other groups (for example, investments in early childhood development), is the potentially broad range of affected outcomes. For example, investments in youth centers may affect school dropout rates, youth unemployment, teen pregnancies, HIV/STD infection, reduced unemployment, and crime, alcohol and drug abuse and tobacco use and reduced suicide rates. Many of these effects are likely to differ significantly between male and female youth (Knowles and Behrman 2003b). Similarly, investments in education may lead to a 55 However, the jump from associations to causality is not always persuasively established and, indeed, gender equality might in part be the result rather than the cause of better economic performance in some settings. 56 On the other hand the Mexican PROGRESA/Oportunidades case that is discussed in Section 5.4 illustrates that gender gaps in enrollments may not be a good indicator of gender gaps in schooling attainment, which presumably is of greater interest ­ see the discussion of targeting in Section 4.3 below. 57 For this point and for the point in the previous bullet, however, there is the question of whether there is any reason for policies to direct more resources into female schooling because the private returns are higher there. 43 similarly wide range of effects, including enhanced labor productivity, reduced probability of unemployment, improved health (and improved health of one's children), reduced fertility, reduced likelihood of HIV/STD infection, and reduced risk of tobacco use, alcohol and drug abuse and involvement in crime. Again, many of these effects of increased education are likely to vary significantly between males and females. 4.2.2 Distribution of project benefits between males and females The distribution of benefits between males and females differs importantly with many investments in youth. In some cases, this is because the problem targeted varies by gender. Examples include crime, alcohol and drug abuse (and in many countries, tobacco use), most forms of violence, suicide and accidents--all of which tend to be more common among males. On the other hand, girls are usually more vulnerable to the effects of unprotected sex and are more likely to be the victims of sexual violence. Additionally, there may be gender differences in access to the benefits of some types of investments. For example, in countries where female school enrollment is much lower than male school enrollment, school-based interventions are unlikely to benefit girls to the same extent as boys. Similarly, military training is unlikely to benefit women as much as men. On the other hand, enforcement of compulsory school enrollment in most of Asia and Africa where, in contrast to Latin America, girls average less schooling than boys and minimum age at marriage laws are more likely to benefit females.58 Gender differences in occupation, including occupational gender segregation, may affect the distribution of the benefits of investments in some types of education and vocational and technical training between men and women. 4.2.3 Gender considerations in project implementation For some types of interventions, it may matter whether the project is implemented by men or women (i.e, the clinic workers, teachers and other implementers of the project) and whether project-provided cash transfers, if any, are given to men or women in the household. As an example of the former, many perceive that females are likely to respond better to programs if the implementers are themselves women. For instance, having more female school teachers is thought to increase female school attendance and having more female health workers is thought to increase use of health services by females, all else equal. As an example of the latter, in the case of conditional cash grants such as in the Mexican PROGRESA/Oportunidades program, outcomes may vary depending on whether the cash is distributed to men or women. 4.2.4 Gender impact Although a shift in some investments toward girls and women may be required to attain gender equality in some countries, some other critically important gender differences may require greater investments in men (for example, problems of predominantly male suicide, alcoholism and crime in some transitional economies). Because gender issues are 58 This statement is made under the wide-spread assumption that somewhat more schooling and somewhat later marriage (at least for those who marry when young) would benefit youth but not their parents and that the parents have important roles in these decisions so the enforcement of policy restrictions, while being welfare-reducing for the parents, are welfare-enhancing for the youth. 44 cross-cutting, investments in sectors other than education and health can have significant gender impacts. In the case of youth, for example, more effective enforcement of child labor, compulsory school attendance and minimum age at marriage laws could have a significant gender impact in many countries. 4.3 TARGETING INVESTMENTS IN YOUTH Economic analysis should consider the cost effectiveness of alternative methods available for targeting investments to poor and other vulnerable youth. Perfect effectiveness in targeting would require that all of the groups targeted receive benefits, while no benefits are received by non-targeted groups. Clearly, perfect targeting is not a practical goal. Instead, targeting effectiveness generally involves finding the best feasible balance between minimizing the risk of benefits being leaked to non-targeted groups and minimizing the risk of not covering all of the targeted groups. Accordingly, targeting effectiveness is usually measured by two indicators: the proportion of benefits reaching the targeted group, and the proportion of the targeted group who receive benefits (Gwatkin 2000). However, effectiveness is an insufficient criterion by which to judge a given targeting method. Some targeting methods may be more effective than others, but their much higher cost renders them less attractive in a given context. In evaluating alternative targeting methods, it is also important therefore to consider an additional tradeoff between cost and effectiveness. In addition, it is important to recognize that targeting may affect the degree of political support for a program, and hence its sustainability (see Section 2.5) because political support for a given governmental program or policy often depends on who receives its benefits (Gelbach and Pritchett 1997). Targeting methods may include: · Self targeting · Geographical targeting · Characteristic targeting · Means testing There are many opportunities for self-targeting investments in youth. One of the most important areas of opportunity occurs with respect to adult literacy, basic education and continuing education investments. These are self-targeting because it is generally the poor (and often women) who would fail to obtain schooling in the absence of such programs. Projects that provide education services of these types are almost guaranteed to affect predominantly the poor, and especially females (except in Latin America and a few countries elsewhere in which female schooling exceeds male schooling). 45 Geographical targeting can be a cost-effective method for targeting some youth interventions.59 The Colombia and Mexico case studies in Sections 5.2 and 5.4 provide examples of geographical targeting (i.e., scholarships targeted to youth residing in predominantly poor neighborhoods of large urban areas for both countries and in poor rural communities in Mexico). However, it is sometimes the case that most poor youth do not reside in predominantly poor areas and are therefore difficult to reach through geographical targeting. Characteristic targeting is also widely used in youth projects. Examples are interventions targeted to girls, unemployed youth, school dropouts, "at-risk youth," and ethnic minority youth (for example, Roma youth in several European countries). The Argentina case study presented in Section 5.7 provides examples of characteristic targeting. Characteristic targeting can also be cost-effective. However, there are two caveats. The first is that some characteristics that are presumed to be related to poverty may not in fact be closely related to poverty in some settings. For example, youth unemployment may be concentrated among the non-poor (poor youth may not be able to afford unemployment and may instead be working at low-paying jobs in the informal sector). As another example, in Mexico's PROGRESA/Oportunidades program (discussed in Section 5.4), higher scholarships are provided to girls because they had lower pre- program enrollment rates. However, it was subsequently learned that the lower enrolment rates of girls were due to their lower failure and repetition rates and that indeed girls averaged higher schooling attainment prior to the program despite lower enrollment rates. The second caveat is that projects need to consider the possible impact that characteristic targeting may have in encouraging the targeted behavior. For example, targeting training or other forms of assistance to school dropouts might encourage some youth to drop out of school. However, this caveat is irrelevant for some types of characteristic targeting, for example, targeting on the basis of sex or age or other characteristics that cannot be changed by behaviors. Means testing is usually a relatively expensive method for targeting investments to youth. Cost-effective means testing may be very difficult in poor rural areas where most families are poor or near poor and where most people are self-employed. Relatively effective means testing is often done in such settings by relying on village-level committees to develop a list of the poor, in some cases involving visits to households to collect information on their housing conditions, asset ownership and other poverty-related characteristics. In some countries, youth investments can be targeted by piggy-backing onto existing systems for identifying the poor, such as those that may exist to determine eligibility for free or subsidized social services or other forms of social protection, such as social funds. Many transitional economies, for example, have extensive social protection programs. In some of these countries, however, social welfare programs may be ineffectively targeted to the poor and other vulnerable groups. 59The increased availability of "poverty maps" in many developing countries should facilitate geographical targeting. 46 SECTION 5. CASE STUDIES This section of the Guide reviews a set of case studies of the economic analysis of youth projects in order to illustrate some of critical points made above, ranging from the nature of the data to concerns about poverty and gender. These case studies are not offered as models of the type of ex ante economic evaluation that is likely to be possible in real world settings. In fact, most of the case studies involve ex post evaluations of actual projects. In addition to illustrating many of the points made above, as previously noted, the main purpose of most of these case studies is to provide examples of careful economic analysis of the type that may yield useful information to those preparing ex ante economic analyses (for example, estimated parameters that may be used to obtain at least rough estimates of the impact of similar interventions in other settings). Another purpose is to illustrate how careful ex post evaluation components should be designed. That said, one of the case studies (Section 5.5) does provide an example of the kind of ex ante economic analysis that is likely to be feasible for the typical youth project.60 This case study describes analysis done for a hypothetical continuing education project in Serbia-Montenegro, the terms of reference for which are presented as a model in Section 6. 5.1 CASE STUDY #1: ANALYSIS OF DE-WORMING INTERVENTION IN KENYAN RURAL PRIMARY SCHOOLS This case study is based on a study in Kenya that was supported in part by the World Bank in order to identify cost-effective interventions to improve schooling outcomes at the primary level (Miguel and Kremer 2004). This case study illustrates several of the key points made in this Guide, including: · That it is important to consider possible outcomes from investments in youth that may occur in sectors different from the sector in which the investment is made (in this case, the effects on schooling of a health intervention). · That there are important market failures in many investments in youth that justify public involvement (in this case, evidence of significant externalities from the de-worming intervention) · That the estimated impacts based on experimental data with random assignment of treatment may differ importantly from previous research based on behavioral data · That even what seem to be well-designed experiments can not be very informative about some important issues (e.g., distributional effects, impacts on those not in school, whether a project affects school enrollment) if they start with selected samples such as those in school rather than population- based samples61 60In addition, the Pakistan case study in Section 5.6 is also an example of the kind of ex ante economic analysis that should be possible in many countries. 61This study considers children aged 6-18 in 75 primary schools in southern Busia in Kenya. Far from all children in this age range are enrolled in primary school in rural Kenya, so using a primary-school-based 47 · That the costs of a pilot may be inappropriate as a basis for estimating the cost of a scaled-up intervention · That there be advantages in using multiple indicators to measure each expected outcome This case study also illustrates the use of both cost-effectiveness and cost-benefit analyses. 5.1.1 Problem and identification of alternatives The schooling interventions that were investigated in the larger study of which this case study is a part included textbook provision, grants to school committees, teacher training, performance-based incentives for teachers (based on student test scores and dropout rates), and mass de-worming.62 Mass de-worming was included among the study's interventions because of evidence from several countries that suggested it had a significant impact on schooling and nutrition outcomes (Knowles and Behrman 2003b).63 The study was conducted as a large randomized experiment in 75 primary schools (covering grades 1-8) with total enrollment of over 30,000 pupils between 6-18 years of age in one relatively poor district of Western Kenya (Busia) that borders on Lake Victoria.64 Apart from the de-worming intervention that is the focus of the case study, the study found that most of the schooling interventions in the study had no or very little impact on schooling outcomes even though there were strong advocates of these other interventions and positive associations between these interventions and school success in cross-sectional data in which adoption was selected rather than randomly assigned.65 5.1.2 Measurement of effectiveness The study used a randomized experimental design to generate the data used to estimate the effectiveness of its interventions. Interventions were in most cases phased in to randomly selected sets of schools over time, with the schools that had not yet received the interventions used as a comparison group. In the case of the de-worming intervention, sample precludes generalization to the broader population in the sample area. Moreover the study provides little information about the sample area compared with some larger area and whether the estimates would be expected to hold only for the sample area or for some larger population such as throughout rural Kenya. 62 Mass de-worming refers to an intervention in which de-worming drugs are provided to everyone in a target group without first conducting diagnostic studies (stool exams) to determine whether those treated are individually infected. However, mass de-worming is preceded by stool exams on a random sample of the target population to determine whether infection levels meet minimum WHO-recommended levels for mass treatment (i.e., 50% prevalence for geohelminths and 30% for schistosomiasis). 63 However, during the course of the study, a careful review (Dickson, et al. 2000) of the existing studies on the impact of de-worming on schooling was published that concluded that there was no evidence that de-worming was a cost-effective investment to improve schooling. 64 Unlike the geohelminthic infections that were also treated in the study (i.e., roundworms, hookworms and whipworms), schistosomiasis is transmitted by bathing or swimming in infected waters. Within the study area, schistosomiasis prevalence was markedly higher in areas bordering on Lake Victoria. 65 The exception was one package of interventions that included free school uniforms, textbooks and new classrooms. The authors concluded that the free uniforms probably had the largest impact since textbooks had been shown in earlier experiments to have little impact, while classroom construction was completed well after impact had been registered. 48 the 75 schools were randomly assigned to one of three groups.66 Most students in the first group of 25 schools began receiving the treatment during 1998, while students in the second group of 25 schools began receiving the treatment during 1999. During 1998, therefore, group one schools were the treatment group while groups two and three served as the comparison group, whereas during 1999, groups two and three were the treatment group while group three was the comparison group. The fact that randomization was done at the level of schools turned out to be an important feature of the study.67 Most previous randomized studies of the impact of de-worming were conducted in a smaller group of schools with randomization done at the level of individual students within the study schools.68 In these studies, the treated students were the treatment group, while the un-treated students were the comparison group. The authors argue that this design biases downward the measured effectiveness of de- worming for two reasons. Firstly, it neglects the impact that the treated students have on the re-infection rates of the untreated students, biasing the observed differences between treated and un-treated students toward zero.69 Secondly, it neglects the effects of reduced infection rates in the treatment schools on rates of re-infection in nearby schools (as well as in the surrounding communities).70 The study estimates that both "externalities" are substantial in magnitude. 5.1.3. Measurement of costs The project collected data on the cost of each intervention studied. However, because the project was a research study, many of the costs incurred were not considered necessary in scaling-up an intervention. For example, in the case of the de-worming intervention, the authors argue that the $0.49 cost per student treated in a much larger Partnership for Child Development (PCD) mass de-worming intervention in neighboring Tanzania 66 Prior to randomization, the schools were stratified according to the interventions that they had previously received through the study. 67 Note that there is an important distinction between the level at which the randomization of the project experiment occurs and the level at which the random sample of study was drawn to evaluate the experiment. If the project had collected data from a random sample at the population level rather than using the selected school-based data, richer inferences could have been made from this experiment. 68 Most previous studies were also conducted in only a few schools or villages and collected information only on cognitive performance (for example, using tests of recall ability), neglecting other potentially important outcomes, such as school attendance, academic test scores, and grade promotion that are included in this study. This study, as do the other case studies reviewed in this section, assumes that such observed fairly short-run effects are, in the terminology of Section 3.2.3.1 above, "sufficient statistics" for all longer-run impacts of interest. However it should be noted that because the sample in this case is school-based rather than population based as, for example, in the Mexican PROGESA/Oportunidades case in Section 5.4, it is not possible to use this sample to examine some possibly important outcomes such as changes in overall school enrollment. 69 Most worms have relatively short lives, so that continual re-infection must occur to maintain infection rates at previous levels. 70 The magnitude of such cross-school externalities may be larger in this setting due to the practice, reported in the study, of neighbors and even siblings attending different schools. Externalities from schistosomiasis treatment, as compared to geohelminth treatment, extend over a wider area because the disease is water borne. In fact, the study found that geohelminth treatment externalities occurred mostly within schools, whereas schistosomiasis treatment externalities occurred mainly across schools. 49 provides a much better estimate of the actual cost of a scaled-up school-based mass de- worming intervention than the actual cost per student in their study ($1.46 per pupil treated, after excluding costs that were obviously related to the research study).71 Because the interventions were expected to increase school participation (i.e., enrollment and attendance), the study also estimated the cost of hiring additional teachers to maintain class size at 30 students. Unfortunately, the study does not report much information on the composition of costs (although it does indicate that nearly one-half of the study's actual cost of $1.46 per pupil treated was for drug purchases). However, it appears that the study's cost estimates do not include any estimate of the cost of collecting the additional tax revenue that would be necessary to support a scaled-up intervention. The study does, however, correctly distinguish between transfers and costs. For example, it recognizes that the cost of providing free school uniforms to children in one of the interventions examined in the larger project might be regarded primarily as a transfer (since it largely substitutes for existing household expenditure on school uniforms), with intervention costs limited to the cost of procuring (transactions costs only) and distributing the uniforms (and the unmeasured cost of collecting the necessary additional tax revenue). In the case of the de-worming intervention, none of the cost was considered a transfer.72 5.1.4 Results and conclusions The study found that de-worming had a significant negative effect both on disease prevalence (i.e., the percentage of moderate-to-heavy infections) and a significant positive effect on enrollment and school attendance rates within the sample of children who were enrolled in school prior to the intervention.73 Nearly one moderate-to-heavy infection was eliminated per treated child, with about one-third of the effect attributable to cross-school externalities. De-worming increased school attendance by 0.14 years of schooling per pupil treated, a 7% gain in school attendance with about one-fifth of the gain attributed to cross-school externalities.74 However, the study did not find a significant effect of de-worming on either cognitive achievement or anemia prevalence.75 71 Although the cost estimates in the PCD programs included administrative and delivery costs and the cost of volunteer labor, it did not include capital costs or an estimate of the distortionary cost of raising the governmental tax revenue needed to finance such a program (Partnership for Child Development 1999). 72 This assessment appears reasonable. The authors argued that the de-worming did not displace a significant level of previous household expenditure on de-worming drugs nor did it displace a significant level of private de-worming services, as evidenced by the fact that only 5% of comparison school pupils reported receiving any medical treatment for worms during the previous year. 73 The de-worming intervention was combined with health education (for example, encouraging children to wear shoes to avoid hookworm infection). However, the study found that the health education had no measurable impact on self-reported behavior. 74 Particularly large gains in school attendance were experienced among younger children, primarily children too young to be considered "youth." School attendance was measured by NGO field workers during unannounced visits to schools. 75 In the case of cognitive achievement, the authors suggest that the increases in school attendance may have been too small to produce statistically significant increases in cognitive achievement. In the case of anemia, they suggest that the widespread practice of geophagy (soil eating) by children in the study area may supply needed iron even in the presence of substantial worm burdens (anemia prevalence was low in the study area even prior to the intervention). 50 In terms of health impact, the study estimated that the intervention resulted in a gain of 649 DALYs, or about $5 per DALY gained using the PCD program cost of $0.49 per treated child.76 Importantly, the measured externality benefits (both within schools and across schools) accounted for 76% of the DALYs gained. As a health intervention, the study concludes the mass de-worming is cost-effective (at least in this particular setting). However, it notes that most of the DALY gain is from schistosomiasis treatment.77 With treatment limited to geohelminths, the cost per DALY gained increases to $280, suggesting that treatment of geohelminths is cost-effective as a health intervention only when combined with treatment for schistosomiasis (or perhaps if combined with another school health intervention that could share in the cost of delivering the albendazole). Among the various interventions studied, the de-worming intervention was by far the most cost-effective in improving school attendance and subsequent enrollment among those children enrolled in school prior to the experiment.78 The cost per additional year of school attendance (based on the $0.49 cost per pupil treated in the Tanzanian PCD intervention) was estimated to be only $3.50 for de-worming, compared to $99 per additional year of attendance with the next most cost-effective intervention (a combined package of free school uniforms, textbooks and new classrooms). Even if the cost of the free school uniforms in this alternative intervention is considered a transfer, implying that uniform-related costs are limited to the cost of purchasing and distributing the uniforms and the cost of raising the necessary tax revenue, the study concludes that de-worming remains the most cost-effective intervention among those studied to increase school attendance. The study also includes a cost-benefit analysis of the de-worming intervention based on its estimated effectiveness in raising school attendance among those children enrolled in school.79 The study assumes that the rate of return to an additional year of school attendance is 7%.80 The study also assumes that 1999 average annual earnings of about 76 The study based its estimates of the number of DALYs gained on Global Burden of Disease estimates of the burden of disease per infected person from each type of infection. The estimates of DALYs lost per infected individual were as follows: schistosomiasis (0.0097), roundworms (0.0004), hookworms (0.0013), and whipworms (0.0005). The Burden of Disease estimates are supposed to reflect mortality, morbidity and disability. However, the reliability of the estimates varies among diseases. The study found that self- reported morbidity rates were significantly lower after treatment among pupils in treated schools (and these simple differences between treatment and control groups mask possible cross-school externalities). 77 Treatment of schistosomiasis is done annually, using praziquantel, whereas treatment of the geohelminths is done at six-monthly intervals, using albendazole. 78 As noted, the sample design precludes investigating the possibly important effect on school enrollment in the general population of school-age children. In some other studies (e.g., the Mexican PROGRESA/Oportunidades study in Section 5.4) enrollment effects are found to be much more important than attendance effects, but that result probably does not carry over to this Kenyan study because of the differences in the interventions. Nevertheless there may have been important enrollment effects so it would have been desirable had the study design permitted investigating such possibilities. 79 The study does not include the intervention's health effects in the cost-benefit analysis, presumably due to the difficulty in attaching a monetary value to the intervention's health effects. 80 This estimate is based on a study of the rate of return to secondary schooling in Kenya that attributes about 40% of the estimated return of 17% to years of schooling, with the balance attributed to cognitive 51 $370 per worker remain constant, that the opportunity cost of the additional time children spend in school is equal to one-half of this average wage, and that children work an average of 40 years after leaving school. Under these assumptions, and using a discount rate of 5%, the study estimates that discounted benefits are over $30 per treated child, implying that the benefit-cost ratio is about 61 (based on the PCD cost of $0.49 per treated pupil). This initial estimate does not consider the possible negative congestion externalities that might occur in schools as the result of increased class size. However, the study calculates that the estimated benefits are large enough to more than offset the cost of hiring additional teachers to maintain class size at 30 pupils per teacher (i.e., the cost of hiring 0.0047 additional teachers for each pupil treated). This cost is estimated to be $9.06 per pupil treated and decreases the benefit-cost ratio from 61 to a little over 3 (i.e., 30 / 9.55 = 3.14). Although not included in the study, it is interesting to consider the effect of including the intervention's estimated health effects in the cost-benefit analysis. For example, if one assumes that each DALY gained has a value equal to the average annual wage of $370 (following the typical practice, as discussed in Section 3.3 above), the estimated benefits increase by about $36 per treated pupil (i.e., [0.49 / 5]*370 = 36.26) and the benefit-cost ratio increases from a little over three (including the cost of hiring additional teachers) to almost 7 (i.e., [30 + 36] / 9.55 = 6.91).81 One of the study's most important findings is that the estimated schooling benefits due to externalities (both within schools and across schools) are large enough by themselves in relation to the intervention's costs to justify a 100% subsidy (and possibly even the payment of an incentive) for the mass de-worming of primary school students in this setting.82 Limited to externalities, the intervention's benefits are estimated to be $15.90 per pupil treated, while the cost is $9.55 per treated child (including the cost of hiring additional teachers). Even if a 10% discount rate is used (instead of 5%), the study reports that the discounted benefits are approximately as large as the intervention's cost. Moreover, the externalities estimated in the study are undoubtedly underestimates because the study design precludes measuring the effects of the intervention on infection rates in children out of school, on adults, or on the population outside the study area.83 performance on tests of literacy, numeracy and reasoning. These estimates may not control for selected school attendance, with greater attainment for those youth with greater ability and motivation and with better family background, and they may not control for endogenous determination of schooling attainment. 81 Over a longer period, continual de-worming might also affect children's height, which has been shown to be associated with increased learning capacity and higher adult earnings in a variety of settings. 82 The study's findings imply that the marginal social benefits of the mass de-worming intervention exceed its marginal social costs even at a price of zero. 83 Consistent with the hypothesis of even broader external effects, the study found that previously untreated pre-school children entering treatment group schools in 1999 had significantly lower infection rates than pre-school children entering comparison group schools. 52 5.2 CASE STUDY #2: SCHOLARSHIP PROGRAM FOR POOR SECONDARY SCHOOL STUDENTS IN URBAN COLOMBIA This case study is based on an evaluation of a targeted secondary school voucher scheme in Colombia that was designed to make it possible for children of the poor to attend private secondary schools in selected urban areas in which public schools were already enrolled to full capacity. This case study illustrates several of the key points made in this Guide, including: · The distinction between income transfers and economic costs · The possibility of obtaining reliable estimates of effectiveness even in the absence of a formal experimental design (in this case, by taking advantage of a "natural policy experiment" to generate appropriate data)84 · That use of instrumental variables based on a random event (in this case, whether a student was or was not a lottery winner to receive a voucher) may improve the estimate of effectiveness · The role of sensitivity analysis in assessing the effects of varying key assumptions · Use of a life-cycle approach in estimating program benefits and costs 5.2.1 Problem and identification of alternatives The Government of Colombia created the Programa de Ampliación de Cobertura de la Educación Secundaria (PACES) voucher program for poor secondary school-age children in 1991 to enable them to attend private secondary schools. By 1996, this partly World Bank-funded program was covering about 100,000 students. At the time the program was launched, enrollment in primary school (covering grades 1-5) was almost universal, while the enrollment rate in secondary school (i.e., covering grades 6-11) was only 73% overall and 55% in the poorest quintile of students. The vouchers were designed to cover about one-half the cost of private secondary schools, and they could be renewed as long as students maintained satisfactory academic performance. The vouchers were targeted to children residing in poor residential areas and who had attended public primary schools. 5.2.2 Measurement of effectiveness The PACES scholarship program (and scholarship programs generally) might be expected to have the following effects: 84Indeed this is an example of a larger reality that has the potential to permit much better project evaluation than currently exists. That is, governments often do not have finances to fund fully programs, particularly new programs in their initial stages. Therefore the programs must be offered to some subset of the eligible population. Choosing who receives the program randomly has the advantage of fairness and of permitting better evaluation of the program impact by collecting information on those randomly assigned to the program and on others who do not (at least initially) have the program option. This is the strategy that the Colombian government followed in this case, and also the strategy that the Mexican government followed in the case considered in Section 5.4 (with random assignment of control versus treatment communities with the former enrolled in the program after a lag). 53 · Increased schooling attainment (however, it is important to consider the possible negative effects of such a program via school crowding) · Earlier completion of given schooling levels by scholarship recipients · Improved cognitive achievement for a given level of schooling attainment (assuming that the private schools attended by some voucher recipients provided better-quality schooling than that provided by the public schools that would have been attended in the absence of the vouchers) · Reduced work effort by other family members (any reduction in work effort by the scholarship recipient is considered to be part of the opportunity cost of schooling) Evaluation of the PACES program was facilitated by the fact that vouchers were initially awarded by lottery in municipalities in which the number of students applying for vouchers exceeded the estimated shortfall in the number of spaces available in public schools that was used by local authorities to decide how many new vouchers to make available in a given school year. This in effect created a "natural experiment" that made it possible to estimate the effect of the vouchers on the target population much as one could do if the voucher scheme had been conducted as a randomized experiment.85 A recent evaluation of the PACES program, partly funded by the World Bank, made effective use of this feature of the program (Angrist, et al. 2003). In fact, the evaluation found that only about 90% of lottery winners actually used their vouchers, while about 24% of lottery users received scholarships from other sources.86 The PACES program evaluation did not find any significant impact on enrollment. However, it did find that lottery winners were 15 percent more likely to attend a private school (Angrist, et al. 2003).87 After three years, lottery winners also had completed 0.12- 0.16 more grades of schooling (primarily due to lower repetition rates) and were about 10 percentage points more likely to have completed the 8th grade at the end of the program's third year. Although the program had no effect on dropout rates, lottery winners scored 0.2 standard deviations higher on standardized tests. In addition, lottery winners reported 85The implicit assumption is that the excess demand for vouchers occurred randomly across municipalities (and therefore was not associated with other municipality-level factors that might affect education). If, for example, excess demand for vouchers more likely occurred where recent expectations were higher regarding the returns to schooling because of greater economic activity than elsewhere, the estimated impacts may not be applicable to other municipalities with lower expectations regarding the returns to schooling. Unfortunately, the study does not provide information with which to assess whether the "excess-demand" municipalities were similar to or different from other municipalities. 86To correct for the effects of such "contamination," the evaluation also used lottery success as an instrument to predict whether a given youth received a scholarship, along the lines suggested as a possibility in Section 3.2.3.2 above. The instrumental variable estimates of the grade completion effect were about 50% greater than the "reduced-form" OLS estimates of being a lottery winner (Angrist, et al. 2003). The estimates of effectiveness used in the published study to obtain the cost-benefit estimates are the reduced-form estimates. In this case, the presumably more accurate instrumental variable estimates should have been used. 87Because of the reported poor quality in some of the new secondary schools that emerged to serve voucher recipients, eligibility to participate in the program was limited to not-for-profit schools from 1996 on (Angrist, et al. 2003). 54 working 1.2 hours per week less than non-winners and were less likely to be either married or cohabiting as teenagers (however, this last difference affected only about one percent of the sample). 5.2.3 Measurement of costs The economic cost of the PACES program includes the following components: · Additional schooling costs (some of which may be incurred directly by households, including the opportunity cost of the additional time scholarship students are engaged in schooling-related activities) · The cost of administering the program · Distortionary costs related to the financing of the program (e.g., the deadweight cost of collecting the necessary taxes to finance the program) or to reduced work effort on the part of other household members (due to a possible income effect) The costs of scholarship programs do not include the cost of the scholarships themselves (i.e., the cost of the voucher, in the PACES program), which is a transfer. Table 5.1 summarizes the information on the costs of the PACES program as reported in the evaluation study. The table indicates that the reported social costs of the program (the last column in the table) are at least US$195 per lottery winner for three years (after adjusting for different rates of voucher take-up in each year of the program). However, several cost items are not reported, including the deadweight cost of financing the additional governmental expenditure of $109 per lottery winner, the possible distortionary cost on adult work effort in lottery-winning households, and the program's administrative costs. Table 5: Three-year costs per lottery winner in the Colombia PACES program (US$) Lottery All Item Government winners households Society Cost of the vouchers -336 336 0 Deadweight cost of government financing ? ? Effect of voucher on adult work effort ? ? Administrative costs ? ? Reduced public expenditure on secondary schooling 227 227 Additional amount spent on schooling by lottery winners -236 -236 Opportunity cost of student timea -186 -186 Totals -109 -86 0 -195 Source: Angrist, et al. (2003). aestimated on the basis of reported reductions in time worked by lottery winners as compared to non- winners. 55 5.2.4 Results and conclusions The PACES program evaluation study estimates that the additional 0.12-0.16 grades of schooling completed by lottery winners would raise their annual incomes by about $36- 48 per year (based on an estimated rate of return to schooling of 10 percent in Colombia and predicted average annual earnings of $3,000). The evaluation does not include a formal cost-benefit analysis, concluding that the benefits would clearly exceed the costs using any plausible discount rate. For purposes of this case study, however, it is useful to illustrate the necessary calculations. A life-cycle approach is used for this purpose, under the following assumptions: · The three-year costs of the program include the estimated project costs of $195 per lottery winner from Table 5.1 but also administrative and distortionary costs of $32.65 per lottery winner (i.e., 30% of the increase in government expenditure from Table 5.1)88 and that all program costs occur at ages 13-15 in equal annual installments · Following findings and assumptions used in the study, it is assumed that lottery winners complete 0.12 additional grades of schooling, with each additional year of schooling resulting in a 10% increase in annual earnings of $3,000 per year · Annual benefits occur at ages 16 to 60 · The discount rate is alternatively 3, 5 or 10% per annum, and benefits and costs are discounted to age 13 Under these assumptions, the benefit-cost ratio is 3.8 with a discount rate of 3%, 2.7 with a discount rate of 5%, and 1.4 with a discount rate of 10%. The discount rate has a strong effect therefore on the estimated benefit-cost ratio because the benefits are spread out over an extended period of time. Table 5.2 presents the estimated benefits and costs at selected ages for illustrative purposes using a discount rate of 5%. The estimated benefits from the increased number of school years completed are presented in column 1, while the program's cost is presented in column 4 (the other columns are discussed below). The calculations show that the benefits are heavily discounted (comparing the last two rows of the table). Table 6: Calculations of discounted benefits and costs (US$) per lottery winner for the Colombia PACES voucher scheme Benefits Increased number of school grades Earlier completion of Age completed schooling Improved test scores Cost 13 0.00 0.00 0.00 75.88 14 0.00 0.00 0.00 75.88 15 0.00 303.60 0.00 75.88 88 The 30% estimate for administrative and distortionary costs is probably conservative, based on the discussion in Section 3.1.3 where it is recommended that an estimate of 20-25% be used as an estimate of the cost of raising additional tax revenue. 56 Benefits Increased number of school grades Earlier completion of Age completed schooling Improved test scores Cost 16 36.00 0.00 360.00 0.00 17 36.00 0.00 360.00 0.00 18 36.00 0.00 360.00 0.00 19-59 36.00 0.00 360.00 0.00 60 36.00 0.00 360.00 0.00 61 0.00 0.00 0.00 0.00 62 0.00 0.00 0.00 0.00 63-69 0.00 0.00 0.00 0.00 70 0.00 0.00 0.00 0.00 Totals 1,620.00 303.60 16,200.00 227.65 Discounted to age 13a 580.38 275.37 5,803.78 216.98 Source: see text. ausing a discount rate of 5% These estimates of the benefit-cost ratio are quite conservative, however, not only because they assume the lower limit to the estimated gain in the number of school grades completed (0.12, as compared to the upper limit of 0.16)89 but also because they neglect the other estimated effects of the intervention, i.e., the finding that lottery winners were 10% more likely to have completed the 8th grade at the time of the evaluation and that they scored 0.2 standard deviations higher on standardized tests.90 Considering first the effect of the voucher scheme on the age of completing the 8th grade, it is assumed that the estimated effect implies that lottery winners gain an additional 0.1 year of full-time work at age 15. This additional benefit is displayed in row 3, column 2 of Table 5-2. Like costs, this benefit is only slightly discounted because it occurs at age 15. The effect of adding this benefit is to increase the estimated benefit-cost ratio from 2.7 to 3.9 (using a discount rate of 5%). This is a significant increase, indicating that in the context of economic analysis earlier completion of a given grade or level of schooling is an important schooling outcome.91 89 If the estimated impact of the voucher scheme on the number of school years completed is assumed to be 0.16, instead of 0.12, the estimated benefit-cost ratio increases from 2.68 to 3.57 with a discount rate of 5%. 90 There may be other biases as well that are not explored here. On one hand, the estimate of a 10% rate of return to schooling attainment may be upward biased due to the failure to control for unobserved endowments (e.g., innate ability, family connections, motivations) in the earnings function estimates. On the other hand, there may be non-labor-market returns that are additional to labor market returns as discussed in the case study on female education in Pakistan in Section 5.6. 91 Moreover, its relative importance increases with the discount rate. For example, with a 5% discount rate, the earlier completion benefit accounts for 32% of total estimated benefits in this example, whereas with a 10% discount rate it accounts for 46% of total estimated benefits. 57 The PACES program evaluation study estimates that the increase of 0.2 standard deviations in standardized test scores among lottery winners is equivalent to about one full additional grade of schooling completed (based on the mean test scores by grade of United States Hispanic students taking the same test). If correct, this translates into an additional annual earnings benefit of $300, compared to the previously included annual earnings benefit of $36 based on the number of additional grades completed. This additional benefit is displayed in column 3 of Table 5.2. Like the benefit associated with the increased number of school grades completed, this benefit is heavily discounted because it occurs during the person's entire assumed working life (ages 16-60). Not surprisingly, including this benefit increases the estimated benefit-cost ratio dramatically, from 3.9 (including the earlier completion benefit and using a discount rate of 5%) to 30.7. 5.3 CASE STUDY #3: HIV, STI AND PREGNANCY PREVENTION IN TANZANIA In Africa, more than 50% of new HIV infections occur in youth.92 Even adolescents 15- 19 are at high risk of HIV infection, STIs and unwanted pregnancies. Sexual health knowledge and skills are limited, and adolescents have poor access to youth-friendly reproductive health services. Since a high proportion of adolescents attend school in many high-prevalence areas, school-based interventions are potentially cost effective. This case study describes an experimental study to test the effectiveness of a comprehensive school- and community-based intervention to prevent HIV and STI infection and teen pregnancies in Tanzania.93 This study was conducted in the Mwanza region of Tanzania, a region with high HIV prevalence bordering on Lake Victoria (adjacent to the region in Kenya in which the case study in Section 5.1 was conducted). Twenty rural communities, each consisting of 5-6 villages with an average of 6 primary schools and two health facilities, were included in the study This case study illustrates the following key points made in this Guide: · That some youth projects may require considerable time before their effects can be correctly measured · That in the meantime, recorded changes in intermediate outcomes, such as knowledge, attitudes and reported practices, may provide misleading information about ultimate impacts · That there may be important gender differences in the timing and the effects of some youth projects · That it may be difficult to measure all longer term effects of a youth project even in well designed randomized trials 92Even in the worst-affected areas, HIV prevalence is very low at ages below 15. However, it increases sharply (particularly among girls) from ages 15-19 and for both sexes from ages 20-24. 93The study is described briefly in Ross and Cleophas-Mazige (2003) and more fully in a presentation available on the internet at http://www.fhi.org/en/Youth/YouthNet/NewsEvents/researchmtg.htm. 58 5.3.1 Problem and identification of alternatives There have been many evaluations of school-based reproductive health interventions (for example, sex education, peer counseling and linked services) designed to prevent HIV infection (Knowles and Behrman 2003b). The results indicate that school-based interventions can be effective in improving reproductive health knowledge, with mixed results in terms of affecting attitudes and intentions, but with few cases of significant effects on reported behavior (for example, sexual initiation, number of partners or condom use). Unfortunately, few of the available studies have measured impact on biological outcomes, such as STI prevalence, and none have measured impact on HIV prevalence. Most of the available studies have been quasi-experimental or randomized trials at the school or classroom level, usually with only a few treatment and comparison groups. The literature on HIV prevention programs targeted to youth suggests that a package of mutually reinforcing interventions is most effective. In this Tanzanian study, the following interventions were supported: · Community activities (described below) · Primary school sexual health education in Standards 5-7 targeted to adolescents aged 12-17+ (teacher led in 10-15 peer-assisted sessions held during normal school hours) · "Youth friendly" sexual health services (with 2-3 health workers per facility specially trained to provide services to youth empathetically and with respect for youth rights, privacy and confidentiality) · Condom promotion and improved access to condoms (4 condom promotion promoters/distributors per village, elected by youth, and trained by PSI) Community activities included one week of initial mobilization (which included formation of an elected community advisory committee consisting of community leaders and youth), subsequent meetings with key groups (e.g., advisory committees, school committees, and religious leaders), and annual youth health weeks. Careful process evaluation (i.e., recording activities, simulated patients, external expert evaluation and external qualitative research on sexual behavior) was conducted to confirm that the interventions were actually and appropriately provided. The study concluded that the program demonstrated that it was feasible to provide sexual education using innovative methods (e.g., edutainment), to address sensitive sexual health issues (e.g., condom use, transactional sex), to improve the youth friendliness of health services, and to implement multiple interventions on a large scale to high standards. 5.3.2 Measurement of effectiveness A community-level randomized trial was used to measure the effectiveness of the program. A baseline survey of 9,445 adolescents aged 15-19 was conducted during August-December 1998 in the 20 study communities, which were then classified into low (6), medium (8) and high (6) HIV risk strata. The 20 communities were then randomly assigned to intervention and comparison groups (within each risk stratum in order to 59 ensure balance in terms of HIV and STI prevalence and geographical districts). Primary school students (9,645) aged 14+ in January 1999 were recruited during the baseline survey from Standards 4-6 of primary school (Standard 7 is the final year of primary school in Tanzania, with the school year extending from January to December).94 A follow-up survey of the students in both the intervention and comparison groups was conducted after 36 months (October 2001 - April 2002).95 At that time, students had been exposed to the school-based interventions for 1-3 years depending on how many grades (Standards 4-6) they had completed at their time of recruitment (for example, students who had completed Standard 6 had only one year of exposure to the school-based interventions, followed by two years out of primary school). 5.3.3 Results and conclusions The study yielded the following findings: · The program produced a significant increase in sexual health knowledge (HIV and STI acquisition, pregnancy prevention) and reported attitudes to sex among both male and female students · There was also substantial impact on some indicators of reported behavior change (sexual debut during follow-up, more than one partner during the past 12 months, first use of condom during follow-up, use of condom at last sex, went to health facility for STI symptoms) · There was also evidence suggesting that program effects were stronger with more years of exposure to the school-based interventions (although this could also represent the effect of starting interventions at a younger age) · There was also some evidence (regular but not statistically significant patterns) of larger program effects among males than females, especially in the case of reported behaviors (possibly because males have more control over their behavior than females, given the gendered power relations existing in the study area) · No consistent biological impacts in either direction were observed (for example, in HIV or STI infection rates or in pregnancy rates) · There was a hint (i.e., a larger but still statistically insignificant difference) of greater biological impact among never married youth The finding of stronger program effects among males is particularly important because of the average age difference among sexual partners (3-5 years). Young men may therefore need to reach ages 20-24 before changes in their own behavior can be expected to have substantial impacts on HIV and STI prevalence in younger women. Such potential effects could not be reflected in data collected after only three years of program interventions (a further follow-up survey is planned for 2005). The fact that the study is community- 94Signed informed consent was given by students. Parents were subsequently provided with an information sheet about the program in an open meeting and were permitted to opt out of the program. Only a few parents opted out of the study, while a few other students effectively opted out by not participating in the data collection (Population Council and FHI 2003). 9572% of the students in the intervention communities (and 73% of those in the comparison communities) were re-interviewed in the follow-up survey. 60 based, rather than school-based, may make it possible to observe a significant share of the longer term impact of the program in subsequent surveys. However, the fact that only 72-73% of the participants could be re-interviewed after only 3 years suggests that it may be difficult to re-interview male participants in 2005, only a fraction of whom may be having sexual relationships with female participants. This study illustrates the problem of measuring longer-term impacts even in well-designd randomized trials. The absence of any biological impacts may indicate that the program only affected knowledge and skills, but not actual risk-taking behavior.96 However, it is possible (as discussed above) that more time is required for the full effects of the intervention to become manifested. It may also be the case that additional interventions need to be added to the program to achieve the desired outcomes (e.g., mass media, interventions for out- of-school youth, interventions for the general community, counseling and voluntary HIV testing, or interventions linked to microcredit to empower women). 5.4 CASE STUDY #4: MEXICO'S PROGRESA/OPORTUNIDADES PROGRAM SCHOLARSHIPS' IMPACT ON SCHOOLING PROGRESA/Oportunidades is a major program of the Mexican Governmen aimed at reducing poverty and developing the human capital of poor households, with substantial emphasis on improving the options for youth. This case study is based on Behrman and Skoufias (2004) and the other references given below, which for the most part build on the Mexican Government-funded and supported evaluation of these programs by outside groups (International Food Policy Research Institute (IFPRI), Instituto de Nutrición y Salud Publica (INSP)). This case study illustrates: · A rich and important systematic evaluation that is an integral part of a large-scale program from the start as designed and undertaken by the government with evaluation by outside groups without a vested interest in supporting the program · A rich menu of data collection-analysis strategies, including a large-scale controlled experiment with random assignment of treatment and controls, "natural experiments," control samples identified by matching and program discontinuity · The importance of collecting data that permit the exploration of externalities because they are central aspects of possible efficiency reasons for public subsidies · An illustration of how structural modeling permits exploration of counterfactual policy experiments and thus can inform policy choices beyond those evaluated by the controlled experiments actually undertaken · An illustration of an important real-world evaluation that had affected not only how the program evolved but also the political sustainability of the program despite a major change in the government 96 It may be the case, for example, that this program and similar interventions affect the way youth respond to questions about attitudes, intentions and risky behavior (because they become better informed about the "right" answers) without affecting their actual behavior. 61 The evaluation of PROGRESA/Oportunidades to date has focused on impact or effectiveness evaluation, with some attention to costs but not an effort yet to integrate the multiple impacts of the multiple program components into overall benefits. 5.4.1 Background The Zedillo government introduced PROGRESA (the Spanish acronym for program for education, health and nutrition) in selected poor rural communities with population 2,500 or less in 1997. PROGRESA's multi-sectoral focus provided an integrated package of education, nutrition, and health services to poor families, conditional on certain behaviors such as attending school. The largest single component of the program is directed toward youth: a scholarship program initially for grades 3-9 (later extended through grade 12) with increasing benefits with higher grades (to reflect the higher opportunity cost of students' time and with somewhat higher benefits for females than for males97). By the end of 1999, PROGRESA covered 2.6 million families in almost 50,000 localities or about 40% of all rural families and one-ninth of all families in Mexico with a budget of approximately $777 million that was equivalent to 0.2% of Mexico's GDP and slightly less than 20% of the Federal Government budget allocated to poverty alleviation. The election of Vincente Fox in 2000 was the first time in almost eight decades that the candidate of PRI (Partido Revolucionario Institucional) had not prevailed. With previous changes in governments, a number of promising programs had been abandoned, as new governments attempted to establish their own mark with their own programs. Therefore it is striking that PROGRESA was not abandoned but instead was modified and expanded -- though renamed Oportunidades (Opportunities). Oportunidades evolved from PROGRESA in regard to some details, such as the extension of the school scholarship program to the preparatoria (upper secondary) school level as suggested in the early evaluation. But the most important change was the expansion into all but the largest metropolitan areas, and with coverage expanded to cover about 20 million poor Mexicans. In 2003 the program was supplemented with Jovenes con Oportunidades (Youth with Opportunties), an effort to provide support for basic activities of youth subsequent to their completion of preparatoria school. 5.4.2 Importance of evaluation and initial large-scale experiment An important distinguishing feature of PROGRESA/Oportunidades since its inception has been serious efforts at collecting necessary information and undertaking systematic evaluation. Many programs pay lip service to the importance of evaluation and have plans on paper that are consistent with systematic evaluation, but often (perhaps usually), these good intentions on paper are not given sufficiently high priority during the pressures of implementation to result in good evaluations. PROGRESA/Oportunidades, in contrast, has established longitudinal population-based (not, for example, school-based as in the Kenya case study discussed in Section 5.1) evaluation samples, including baseline data, with random assignment to treatment and control groups and systematic evaluations by outsiders that have been subject to considerable examination. It appears that an important reason that PROGRESA was sustained with the change of the 97 See discussion of this gender difference in Section 4.3 above. 62 government was the widespread perception that the program had been evaluated seriously and found to have substantial merits, which gave the program considerable credibility. The primary initial PROGRESA evaluation strategy was an experimental design with random assignment of 506 communities to receive program "treatment" (320 communities) versus "controls" (186 communities) that did not initially receive program benefits nor any commitments to subsequently receive program benefits (though they did receive them after delays of about two years). To provide information to evaluate the impact of PROGRESA in rural areas, PROGRESA conducted formal surveys of all residents in the roughly 24,000 households in the evaluation communities (which permits evaluation of possible spillover effects on others, as well as estimation based on program discontinuity) about every six months over the 1997-1999 period (including pre-program baseline data and with further data collected in 2003), as well as structured and semistructured observations and interviews, focus groups, and workshops with a series of stakeholders, including beneficiaries, local leaders, local and central PROGRESA officials, health clinic doctors, nurses and assistants, and schoolteachers. While the estimation of program impacts have depended on the quantitative data collected, the focus groups and workshops have provided input into what questions should be asked on the quantitative surveys and insights into operational aspects of the program (see Behrman and Skoufias 2004). 5.4.3 Impact on schooling in rural communities ­ enrollment, attainment, drop- outs, re-entry, grade progression, spillovers within families and within communities PROGRESA's scholarship program has been its largest component and is the component most focused on youth. Mexico's children typically have a high enrollment rate in primary school of about 93% (Figure 1). For the rural poor, however, schooling enrollment then drops off, declining to 55% after completing the sixth grade. PROGRESA provides scholarships conditional on school attendance by a child from an eligible household (determined to be below the poverty line) for grades 3-9 (later, as noted, increased to 12), with the amount of the scholarship increasing by age and about 15% higher for girls than boys at the secondary school level (see Section 4.3). 63 Figure 1: School enrollment and labor force participation of children in PROGRESA communities prior to program implementation School Enrollment and Labor Force Participation of School Enrollment and Labor Force Participation of Girls in Boys in PROGRESA PROGRESA Communities Prior to Program Implementation Communities Prior to Program Implementation 100 90 100 80 80 70 60 t 60 rcent 50 Pe 40 ercenP 40 20 30 20 0 10 8 9 10 11 12 13 14 15 16 17 0 Age (Years) 8 9 10 11 12 13 14 15 16 17 Age (Years) Enrolled In School Working Enrolled In School Working Source: Parker and Skoufias (2000). PROGRESA's effect on school enrollment has been evaluated at both the aggregate community and individual level and exploiting the experimental design and the program discontinuity (as well as using structural models ­ see Section 5.4.5). Based on a number of tests with different approaches and different samples to test for robustness, it has been concluded that PROGRESA has had a positive enrollment effect for both boys and girls at the primary and secondary levels (Figure 2). At the primary school level ages, where Figure 2: Positive enrollment effect for children at primary and secondary school levels All Children 12-17 Years Old ng 0.7 0.6 ndiet lo 0.5 AttnecreP ho 0.4 Sc0.3 0.2 0.1 1997 1998 1999 Survey Round (Year) PROGRESA Control Source: Behrman and Skoufias (2004, Figure 2). 64 enrollment rates before PROGRESA were between 90 and 94 per cent, PROGRESA increases the enrollment rate of boys by 0.74 to 1.07 percentage points and of girls by 0.96 to 1.45 percentage points (Schultz, 2000, 2004). At the secondary school level ages, where the initial enrollment rates before PROGRESA were 67% for girls and 73% for boys, the increases in enrollments for girls ranged from 7.2 to 9.3 percentage points and for boys from 3.5 to 5.8 percentage points. This represents a proportional increase of boys from 5-8% and of girls 11-14% (Schultz, 2000, 2004). If these program effects could be sustained over the period in which a child is of school age, the cumulative effect on educational attainment for the average child from a poor household would be an increase in schooling attainment of 0.72 years for girls and of 0.66 years for boys (Schultz 2000b, 2004). Given that the average youth aged eighteen achieved about 6.2 years of completed schooling prior to the program, these estimates suggest an overall increase in schooling attainment of about 10%. Participation in the program not only increases enrollment rates, but also causes earlier ages of school entry, less grade repetition and better grade progression, lower dropout rates, higher school reentry rates among dropouts and lower mean ages for every grade completed ­ though there is not evidence of much impact on attendance or cognitive achievement. (Behrman, Sengupta and Todd 2000, 2005; Schultz 2000a). The program is especially effective in reducing dropout rates during the transition from primary to secondary school. The increased demands generated by the program have not led to a degeneration in the quality of education services, suggesting that previously resources were idle or used ineffectively or that resources have been increased. For instance, Behrman, Sengupta and Todd (2005) compare the impact on children in noneligible families between treatment and control areas and find no difference, contrary to what might be expected if, for example, the increased enrollment of eligible children caused congestion or severely strained school resources. In many cases, there seems to have been an improvement. This view is also consistent with evidence from the quantitative survey of directors, with most schools reporting some improvements in infrastructure and other resources, albeit from a poor initial position 5.4.4 Impact on child and adult labor The results also show very clear impacts of PROGRESA on reducing children's labor market participation. Estimates based on double-difference models (see Section 3.2.3.2) of labor force participation before and after the implementation of PROGRESA show important reductions in children's labor force participation for both boys and girls, in both salaried and nonsalaried activities. Labor force participation for boys shows reductions as large as 15-25% relative to the probability of participating prior to the program. For girls, in spite of their overall lower participation level prior to the program, there are also significant reductions associated with PROGRESA. Also the lower incidence of child work due to the PROGRESA program is found to be consistent with from 65% (in November 1999) to 82% (in November 1998) of the increase in the enrollment of boys in school. 65 While PROGRESA reduces child and youth labor, it does not appear to create negative incentives for work (Parker and Skoufias, 2000). Analysis of before- and after-program data shows no reduction in the labor force participation rates of either men or women. These results may, in part, reflect the design of PROGRESA, where benefits are provided to families for three years, irrespective of subsequent family income, so that there is not much disincentive effect on work in order to maintain program eligibility (though there still may be wealth effects), as opposed to transfer programs in other countries that often reduce benefits with additional work income. The conventional wisdom is that there are trade-offs between providing benefits to a population in need and stimulating work; the analysis here shows that, thus far, there is not necessarily any such empirically important trade-off in PROGRESA. 5.4.5 Counterfactual evaluation of experimental variants of the PROGRESA program As noted in Section 3.2.3.1, controlled experiments are thought in certain respects to be the "Gold Standard" for evaluation of program impacts, but they are not without their limits. One important limitation discussed there is that they do not permit assessing counterfactual policy experiments for policies that differ in important respects from those covered by the experiment. Structural models permit such assessments, conditional on the assumptions necessary to specify and estimate such models. An important illustration of such possibilities is provided in the case of PROGRESA by Todd and Wolpin (2003). They use PROGRESA data to: (i) estimate and validate a dynamic behavioral model of parental decisions about fertility and child schooling98, (ii) forecast long-term program impacts that extend beyond the two-year life of the initial experiment, and (iii) assess the impact of a variety of counterfactual policies: The topics covered by their counterfactual simulations include: (1) Long-run effects on schooling attainment and fertility: If PROGRESA were in place for all school ages of all children in the sample, the long-run impact would be to increase mean schooling attainment to 8.7 grades, which is about a grade more than suggested by estimates based on non-structural extrapolations of the two years of existing post- program initiation data in Section 5.4.3. Though the designers of PROGRESA had been concerned that the program might induce increased fertility because of the large cash payments for children attending school, the simulations indicate that completed fertility would change relatively little with subsidy levels, increasing but slightly as subsidy levels rise. (2) Changing the amount of the scholarships: The program has a fixed scholarship schedule from which it is not possible to tell if there are important nonlinearities in response to scholarships. This study finds that mean completed schooling increases at a linear rate with increments in subsidy amounts up to the original amount and then at a diminishing rate. For example, for girls the increases in mean schooling between the 98To elaborate, they use pre-program and control data to estimate the model, but do not use the experimental data for estimation. Then they test the model against the experimental results and find that it predicts well the outcomes. 66 baseline and one-half of the original subsidy and between one-half and the full original subsidy amount are both 0.27 years, while the increase from doubling the subsidy amount is 0.47 years. (3) Eliminating payments for grades 3-5: PROGRESA starts making positive payments in grade 3. Because the great majority of children complete at least grade 5, the subsidy to the earlier grades acts mainly as a direct income transfer program and might have only a small effect on schooling although a large effect on the cost of the program. Restricting the subsidy to grade 6 and higher (with this same subsidy schedule for grades 6-9) reduces the per-family expenditure of the program considerably, from around 26,000 pesos to less than 16,000 pesos. Perhaps surprisingly, however, the fall in completed schooling also is significant, with approximately 30% of the gain in mean schooling for girls and 33% for boys lost. The reason for such declines lies in the interdependence of parental decisions among children within the family. If there are multiple children of school age in the household, providing a subsidy to attendance for children at lower grades, because it increases family income, increases the incentive for older children to attend higher grades rather than to produce additional income through work. In an alternative experiment, the subsidy again is limited to grades 6-9, but the subsidy schedule is increased by 1.43 so that the program financial cost remains the same as in the original program. Relative to the original subsidy, the gain in mean completed schooling is predicted to be 0.14 years for girls and 0.11 years for boys, an increase of about 25% over the original gain. This restricted subsidy program would appear to be more effective than the actual program in producing higher completed schooling levels in this population. (4) Replacing current scheme with bonus for ninth-grade graduation: An alternative subsidy scheme would reward lower-secondary school completion rather than attendance. An experiment considers the impact of a ninth-grade graduation bonus in the form of a payment of 30,000 pesos to families when a child graduates from lower secondary school. Clearly, the effect of such a bonus scheme requires, as the model assumes, that families be forward-looking. The simulations show that the bonus increases the percentage of children completing junior secondary school significantly, by about ten percentage points for both girls and boys, but has a relatively small impact on average schooling. In fact, the increase in average schooling is not as large as the effect of the original subsidy even though the cost of this bonus program is about 50% higher. Interestingly, the proportion of children who complete at least 6 grades actually falls below the non-subsidy level, suggesting that families are substituting more schooling for some children and less for others. Thus, the effect of the bonus is largely to induce children who were already attending junior secondary school to complete ninth grade. (5) Replacing scholarships with unconditional cash transfers: Another experiment is to investigate the impact of a pure income transfer program that pays 5,000 pesos per year to families without any school attendance requirement. This amount is close to the maximum benefit that families may currently receive under the program in any year, and represents about a 50% increase in annual family income. In this experiment, mean schooling increases due to the subsidy as schooling is a normal good, but the increase is 67 only about 20% as large as the original attendance-based subsidy even though the cost per family is an order of magnitude larger. Thus making the transfer conditional on schooling attendance apparently (as compared with the unconditional transfer) (a) lowers the welfare of household decision makers and (b) transfers resources from current consumption to investment in the schooling of youth. They also consider some additional interventions, all of which have rather small effects on schooling. In particular, enforcing a child labor law prohibiting children under the age of 16 from working and building a junior secondary school in each village where it is absent each would raise mean schooling by 0.1 years or less. These simulations illustrate how counterfactual experiments can, under the assumptions necessary to estimate the underlying behavioral models, be quite illuminating regarding the impact of policies on outcomes for counterfactual policies that last longer and/or differ from those investigated in experiments and that in the specific case of PROGRESA, there may be some opportunities for considerably improving some details of the program (e.g., by concentrating the scholarship funds on grades 6-9). 5.4.6 Ongoing data collection and evaluation As noted, while the currently available evaluation is limited to rural areas for the original PROGRESA program for 1998-2000 that is summarized above, under the Fox Government the program has continued and evolved under the name "Oportunidades." As also noted, though many of the details of the program have remained the same, there have been some modifications, in particular the extension of the educational subsidy component of the program to grades 10-12 (upper secondary or preparatoria). But the larger changes relate to program participation. First of all, all the rural control families eligible for the program were enrolled into the program. Second, the program was expanded into all but the largest metropolitan areas, with enrollment open in neighborhoods for eligible families who attended local enrollment centers during specified enrollment periods that Oportunidades tried to advertise widely. These developments raise two broad questions for ongoing evaluation. Firstly, what are the medium-term impacts of the rural PROGRESA/Oportunidades program. The above discussion summarizes the impacts of the program during the first two years. But by 2003 one could ask what are the effects after six years. Secondly, what are the impacts in urban areas? To what extent are the impacts similar and to what extent are they different from those in rural areas? Given the much different markets and other institutions in the urban as compared to the rural areas, the program effects might be much different. To answer these two broad questions ­ and the many detailed questions underlying them ­ Oportunidades has continued with evaluation of both the rural and urban programs. The results of these evaluations soon will be available (e.g., Angelucci, Attanasio and Shaw 2004, Behrman, Parker and Todd 2004, Parker, Behrman and Todd 2004, Todd, et al. 2004). But it is useful here to summarize some aspects of the information that is being collected for the evaluation because in some important respects the information basis differs from that for the original rural evaluation. 68 For the rural areas, as noted, the eligible controls were incorporated into the program after two years. Though household survey data were collected from the original treatment and control households in 2003 six years after program initiation, the implications of the original experimental design have changed. Comparisons of the original treatment and control groups will permit comparison within an experimental framework of being exposed to treatment for six years versus four years. For a few topics for which program exposure may be critical during a certain lifecycle period, it will permit comparison six years later of the effects of being in the treatment versus the control groups during that critical lifecycle stage. The most promising examples of lifecycle stages during which interventions may be critical are the introduction-of- complementary-food period for nutritional interventions (roughly six-24 months) and the age range for completion of primary school (about 12-13) at which critical decisions are made for continuation into secondary school. The 2003 data will permit the comparisons in the medium term, after six years, of treatment versus control in 1998-1999 at those critical lifecycle stages. But they will not permit more general medium-term evaluations of program effects after six years by comparing the original treatment and control groups because the control group received treatment after the first two years. Therefore, for most questions, alternative strategies need to be employed to obtain estimates of medium-term program effects in rural areas. Two such strategies are being employed to attempt to learn what would happen to the same persons with and without the program. The first strategy is to compare those close to but below the eligibility cutoff with those who are close to but above the eligibility cutoff ­ that is to exploit the fact that households close to the eligibility cutoff are very similar, but some received and others did not receive the program benefits ("discontinuity design" ­ see Sections 3.2.3.1-2). This is an effective strategy for obtaining estimates of the medium-term impacts for those close to the eligibility cutoff (that depends importantly, of course, on having data on those above as well as below the eligibility cutoff), though if there are nonlinearities, the estimates obtained may not be informative about program impacts on the very poor who are far below the cutoff.99 The second strategy is to compare the original treatment group with a new control group that has not received the program. Indeed, as part of the 2003 data collection there was an effort to collect data on such a sample, using nonparametric matching techniques to locate households and communities as similar as possible to the treatment localities prior to the program initiation in 1998 (see Section 3.2.3.1). Comparisons then can be made between the original treatment group and this new control sample. But such comparisons must be interpreted with care because the new control group is not a random sample of localities, but a sample selected from those who had not received the original program after six years. Therefore they may be different in some fundamental ways from the original treatment localities. Even if such differences can be controlled, finally, the information with which to make the comparisons is much less rich for the new controls (just starting in 2003, though with as much retrospective information as possible) than for the original treatment and controls (with baseline-longitudinal data since 1997). 99 This strategy has been used for the rural data by Buddelmeyer and Skoufias (2003). 69 For urban areas there were two important differences from the start, as compared with the start of the rural evaluation sample. Firstly, it was not thought possible to randomly assign treatment to communities, in part because in urban areas communities are not well-defined separate entities but blend together in larger metropolitan areas. Secondly, enrollment in the urban areas required going to an Oportunidades enrollment center and providing information to apply for enrollment, rather than being selected based on previous information as in the rural sector. This means that not all eligible households in a locality that was selected for the program actually received the program benefits. Therefore, for urban localities, there are several comparison groups that are possible for those who received treatment. Firstly, nonparametric matching was used with census data to identify control groups among localities that had not yet received the option of enrolling in the program. Within such localities, eligible and non-eligible households can be identified by the standard Oportunidades criteria. Secondly, as for the rural areas, comparisons can be made between those close on both sides of the eligibility line. Thirdly, comparisons can be made between eligible households in treatment areas who elect to participate in the program and those who do not because they did not know of the program. All of these approaches have their limitations, but the combination of approaches will be informative about how robust the estimates of program effectiveness are to various approaches. 5.5 CASE STUDY #5: CONTINUING EDUCATION IN SERBIA AND MONTENEGRO This case study is included as an example of the kind of economic analysis that is often feasible with locally available data. This case study uses existing household survey data to estimate the benefits and costs of a hypothetical continuing education project in Serbia and Montenegro.100 This study involves strong assumptions in several areas, as pointed out below. The project's costs are estimated on the basis of cost estimates prepared for a similar intervention in Hungary (no comparable cost estimates were available for Serbia and Montenegro). This case study illustrates the following key points made in this Guide: · That returns to investments in youth may vary significantly by gender · That including a wider range of benefits can have a significant effect on the resulting benefit-cost estimates · That the results obtained in an economic analysis (including estimated gender differentials) can be very sensitive to the assumptions used, as revealed by sensitivity analysis 5.5.1 Problem and identification of alternatives The project analyzed would provide financial support (scholarships) to enable poor youth to complete secondary school (academic high school), including those who have previously dropped out of school. The project is designed to address the problem of high 100Sample terms of reference referring to this case study have also been prepared and are provided in Section 6 as model terms of reference for an ex ante economic analysis. 70 dropout rates from secondary school, together with high rates of youth unemployment in Serbia-Montenegro (see Section 6 for additional background information). Dropout rates are particularly high among the rural poor and among some ethnic groups (for example, the Roma). One option would be for the project to try to encourage youth to return to public secondary schools. However, the public schools have already failed these youth for one reason or another, and there are numerous obstacles to improving their performance at this time. Accordingly, the hypothetical project envisions providing continuing education through non-formal, NGO-operated schools that have been shown to be effective in other settings (for example, among the Roma in Hungary). 5.5.2 Measurement of effectiveness The benefits of secondary school completion are likely to include increased labor productivity as well as possibly some other benefits, for example, reduced likelihood of unemployment, improved health and, eventually, increased schooling of the youth's own children. The effect of secondary school completion on labor productivity can be estimated on the basis of the observed relationships between annual earnings and the number of grades of schooling completed (as a crude proxy for cognitive achievement and other skills that are believed to increase labor productivity), the length of work experience, and other relevant characteristics (for example, gender and place of residence).101 Table 5.3 presents estimates of the mean annual benefits from completing high school by sex and level of work experience, based on the estimation of age-earnings relationships using data from the 2002 Serbia Living Standards Measurement Survey (LSMS) (Lokshin and Jovanovic 2003).102 These estimates are used to obtain estimates of the percentage increase in earnings by sex and level of work experience due to high school completion, which are then multiplied by mean monthly earnings (4,418.3 dinars for males and 3,339.2 dinars for females), annualized and converted into United States dollars at the market rate prevailing during the month to which the earnings data refer (October 2000, i.e., $1=13.96 dinars). These estimates of the benefits of high school graduation are limited to increases in labor productivity that may be reflected in private sector earnings differentials. In particular, they neglect possible benefits associated with the effect of high school completion on the probability of finding employment, on improved health and, eventually, on the level of schooling completed by the youth's own children. The implications of including these other benefits are explored below. The differences between males and females with more than five years of experience are striking and difficult to explain, as is the negative earnings differential for females with more than 15 years of experience. Some of these differences are likely to reflect selectivity and cohort effects, as discussed in Section 3.2.3. 101 Again, as in Sections 5.1 and 5.2, assuming that there are not substantial biases due to unobserved endowments such as ability, motivation and family connections and assuming that there are not important differences in rates of return to schooling between labor markets and other time uses. 102 Private sector wage rate earnings differentials by level of schooling completed are used initially to estimate earnings benefits, but this assumption is also relaxed in the discussion below. 71 Table 7: Estimates of the annual earnings benefits from completing high school in Serbia, based on private sector earnings differentials by level of work experience and sex, 2000 (October) Mean annual benefits (US$) of high school completion Level of experience Male Female Less than 5 years 704 706 5 to 15 years 38 1,203 More than 15 years 0 -194 Source: Lokshin and Jovanovic (2003). 5.5.3 Measurement of costs There are no data available on the cost of non-formal secondary schooling in Serbia- Montenegro. Accordingly, the study uses estimates of the cost per student and cost per secondary school graduate of continuing education programs leading to secondary school completion targeted to the Roma population in Hungary (Table 5.4). The estimates of the cost per secondary school graduate in column 4 can be used as cost-effectiveness estimates if it is assumed that students would not have graduated from secondary school if they had not attended these (or similar) schools. The estimates in column 4 range from US$6,479 to US$29,327 per secondary school graduate. Because the Roma are a disadvantaged minority in terms of schooling in Hungary as in SEE countries and because the cost of living is higher in Hungary than in SEE countries, cost per secondary school graduate may be higher in this case than with similar programs in Serbia- Montenegro that are not targeted exclusively to the Roma. Table 8: Cost per secondary graduate in selected nonformal schools, Hungary 1998-99 School Cost per student Number of Number of Cost per secondary (US$) secondary students secondary school graduate graduates (US$) Don Bosco a1,110 180 24 8,322 Martineum 7,821 30 8 29,327 Gandhi 3,093 170 18 29,209 Jozefvaros a864 15 2 6,479 Source: World Bank (2001b). aincludes both academic and vocational students. 5.5.4 Results and conclusions The estimates of unit benefits in Table 5.3 and of cost per secondary school graduate in Table 5.4 can be used to develop cost-benefit estimates for the continuing education project in Serbia-Montenegro, using the following additional assumptions (some of which are subsequently relaxed in sensitivity analysis) within a life-cycle framework: · The cost of continuing education per secondary school graduate is US$ 7,658 (i.e., direct costs of $6,479 from Table 5.4 plus 30% in administrative and 72 distortionary costs) 103 and this total cost is incurred in equal installments at ages 13-17 (i.e., 5 equal installments of $1,531.60). · Benefits begin to occur at age 18, when graduates are assumed to begin working · Annual benefits at ages 18-22 are the same as those for workers with 1-5 years of experience in Table 5.34, those at ages 23-32 are the same as those for workers with 5-15 years of experience, and those at ages 33 to 60 (the assumed retirement age) are the same as those with more than 15 years of work experience · Benefits are limited to increased earnings, as reflected in Table 5.3 · The benefits and costs are discounted to age 13 (the age of the initial investment), using an annual discount rate of 5% Under the above assumptions, the estimated benefit-cost ratio for males is 0.4 and 1.0 for females. The estimated benefit-cost ratio is higher for females because of the sharply higher benefits that female high school graduates with 5-15 years of experience are estimated to receive in the private sector compared to those received by male high school graduates with the same work experience (Table 5.3). Reducing the discount rate from 5% to 3% leaves the benefit-cost ratio unchanged (to one decimal place) for males while increasing it slightly (from 1.0 to 1.1) for females. Increasing the discount rate to from 5% to 10% reduces the benefit-cost ratio from 0.4 to 0.3 for males and from 1.0 to 0.7 for females. In contrast, retaining the 5% discount rate but assuming that the direct cost per secondary graduate is $8,322 per secondary school graduate, instead of $6,479 (i.e., the cost for Don Bosco school in Table 5.4, instead of the cost for Josefvaros school) lowers the estimated benefit-cost ratio from 0.4 to 0.3 for males and from 1.0 to 0.9 for females. Relaxing the assumption that high school completion has no effect on unemployment significantly affects the estimated benefit-cost ratios. For example, if the effect of high school completion reduces the unemployment rate by 10 percentage points (for example, from 25 to 15%) at ages 18-22 for both males and females and thereafter by 2 percentage points for males and 1 percentage point for females (these assumptions are broadly consistent with the estimated relationships between unemployment and years of schooling in Serbia that are presented in Behrman and Knowles 2003),104 the benefits in terms of increased earnings increase significantly (Table 5.5, compared to Table 5.3).105 Under these revised assumptions, the benefit-cost ratio rises from 0.4 to 0.7 for males and from 1.0 to 1.2 for females with a discount rate of 5%. 103The 30% estimate for administrative and distortionary costs is probably conservative, based on the discussion in Section 3.1.3 where it is recommended that an estimate of 20-25% be used as an estimate of the cost of raising additional tax revenue. 104Use of these estimates as a basis for estimating the benefits of the effect of increased schooling on the probability of employment involves the strong assumption that the observed relationship is causal. 105The revised benefits estimates in Table 5.5 are calculated as the sum of the benefits in Table 5.3 plus the product of the assumed change in the proportion unemployed with average earnings adjusted for high school completion. 73 Table 9: Estimates of the annual earnings benefits (including those from reduced unemployment) from completing high school in Serbia, based on private sector earnings differentials by level of work experience and sex, 2000 (October) Mean annual benefits (US$) of high school completion Level of experience Male Female Less than 5 years 1,154 1,064 5 to 15 years 115 1,244 More than 15 years 76 -167 Source: Behrman and Knowles (2003). The estimates of benefits to this point have been limited to direct labor market gains. Possible additional benefits due to high school completion might also include: the improved health experienced by youth who have completed secondary schooling and the eventual increase in the level of schooling completed by their own children. In the case of improved health, the estimated relationship between the occurrence of chronic illness and years of schooling completed in the 2002 Serbian LSMS implies that four additional years of schooling are associated with an approximately 5% lower probability of chronic illness (Behrman and Knowles 2003). Assuming on average that each chronic illness results on average in the loss of 10 disability-adjusted life years (DALY), this estimated relationship suggests that there is a gain of about one DALY associated with secondary school completion. This one DALY gain is valued at one year of earnings (see discussion in Section 3.3). Inclusion of this health benefit (with the reduced unemployment benefit already included) raises the benefit-cost ratio from 0.7 to 1.1 for males and from 1.2 to 1.5 for females with a discount rate of 5%. The estimated relationship between the number of years of schooling completed by parents and the corresponding number completed by their children suggests that four additional years of schooling on the part of both mothers and fathers is associated with about 0.8 additional years of schooling completed by each of their children. It is assumed that this benefit106 occurs on average at age 45 for males and at age 40 for females and that both males and females have two children. Inclusion of this additional benefit leavess the estimated benefit-cost ratio unchanged for males at 1.1 (including all previously defined benefits) but changes it from 1.5 to 1.6 for females with a discount rate of 5%. 5.6 CASE STUDY #6: INVESTMENTS IN BASIC EDUCATION IN PAKISTAN This case study is based on several studies of investment in basic education in Pakistan.107 It illustrates the following key points made in this Guide: 106In this case, the benefit is defined as one-quarter of the value of all previously estimated benefits of secondary school completion, discounted to age 13 (since 0.8 additional years of schooling is equivalent to about one fourth of the additional schooling required to complete secondary school). 107The use of results from several different studies in a given country is fairly typical of how an actual ex ante economic analysis is likely to be prepared. 74 · That cost-benefit analysis can be used both to evaluate alternative interventions in basic education in rural areas of Pakistan and to compare the economic returns from these interventions to those from investments in other sectors · That including a wider range of benefits in cost-benefit analysis is feasible and can have a significant effect on the results · That returns to investments in youth may vary significantly by gender 5.6.1 Problem and identification of alternatives Increasing the quantity of schooling an individual receives is likely to raise his or her cognitive skills. Improving school quality is likely to have the same effect. Increasing the quantity of schooling -- by providing a primary education to children who otherwise would not go to school, or by providing a middle school education to children who otherwise would leave school upon the completion of primary school -- entails substantial costs. Similarly, improving the quality of schools has costs. However, in the case of quality improvement, there is little or no change in the opportunity cost of student time -- a large component of the total resource cost of schooling. 5.6.2 Estimation of effectiveness Economic evaluation of investments to increase the quantity versus the quality of schooling is difficult because it requires considerable data: information on what schools are producing ­ e.g., cognitive achievement, what are the effects of that product on outcomes of interest such as wage rates, what are the inputs into the production of that outcome ­ such as student time in school, teacher quality, student-teacher ratios, what are the costs of improving schooling quality and of the time that children spend in school, and what are the determinants of a number of behavioral decisions such as how much time an individual spends in school and whether an individual subsequently participates in the labor market. Behrman, Ross and Sabot (2004) developed a conceptual framework for evaluating the rates of return to increases in the quantity versus the quality of schooling. They collected most of the necessary data for rural Pakistan, and made estimates with methods that control for the key behavioral choices and for unobserved determinants of education. Alderman, et al. (1996) find that higher cognitive skills are rewarded with higher wages in rural Pakistan, presumably because more skilled workers are more productive. Because they are more skilled, graduates of even low-quality primary schools earn more than uneducated workers. In like manner, graduates of high-quality primary schools and graduates of middle schools who attended low-quality primary schools earn more than students who complete only low-quality primary schools. 5.6.3 Results and conclusions Behrman, Ross and Sabot (2004) estimate that the "social" rate of return108 to enabling the graduate of a low-quality primary school to complete middle school -- 2.8 percent -- 108 These are "social" rates of return with "social" in quotation marks because, as in most of the economics literature on the rates of return to schooling (e.g., surveys such as in Psacharopoulos 1994), these estimates are based on private labor market benefits (not including any externalities that might distinguish private 75 is low compared to improving school quality -- 13.0 percent -- or providing access to a low-quality primary school -- 18.2 percent. The benefits are in terms of the estimated market returns to better cognitive achievement. The costs include the resource costs of the schools109 and the private costs, including the opportunity costs of students' time. The relatively high rate of return to improving quality reflects the absence of any additional opportunity cost to the students and the absence of higher capital costs for students already enrolled in school. In this context, it appears that productivity and equity concerns both point towards expanding primary schools, even if they are of lower quality. Because few boys now lack access to basic schooling, girls would benefit disproportionately from this recommended policy. This study again points to some of the difficulties in undertaking such evaluations. Even with the special data collected for the study, for example, it was not possible to identify with confidence the relative importance of the various components of teacher quality (i.e., the relative importance of factors such as teacher experience, teacher schooling, teacher training). This study also limits the measurement of the effects of changes in schooling to the value of labor market outcomes, and provides no information on possible efficiency reasons for governmental interventions. Further, as in the other studies reviewed in this section, there is no attention to some possibly important distortionary costs, such as those incurred to raise governmental revenues. 5.6.4 Extensions to include additional benefits Another schooling study in Pakistan (Summers 1992) suggests that benefits other than labor market outcomes may be large enough by themselves to justify investment in girls' education. Summers assumed that an additional year of female education reduces by 7.5% both the total fertility rate (assumed to be 6.6 live births per woman initially) and the child mortality rate (assumed to be 121 deaths per 1,000 live births initially).110 Accordingly, if one thousand girls receive an additional year of schooling, this implies 495 births averted, 60 child deaths averted (in addition to those averted among unborn children), and 3 maternal deaths averted (due to reduced exposure to the risk of maternal mortality as the result of fewer births). from social benefits) and private plus public costs. The estimates used for the other cases studies above in Sections 5.1, 5.2, 5.4 and 5.5 also make no effort to adjust for differences between private and social benefits due to schooling externalities, though Sections 5.1, 5.3 and 5.4 consider other aspects of externalities and Section 5.5 considers distortions in wages and salaries due to civil service pay scales. 109 They estimates of primary and middle school resource costs per student and of how primary school resource costs vary with quality were derived from the cost information obtained from a sample of 75 primary schools and 36 middle schools. A hedonic cost function was estimated with these data in which primary school resource cost per student is a quadratic function of number of teachers, number of students, the school quality indices, along with gender and regional dummies. The authors lacked data on the annual amortized capital costs attributable to each student, but they suggest that the physical characteristics of the primary schools in their sample imply that these costs are small. 110 These assumptions appear to be based on advice from World Bank experts, but it is not clear that there were careful estimates underlying them that controlled for unobserved characteristics and endogenous choices to identify causality from associations (see Section 3.2.3). 76 Summers assigned monetary values to these assumed outcomes on the basis of the cost of the least-cost alternative means that Pakistani society used at the time to obtain the same outcomes, i.e., $800 to avert one child death, $65 to avert one birth, and $2,500 to avert one maternal death. He calculated the discounted sum of these benefits, which he assumed would occur after a delay of 15 years, to be $42,600 using a discount rate of 5% and compared it to the cost of providing one more year of schooling to one thousand girls ($30,000, or $30 per girl). Summers concluded that the resulting benefit-cost ratio of 1.42 (which does not include any estimate of the increased value of productivity gains in or outside of the labor market that would result from such an investment) constitutes a strong economic case for investing in girls' schooling. Since most of these additional benefits are private (i.e., accrue to the girls or their children), these results do not by themselves bolster the case for governmental intervention in schooling markets. But they do raise questions regarding why such investments are not undertaken privately and whether failure to undertake them reflects some market or policy failures that should be rectified. For example, it is sometimes suggested that in South Asia there is under investment in girls because the returns to such investments are reaped primarily by in- laws not by the parents themselves even though some studies (e.g., Rao 1993) find significant marriage market returns to girl's schooling. Table 10: Non-earnings benefits from educating 1,000 girls in Pakistan, 1992 Effects Number Unit value Total benefits Births averted 495 65 32,175 Child deaths averted 60 800 48,000 Maternal deaths averted 3 2,500 7,500 Total undiscounted benefits 87,675 Present value of total benefitsa 42,600 Costs 1,000 30 30,000 Benefit-cost ratio 1.42 Source: Summers (1992), as reported in Belli, et al.(1998). ausing a discount rate of 5%. 5.7 CASE STUDY #7: YOUTH TRAINING PROGRAM (PROGRAMA JOVEN) IN ARGENTINA This case study is based on an evaluation of the Programa Joven job training program in Argentina (Aedo and Nuñez 2001). The evaluation was designed to answer the following questions: · Does the program increase the labor income of trainees? · Does the program increase the probability that trainees will be employed? · What is the rate of return to dollars spent on the program? This case study illustrates the following key points made in this Guide: · That it is often possible to obtain estimates of project effectiveness using statistical techniques such as matching that may provide better estimates than 77 cross-sectional multiple regression estimates (such as those used in the Serbia- Montenegro case study reported in Section 5.5 and discussed in Section 3.2.3.2 as "Comparisons of indicators for youth program participants versus non- participants with multivariate controls for observed characteristics") · That returns to investments in youth may vary significantly with gender and age · That conclusions regarding the economic returns from youth projects may be very sensitive to the assumptions used (including the choice of discount rate) · That targeted investments in youth may provide higher economic returns than untargeted investments · That project costs are often incorrectly estimated in existing economic analyses 5.7.1 Problem and identification of alternatives Youth unemployment is a serious problem in most developing countries. Globally, youth account for an estimated 41% of all unemployment, with rates typically 2-3 times higher than those of adults (World Bank 2004). Youth unemployment represents a waste of productive resources, loss of an opportunity for additional human capital formation at relatively low cost through work experience, and a risk that existing human capital obtained from formal schooling may decrease in value over time due to non-use. Unemployed youth are also more prone to engage in individually and socially harmful activities, such as crime, violence, risky sexual behavior and drug abuse. For example, recent studies have found that developing countries with the highest shares of youth in their total population (above 40%) and with high levels of youth unemployment were more than twice as likely to experience an outbreak of civil conflict during the 1990s (World Bank 2004). The target population of Programa Joven is the large number of poor youth, male and female, with limited education and without work experience and who are unemployed, under-employed or inactive. The selection criteria for the program are: minimum age of 16 years, no more than secondary education completed, member of a poor household, and not currently employed. The program provides intensive training (200 hours over a period of 6 to 12 weeks) for positions in the productive sectors of the economy, including reimbursement of transportation expenses, a stipend for females with children under 5, medical checkups, books, materials, work clothing and an 8-week internship in a firm. 5.7.2 Measurement of effectiveness The impact of Programa Joven is estimated in this study by matching program participants with several "similar" non-participants. The matching process requires two steps. Firstly, a logit model is estimated to predict whether or not an individual participated in the program. A "propensity score" is calculated for each person in the sample from the estimated logit function and is used to match each participant with 5 (or alternatively, 10, 20 or 30) non-participants. Secondly, the value of an outcome variable for each participant (earnings or the probability of employment) is compared with the (mean) value of the same outcome variable for the matched non-participants. The mean difference between these participant and non-participant values is used as an estimate of program impact on a given outcome. 78 In this study, three different samples/sources of information were used: 1. Administrative data on all individuals who registered and qualified to take training programs during the period March 1996 to December 1997 (about 140,000 persons) 2. The same administrative data as above but restricted to a sample of 3,340 individuals consisting of equal numbers of persons extracted from the above group in each of the following two groups: 1) persons who have completed the program training, selected to be representative in terms of sex and region of residence, and 2) a "comparison" group of persons who were qualified to enter the training programs but who did not participate and who resembled those who did participate in terms of age, sex, level of education, labor force participation, socioeconomic level and whether they had children under age 5 years 3. Additional survey data available only for the restricted sample of 3,340 persons in the second data base Separate logit functions explaining participation (with participation defined as successful completion of the technical knowledge phase of the training) were estimated using the three different samples/information sources described above to obtain three different sets of propensity scores.111 The authors conjectured that use of different propensity scores would result in significantly different estimates of program impact. The estimation was done separately for the following four groups: adult males (21-35), young males (16-20), adult females (21-35) and young females (16-20)--as well as for all four groups combined. The participants in each group were matched with alternative numbers of non- participants (i.e., 5, 10, 20, 30) in the same group, using the nearest neighbor matching method (i.e., those with the lowest absolute differences between their propensity score and that of the participant). The outcome variables considered were earnings and employment in the 12th month following completion of the program. The estimator used was the "nearest neighbor cross-sectional simple average matching estimator" (Todd 1999).112 Estimates of sample standard deviations of the impact estimates were obtained using bootstrapping. 5.7.3 Measurement of costs The study separated the program's costs into direct and indirect components. Direct costs were defined to include the cost of program training services provided by competitively selected providers, the cost of employee insurance during the internships, and the cost of stipends provided to the trainees (for example, the stipends paid to female participants 111 The samples were re-weighted prior to estimation with the sample of 3,340 to correct for choice-based sampling (i.e., the fact that the sample was constructed to have equal numbers of participants and non- participants). 112 As discussed in Section 3.2.3.2, it would be preferable to use a double-differencing (difference-in- difference) estimator to control for fixed effects. However, this would require longitudinal data, which were not available in this study. 79 with children under 5). (As discussed in Section 3.1.2, this last category of direct costs is actually a transfer and should not have been included in the program costs.) The study assumes that the opportunity cost of trainee time is zero, which is a strong assumption.113 The study also assumes a zero deadweight loss for the governmental resources used to fund the program (although a 50% deadweight loss is alternatively assumed in sensitivity analysis reported in the study). Indirect costs were estimated to be 32.7% of direct costs (with an indirect cost ratio of 15% used in sensitivity analysis). 5.7.4 Results and conclusions Estimates of program effects on earnings were statistically significant only for young males (16-20) and adult females (21-35). For young males, the estimated impact of the program on monthly earnings varied from $17.17 to $23.75 depending on the assumptions.114 For adult females, the corresponding estimates varied from $23.40 to $32.40. Estimates of the program's impact on monthly earnings for all four groups combined varied from $15.67 to $21.26. Estimates of program effect on the probability of employment were statistically significant only for adult females, with the estimated effect varying from 0.1035 to 0.1346. The authors conclude that these differences in estimated program impact among groups probably reflect mainly the different labor market conditions facing each group. A cost-benefit analysis (CBA) was done for the program. The CBA was done for two groups of participants, young males and adult females (representing a targeted program) and all four groups combined (representing the actual program). The estimated effect of the program on monthly earnings (the average of the estimates obtained for each group) was assumed to be the program's only benefit and was assumed to remain constant during a period of time that was alternatively assumed to last for 1, 3, 6, 9, 12, 15 years or for an unlimited period of time. Two alternative discount rates were used (5% and 10%). In the case of the four groups combined, representing the actual program, the estimated net present value of benefits was positive with a discount rate of 5% after 12 years (but was still negative with a discount rate of 10% after 15 years).115 With the CBA restricted to young males and adult females, representing a program targeted116 only to these two groups, the estimated net present benefits are positive after 9 years with a discount rate of 5% (or with an indirect cost ratio of 15%) and after 12 years with a discount rate of 10%. 113Presumably, many of the trainees would have been employed in the informal sector or in housework if they had not participated in the program. This assumption also implicitly assumes that interns did not contribute positively to the output of the firms in which they were working for 8 weeks. 114Within the statistically significant groups, the impact estimates were not very sensitive either to the sample/information used to estimate the propensity score functions or to the number of nearest neighbors used (5,10, 20, or 30). This result was surprising in the case of the propensity scores since the three sets of propensity scores were not very highly correlated. 115However, with an indirect cost rate of 15% estimated net present benefits were positive after 15 years even with a discount rate of 10%. In addition, under the basic assumptions used, the program had estimated positive net benefits over an infinite time period. 116This is an example of "characteristic targeting" in the context of the discussion in Section 4.3. 80 If a deadweight loss of 50% of the program's total expenditure is assumed as the distortionary cost of governmental financing, estimated net present benefits of the existing program are non-positive after 15 years, even with a discount rate of 5%, while for a targeted program (i.e., one restricted to young males and adult females), they are positive only after 15 years with a discount rate of 5% (but only when an unlimited time period is assumed with a discount rate of 10%). SECTION 6. MODEL TERMS OF REFERENCE FOR AN ECONOMIC ANALYSIS OF A CONTINUING EDUCATION PROJECT IN SERBIA-MONTENEGRO This section of the Guide considers model terms of references, using as an explicit example for concrete illustration the economic analysis of a hypothetical continuing education project in Serbia and Montenegro. The case study presented in Section 5.5 above is an example of the economic analysis that might be produced in accordance with these model terms of reference. 6.1 PROJECT BACKGROUND AND OBJECTIVES 6.1.1 Country background Although conflict has ended in Serbia-Montenegro and neighboring countries, ethnic tensions remain in most areas and both physical and social capital have been heavily damaged. There are large numbers of refugees and internally displaced persons, and there are also large numbers of recently demobilized military. Many families have lost or suffered damages to their housing, land and other assets during recent conflicts. Property titles are uncertain in many cases. Serbia is in the early stages of trying to rebuild its economy after more than a decade of conflict and economic sanctions. Economic growth is relatively slow compared to some other regions (for example, East, South and Southeast Asia), although progress has been made in stabilizing the economy and in many areas of economic reform. Fiscally, Serbia-Montenegro currently has very limited resources to undertake substantial public investments and is still highly dependent on donor funding of its investment budget. Unemployment rates are currently very high, particularly among youth, the internally displaced, and recently demobilized military. The analysis of the 2002 Serbian Living Standards Measurement Survey (LSMS) indicates that less than one-fifth (17.4%) of youth are working, though the probability of employment increases sharply with age as youth leave school (about 4.0% higher probability with every additional year of age). Among those working, about a quarter have public sector jobs. The low percentages of youth who are working would not be troublesome if it were due to high schooling enrollment rates (and schooling is indeed the leading main activity of Serbian youth). However, that the percentages of youth who are working is substantially below the percentages unemployed (22.1% in Serbia) is indeed cause for concern. 81 Much hope is placed on Serbian youth (and on SEE youth generally) as a possible resource for reconciliation. However, the schools and other public institutions currently appear to exacerbate ethnic tensions rather than heal them. For example, separate sessions are still operated in some public schools for children from different ethnic groups. Public schools are poorly funded and inefficient. Serbia-Montenegro spends only 2.5% of its GDP on education, compared to 5% in Hungary. Because of the large number of personnel, and despite very low salaries, wages and salaries still absorb the major share of education budgets. Relatively little is available for materials, supplies, equipment or maintenance. Household out-of-pocket spending on education is high and exacerbates inequalities in human capital formation by income. Primary schooling is practically universal throughout the region (except among some ethnic minority groups), but secondary enrollment rates fall off steeply with age (and more steeply for some ethnic groups, girls, rural and poor children). Most secondary enrollment is in technical and vocational schools (particularly in poor and rural areas), rather than academic schools. However, the curriculum in vocational and technical schools does not meet employers' needs, while enrolling in these high schools does not facilitate access to higher education. A substantial share of public education budgets is allocated to higher education (including technical and vocational education), but the quality of instruction is poor at this level, financing is inequitable, and institutions are poorly managed. 6.1.2 Project description The project proposes to provide demand-side financial support to poor and vulnerable Serbian youth to enable them to complete secondary school. The project is designed to increase the secondary school completion rate among children aged 20-24 from current levels of 55% to 65% at the end of five years. Vouchers will be provided to poor youth who have completed primary school, including youth under age 18 who have dropped out of school before completing their secondary schooling. The vouchers can be used to pay for tuition and fees at private non-formal (mostly NGO-operated) secondary schools. The project is expected to cost $5 million over 5 years. 6.2 PURPOSE OF THE ECONOMIC ANALYSIS The purpose of the economic analysis is to determine whether the project provides net present benefits comparable to or better than alternative uses of the same resources both within the education sector (for example, quality improvements in formal schooling) and in other sectors (for example, infrastructure and health). The economic analysis should also address whether the approaches used in the project are the most cost-effective alternatives to achieve the desired results. 6.3 PROFILE OF SKILLS REQUIRED The consultant should have a Ph.D. in economics or comparable formal educational qualifications, and should have had some prior experience in the economic evaluation of education or youth projects. The consultant should be fluent in English and have good English-language writing skills. Some prior experience in Serbia-Montenegro or in another South-Eastern European country is desirable but not essential. 82 6.4 RESOURCES 6.4.1 Key reference documents The following documents are expected to be particularly helpful in completing this assignment: Aoki, Aya, Luis Crouch, Peter Easton and Maman Sidikou, 2003, "Rethinking World Bank Policy and Practice in Support of Adult and Nonformal Education," Working Paper, Washington, DC: The World Bank (May). Behrman, Jere R. and James C. Knowles, 2003, "Economic Evaluation of Investments in the Youth in Selected SEE Countries," Report prepared for the World Bank/ECA Social Development Initiative, Washington, DC. Processed (December). Belli, Pedro, J. Anderson, H. Barnum, J. Dixon and J.P. Tan, 1998, Handbook on Economic Analysis of Investment Operations, Washington, DC: The World Bank (January). Knowles, James C. and Jere R. Behrman, 2004, "A Practical Guide to Economic Analysis of Youth Projects," Washington, DC: Human Development Network of the World Bank. Kolev, Alexandre, 2003, "Addressing the Problem of Youth Unemployment in South- East Europe: Evidence and Practices," Background Note for the Regional Study on Youth Empowerment and Social Inclusion in SEE, ECSPE. Lokshin, Michael and Branko Jovanovic, 2003, "Wage Differentials and State-Private Sector Employment Choice in the Federal Republic of Yugoslavia," Policy Research Working Paper 2959, Washington, DC: The World Bank (January). Palacios, Miguel, 2003, "Options for Financing Lifelong Learning," Policy Research Working Paper 2994, Washington, DC: The World Bank (March). Ringold, Dena, Mitchell A. Orenstein and Erika Wilkens, 2003, "Roma in an Expanding Europe: Breaking the Poverty Cycle," Washington, DC: The World Bank. World Bank, 2001, "Alternative Schools and Roma Education: A Review of Alternative Secondary School Models for the Education of Roma Children in Hungary," World Bank Regional Office Hungary, NGO Studies No. 3 (September). World Bank, 2003, "Serbia and Montenegro: Public Expenditure and Institutional Review," Report No. 23689-YU, Washington, DC (February). World Bank, 2003, "Education: Policies to Combat Poverty," Chapter 4, Serbian Poverty Assessment (draft), Washington, DC. 6.4.2 Description of key data sets The key data set for this assignment is expected to be the 2002 Serbian Living Standards Measurement Survey (LSMS), which can be downloaded from the World Bank's Living Standards Measurement Survey (LSMS) website. This survey includes 2,555 youth aged 15-24 (62% residing in urban areas) as well as 10,715 adults aged 25-64 (58% urban). The Serbian LSMS includes a wide range of data, including demographic characteristics (e.g., age, sex, marital status, urban-rural residence, place of residence in 1991), education (e.g., current school enrolment, grades completed, vocational/technical schooling, and expenditure on schooling), health (for example, chronic and recent acute 83 illnesses, visits to health providers during the past month, and expenditure on health care), social protection (e.g., amount received monthly in social and child assistance and humanitarian aid), labor market activities (e.g., main activities, hours worked and earnings per month, and sector of employment). The main activities of sample youth are student (56%), unemployment (22%), work (17%), housework (2%), and other (2%); while those of sample adults are work (58%), other (17%), unemployment (13%), housework (10%), and student (2%). 6.4.3 Key contacts The consultant will participate as a team member of the project design team under the direct supervision of the Task Team Leader (TTL). The consultant is expected to work closely with other team members (particularly with the team's financial and M&E specialists) and with the Project Preparatory Team (PPT) based at the Ministry of Education. The PPT will assist the consultant in arranging special meetings related to his/her assignment in addition to those already arranged for the project design team. 6.5 SCOPE OF WORK During the consultant's initial mission, the timing of which should be agreed with the TTL, he/she should review relevant documents and existing data sets and prepare a detailed outline/analysis framework for the final report that includes a description of the data and methodologies to be used to measure and compare the project's costs, effectiveness and benefits (see suggested outline below). The analysis framework should also include a detailed description of any field work and data collection that will need to be done by the PPT. A draft of the analysis framework will be due at the end of the consultant's initial mission. During the consultant's initial mission, he/she is also expected to discuss informally with the TTL and other team members any ideas for how the project might be modified to enhance its economic value. The draft economic analysis will be prepared during the consultant's second mission, the timing for which should be agreed with the TTL. The draft economic analysis will address the points in the approved outline and will conform to World Bank guidelines concerning the preparation of economic analyses (for example, Belli, et al. 1998 cited above). The draft economic analysis should also describe any adjustments that have been made during the project design to enhance its economic value. Following the receipt of comments from the World Bank, the consultant will finalize the economic analysis within 15 days. 6.6 SUGGESTED OUTLINE OF THE ECONOMIC ANALYSIS The following outline (which can be modified by the consultant, subject to the approval of the Task Team Leader) is suggested for the economic analysis: 1. The project's relationship to the Country Assistance Strategy (CAS), the country context and previous economic and sector work 2. Description of the project and analysis of alternative project designs 84 3. The project's overall economic evaluation (based on cost-benefit analysis and/or other selection criteria) 4. The project's fiscal impact and cost recovery options 5. Sensitivity/risk analysis 6. Institutional capacity 7. Poverty and gender analysis 8. Economic performance criteria (including indicators and targets) and M&E plan during implementation 6.7 LEVEL OF EFFORT AND BUDGET It is expected that the assignment will require 45 days of the consultant's time, including 15 days for an initial mission and 22 days for the second mission (and including eight days for initial review of documents and finalization of the report) 85 86 REFERENCES Adhikari, Ramesh, Paul Gertler and Anneli Lagman, 1999, "Economic Analysis of Health Sector Projects--A Review of Issues, Methods, and Approaches," Manila: Asian Development Bank (March). Aedo, Cristián and Sergio Nuñez, 2001, "The Impact of Training Policies in Latin America and the Caribbean: The Case of `Programa Joven,'" Processed, Georgetown University (May). Alderman, Harold, Jere R. Behrman, David Ross, and Richard Sabot, 1996, "The Returns to Endogenous Human Capital in Pakistan's Rural Wage Labor Market," Oxford Bulletin of Economics and Statistics 58:1, 29-55. Alva, Soumya, Edmundo Murrungarra and Pierella Paci, 2002, "The Hidden Costs of Ethnic Conflect: Decomposing Trends in Educational Outcomes of Young Kosovars," Policy Research Working Paper 2880, Washington, DC: The World Bank (August). Angelucci, Manuela, Orazio P. Attanasio and Jonathan Shaw, 2004, "The Effect of Oportunidades on the Level and Composition of Consumption in Urban Areas, " London: University College London (Technical Document on the Evaluation of Oportunidades 2004). Angrist, Joshua D., Eric Bettinger, Erik Bloom, Elizabeth King and Michael Kremer, 2002, "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review 92:5 (December), 1535-59. Aoki, Aya, Luis Crouch, Peter Easton and Maman Sidikou, 2003, "Rethinking World Bank Policy and Practice in Support of Adult and Nonformal Education," Working Paper, Washington, DC: The World Bank (May). Ashenfelter, Orley and Michael Greenstone, 2004, "Using Mandated Speed Limits to Measure the Value of a Statistical Life," The Journal of Political Economy 112(11):S226-68. Attanasio, Orazio, Costas Meghir and Ana Santiago, 2002, "Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to evaluate Progresa," London: University College London, mimeo. Ballard, Charles, John Shoven, and John Whalley, 1985, "General Equilibrium Computations of the Marginal Welfare Costs of Taxes in the United States," American Economic Review 75(1):128-38. Barnum H., 1995, "Economic Evaluation of Health Projects," Washington, DC: World Bank. Becker, Gary S., 1967, "Human Capital and the Personal Distribution of Income: An Analytical Approach," Ann Arbor: University of Michigan, Woytinsky Lecture, republished in Gary S. Becker, Human Capital, New York: NBER, 2nd edition 1975, 94-117. Behrman, Jere R., Harold Alderman and John Hoddinott, 2004, "Hunger and Malnutrition" in ed., Bjorn Lomborg, Global Crises, Global Solutions, Cambridge, UK: Cambridge University Press, forthcoming (draft available at www.copenhagenconsensus.com). 87 Behrman, Jere R., Yingmei Cheng and Petra Todd, 2004, "Evaluating Preschool Programs when Length of Exposure to the Program Varies: A Nonparametric Approach," Review of Economics and Statistics 86:1 (February 2004), 108-132. Behrman, Jere R. and Anil B. Deolalikar, 1995, "Are There Differential Returns to Schooling by Gender? The Case of Indonesian Labor Markets," Oxford Bulletin of Economics and Statistics 57:1 (February), 97-118. Behrman, Jere R. And John Hoddinott, 2005, "Program Evaluation with Unobserved Heterogeneity and Selective Implementation: The Mexican Progresa Impact on Child Nutrition," Oxford Bulletin of Economics and Statistics. Behrman, Jere R., John Hoddinott, John A. Maluccio, Agnes Quisumbing, Reynaldo Martorell and Aryeh D. Stein, "The Impact of Experimental Nutritional Interventions on Education into Adulthood in Rural Guatemala: Preliminary Longitudinal Analysis," Philadelphia-Washington-Atlanta: University of Pennsylvania, IFPRI, Emory, processed, 2003. Behrman, Jere R. and James C. Knowles, 2003, "Economic Evaluation of Investments in Youth in Selected SEE Countries," Report prepared for the Social Development Initiative/ECA, Washington, DC: The World Bank. Processed. Behrman, Jere R., Susan W. Parker and Petra E. Todd, 2004, "Medium-Term Effects of the Oportunidades Program Package, including Nutrition, on Education of Rural Children Age 0-8 in 1997," Philadelphia, PA: University of Pennsylvania (Technical Document Number 9 on the Evaluation of Oportunidades 2004). Behrman, Jere R., David Ross and Richard Sabot, 2004, "Improving the Quality Versus Increasing the Quantity of Schooling: Evidence from Rural Pakistan," Topics in the Economics and Growth of Developing Areas of The B.E. Journals of the Economics and Growth of Developing Areas (forthcoming). Behrman, Jere R., Piyali Sengupta and Petra Todd, 2000, "The Impact of PROGRESA on Achievement Test Scores in the First Year," Washington, DC: International Food Policy Research Institute, processed. Behrman, Jere R., Piyali Sengupta and Petra Todd, 2005, "Progressing through PROGRESA: An Impact Assessment of Mexico's School Subsidy Experiment," Economic Development and Cultural Change (July). Behrman, Jere R. and Emmanuel Skoufias, 2004, "Evaluation of PROGRESA/Oportunidades: Mexico's Anti-Poverty and Human Resource Investment Program," in Jere R. Behrman, Douglas Massey and Magaly Sanchez R, eds., The Social Consequences of Structural Adjustment in Latin America, book manuscript. Belli, Pedro and Lant Pritchett, 1995, "Does Good Economic Analysis Lead to Better Projects?" HCO Dissemination Notes No. 53, Washington, DC: The World Bank. Belli, Pedro, J. Anderson, H. Barnum, J. Dixon and J.P. Tan, 1998, Handbook on Economic Analysis of Investment Operations, Washington, DC: The World Bank (January). Buddelmeyer, H., and E. Skoufias, 2003, "An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA," IZA Discussion Paper No. 827 (July), Institute for the Study of Labor (IZA), Bonn, Germany. 88 Claeson, M, T. Mawji and C. Walker. 2000. Investing in the Best Buys: A Review of the Health, Nutrition and Population Portfolio, FY1993-1999. Washington, DC: The World Bank. Coady, David P. and Rebecca L. Harris, 2001, "Evaluating Transfer Programs within a General Equilibrium Framework," Washington, DC: International Food Policy Research Institute, mimeo. Cohen, M.A., 1998, "The monetary value of saving a high risk youth," Journal of Quantitative Criminology, 14(1), 5-33. Commission on Macroeconomics and Health, 2001, Macroeconomics and Health: Investing in Health for Economic Development, Geneva: World Health Organization. Curtain, Richard, 2004, "The Case for Investing in Young People as Part of a National Poverty Reduction Strategy," Paper commissioned by the United Nations Population Fund, New York. Devarajan, S., L. Squire and S. Suthiwart-Narueput, 1997, "Beyond Rate of Return: Reorienting Project Appraisal, The World Bank Research Observer 12(1):35-46. Deininger, Klaus, Lyn Squire and Swati Basu, 1998, "Does Economic Analysis Improve the Quality of Foreign Assistance?" World Bank Economic Review 12(3):385-418. Dickson, John and Stefano Pagiola, 1998, "Economic Analysis and Environmental Assessment," Environmental Assessment Sourcebook Update, Number 23, Environmental Department. Washington, DC: The World Bank (April). Dickson, Rumona, Shally Awasthi, Paula Williamson, Colin Demellweek, and Paul Garner, 2000,. "Effect of Treatment for Intestinal Helminth Infection on Growth and Cognitive Performance in Children: Systematic Review of Randomized Trials," British Medical Journal, 320 (June 24), 1697-1701. Duflo, Esther, 2001, "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment," American Economic Review 91:4 (September), 795-813. Feldstein, Martin, 1995, "Tax Avoidance and the Deadweight Loss of the Income Tax," Cambridge, MA: National Bureau of Economic Research Working Paper No. 5055. Gelbach, Jonathan B. and Lant H. Pritchett, 1997, "More for the Poor Is Less for the Poor: The Politics of Targeting," Policy Research Working Paper 1799, Washington, DC: The World Bank (July). Gwatkin, Davidson, 2000, "Targeting Health Programs to Reach the Poor," Washington, DC: The World Bank (February). Hammer, Jeffrey S., 1996, "Economic Analysis for Health Projects," Policy Research Working Paper No. 1611, Washington, DC: The World Bank (May). Harberger, Arnold C., 1997, "New Frontiers in Project Evaluation? A Comment on Devarajan, Squire and Suthiwart-Narueput," The World Bank Research Observer 12(1):73-79. King, Elizabeth M. and Andrew Mason, 2000, Engendering Development: Gender Equality in Rights, Resources, and Voice, New York: Oxford University Press for the World Bank. Kolev, Alexandre, 2003, "Addressing the Problem of Youth Unemployment in South- East Europe: Evidence and Practices," Background Note for the Regional Study on Youth Empowerment and Social Inclusion in SEE, ECSPE. 89 Knowles, James C. and Jere R. Behrman, 2003a, "Assessing the Economic Returns to Investing in Youth in Developing Countries," Bangkok/Philadelphia, PA: University of Pennsylvania, mimeo, for World Bank Human Development Network. Knowles, James C. and Jere R. Behrman, 2003b, "The Economic Returns to Investing in Youth in Developing Countries: A Review of the Literature," Bangkok/Philadelphia, PA: University of Pennsylvania, mimeo, for World Bank Human Development Network. Lokshin, Michael and Branko Jovanovic, 2003, "Wage Differentials and State-Private Sector Employment Choice in the Federal Republic of Yugoslavia," Policy Research Working Paper 2959, Washington, DC: The World Bank (January). McGuire, Judith S., 1996, "The Payoff from Improving Nutrition," Unpublished mimeo. Miguel, Edward and Michael Kremer, 2004, "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica 72(1):159-217. Miguel, Edward and Michael Kremer, 2002, "Why Don't People Take Their Medicine: Experimental Evidence from Kenya," Harvard University and University of California-Berkeley, mimeo. Morley, Samuel A. and David Coady, 2003, From Social Assistance to Social Development: Targeted Education Subsidies in Developing Countries , Washington, DC: International Food Policy Research Institute. National Academy of Science-National Research Council (NAS-NRC), 2005, Transitions to Adulthood in Developing Countries (preliminary title), Washington, DC: National Academy of Science Press. Over, Mead, 1991, "Economics for Health Sector Analysis: Concepts and Cases," Economic Development Institute, Washington, DC: The World Bank. QED Group, 2003, "Study on Youth Risky Behavior in South Eastern Europe," Report submitted to the World Bank, Washington, DC (June 5). Palacios, Miguel, 2003, "Options for Financing Lifelong Learning," Policy Research Working Paper 2994, Washington, DC: The World Bank (March). Parker, Susan W., 1999, "Explaining Differences in Returns to Education in 39 Mexican Cities," Mexico City: unpublished manuscript, PROGRESA Parker, Susan and Emmanuel Skoufias,. 2000, "The Impact of PROGRESA on Work, Leisure, and Time Allocation," Washington, DC:. International Food Policy Research Institute. Parker, Susan W., Jere R. Behrman and Petra E. Todd, 2004, "Medium-Term Effects on Education, Work, Marriage and Migration in Rural Areas," Philadelphia, PA (Technical Document Number 1 on the Evaluation of Oportunidades 2004 conducted by INSP). Partnership for Child Development, 1999, "The cost of large-scale school health programs which deliver antihelminthics to children in Ghana and Tanzania," Acta Tropica 73: 183-204. Phillips, Margaret and Tina G. Sanghvi, 1996, "The Economic Analysis of Nutrition Projects: Guiding Principles and Examples," Washington, DC: The World Bank (May). Population Council and Family Health International, 2003, "New Findings from Intervention Research: Youth Reproductive Health and HIV Prevention," Meeting Report (September 9), Washington, DC, FRONTIERS, Horizons and YouthNet projects. 90 Psacharopoulos, George, 1994, "Returns to Investment in Education: A Global Update," World Development 22:9 (September), 1325-1344. Rao, Vijayendra, 1993, "The Rising Price of Husbands: A Hedonic Analysis of Dowry Increases in Rural India," Journal of Political Economy 101:4 (August), 666-677. Ravallion, Martin and Quentin Wodon, 2000, "Does child labor displace schooling? Evidence from behavioral responses to an enrollment subsidy," Economic Journal 110(462). Rosenzweig, Mark R. and Kenneth J. Wolpin, 1986, "Evaluating the Effects of Optimally Distributed Public Programs," American Economic Review 76:3 (June), 470-487. Ross, David and Bernadette Cleophas-Mazige, 2003, "Results from a Community Randomized Trial in Rural Tanzania: the MEMA kwa Vijana Project," in Population Council and FHI (2003), pp. 21-22. (Also see presentation at http://www.fhi.org/en/Youth/YouthNet/NewsEvents/researchmtg.htm). Rubalcava, Luis, Graciela Teruel and Duncan Thomas, 2002, "Welfare Design, Women's Empowerment and Income Pooling," Mexico City: CIDE, mimeo. Schultz, T. Paul, 1988, "Education Investment and Returns," in Handbook of Development Economics, Vol. I, eds. H. Chenery and T. N. Srinivasan, Amsterdam: North Holland Publishing Co. Schultz, T. Paul, 1993, "Investments in the Schooling and Health of Women and Men: Quantities and Returns," Journal of Human Resources 28:4 (Fall), 694-734. Schultz, T. Paul, 2000a, "The Impact of PROGRESA on School Attendance Rates in the Sampled Population," Washington, DC: International Food Policy Research Institute, processed. Schultz, T. P., 2000b, "School Subsidies for the Poor: Evaluating a Mexican Strategy for Reducing Poverty," Washington, DC: International Food Policy Research Institute. Schultz, T. Paul, 2004,. "School Subsidies for the Poor: Evaluating the Mexican Progresa Poverty Program," Journal of Development Economics 74:2 (June) Skoufias, Emmanuel and Susan W. Parker, 1991, "Conditional Cash Transfers and Their Impact on Child Work and Schooling: Evidence from the PROGRESA Program in Mexico" Economia 2:1(Fall 2001). Summers, Lawrence H., 1992, "Investing in All the People," Pakistan Development Review 31:4, 367-406. Summers, Lawrence H., 1994, "Investing in All the People: Educating Women in Developing Countries," Washington, DC: World Bank, Economic Development Institute Seminar Paper No. 45. Sundaram, Sandhya, J. Epp, N. Ooman and J. Rosen, 2004, "A Review of Population, Reproductive Health, and Adolescent Health & Development in Poverty Reduction Strategies," Draft report, Washington, DC: The World Bank (January). Todd, Petra A., 1999, "A Practical Guide to Implementing Matching Estimators," Processed presentation at IADB meetings in Santiago, Chile (October). Todd, Petra A., 2004, "Technical Note on Using Matching Estimators to Evaluate the Opportunidades Program For Six Year Follow-up Evaluation of Oportunidades in Rural Areas," Philadelphia: University of Pennsylvania, mimeo. Todd, Petra E., Jorge Gallardo-Garcia, Jere R. Behrman and Susan W. Parker, 2004, "Oportunidades Impacts on Education in Urban Areas," Philadelphia, PA: University of Pennsylvania (Technical Document Number 2 on the Evaluation of Oportunidades 2004). 91 Todd, Petra and Kenneth Wolpin, 2003, "Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility: Assessing the Impact of a School Subsidy Program in Mexico," Philadelphia, PA" University of Pennsylvania, Processed. UNFPA, 2004, "Programming for Adolescent Sexual and Reproductive Health in RCH 2," Draft report for MOHFW, New Delhi (April). Van der Gaag J. and J.P. Tan, 1997, "The benefits of early child development programs, an economic analysis," Human Development Network, Washington DC: The World Bank. Vawda, Ayesha, Peter Moock, J. Price Gittinger and Harry Patrinos, 1999, "Economic Analysis of World Bank Education Projects and Project Outcomes," Economics of Education Thematic Group, Washington, DC: The World Bank. Were, Maureen and Nafula, Nancy N., "An Assessment of the Impact of HIV/AIDS on Economic Growth: The Case of Kenya" (September 2003). CESifo Working Paper Series No. 1034. http://ssrn.com/abstract=449241 World Bank, 1995, World Development Report 1995. Washington, DC and New York: Oxford University Press. World Bank, 1997, "Violence and Urban Poverty in Jamaica: Breaking the Cycle," Report No. 15895-JM, Washington, DC (January). World Bank, 1999, Curbing the Epidemic: Governments and the Economics of Tobacco Control, Washington, DC: The World Bank. World Bank, 2000a, "Trinidad and Tobago Youth and Social Development," Report No. 20088-TR, Washington, DC (June). World Bank, 2000b, "HIV/AIDS in the Caribbean: Issues and Options," Report No. 20491-LAC, Washington, DC (June). World Bank, 2001a, "The Dominican Republic Poverty Assessment," Report No. 21306- DR, Washington, DC (January). World Bank, 2001b, "Alternative Schools and Roma Education: A Review of Alternative Secondary School Models for the Education of Roma Children in Hungary," World Bank Regional Office Hungary, NGO Studies No. 3 (September). World Bank, 2001c, "Engendering Development: Through Gender Equality in Rights, Resources, and Voice," Policy Research Report, Washington, DC: The World Bank. World Bank, 2002a, School Health Toolkit. Washington, DC: The World Bank. World Bank, 2002b, "Review of World Bank supported projects in adult basic education and literacy, 1997-2002." Washington, DC: The World Bank. World Bank, 2002c, "Optimizing the allocation of resources among HIV prevention interventions in Honduras," Washington, DC: The World Bank. World Bank, 2002d, "Integrating Gender into the World Bank's Work: A Strategy for Action," Washington, DC (January). World Bank, 2002e, "A Review of Gender Issues in the Dominican Republic, Haiti and Jamaica," Report No. 21866-LAC, Washington, DC (May). World Bank, 2002f, "Integrating Gender into the World Bank's Work: A Strategy for Action," Washington, DC: The World Bank (January). World Bank, 2002g, "Skills and Literacy Training for Better Livelihoods: A Review of Approaches and Experiences," Africa Region Human Development Working Paper Series, Washington, DC: The World Bank. 92 World Bank, 2003, "Caribbean Youth Development: Issues and Policy Directions," Draft report, Caribbean Country Management Unit, Washington, DC (February). World Bank, 2004, "Children and Youth: A Framework for Action," Draft, Washington, DC. 93 About this series... This series is produced by the Health, Nutrition, and Population Family (HNP) of the World Bank's Human Development Network. The papers in this series aim to provide a vehicle for publishing preliminary and unpolished results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual authors whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Managing Editor Rama Lakshminarayanan (rlakshminarayana@worldbank.org) or HNP Advisory Service (healthpop@worldbank.org, tel 202 473-2256, fax 202 522-3234). For more information, see also www.worldbank.org/hnppublications. THE WORLD BANK 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org