63882 Human Development: Lessons from Evaluation Global Program Review of the Global Fund to Fight AIDS, TB, and Malaria Contents IEG is working on the review of Global Fund to Fight AIDS, TB, and Malaria as part of its ongoing reviews of global partnership programs in which the Bank is involved. Previous reviews have included the Stop TB Partnership, the Global Forum for Health Research, and the 1. What’s happened to World Bank support to 5. What can we learn from nutrition impact evalua- Medicines for Malaria Venture–which are available at www.globalevaluations.org. The global program review focuses on lessons about education over the past decade? tions? the Bank’s engagement with the Global Fund at the country level. The �ndings are based on extensive interviews at the Global Fund 2. What are the longer-term effects of conditional cash 6. Raising learning while expanding access to and the World Bank, missions to seven countries (already completed), and a survey (currently being administered) of task team leaders transfers on human capital? schooling in Tanzania: success for primary but a of World Bank health projects in countries in which the Global Fund has been active. A similar survey is also being administered to fund 3. Poverty Reduction Support Operations support, but challenge for secondary education portfolio managers at the Global Fund. Both surveys seek the views of task team leaders (or fund portfolio managers) on the breadth of rarely replace, education lending 7. Upcoming Evaluations the World Bank-Global Fund engagement since the Global Fund was established in 2002. 4. Do health sector-wide approaches (SWAps) achieve results? What’s happened to World Bank support to declining from 82 to 69 percent of projects. Bank support education over the past decade? has been most successful at increasing access to education and improving its equity, whereas fewer than half of projects From 2001–10, World Bank commitments to education have achieved education quality or labor force, management, totaled $23 billion and doubled on an annual basis. The Bank’s learning, or efficiency objectives. Improving the quality of education strategy has evolved from a focus on basic education education inputs has not necessarily improved learning. at the beginning of the decade to a dual focus on universal Support to post-primary education has expanded, but the share primary completion and post-primary education “for the of projects with labor market objectives has not, suggesting knowledge economy� during the second half of the decade. The possible issues in the labor market relevance of post-primary most recent Bank strategy, just launched, focuses on Learning for efforts. All. Achieving the new strategy’s objective of Education for All will An IEG review of the education lending portfolio found that involve substantial challenges. Only about a third of the recently Education Sector projects most often aimed to improve the closed projects with learning outcome objectives substantially quality of education inputs, increase access to schooling, and achieved them. More information can be found in World Bank improve the equity and efficiency of education; one in �ve Support to Education Since 2001: A Portfolio Note. projects had an explicit objective to improve learning. There was no clear trend in learning objectives, although projects have increasingly �nanced learning assessments over the decade. Most projects with substantial education components that were Photos courtesy of Curt Carnemark / World Bank, John Issac / World Bank and Jouni Martti Eerikainen / IEG managed by other sectors did involve Education Sector staff in their preparation or supervision, especially if there was an education objective. The Social Protection Sector had the largest portfolio of projects with large education components. The review found, however, that the performance of Education Sector worsened over the decade, with satisfactory ratings www.worldbank.org/ieg localized shocks, such as after Hurricane Ivan. For more information, and, third, better documenting the costs and effectiveness of Poverty Reduction Support Operations support but rarely replace human development sector lending download the evaluation “Jamaica Social Safety Net Project and interventions. In sum, in approaching the impact evaluation Poverty Reduction Support Operations (credits and grants) provide budget support to IDA (International Development Association) National Community Development Project�. literature on nutrition, the question should not simply be, “What countries and are anchored in country-owned development strategies. They provide broad-based programmatic support with an works?� but rather, “Under what conditions does it work, for whom, emphasis on poverty-reducing growth and pro-poor service delivery, including education. IEG’s 2010 evaluation of PRSOs found that Also in Jamaica, the National Community Development Project what part of the intervention works, and for how much?� To learn more than 95 percent of the 72 PRSO operations approved from FY01 to 2007 had health and education objectives or actions in their (NCDP) complemented a Bank emergency rehabilitation loan more, download the report on What Can We Learn from Nutrition policy matrix. It was expected that PRSOs would eventually replace education sector lending in countries like Benin, Cape Verde, and was implemented by the Jamaica Social Investment Fund. Impact Evaluations? Madagascar, Mozambique, Tanzania, and Uganda. However, as of 2008, education sector lending had been replaced only in Cape It aimed to provide basic services and temporary employment Verde. PRSOs were expected to replace health lending in nine countries, but as of 2008 this had occurred in only two. Most of the in low-income communities, and promoting greater social and Do health sector-wide approaches (SWAps) countries that experimented with funding education or health projects only through PRSOs eventually went back to launching parallel community development among the poor. The project provided investment projects. achieve results? well-targeted and good quality basic services. Fifty-nine percent of subprojects were in the poorest communities. The project More than a decade ago, the World Bank and other donors The evaluation found that PRSOs have played a strong supporting role in sector investment lending but are limited as an instrument increased access of poor communities to social services and proposed a new way of working with developing country to replace sector lending. They have strengthened the dialogue between central ministries and sector agencies and been able to roads. However, maintenance of the infrastructure was poor governments to overcome inefficiencies, lack of government promote high-level actions that were sometimes difficult to pursue within the sector. They also have raised cross-cutting issues and because of the lack of training. Although it was an objective to ownership, and other problems constraining the impact of brought attention to sector budgets. However, surveys of PRSO and sector operations staff suggest that the depth of the engagement create employment, data on employment opportunities were development support. This new sector-wide approach (SWAp) and dialogue may have been limited and that they are not a good instrument for tackling sector details. not collected. Achievement of the second objective, to promote embraced many of the principles of harmonization and alignment. greater social and community development, was modest. For more The evaluation recommended that PRSO sector content focus on cross-sectoral or central ministry issues critical to facilitating key information, download the evaluation “Jamaica Social Safety Net A comprehensive review of lessons learned from Bank-supported sector reforms and strengthening sector budget processes. Complementary parallel sector lending remains important for addressing Project and National Community Development Project�. health SWAps in six countries -- Bangladesh, Ghana, Kyrgyz detailed technical issues and facilitating program ownership by line ministries. Republic, Malawi, Nepal, and Tanzania -- posed four questions: (1) Were the anticipated bene�ts of the approach realized? (2) Source: IEG. 2010. Poverty Reduction Support Credits: An Evaluation of World Bank Support. World Bank, Washington, DC What can we learn from nutrition impact Were the objectives of the national health strategies achieved? evaluations? (3) Did the approach facilitate the achievement of national health What are the longer-term effects of conditional In Colombia, Familias en Acción was started in 2001–02 as an objectives? and (4) In what ways did working through a SWAp cash transfers on human capital? emergency safety net program in response to a major economic High levels of malnutrition in developing countries contribute to affect the World Bank’s efficacy? crisis. It then became a key element in the country’s poverty mortality and have long-term consequences for children’s cognitive Conditional cash transfers (CCTs) are programs under which poor reduction strategy by improving children’s education and health. development and earnings as adults. In 2010 IEG reviewed 46 The review found that health SWAps in these countries have been families get a stipend for taking certain actions, for example, IEG’s impact evaluation found that the program helped increase recent impact evaluations of interventions and programs to largely successful in putting in place critical tools and processes keeping their children in school or taking them for health checks. the likelihood that participant children complete high school, on improve child anthropometric outcomes – height, weight, and for improved sector coordination and oversight. They helped There is signi�cant evidence showing that CCTs have positive average, by 4 to 8 percentage points. Additionally, the program birth weight. Speci�cally, the review asked: establish new, country-led partnerships and dynamics between short-term impacts on school participation, but little is known appears to have had larger impacts on girls’ and rural students’ government and a consortium of development partners. However, about their long-term impacts on human capital. In 2010, IEG high school completion. The evaluation also found that the test • What can be said about the impact of different interventions in most of the countries, national health objectives were only evaluated the impact of Bank-supported CCT projects in Pakistan scores of program recipients who graduated from high school on children’s anthropometric outcomes? modestly achieved under the SWAp. The effect of the SWAp on and Colombia and conducted a �eld evaluation of a CCT project in were similar to the ones of poor non-recipient graduates. More • How do these �ndings vary across settings, and what accounts achieving national health objectives depended on four critical Jamaica. information can be found in IEG’s impact evaluation “Assessing the for this variability? Long-Term Effects of Conditional Cash Transfers on Human Capital: • What is the evidence of cost-effectiveness of the In Pakistan, IEG evaluated the Female School Stipend Program Evidence from Colombia.� interventions? (FSSP), which was supported by the Bank and other donors • What have the lessons from impact evaluations of Bank- under the Punjab Education Sector Reform Program to raise girls’ To strengthen an existing social safety net that suffered from supported programs with anthropometric impacts been? participation in schools and decrease gender inequities, especially weak targeting and high overheads, the Jamaican government at the middle school level. It was implemented in 15 of the lowest launched a reform program in 2000 focused on children. The World The review found that distinct interventions can improve children’s literacy districts in Punjab in late 2003. The FSSP provided quarterly Bank supported the reforms through the Social Safety Net Project, anthropometric outcomes, but that similar interventions can have subsidies of about $10 to the families of girls enrolled in middle which introduced the Program of Advancement through Health widely different results, depending on differences in local context, school if the girls attended at least 80 percent of classes. In 2006, and Education (PATH). PATH provided cash transfers to families the causes and severity of malnutrition, and the capacity for the stipend was extended to girls enrolled in grades 9 and 10. IEG’s with children up to 17 years old, pregnant and lactating women, program implementation. 2010 evaluation found that girls who had been in the program and the elderly and disabled, conditional on regular school for up to four years were more likely to progress through and attendance and health check-ups. IEG’s assessment of the project The review concluded that the relevance, policy, and program complete middle school, delay going into the labor force, increase found that the project achieved its objective to improve social impact of nutrition impact evaluations could be enhanced in three marriage age, and have fewer children than girls in non-stipend assistance through better-targeted cash transfers. It also provided ways: �rst, by collecting rich data on service delivery, behavioral districts. Visit IEG’s Web site to download Do Conditional Cash a more cost-effective social assistance system and reached the outcomes, and implementation processes to better understand Transfers Lead to Medium-Term Impacts? Evidence from a Female poor and vulnerable. The results showed that CCT programs the causal chain and what part of the chain is weak; second, by School Stipend Program in Pakistan report. such as PATH can be useful tools for protecting the poor after analyzing the distribution of impact, especially among the poor; www.worldbank.org/ieg factors – prioritized and phased programs of work; the strength Raising learning while expanding access to Upcoming evaluations of interest to human development of local capacities and systems for common implementation schooling in Tanzania: success for primary but a arrangements; the character of the partnerships within the SWAp; Evaluation of World Bank Support for Social Safety Nets and the predictability, flow, and use of health sector resources. The challenge for secondary education IEG is �nalizing an evaluation of the Bank’s support to social safety nets over the last evaluation highlights that the World Bank has an opportunity to decade—FY00-10. Social safety nets—programs designed to protect the poor and Tanzania managed to raise both primary school enrollment and further strengthen and exploit its areas of comparative advantage vulnerable from shocks and help reduce long-term poverty—are important elements of school quality, resulting in improved pass rates, but this proved crucial to the success of SWAps. For more information and to a country’s poverty reduction and growth strategy. The critical need for such programs much more difficult for secondary education, according to IEG’s download the working paper, please visit IEG’s Web site. in all countries, especially in times of crisis, has been underscored over the past decade. 2010 �eld assessment of two projects. It was particularly evident following the food, fuel, and �nancial crises, which pushed an The Tanzania Primary Education Development Program (2001–04) additional 65 million people into extreme poverty. The evaluation looks at how relevant Components of a Good Partnership under a SWAp aimed to improve the quality of primary education, expand access, and effective the Bank has been in helping countries build safety nets that can protect and increase the school retention rate. It provided budget support the poor from shocks, alleviate poverty, and help build human capital. The evaluation Who is in the partnership? to institutionalize capitation and development grants to schools. report is forthcoming in summer 2011. National actors and stakeholders: • Government (central, regional, peripheral levels, Among other things, primary school fees were eliminated, and the government provided a capitation grant for books and learning Social Safety Nets: What Does Impact Evaluation Show? Parliament) materials. As a results of multiple reforms, enrollment rates rose IEG conducted a comprehensive review of the existing impact evaluation literature • Ministry of Health on social safety nets. The review �nds that many safety net interventions, including • Cross-cutting ministries: �nance, civil service, local drastically, the pupil to textbook ratio dropped, the pupil to teacher ratio rose slightly, and the pass rate on the Primary School Leaving conditional and unconditional cash transfers as well as workfare programs, have government, etc. Examination doubled. achieved their primary objectives of raising households’ immediate consumption and • Other relevant sectors: nutrition, education, social income and reducing poverty. In some cases they have also enhanced households’ development, etc. This was not the case for secondary education. The Secondary ability to mitigate the negative effects of shocks. In addition, programs with explicit • Non-governmental human development goals were found to consistently improve the use of educational • For-pro�t and not-for pro�t service providers Education Development Program (2004–08) aimed to increase the secondary school completion rate, improve learning outcomes— and health services and to reduce the burden of labor for children. The evidence is • Other civil society organizations thinner on longer-term human capital improvements. The impact evaluation evidence especially of girls—and improve the management of secondary education. It reduced tuition in government secondary schools by is scarce regarding the contributions of program components, implementation All external development partners, no matter the half and �nanced capitation grants per pupil for quality inputs. The processes, and local contexts to impacts and largely concentrated on conditional cash modality of their support program established scholarship grants to schools for poor families transfers. The report will be available in spring 2011. To do what? ===> To what end? • Negotiate evidence-based policy/strategy ===> and introduced a new curriculum and incentives to attract teachers to under served areas. The number of government-supported Comparing the Effects of Financial Incentives for Primary Health Care in Coherent, coordinated sector policy secondary schools rose almost fourfold and secondary enrollment Argentina and Brazil • Allocate resources to sector priorities ===> Rational/ This evaluation presents evidence on the achievement of objectives in three World flexible resource allocation nearly tripled in four years, leading to shortages in teaching inputs and teachers. Although the number of secondary graduates Bank health operations in Argentina and Brazil. The operations have supported the two • Review sector performance and outcomes ===> Greater governments over the past decade in developing and implementing innovative health focus on results increased, the pass rate necessary to continue study declined substantially and the large gender gap in outcomes persisted. �nancing approaches. They have changed the �nancial incentives at three levels in decentralized health systems, namely (i) from the World Bank to the country’s health How do they interact? Tanzania’s experience points to three lessons in attempting to sector budget, (ii) from the central government to the provincial/municipality level, and • Government in a leadership position, with adequate (iii) from the local government to primary health care providers. Argentina and Brazil capacity and stability to ful�ll its role both expand and improve the quality of secondary education. First, expansion must take into account �scal constraints and the implemented health policy reforms to change health �nancing incentives and the • Development partners in a supportive position, with provision of primary health care, introduced monitoring of patient care data and regular clearly de�ned roles, responsibilities, accountabilities and needs of other subsectors. Second, secondary education is more complex and thus difficult to expand rapidly, involving higher provider performance analysis and independent audits of performance indicators. In capacities teacher quali�cations and specialization in different subject addition, both countries link the intergovernmental �scal transfer to indicator results, • Dialogue and decisions based on the generation of and Argentina introduced fee-for-service payments to contracted providers for evidence rather than conditionality matter; coordination of curriculum, textbooks, and examinations; and inherent management challenges. Finally, rapid expansion of selected maternal and child health care services. Results show the effect on institution • Mutual accountability of all partners for results based on building and governance, and changes in provider behavior. The project performance performance benchmarks for each partner (or group of relatively low-quality secondary schools risks creating a pool of unemployed graduates who are less likely to contribute to poverty assessment will be available by summer 2011. partners) • Clear guidance and mechanisms for managing and reduction and economic growth. For more information, download the evaluation “Tanzania: Human Resource Development Pilot Learning from Post-Primary Education Projects resolving disputes Project, Primary Education Development Program, and Secondary IEG is currently �elding project performance assessments on higher education • Coordination/collaboration around one national health and skills development projects in MENA, Zambia, Chile and India to help inform plan, one M&E framework, one review process. Education Development Program�. upcoming evaluations of World Bank support for post-primary education and youth Source: Denise Vaillancourt. 2009. “Do Health Sector-Wide Approaches unemployment. Findings and lessons from the evaluation of these projects will be Achieve Results?� IEG Working Paper 2009/4. available by June 2011.