Indonesia Long-Term Generasi Impact Evaluation THE WORLD BANK OFFICE JAKARTA Indonesia Stock Exchange Building Tower II/12th Floor Jl. Jend. Sudirman Kav. 52-53 Jakarta 12910 Tel: (6221) 5299-3000 Fax: (6221) 5299-3111 Website: www.worldbank.org/id THE WORLD BANK 1818 H Street NW Washington, DC 20433, USA Tel: (202) 458-1876 Fax: (202) 522-1557/1560 Website: www.worldbank.org Printed in November 2018 The Long-Term Generasi Impact Evaluation is a product of the staff of the World Bank. The findings, interpretations, and conclusions expressed herein do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the Government they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. For any questions regarding this report, please contact Audrey Sacks (asacks@worldbank.org). Copyright Statement: The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/ The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, http://www.copyright.com/. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. ACKNOWLEDGMENTS This Impact Evaluation (IE) was prepared by a core team led by Benjamin Olken (J-PAL/MIT) and Audrey Sacks (Senior Social Development Specialist, World Bank). The authors thank the World Bank’s Generasi Team, including Samuel Clark, Gerda Gulo, Sadwanto Purnomo, Ali Subandoro, Budi Wijoyo, and Rob Wrobel. Special thanks go to Dwitri Amalia, Yuanita Christayanie, Kelik Endarso, Findi Firmanliansyah, Yulia Herawati, Masyhur Hilmy, Donghee Jo, Fera Miasari, Natasha Plotkin, Samuel Solomon, Iis Surtina, Inge Tan, Juan Tellez, Nurul Wakhidah, Fazla Zain, and Siti Zulva for their excellent effort in survey preparation, training, oversight, and data preparation. This report was edited by Kelley Friel. The Government of Indonesia—National Development Planning Agency (Bappenas), Coordinating Ministry of Human Development and Cultural Affairs, Ministry of Villages, Disadvantaged Areas and Transmigration, Ministry of Health, Ministry of Education and Culture, and Ministry of Home Affairs—provided support and input through the Generasi IE Steering Committee. Special thanks to Ibu Vivi Yulaswati (Bappenas) who has supported this IE from the beginning. The University of Gadjah Mada (UGM), Center for Population and Policy Studies implemented the 2016 survey round as well as the other previous three rounds, which were collected between 2007 and 2009. This report was produced under the overall guidance of Kevin Tomlinson and Susan Wong. Key comments were provided by Deon Filmer, Rema Hanna, Junko Onishi, and Emmanuel Skoufias. The Long-Term Generasi Impact Evaluation is a product of Social, Urban, Rural and Resilience Global Practice’s team in the World Bank Office, Jakarta. Support for this report has been generously provided by the Department of Foreign Affairs and Trade of the Australian Embassy. LONG-TERM GENERASI IMPACT EVALUATION i LIST OF ABBREVIATIONS Bappenas Indonesian Ministry of National Development Planning CCT Conditional cash transfer GoI Government of Indonesia IE Impact evaluation ITT Intent to Treat MIS Management information system MoHA Ministry of Home Affairs MoV Ministry of Villages, Disadvantaged Areas and Transmigration NTT Nusa Tenggara Timur PAUD Early Childhood Education and Development PKH Hopeful Family Program (Keluarga Harapan Program) PNPM  National Community Empowerment Program—Healthy and Smart Generation (Program Nasional Pemberdayaan Masyarakat) PMT Supplementary food SD Elementary School SMP Junior High School SMA Senior High School ii INDONESIA TABLE OF CONTENTS ACKNOWLEDGMENTS i LIST OF ABBREVIATIONS ii TABLE OF CONTENTS iii EXECUTIVE SUMMARY 1 INTRODUCTION 5 Background 5 Results from the 2009 IE 6 Motivation 6 The Generasi Program 7 Village-Level Block Grants 10 Spending Choices 12 Experimental Design 13 EVALUATION DESIGN 15 Methodology 16 Regression Specification 17 Heterogeneity 18 Balance Tests 20 Preanalysis Plan 20 LONG-TERM GENERASI IMPACT EVALUATION iii TABLE OF CONTENTS MAIN RESULTS 21 Direct Benefits of Generasi Funds 21 Program Impact on Main Targeted Indicators 24 Heterogeneity 27 Program Impact on Long-Term Outcomes 28 Program Impact on Secondary Final Outcomes and Nontargeted Outcomes 28 UNDERSTANDING CHANGES SINCE 2009 32 Increase in Other Health and Education Programming 32 Why No Continued Program Impact on Malnutrition Outcomes? 35  Hypothesis 1: General Improvements in Stunting 35  Hypothesis 2: Crowd-In/Crowd-Out Effects 36  Hypothesis 3: Implementation Delays 37 Hypothesis 4: Full Suite of Complementary Interventions    Needed to Address Stunting Were not Fully Implemented 39 CONCLUSION 42 Policy Implications 43 REFERENCES 45 APPENDIX TABLES 47 Appendix Table 1. Questionnaire Modules and Sample Size 47 Appendix Table 2. Direct Benefits 50 Appendix Table 3. Direct Benefits, Provincial Breakdown 52 Appendix Table 4. Program Impact on Main Targeted Indicators (With and Without New Indicators) 54 Appendix Table 5. Program Impact on Main Targeted Indicators, Provincial Breakdown 56 Appendix Table 6. Program Impact on Longer-Term Outcomes 58 Appendix Table 7. Program Impact on Longer-Term Outcomes, Provincial Breakdown 60 Appendix Table 8. Program Impact on Main Targeted Indicators, Interactions with Preperiod Subdistrict-Level Variables, Wave IV 62 iv INDONESIA TABLE OF CONTENTS Appendix Table 9. Program Impact on Longer-Term Outcomes, Interactions with Preperiod Subdistrict-Level Variables, Wave IV 63 Appendix Table 10. Results for Service Provider Quantities 65 Appendix Table 11. Results for Service Provider Quality (Health and Education Infrastructure Availability) 67 Appendix Table 12. Results for Service Provider Level of Effort 69 Appendix Table 13. Results for Community Efforts in Service Provision, Monitoring, and Participation 71 Appendix Table 14. Service Prices and Supply 74 Appendix Table 15. Main Targeted Indicators, Heterogeneity Based on Areas Most in Need 79 Appendix Table 16. Stunting Difference-in-Differences Analysis 81 ANNEX: SUPPLEMENTARY MATERIAL 85 Annex: Table 1. Do Attrition Rates Vary Between Treatment and Control Areas? 85 Annex: Table 2. Program Impact on Main Targeted Indicators, Separated Based on 2007–09 Incentive/Nonincentive Randomization 86 Annex: Table 3. Program Impact on Longer-Term Outcomes, Separated Based on 2007–09 Incentive/Nonincentive Randomization 87 Annex: Table 4. Program Impact on Main Targeted Indicators Limited to Repeated Cross-Sectional Households 88 Annex: Table 5. Program Impact on Longer-Term Outcomes Limited to Repeated Cross-Sectional Households 89 Annex: Figure 1. Visualization of Program Impact on Main Targeted Indicators Showing Trends Over Time and Treatment Effects 90 Annex: Figure 2. Visualization of Program Impact on Longer-Term Outcomes Showing Trends Over Time and Treatment Effects 92 LONG-TERM GENERASI IMPACT EVALUATION v EXECUTIVE SUMMARY Indonesia has made remarkable strides in key human development indicators over the past few decades. Primary school enrollment is close to universal for both boys and girls, and the child mortality rate has declined rapidly (World Bank 2017). Nevertheless, infant mortality, child malnutrition, maternal mortality, and educational learning quality have all remained challenges in Indonesia compared with other countries in the region (World Bank 2015). Furthermore, achievements in these indicators reveal large geographical disparities within Indonesia, with poorer outcomes in rural and remote provinces and districts. Improving access to basic quality health and education services is a key component of the country’s overall poverty reduction strategy. In 2007, the government of Indonesia (GoI) launched two large-scale pilots of programs designed to tackle these issues: 1) conditional cash transfers (CCTs) to households, known as the Hopeful Family Program (Keluarga Harapan Program/PKH), and 2) an incentivized community block grant program, known as the National Community Empowerment Program—Healthy and Smart Generation (Generasi). In 2014, the Generasi program was renamed Bright Healthy Generation when its administration was transferred from the Ministry of Home Affairs (MoHA) to the Ministry of Villages, Disadvantaged Areas and Transmigration (MoV). These two complementary pilot projects began in six provinces and are designed to target the same health and education indicators. They are consistent with both GoI priorities and the Sustainable Development Goals: reduce poverty, maternal mortality, and child mortality and ensure universal coverage of basic education. The initial PKH locations focused more on supply-side ready areas, including both urban areas and more developed rural areas, whereas Generasi operated exclusively in rural areas. This study reports on the long-term LONG-TERM GENERASI IMPACT EVALUATION 1 EXECUTIVE SUMMARY evaluation of Generasi, conducted nine years after the assignment permits an unusually long-term impact program’s launch. evaluation (IE) of a community-driven development program. With over 2,100 villages randomized to receive Under the Generasi program, treatment villages receive either the incentivized or nonincentivized version of the a block grant each year. With the assistance of trained program (plus over 1,000 villages in control subdistricts) program facilitators and local service delivery workers, and over 1.8 million target beneficiaries in treatment villagers undertake a social mapping and participatory areas, this IE represents one of the largest randomized planning exercise to decide how best to use these social experiments ever conducted. funds to meet 12 education and health targets related to maternal and child health behavior and education In 2009, a rigorous IE using the random assignment behavior. These 12 targets initially related to pre- and found that the program had achieved substantial postnatal care, child immunizations, and primary and improvements in health and education targets after junior secondary school enrollment and attendance; they 30 months. Generasi had particular success at improving were revised slightly in 2010 to accommodate changing participation in community health posts (posyandu), local needs. To incentivize communities to focus on increasing the frequency of weight checks for infants, the most effective policies, GoI bases the size of the and increasing school enrollment rates. It was also village’s block grant for the subsequent year partly on found to produce significant long-term reductions in its performance on each of the targeted indicators. The malnutrition rates (2.2 percentage points). Improvements project therefore applies CCT program-style performance in malnutrition outcomes were especially large in low- incentives at the community level in a way that gives performing provinces such as Nusa Tenggara Timur communities the flexibility to address supply and/or (NTT), where underweight rates were reduced by demand constraints. Generasi is the first health and 8.8 percentage points (20% decline compared with education program in the world to combine community control areas) and severe stunting was reduced by block grants with explicit performance bonuses for 6.6 percentage points (21% decline compared with communities. control). The evaluation further found evidence that making block grants conditional on prior performance To allow for a rigorous, randomized evaluation of yielded significantly faster improvements in health Generasi, GoI incorporated random assignment into indicators, particularly at 18 months. the selection of Generasi locations (Olken, Onishi, and Wong 2011). Within the districts selected by GoI for the The health and education contexts in Indonesia as a program, entire subdistricts were randomly assigned whole, as well as in the Generasi program, have changed to either participate in the program or be in a control substantially since the 2009 IE was conducted. The group. Each Generasi location was further randomly report documents that Indonesia has made remarkable allocated to one of two versions of the program: 1) an strides in continuing to improve access to education and “incentivized” treatment with the pay-for-performance basic health. The Generasi program has also undergone component (treatment A) described above, or 2) an significant changes since the 2009 IE, including a revision otherwise identical “nonincentivized” treatment without of the program’s target indicators in 2014 to include pay-for-performance incentives (treatment B). The nutrition and prenatal counseling and school participation randomized assignment of subdistricts into treatment or for students with disabilities, as well as expanding control has remained remarkably intact after nine years the performance incentive condition into all Generasi of programming; only a handful of locations originally programming areas in 2010. These developments raise assigned to the control group have received treatment in questions about the program’s long-term impact as well as the intervening period. This preservation of randomization its ability to yield improvements on the revised indicators. 2 INDONESIA EXECUTIVE SUMMARY This document describes the findings from an evaluation system. The posyandu are monthly clinics for carried out in 2016/17 to determine Generasi’s long- mothers and children that distribute snacks and term impact. It represents the fourth and final wave vitamin A tablets, measure children’s height and of evaluations; the first three waves were carried out weight, immunize children, and provide nutrition between 2007 and 2010. The baseline survey took and health advice. This system has been central place from June to August 2007. The second wave to GoI’s efforts to curb infant/child mortality and was conducted from October 2008 to January 2009, provide citizens with family planning services after 15 to 18 months of Generasi implementation. The since the early 1980s (Leimena 1989). By the late third survey was implemented from October 2009 1990s attendance at posyandu had decreased to January 2010 after 27 to 30 months of project from 52% to 40% in both urban and rural areas but implementation. The most recent survey was carried with a greater decline in rural locations. Reasons out between October 2016 and February 2017 after nine for the decline include a loss of support from years of program implementation. Over 46,000 household nongovernmental organizations and changing members, village heads, and school and health facility preferences for private providers (Marks 2007). staff were surveyed in the final round. Despite these setbacks, community participation in posyandu activities continues to improve nine years The main findings of the Generasi IE are as follows. after program implementation. This participation 77 Since 2009, the overall health and education has been sustained in part by communities environment in Generasi IE districts has improved choosing to allocate portions of their Generasi dramatically, even in control areas. Vital health block grants to fund interventions that incentivize indicators, such as deliveries attended by a doctor participation at the posyandu, such as providing or midwife, have increased substantially since nutritional supplements to mothers who attend, 2009 and now account for over 92% of births in the funding subsidies for pre- and postnatal care, and sample area. Similarly, school participation rates remunerating posyandu volunteers. have risen significantly since 2009: enrollment 77 Specifically, Generasi still helps mobilize community for school years 7–12 was 98% in 2016. These members to attend the posyandu for infant weighing improvements likely reflect both substantial and maternal health and parenting classes. Treatment policy changes and improved household incomes areas experienced 0.13 more weight checks, on throughout Indonesia. average, for young children in control areas (a 6% 77 There is now significantly less room for improvement increase compared with control areas), as well as a in many Generasi target areas. For example, 73% increase (8.5 percentage points) in attendance Generasi’s impacts on malnutrition and school of parenting classes compared with control areas, enrollments that were present in Wave III are no particularly among mothers of young children. Prenatal longer observed in Wave IV. The IE also documents class attendance also increased by 8 percentage that there have been substantial improvements in points (25% increase compared with control areas) precisely those indicators in both treatment and in treatment areas. The frequency of prenatal control areas compared with 2009. attendance increased by 0.28 classes on average. 77 One of Generasi’s greatest accomplishments is the 77 In the lowest-performing districts, Generasi has sustained revitalization of the posyandu, which continued to be effective at encouraging community was accomplished through program facilitation, members to attend the posyandu and increasing community participation, and a targets/incentive immunizations and vitamin A distribution. Nine LONG-TERM GENERASI IMPACT EVALUATION 3 EXECUTIVE SUMMARY years after Generasi implementation, treatment participation without incentives. The disruption areas in the lowest-performing tercile continue meant that funding could not be spent on nutritional to experience a 0.19 increase in weight check supplements, which based on qualitative field frequency. In the same tercile, immunization rates reports led to a reduction in posyandu attendance. increased by 3 percentage points (roughly 4% The future of posyandu success depends on higher than control areas), whereas vitamin A villages continuing to support participation after uptake increased by 0.15 supplements (11% increase the end of Generasi. Across Indonesia, village compared with control areas). governments could use Village Law funds to support the posyandu and continue to ensure 77 Generasi’s initial impact on stunting, concentrated they are sufficiently staffed and that volunteers in NTT province, has not been sustained beyond are compensated appropriately. The GoI could the 2009 IE. There are four possible reasons for encourage village governments to use Village Law this. First, the overall substantial improvements in funds to support posyandu either by prioritizing the stunting in NTT that occurred in both control and health clinics at the central and district levels and/ treatment areas may have exhausted the “low- or incentivizing village governments to allocate hanging fruit” that Generasi was able to address resources for the clinics. in earlier periods. Second, Generasi funding produced crowd-in/crowd-out effects on other 77 The results show that Generasi is effective program resources that undercut the efficacy of at increasing basic service utilization in poor the intervention. Third, implementation issues and contexts, where baseline service delivery and delays in the maternal health and parenting classes health indicator levels are low but there are at may have weakened any potentially positive impacts least some elements of a functioning supply this intervention may have had on behavioral side. Generasi was more effective in 2009, when change and malnutrition. Fourth, Generasi’s effects baseline levels of service delivery were much on stunting were limited because the full suite lower, and even in 2009 it was most effective in of complementary demand- and supply-side provinces and districts with the lowest levels of interventions needed to address stunting were not baseline service delivery. Today, Generasi remains fully implemented. most effective at improving the frequency of weight checks, immunization rates, and vitamin The evaluation results have three policy implications.1 A distribution in the bottom third of districts in 77 Future GoI health-related programming needs to terms of predicted levels of achievement in the consider how to sustain the posyandu and ensure absence of the program. This suggests that GoI that mothers continue to bring their children and other governments worldwide that are trying for weight/height measurement, participation to accelerate the achievement of basic health and in Early Childhood Education and Development education indicators could consider applying the programs, and basic maternal and infant health Generasi model in contexts where baseline levels services. An implementation disruption in Generasi of health service delivery are low. programming that occurred in 2015 when the 77 As this IE demonstrates, short- and long-term IEs Generasi program transferred from MoHA to MoV are essential to ensuring that government programs underscores the difficulty of maintaining posyandu continue to have an impact as the programs and 1 The policy and operational recommendations are elaborated in the contexts change. IEs can also inform governments complementary report Long-Term Generasi Qualitative Study. about how to adjust targets appropriately. 4 INDONESIA INTRODUCTION BACKGROUND Indonesia has made remarkable strides in key human development indicators over the past few decades. Primary school enrollment is close to universal for both boys and girls, and the child mortality rate has declined rapidly (World Bank 2006, 2008). Nevertheless, infant and maternal mortality, child malnutrition, junior secondary school enrollment, school transition rates, and learning outcomes are lower in Indonesia than in other countries in the region (World Bank 2006, 2008). Furthermore, there are substantial geographical disparities in these outcomes, with poorer outcomes in rural and remote provinces and districts. In 2007, the government of Indonesia (GoI) launched two programs designed to tackle these issues: 1) the Hopeful Family Program (Program Keluarga Harapan/PKH), a conditional cash transfer (CCT) to households; and 2) the National Community Empowerment Program—Healthy and Smart Generation, known as Generasi, an incentivized community block grant program. In 2014, the Generasi program was renamed Bright Healthy Generation when it transferred administration from the Ministry of Home Affairs (MoHA) to the Ministry of Villages, Disadvantaged Areas and Transmigration (MoV). These two pilot projects began in six provinces and were designed to achieve the same objectives and goals.2 These goals are consistent with GoI’s priorities 2 Indonesia is divided into provinces (the highest administrative unit). Below provinces are regencies (generally rural) and cities (generally urban). Regencies and cities are further divided into subdistricts (common in most of Indonesia) and districts (only present in Papua and West Papua). Finally, subdistricts and districts are divided into villages and urban communities. The neighborhoods within villages are called hamlets. LONG-TERM GENERASI IMPACT EVALUATION 5 INTRODUCTION and the Sustainable Development Goals: to reduce RESULTS FROM THE 2009 IE poverty, maternal mortality, and child mortality, as well In 2009, a rigorous randomized IE using data from three as ensure universal coverage of basic education. PKH survey waves (Wave I at baseline, Wave II 18 months focused more on supply-side ready areas, including after implementation, and Wave III 30 months after urban areas, whereas Generasi operated in rural areas. implementation) showed that the Generasi program had This study reports on the long-term evaluation of produced significant improvements in target health and Generasi, conducted nine years after the program’s education indicators (World Bank 2011). launch in 2007. Strong improvements were made in the frequency Generasi differs from conventional CCT programs in that of weight checks for young children, primary school block grants are allocated to communities rather than participation rates, and malnutrition rates. Other to individual targeted households. Generasi focuses indicators showed improvement in access to maternal, primarily on rural areas, building on a preexisting GoI neonatal, and child health care services, such as an community program known as PNPM Rural. Under increase in mother and child participation in community Generasi, over 1,600 rural villages received an annual health post (posyandu) activities. Overall, the IE found a block grant during the first year. Each village could substantially positive impact on average across the use the grant for any activity that supported one of 12 indicators it was designed to address. 12 indicators related to health and education service delivery (such as pre- and postnatal care, childbirth These improvements were especially marked in the assisted by trained personnel, immunization, school lowest-performing areas. On average, the program enrollment, and school attendance). was approximately twice as effective in areas in the 10th percentile of service provision (very low health To incentivize communities to focus on the most effective and education status) at baseline. In Nusa Tenggara policies, GoI bases the size of the village’s grant for the Timur (NTT) province, for example, Generasi reduced subsequent year partly on its performance on each of underweight rates by 8.5 percentage points and severe the 12 health and education targets. The Generasi project stunting rates by 6.3 percentage points. thus applies CCT program-style performance incentives to communities, in a way that gives communities the flexibility to address supply and/or demand constraints. MOTIVATION To allow for a rigorous, randomized evaluation of This report discusses the results of an IE of the long-term Generasi, GoI incorporated random assignment into the effects of the Generasi program. Evaluating the impact of selection of Generasi locations. Each Generasi location programs over the long term is valuable for both policy was further randomly allocated to one of two versions makers and practitioners, yet long-term evaluations remain of the program: 1) an “incentivized” treatment with the rare. As Wong (2012) notes, there are few longitudinal IEs pay-for-performance component (treatment A) or 2) an in general, and those reviewed in the study measure, otherwise identical “nonincentivized” treatment without on average, only 3.1 years of project interventions. pay-for-performance incentives (treatment B). Starting in 2010, however, all Generasi locations shifted to using One of the reasons long-term evaluations are so rare is the incentivized version of the program based on results that in many cases, after a few years of implementation, from the 2008 (18-month) wave of the impact evaluation the participating government expands the program into (IE) (described below), which provided evidence that the control areas. However, GoI chose to expand the program incentivized grant model was more effective. over time into new provinces rather than to control areas 6 INDONESIA INTRODUCTION in treatment provinces. This decision created a virtually FIGURE 1: Photograph of Generasi unprecedented opportunity for a long-term evaluation of Activities. Note the Generasi Board Generasi interventions. in the Background The current IE measures the effects of Generasi interventions over a comparatively long period of nine years. Using four waves of evaluation data, the report estimates effects over the medium and long terms and evaluates how programming and intervention impacts have changed over time. Combining the long-term scope of the evaluation with the program’s large scale (a baseline sample of more than 12,000 households, with 1.8 million target beneficiaries in treatment areas), the current evaluation is rare among both health and education evaluations in developing countries. This evaluation is also applicable to several of GoI’s key policy priorities, the most significant of which is the enactment of the Village Law in 2014, a massive THE GENERASI PROGRAM4 decentralization effort that substantially increases direct Generasi began in mid-2007 in 164 pilot subdistricts transfers to villages.3 The IE will help inform how village spread across five provinces selected by GoI: West Java, governments spend Village Law funds, as well as efforts East Java, North Sulawesi, Gorontalo, and NTT. By the to align village investments with investments made time of the first IE in 2009, the program was operating in by other levels of government to address health and 264 subdistricts across these five provinces. It currently education challenges. Further, the Indonesian Ministry operates across 499 subdistricts in 11 provinces. However, of National Development Planning (Bappenas) has been the current report and analysis focus on the 264 subdistricts developing a strategy called “Improving Basic Services for considered in the 2009 IE. the Poor and Vulnerable,” which will focus on enhancing the accountability of public service provision through The Generasi project focuses on 12 indicators of maternal/ community participation and engagement. This IE will child health and educational behavior. These indicators are inform the design of Bappenas’ service delivery programs. in line with Ministry of Health priorities and protocols and In 2017, GoI launched a Presidential National Action Plan for GoI’s constitutional obligation to ensure nine years of basic reducing stunting with a multi-sectoral response. Beginning education for all Indonesian children. GoI chose these in 2018, the plan directs national ministries to focus their indicators to be as similar as possible to the conditions stunting-related programs and activities on 100 districts for the individual household CCT program piloted at the with a high stunting prevalence and incidence. The IE results same time as Generasi (but in different locations). These will contribute to this program and a related World Bank 12 indicators relate to seeking health and educational operation, Investing in Nutrition and Early Years. services that are within the direct control of villagers, such as the number of children who receive immunizations, 3 Village transfers will be scaled over time. The national government allocated Rp 280 million (US$20,000) in 2015, and district governments are estimated to allocate around Rp 500 million (US$40,000). Each 4 Portions of the description of the Generasi program in this section, as village will receive approximately Rp 1.4 billion (US$122,000) on average well as the experimental and evaluation design sections, draw directly each year. from Olken, Onishi, and Wong (2011). LONG-TERM GENERASI IMPACT EVALUATION 7 INTRODUCTION pre- and postnatal care, and the number of children Table 1). Through social mapping and in-depth discussion enrolled and attending school, rather than long-term groups, villagers identify problems and bottlenecks outcomes, such as test scores or infant mortality. in reaching the indicators. Intervillage meetings and consultation workshops with local health and education School enrollment rates improved significantly across service providers allow community leaders to obtain control and treatment areas over the past decade, and in information, technical assistance, and support from the 2014 Generasi revised its education targets to better focus local health and education offices and coordinate the use of investments on the neediest populations. The new education Generasi funds with other health and education interventions targets include participation rates for children with disabilities in the area. After these discussions, the elected management and transition rates from primary to junior secondary school. team makes the final Generasi budget allocation. In addition, Generasi introduced indicators to measure community participation in enhanced nutrition counseling Table 1 provides descriptive statistics on program sessions delivered through the posyandu. facilitators. Most facilitators have a high school or postdiploma-level education, with subdistrict facilitators Under the Generasi program, all participating villages holding much higher education levels than those at the receive a block grant each year to improve education village level. Facilitators have an average of five years and maternal and child health. For example, these grants of relevant facilitation experience before starting their can be used for a wide variety of purposes, including current post, and those working at the subdistrict level hiring extra midwives for the village, subsidizing the costs tend to be more experienced. Facilitators also report of pre- and postnatal care, providing supplementary an average gap of 5.2 months between the departure feeding, hiring extra teachers, opening a branch school of a facilitator and the arrival of a replacement. Of the in the village, providing scholarships or school supplies, facilitators who change jobs, about 16% go on to work providing transportation funds for health care or school as facilitators in other villages, whereas another 11% attendance, improving health or school buildings, or work in village administration. Approximately 60% of rehabilitating a road to improve access to health and facilitators pursue other miscellaneous jobs, including education facilities. entrepreneurship, farming, teaching, and working as Trained facilitators help each village elect an 11-member posyandu staff. village management team and select local facilitators and In 2016, communities used the bulk of their block grants for volunteers to decide how to allocate the block grants (see health activities (see Figure 3). Communities chose to use most of their funds allocated to education for “individual goods,” such as school materials, equipment and uniforms, FIGURE 2: A Generasi Facilitator Talks and school financial assistance. Most of the health funds During a Community Meeting were used for supplementary food (PMT) and training. Performance incentives are a critical (and unique) element of the Generasi approach. The size of a village’s block grant depends in part on its performance on the 12 targeted indicators in the previous year. The incentive is designed to encourage a more effective allocation of Generasi funds and stimulate village outreach efforts to encourage mothers and children to obtain appropriate health care and increase educational enrollment and attendance. 8 INDONESIA TABLE 1:  Descriptive Statistics, Generasi Program Facilitators Facilitator characteristics Characteristic Overall Subdistrict facilitators Village facilitators Average age (years) 38.91 38.79 38.93 Educational attainment SD (elementary school) incomplete 0.11% 0% 0.13% SD (Islamic elementary school) or equivalent 5.54% 0% 6.76% SMP (Islamic junior high school) or equivalent 16.09% 0.60% 19.50% SMA (Islamic senior high school) or equivalent 45.65% 0.60% 55.57% D1/D2/D3 (diploma) 2.50% 4.22% 2.12% D4/S1 (postdiploma) 29.24% 92.77% 15.25% S2/S3 (master’s) 0.54% 1.81% 0.27% Not yet/never attended school 0.11% 0% 0.13% Years of experience 5.02 6.55 4.68 Post-Generasi careers � 15.6% facilitator elsewhere � 11.3% village administrator �4 .2% civil servant, 2.1% government � 0.8% students � 57.5% other Average gap between facilitators 5.22 months 5.92 months 5.05 months FIGURE 3:  Village Funding Allocations, 2016 Education Activities: 25% Health Activities: 75% Incentives Supplementary 5% Feeding Activities Infrastructure Financial 38% 6% Incentives 3% Financial Infrastructure Financial Assistance 11% Assistance 38% 10% Training 16% Facilities and Equipment 16% School Materials, Equipment, Uniforms 32% Training 25% Note: In 2016, communities allocated most of Generasi’s block grants for health rather than education. Within health, communities allocated most of the grants for supplementary food that facilitators distribute to mothers and children at the monthly posyandu and training for posyandu staff. Within education, communities allocated most of the funds for financial assistance and school materials, equipment, and uniforms for students. Source: Generasi Project management information system (MIS) data. INTRODUCTION The performance bonus is structured as a relative ratchet effect (Weitzman 1980); the minimums, mvi, are competition among villages within the same subdistrict. determined based on historical national data sets. GoI used this approach to minimize the impact of The Generasi project design built on GoI’s PNPM Rural unobserved differences in the capabilities of different program, which along with its predecessor program areas on the performance bonuses (Lazear and Rosen (Kecamatan Development Project), funded over 1981; Mookherjee 1984; Gibbons and Murphy 1990). The US$2 billion in local infrastructure and microcredit fixed allocation to each subdistrict also ensures that the programs in approximately 61,000 Indonesian villages bonus system does not result in the unequal geographic from 1998 to 2014. The Generasi project is implemented distribution of funds. by MoV and funded through GoI resources and in part The size of the overall Generasi allocation for the subdistrict by loans from the World Bank and grants from several is determined by the subdistrict’s population and poverty bilateral donors. Technical assistance and evaluations level. Within a subdistrict, in year 1 of the project, funds have been supported by a multidonor trust fund with were divided among villages in proportion to the number contributions from the World Bank; embassies of the of target beneficiaries in each village (that is, the number Netherlands, Australia, United Kingdom, and Denmark; of children of varying ages and the expected number of and the World Bank–managed Spanish IE Fund. The pregnant women). Starting in year 2, 80% of Generasi’s 2016 IE was supported by the Australian Department of allocation to the subdistrict continued to be divided among Foreign Affairs and Trade. villages in proportion to the number of target beneficiaries; the remaining 20% formed a performance bonus pool that VILLAGE-LEVEL BLOCK GRANTS is to be divided among villages based on their performance on the 12 indicators. Generasi originally included two This section describes the allocation of Generasi block distinct treatment arms to separate the impact of the grants and how villages have chosen to spend grant performance bonuses from the overall impact of the block funds over time. Unfortunately, although data on the grant program. From 2010, all treatment areas received the annual block grant allocation and planned expenditures block grant program with performance bonuses. are available at the village-year level, data on actual realized expenditures are available only at the provincial The performance bonus pool is allocated to villages level. Although the IE team expects planned and actual in proportion to a weighted sum of each village’s expenditures to correspond closely, there are limited performance above a predicted minimum achievement opportunities to analyze village-level expenditures. level. Specifically, each village’s share of the performance Overall, annual Generasi allocations have declined steadily bonus pool is determined by: over time from a peak in 2009 (see Figure 4). However, yearly allocations ignore the disbursement of multiyear Pv Share of bonus = , where Pv = ∑  wi X ( yvi − mvi ) , grants, and do not take into account that unspent funds ∑ Pj are carried forward into the next programming year. where yvi represents village v’s performance on indicator i, Data on annual and multiyear planned expenditures for the wi represents the weight for indicator i, mvi represents the available time period of 2013–16 show that disbursements predicted minimum achievement level for village v and increased in 2015 and 2016 (see Figure 5). The sharp indicator i, and Pv is the total number of bonus “points” increase in 2016 was a function of a programming delay in earned by village v. Generasi uses performance relative 2015, which meant some disbursements scheduled for to a constant predicted minimum attainment level, rather 2015 were held until 2016, as well as a new regulation that than improvements over an actual baseline, to avoid the pushed villages to spend unused funds by the end of 2016. 10 INDONESIA INTRODUCTION FIGURE 4:  Average Block Grant Size per IE Village 350 300 250 IDR, Millions 200 150 100 50 0 2002 2004 2006 2008 2010 2012 2014 2016 2018 Year Note: Beginning in 2009, there was a steady decline in annual Generasi allocations. This figure does not show the disbursement of multiyear grants and does not account for unspent funds being carried forward into the next programming year. The disbursements of Generasi block grants are likely to be higher in 2015 and 2016 than the figure indicates. Source: MIS data. FIGURE 5:  Total Block Grants Disbursed from Subdistrict Implementation Unit Accounts to Generasi Village Activity Implementers (Including Multiyear Accounts) to All Generasi Villages 700,000 600,000 500,000 IDR, Millions 400,000 300,000 200,000 100,000 0 2011.5 2012 2012.5 2013 2013.5 2014 2014.5 2015 2015.5 2016 2016.5 Year Note: This figure shows annual and multiyear planned expenditures for the available time period of 2013–16. In 2015 and 2016, there was an increase in disbursements. Realized expenditure data are not available. LONG-TERM GENERASI IMPACT EVALUATION 11 INTRODUCTION The most significant shift in how Generasi funds have that the respective shares of health and education been spent over time has been a substantial decrease in programs as a percent of total expenditures was relatively infrastructure spending. This decrease is the result of the similar at the start of the program but gradually diverged expansion of the PNPM Rural program into Generasi areas, as the share of health expenditures grew rapidly. By leading GoI to advise Generasi not to use block grant 2016, roughly 80% of village expenditures were allocated funds for infrastructure costs because such costs are now to health programming, leaving the remaining 20% for borne by PNPM Rural. Figure 6 shows that the level of education. block grant spending on noninfrastructure-related health This shift in spending was partly caused by changing activities has remained roughly steady over time and was priorities within the national implementing agency, virtually constant from 2010 to 2015. Specifically, in 2008, which in turn reflect the dramatic expansion in non- the average village-level allocation for health activities was Generasi education expenditures at the national level. Rp 81.15 million. This figure jumped to Rp 125.12 million in As primary and secondary school enrollment rates 2009 and declined to Rp 54.67 million by 2010. However, improved significantly over the past decade, the the share of spending on education has decreased Directorate for Village and Community Empowerment significantly over time, reflecting changing program reformulated education targets to shift communities’ priorities. focus toward identifying and assisting hard-to-reach, out-of-school children, including those with disabilities, and the transition phase from primary to junior secondary SPENDING CHOICES school. This resulted in fewer education target indicators Treatment communities have changed how they allocate and potentially fewer incentives for communities funds from the block grants over time to prioritize health to use the Generasi funds for education-related over educational interventions. Figure 7 demonstrates purposes. FIGURE 6:  Average Village-Level Expenditures (Excluding Infrastructure) 250 200 150 IDR, Millions 100 50 0 2002 2004 2006 2008 2010 2012 2014 2016 2018 Health and Education Expenditures Health Expenditure Education Expenditure Note: The level of block grant spending on noninfrastructure-related health activities has remained relatively steady over time and was virtually constant from 2010 to 2015. Yet, over time, the share of spending on education has decreased, reflecting changing program priorities. 12 INDONESIA INTRODUCTION FIGURE 7:  Average Village-Level Shares 100% 80% 60% Share 40% 20% 0% 2002 2004 2006 2008 2010 2012 2014 2016 2018 Health Share Education Share Note: At the program’s start, the respective shares of health and education programs as a percent of total expenditures was relatively equal. Over time, communities shifted spending from education to health. By 2016, almost 80% of village expenditures were allocated to health programming, with the remaining 20% allocated to education. EXPERIMENTAL DESIGN such as community health centers and junior secondary schools, provide services to multiple villages within a To evaluate the program’s overall impact and separately subdistrict. Thus, an increased demand for services from identify the impact of its performance incentives, Generasi one village within a subdistrict could potentially crowd locations were originally selected by lottery to form a out the services provided to other villages within the randomized, controlled field experiment. Randomized same subdistrict; alternatively, an effort by one village to evaluation techniques are considered the gold standard improve service provision at the community health center for evaluating the impact of clinical and public health could benefit other villages in the same subdistrict. By interventions (Gordis 2004) as well as development randomizing at the subdistrict level so that all villages programs more generally (Duflo, Glennerster, and Kremer in the subdistrict receive the same treatment status, the 2007). They have formed the basis of a number of high- evaluation design ensures that the total net effect of profile social policy experiments in the United States (see the program is captured because any within-subdistrict Newhouse et al. 1993; Kling, Liebman, and Katz 2007) spillovers would also be captured in other treatment and internationally (see Gertler 2004; Miguel and Kremer villages. This type of cluster-randomized design is 2004; Schultz 2004; Skoufias 2005). common in program evaluations, where there might be local spillovers from the treatment (Miguel and Kremer The Generasi randomization was conducted at the 2004; Olken 2007). subdistrict level so that all villages within the subdistrict either received the same treatment of Generasi or were in The Generasi locations were selected using the following the control group. Randomizing at the subdistrict level is procedure. First, 300 target subdistricts were identified, important because many health and education services, targeting poor, rural areas that had an existing community- LONG-TERM GENERASI IMPACT EVALUATION 13 INTRODUCTION FIGURE 8:  Status of Subdistricts’ Treatment Assignment in Waves III and IV Wave III Wave IV Status in 2009 Status in 2016 Randomization Control Treatment Total Randomization Control Treatment Total Control 83 0 83 Control 78 5 83 Treatment 20 161 181 Treatment 18 163 181 Total 103 161 264 Total 96 168 264 Note: The control and treatment assignments remained markedly intact from 2009 to 2016. In Wave IV, five control subdistricts were participating in the program, whereas two of the original 20 subdistricts that failed to receive treatment in 2009 began participating in Generasi between 2009 and 2016. driven development infrastructure. Each subdistrict was then The 2009 IE relied on the original lottery assignment randomly assigned by computer into one of three equal- for its analysis, focusing on the 264 eligible subdistricts size groups: treatment A, incentivized (100 subdistricts); and interpreting results as intent-to-treat (ITT) treatment B, nonincentivized (100 subdistricts); or control estimates (Imbens and Angrist 1994). The current (100 subdistricts). Within a subdistrict, all villages received evaluation focuses on these 264 subdistricts and also the same treatment. The randomization was stratified by interprets results as ITT estimates. Figure 8 depicts district to ensure a balanced randomization across the the status of treatment assignment in Wave III (2009) 20 districts in the study. The tests for balance confirm and the focus of the current report—Wave IV (2016). that the three groups of subdistricts appear similar on At the time of Wave III, no subdistrict assigned to preperiod characteristics (World Bank 2008). Note the control group incorrectly received treatment. that 36 of the 300 subdistricts should not have been However, 20 subdistricts assigned to the treatment included in the randomization since they were ineligible had still not begun participating in Generasi at the for Generasi because they had been selected (before the time of the survey. By Wave IV, five control villages randomization) to receive other programs or had had were participating in the program, whereas two of the prior implementation problems with previous programs. original 20 subdistricts that failed to receive treatment Because the eligibility decision was made on the basis of began participating. Thus, randomization assignment lists determined before the randomization and because has remained remarkably intact over nine years. Only a those lists were obtained for treatment and control areas, handful of the original control subdistricts gained access ineligible subdistricts in both treatment and control to treatment, whereas two treatment subdistricts began groups were excluded from the main analysis. receiving programming after a delay. 14 INDONESIA EVALUATION DESIGN The main data for the impact analysis is from a set of surveys of households, village officials, health service providers, and school officials. Three waves of the survey were planned as part of the original evaluation series. Wave I, the baseline round, was conducted from June to August 2007 before program implementation. Wave II, the first follow-up survey round, was conducted from October to December 2008. Wave III, a medium-term follow-up round, was conducted from October 2009 to January 2010. Finally, Wave IV was conducted between October 2016 and February 2017. These surveys were designed by the World Bank, Abdul Latif Jameel Poverty Action Lab/ Massachusetts Institute of Technology (J-PAL/MIT), and GoI and conducted by the Center for Population and Policy Studies of the University of Gadjah Mada, Yogyakarta, Indonesia. The final evaluation is based on data collected from all four rounds. This IE round examines the 264 subdistricts sampled across five provinces (West Java, East Java, NTT, Gorontalo, North Sulawesi) that were included in the 2009 IE. In the original evaluation, eight villages were selected at random within each subdistrict (if the subdistrict contained fewer than eight villages, all were selected). In the current evaluation, four of the eight villages within each subdistrict were chosen to be panel villages (in these villages, households that had been sampled in the previous evaluation were recontacted), whereas the other four represent a new cross section of households (households not surveyed in the previous evaluation). Teams tracked and reinterviewed migrated or split households that provided information for any of the married women or children modules, as long as they were within the same subdistrict. In panel areas, 99% of target households were reinterviewed in Wave 3, and 94% of the target households from the baseline survey were reinterviewed LONG-TERM GENERASI IMPACT EVALUATION 15 EVALUATION DESIGN in Wave 4 (Appendix Table 1).5 This sampling design in the household at the time had their anthropometric provides a cross section of the current cohort of measurements (height and weight) taken. pregnant and new mothers, a panel of pregnant and Separate instruments were administered for household new mothers who received program benefits in an heads, pregnant women, infants (0–2 years), and young earlier pregnancy, and a panel of existing children, as children (6–15 years). For service providers, enumerators well as new children within the same family. collected data from community health center workers, Surveys targeted both beneficiary and provider midwives, school officials, and community health post populations: households, service providers, and (posyandu) volunteers. Finally, sampled facilitators governance personnel. The sampling design for include subdistrict heads, village heads (elected by their households was chosen to ensure adequate coverage communities), and programming facilitators. of the key Generasi demographic groups: new mothers, Data from these surveys were supplemented with children under three, and school-age children. Within detailed administrative data from the Generasi project’s each of the new cross-sectional villages, one hamlet internal MIS. This included detailed budget allocations was randomly selected, and a list of all households was for the block grants, performance data on the Generasi obtained from the head of the hamlet. Five households indicators, and data on participation levels in Generasi were randomly sampled from that list to be interviewed, village meetings. In addition, a joint team made up of stratified to fulfill the following criteria: representatives from J-PAL, Kompak, Bappenas, and 77 Type 1 (three households): Household with at least the World Bank conducted a qualitative study to assess one child under age two, a pregnant mother, or a the impact of a program disruption in 2015 on service mother who was been pregnant in the last delivery and target outcomes. This qualitative analysis two years; allowed the study team to contextualize some of the decision-making and implementation challenges behind 77 Type 2 (one household): Household with at least one the quantitative results. child under 15 but not included as Type 1; and 77 Type 3 (one household): Household does not fit the criteria of Type 1 or Type 2 households. In the METHODOLOGY panel villages, households were chosen in 2007 This section describes the 12 original target indicators, to have two households with children of Type 1, eight of which are associated with health outcomes and two households of Type 2, and one household of four with education, as well as the revised indicators Type 1 based on the ages of children at that time. that followed from the 2014 revision of the program All of these households were followed up in panel (Figure 9). Target health outcomes consider both health villages. care-related behaviors (for example, pre- and postnatal In cross-sectional households, additional households care visits) and outcomes (for example, rate of underweight were sampled for a short module that focused on a children in village). Education indicators focus largely on few key outcomes: underweight, stunting, wasting, and participation, tracking enrollment and attendance rates for infant mortality. Four Type 1 households were selected primary and junior secondary students. from the household listing, and all children aged 0–12 The 2014 revised health indicators track participation rates for pregnant women and male partners in nutrition counseling sessions as well as participation rates for 5 There are no differences in attrition rates between the treatment and control areas (see Annex Table 1). parents of infants in nutrition counseling sessions. The 16 INDONESIA EVALUATION DESIGN FIGURE 9: Generasi Program Target Indicators Health Indicators 1. Four prenatal care visits 2. Taking iron tablets during pregnancy 3. Delivery assisted by a trained professional 4. Two postnatal care visits 5. Complete childhood immunizations 6. Adequate monthly weight increases for infants 7. Monthly weighing for children under three and biannually for children under five 8. Vitamin A twice a year for children under five Education Indicators 9. Primary school enrollment of children 6 to 12 years old 10. Minimum attendance rate of 85% for primary school-aged children 11. Junior secondary school enrollment of children 13 to 15 years old 12. Minimum attendance rate of 85% for junior secondary school-aged children Indicators 9-12 have been revised to (post-2014) 1. Participation of pregnant women and male partners in nutrition counseling o ered through maternal health classes 2. Participation of parents (and/or caregivers) in nutrition counseling o ered through classes for infants. 3. Enrollment of all primary and junior secondary aged children who have not enrolled in school or have dropped out, including children with disabilities. 4. All children who graduate from primary school, including children with disabilities, enroll in junior secondary school. Note: This table shows the 12 original target indicators, eight of which are associated with health outcomes and four with education, as well as the revised indicators that followed from the 2014 program revision. new education indicators include enrollment rates for 2010, all subdistricts assigned to treatment have received children at risk of dropping out or not being enrolled in the incentivized version of the program, so there is no school at all, as well as transition rates from primary to longer an unconditional grant program to evaluate. As a junior secondary school. result, treatment effects reflect differences in outcomes from receiving the performance-incentivized block grants versus being in the control group.6 REGRESSION SPECIFICATION Defining treatment status in this way exploits only the Given that treatment assignment was randomized variation in program exposure that is attributable to in the Generasi program, the IE is econometrically chance. This captures the ITT effect of the program, straightforward: a comparison of outcomes in treatment and because the lottery results were closely followed— and control subdistricts, controlling for outcome levels at they predict true program implementation in 90% of baseline. The sample is restricted to the 264 subdistricts that were 6 The authors also checked whether there are any differences in achievement of these targets and final outcomes between the analyzed in the 2009 IE. Following the methodology used subdistricts that were incentivized and subdistricts that were not in that evaluation, treatment status (Generasi) is defined incentivized during the period up to 2009. The results do not differ among the subdistricts that were part of one of the two treatment arms here as an indicator variable that takes a value of 1 if the between 2007 and 2009. These results are available in Annex Tables 2 subdistrict was randomized to receive Generasi. Since and 3. LONG-TERM GENERASI IMPACT EVALUATION 17 EVALUATION DESIGN subdistricts in 2016/17 (according to Figure 6)—they will Because of the large number of indicators, to calculate joint be close to the true effect of the treatment on the treated significance, average standardized effects are calculated (Imbens and Angrist 1994). for each family of indicators, following Kling, Liebman, and Katz (2007). Specifically, for each indicator i, define s i2 to All regression specifications control for the baseline be the variance of i. Equation 1 is then estimated for each value of the outcome variable. This includes controls for indicator, but the regressions are run jointly, clustering the outcome’s average baseline value for the subdistrict, the standard errors by subdistrict to allow for arbitrary individual-specific preperiod panel data values for those correlation among the errors within subdistricts, both who have it, and a dummy variable that corresponds to between and across indicators. The average standardized having nonmissing, preperiod variables. All household effect is then defined as: survey regressions further include dummies for the three different sample types interacted with whether ˆ a a household came from a panel or nonpanel village. ∑ σi i i Finally, because many of the indicators for children vary naturally as the child ages, all child-level variables include Finally, note that all reported P values are calculated age dummies. using a randomization inference procedure (Athey and For each indicator of interest, the following regression Imbens 2017). was run: (1) y pdsi 4 = α d + β 1GENERASIpds 4 + γ 1 y pdsi 1 + γ 2 1{ ypdsi 1≠ missing} HETEROGENEITY + γ 3 y pds 1 + SAMPLE pdsi + α p × Ps + ε pdsi Part of the analysis explores the existence of heterogeneous treatment effects based on either Where p is a person, d is a district, t is the survey wave preexisting conditions or province-level differences. (t = 4 in the regression for Wave IV), ypdsi4 is the outcome This analysis focuses on the 10 target health and in Wave IV, αd is a district fixed effect, ypdsi1 is the baseline education indicators (see Table 2) as well as the final value for individual i (assuming this is a panel household and baseline values are nonmissing; 0 otherwise), outcomes detailed previously. 1{ypdsi1≠missing} is a dummy that indicates the baseline value To detect heterogeneous treatment effects related to is missing, and y pds 1 is the average baseline value for the preexisting conditions, the authors generate predicted subdistrict. SAMPLE includes dummies indicating how outcomes in the absence of treatment for both treatment the household was sampled interacted with being a panel and control areas by regressing outcome indicators or cross-sectional household, and αp × Ps are province- on district dummies. The authors then group districts specific dummies for being in the previous Kecamatan into terciles of predicted performance and estimate Development Project sample. Standard errors are the impact of the program separately for each tercile. clustered at the subdistrict level in all specifications.7 The authors follow Abadie, Chingos, and West (2013) in using a repeated split-sample estimation strategy, which yields unbiased heterogeneous treatment effects in this 7 For each regression on the target intermediate and final outcomes, context. This approach allows a proper estimation of the authors checked whether the results are consistent with estimating whether the program was more effective in areas that only the models using the households that were newly sampled repeated cross sections in each survey wave (that is, dropping panel households). would have done worse in the absence of the program, The results do not differ between the models estimated using the full and allows for the possibility that the districts most in data set and those estimated using only the cross-sectional data. See Annex Tables 4 and 5. need changed since baseline. 18 INDONESIA EVALUATION DESIGN TABLE 2: Wave IV Indicators Primary Secondary Targets Prenatal care (number of prenatal visits by all women who gave �  7 to 12 participation rate �  (Intermediate) birth in last 24 months) (enrollment dummy for Outcomes Delivery (delivery by trained midwife/doctor for all women who �  ages 7–12 in school year gave birth in last 24 months) 2016–17) Postnatal care (number of postnatal visits within 42 days after �  13 to 15 SMP participation �  delivery by all women who gave birth in last 24 months) rate (enrollment dummy for Iron pills (number of iron tablet sachets during pregnancy for all �  ages 13–15 in SMP in school women who gave birth in last 24 months) year 2016–17) Immunizations (percent of recommended immunizations up to �  11 months, for all children 23 months old and younger) Weight checks (number of weight checks in past three months, �  for all children younger than three, using mother’s recall of the number of posyandu visits in last three months, but 0 if child was not weighed at last visit) Vitamin A (number of vitamin A supplements in past 18 months, �  for all children aged six months to two years) Underweight (% underweight, weight-for-age less than two �  standard deviations, for all children younger than three) Targets added Parenting classes (attendance, frequency, mother with child �  in 2014 (not in under five) pre-analysis Prenatal (maternal) classes (attendance, women who have been �  plan) pregnant in the last 24 months) School participation rate for children with special needs (enroll- �  ment dummy for special needs in school year 2016–17) Final Outcomes Underweight (weight-for-age less than two standard deviations, �  Neonatal mortality (death �  all children younger than age three) of child aged 0–28 days, all Severe underweight (weight-for-age less than three standard �  births since 2010) deviations, all children younger than age three) Infant mortality (death of �  Wasting (weight-for-height less than two standard deviations, �  child aged 0–11 months, all children younger than age three) all births since 2010) Severe wasting (weight-for-age less than three standard �  Language score �  deviations, all children younger than three) (age-adjusted Z-score) Stunting (height-for-age less than two standard deviations, all �  Math score (age-adjusted �  children younger than three) Z-score) Severe stunting (weight-for-age less than three standard �  Total test score: sum of �  deviations, all children younger than three) language and math score Raven’s test of cognitive ability (cognitive assessment, �  (age-adjusted Z-score) age-adjusted Z-scores) Note: Wave IV indicators, broken down by primary or secondary outcome, intermediate or final outcomes, and in or not in the preanalysis plan. LONG-TERM GENERASI IMPACT EVALUATION 19 EVALUATION DESIGN The IE also explores whether there are heterogeneous PREANALYSIS PLAN effects across the five provinces in the sample by All of the analyses presented here (regression interacting treatment status with an indicator for each specifications, outcome variables, and aggregate effects) of the specified provinces. This analysis is of particular follow a plan that was finalized before examining the interest to the Ministry of Health, MoV, and Bappenas, unblinded Wave IV data.8 In conjunction with GoI, the given that the previous analysis found that Generasi evaluation team agreed on two sets of primary outcomes had a substantial impact on reducing severe stunting for the analysis that were registered in the preanalysis plan only in NTT. (see Table 2). One set of primary outcomes is composed of the eight original Generasi target health indicators. BALANCE TESTS The second set of primary outcomes is composed of long-term health indicators that effect malnutrition and Determining whether randomization was carried out cognitive capacity. The rest of the outcome variables properly is key to drawing inferences about program are relegated to secondary status. Results using these effects. Balance tests using baseline data for the 12 major variables are presented as additional analysis. indicators and the average standardized effect outcomes were carried out in the 2011 IE and are described more fully 8 This hypothesis document was registered with the American Economic in that report (Olken et al. 2011). Results from the balance Association Social Science Registry (https://www.socialscienceregistry. tests are consistent with a balanced sample of treatment org/trials/332) on April 26, 2017, before any data were analyzed from this wave separately by treatment and control (that is, the data were and control groups, and confirm that randomization was examined without any identifiers marking treatment versus control carried out properly. areas) and is available upon request. 20 INDONESIA MAIN RESULTS This section describes the main results of Generasi after nine years of programming interventions. The results are reported in terms of the types of support beneficiaries received, the impact of the program on the main target indicators (both primary and secondary), long-term final outcomes (primary and secondary), and nontargeted indicators. In the figures that follow, bar plots depict the estimated coefficient on the Generasi variable from estimating Equation 1 for Waves III (2009) and IV (2016). This is interpretable as Generasi’s average impact on the outcome variable for each wave. Error bars depict the corresponding 95% confidence intervals for the coefficient estimates. The bars of coefficient estimates that are statistically significant at P < 0.10 (using randomization inference) are depicted in yellow, whereas those that are insignificant at this level are shown in red. The corresponding results are also shown in tables. DIRECT BENEFITS OF GENERASI FUNDS This section describes the impact of Generasi programming on the types and quantities of direct benefits received by children under three, school-aged children, and pregnant women. The results show slightly smaller Generasi effects overall in Wave IV and much smaller effects on education-related targets. The decline in education subsidies later in the program reflects the previously discussed shift in emphasis from education targets focused on boosting enrollment and participation. Figure 10 shows the change in the probability of receiving health subsidies in treatment regions. Households in treatment regions across both waves are significantly more likely to receive health subsidies for pre- and postnatal care and childbirth than LONG-TERM GENERASI IMPACT EVALUATION 21 MAIN RESULTS FIGURE 10: Impact on Health Subsidies Wave III Wave IV Received health subsidies for pre-/postnatal care Received health subsidies for childbirth –0.1 –0.05 0 0.05 0.1 0.15 –0.1 –0.05 0 0.05 0.1 0.15 E ect size E ect size Statistically significant at 10% Not statistically significant at 10% Error bar shows confidence interval at 95% Note: This figure shows the amount of health subsidies women receive for pre- and postnatal care and childbirth. Compared with Wave III, women received substantially less in health subsidies in Wave IV. are control regions, although by Wave IV the effect is Village expenditures for both intensive and nonintensive substantively smaller, particularly for childbirth. One PMT have declined since 2009 (Wave III). potential reason for this decrease is the expansion of Given the overall shift in programming priorities from GoI’s national health insurance program during this time, increasing school enrollment and participation rates, it is which led communities to increasingly choose not to not surprising to see in Wave IV that Generasi is producing spend block grant funds on health-related subsidies. substantially weaker effects on spending geared toward Wave IV demonstrates a significant Generasi effect in enrollment and participation-boosting activities (Figure 14). communities receiving supplementary food (PMT) at the Generasi areas are significantly less likely to receive posyandu,9 although the magnitude is half of what was education scholarships than are control areas, and the found in Wave III (Figure 12). The effect of Generasi on positive effects on receipt of uniforms, supplies, and other intensive PMT (receiving supplementary food at least four types of support in Wave III become small or disappear times a month) decreases substantially from Wave III to Wave IV and is not significantly different from zero. For FIGURE 11: A Mother and Infant Receive the new Wave IV indicators (number of days receiving Health Services at the Posyandu PMT in the past three months for underweight and unrestricted samples), there is a small but significantly positive effect among all children and a larger and significant effect among underweight children. The slight decrease in PMT access from Wave III to Wave IV is reflected in expenditure data (Figure 13). 9 There are two types of PMT: PMT that is distributed at the posyandu, which is often a low–nutritional-content snack used to incentivize attendance at the posyandu, and nutritious PMT that is distributed at community health centers to treat malnutrition. 22 INDONESIA MAIN RESULTS FIGURE 12: Impact of Generasi on Receipt of PMT Wave III Wave IV Receive PMT posvandu Receive PMT intensive Receive PMT (if underweight) Receive PMT (unrestricted) -0.2 -0.1 0 0.1 0.2 0.3 0.4 -0.2 -0.1 0 0.1 0.2 0.3 0.4 E ect size E ect size Statistically significant at 10 % Days receive PMT (if underweight) Not statistically significant at 10 % Error bar shows confidence interval at 95 % Days receive PMT (unrestricted) 0 2 4 6 8 10 12 E ect size Note: Mothers are receiving more PMT at the posyandu in Generasi than in control villages, although the magnitude is half of what was found in Wave III. By comparison, there are no statistically significant differences in the amount of intensive PMT that households in Generasi and control villages are receiving. There is a small but significant difference in the number of days children (unrestricted sample) receive PMT. FIGURE 13: PMT Expenditure According to MIS Data 10 IDR, Millions 0 2006 2008 2010 2012 2014 2016 Year Received non-intensive PMT Received intensive PMT Note: The slight decrease in PMT access from Wave III to Wave IV is reflected in this expenditure data. Village expenditures for intensive and nonintensive PMT have declined since 2009 (Wave III). LONG-TERM GENERASI IMPACT EVALUATION 23 MAIN RESULTS FIGURE 14: Impact on Education Benefits Wave III Wave IV Received scholarship Received uniform Plot Area Received other supplies Received transport subsidy Received other school support -0.1 -0.05 0 0.05 0.1 0.15 -0.1 -0.05 0 0.05 0.1 0.15 E ect size E ect size Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Note: In Wave IV, Generasi is producing substantially weaker effects on spending geared toward school enrollment and participation-boosting activities than what was observed in Wave III. Children in Generasi areas are significantly less likely to receive education scholarships than are children in control areas. The positive effects on receipt of uniforms, supplies, and other types of support in Wave III are either small or disappear in Wave IV. entirely in Wave IV; the average standardized effects for is examined on an analogous indicator from the household direct education benefits are not statistically significantly survey. The average standardized effect is assessed first different from zero (see Appendix Table 3). because, as discussed, it represents a statistically efficient way of pooling all the effects to maximize statistical power given that there is insufficient statistical power to detect PROGRAM IMPACT ON MAIN effects on individual indicators. Although Generasi may have TARGETED INDICATORS affected the average of the indicators, this does not mean it This section describes the impact on the primary health affected all of them individually. Conversely, given that some indicators and secondary education indicators after nine indicators used in the study have weak statistical power, years of program implementation. For each indicator it is possible that Generasi is affecting more than just the provided to the villagers for improvement, Generasi’s impact indicators that are individually statistically significant. Overall, improvements on target health indicators in Wave III FIGURE 15: Students in School are found to be broadly similar in magnitude for Wave IV but often do not reach the same level of statistical significance. Specifically, the program’s average standardized effects (Figure 16) on health are slightly smaller in Wave IV than Wave III and fall just below statistically significant levels— the average standardized effect for health in Wave IV is 0.025 standard deviations (P value 0.162), compared with 0.039 in Wave III. There is also a large change in target education indicators from Wave III to Wave IV. Although the average standardized effect for education is large and statistically significant in Wave III, the same metric is effectively zero in Wave IV. 24 INDONESIA MAIN RESULTS FIGURE 16: Average Standardized Effects Wave III Wave IV ASE total ASE health ASE education -0.05 0 0.05 0.1 0.15 -0.05 0 0.05 0.1 0.15 E ect size E ect size Statistically significant at 10 % Note: The program’s average standardized effects on health are slightly smaller in Wave IV than Wave III and fall just below statistically significant levels. The average standardized effect for education is large and statistically significant in Wave III, but the same indicator is effectively zero in Wave IV, ASE = average standardized effects For the individual target health indicators, there are strong the main change is that the reduction in underweight effects on growth monitoring in Wave IV (Figure 17): (weight for age) that was associated with Generasi in Generasi led to about 0.13 more weight checks for Wave III is no longer present in Wave IV. The indicators children, an increase of about 6% compared with the that were not found to have significant changes in control group. This is similar to the Wave III effect of Wave III (for example, iron pill uptake) continued to about 0.19 more weight checks (8.3% increase). However, show no significant improvement in Wave IV. FIGURE 17: Impact on Health Targets Wave III Wave IV Prenatal care Delivery Postnatal care Iron pills Immunization Weight checks Vitamin A Underweight -0.5 -0.3 -0.1 0.1 0.3 0.5 -0.5 -0.3 -0.1 0.1 0.3 0.5 E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: The effect of Generasi on growth monitoring (0.13 more weight checks for children, an increase of about 6% compared with the control group) is similar to the Wave III effect (about 0.19 more weight checks, a 8.3% increase). In Wave III, Generasi reduced underweight (weight for age), but the effect is no longer present in Wave IV. Alternative visualizations of these results, which show the trends over time in control and treatment areas relative to the treatment effects, are available in Annex Figure 1. LONG-TERM GENERASI IMPACT EVALUATION 25 MAIN RESULTS FIGURE 18: Impact on Education Targets Wave III Wave IV 7–12 participation rate Participation rate for 13-15 in SMP -0.05 0 0.05 0.1 -0.05 0 0.05 0.1 E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: The positive effects of Generasi on the school participation rates of primary and junior secondary school students that were present in Wave III are no longer present in Wave IV. The positive effects on secondary education indicators Figure 19 presents results corresponding to the new detected in the Wave III evaluation disappeared in indicators that were added in 2014. Overall, Generasi Wave IV (Figure 18). The current evaluation round increased the rate of participation among mothers and found no significant improvements in school participation pregnant women in parenting and prenatal classes, rates among primary or junior secondary school respectively, but did not change the rate of enrollment of students. special needs children. Specifically, for mothers of young FIGURE 19: Impact on New Indicators Enrollment for children Parenting Class – Wave IV Prenatal Class – Wave IV with special needs – Wave IV Attendance, mothers with kids under 5 Attendance, all mothers Attendance, women who have been pregnant in the Enrolled in school, Frequency of past 24 months special needs children attendance, mothers with kids under 5 Frequency of attendance, all mothers 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 -0.1 -0.05 0 0.05 0.1 E ect size E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: This figure presents results corresponding to the indicators that were added in 2014. Generasi increased the rate of participation among mothers and pregnant women in parenting and prenatal classes, but did not change the rate of school enrollment of special needs children. 26 INDONESIA MAIN RESULTS FIGURE 20: Average Standardized Effects (Including New Indicators) Wave IV ASE total ASE health ASE education -0.15 -0.1 -0.05 0 0.05 0.1 0.15 E ect size Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Note: The average standardized effect for the revised set of health indicators and the revised overall indicators is positive and statistically significant. The average standardized effect for education is statistically insignificant. ASE = average standardized effects children the likelihood of attending a parenting class HETEROGENEITY increased by 8 percentage points (73% increase compared Heterogeneity in program effectiveness was compared with control areas), whereas the frequency of attendance with control group levels of the target indicators. Generasi increased by 0.28 classes on average. For pregnant women, locations were stratified into three terciles based on the rate of participation in prenatal classes10 is roughly control group performance levels in the same district, and 0.08 visits higher in Generasi programming areas (25%). the program’s impact for each tercile was reestimated Incorporating the largely positive effects of the new using the “endogenous stratification” method of Abadie indicators into the calculation of average standardized et al. (2013) to group subdistricts into three terciles based effects yields significantly positive changes overall and on predicted levels of the outcome variable in the control for health (Figure 20). As expected, given the null effects for the new education targets, education effects remain FIGURE 21: Mothers Attend a Parenting statistically insignificant. Class at the Posyandu To summarize, Generasi is mobilizing community members to attend the posyandu for infant weighing and prenatal and parenting classes. The main change between Waves III and IV is that the reduction in underweight (weight for age) that was associated with Generasi in Wave III is no longer present in Wave IV. Further, unlike in Wave III, the current evaluation round does not find any significant improvements in school participation rates among primary or junior secondary school students. At the time of evaluation, parenting classes were held in only 10 approximately 20% of treatment areas. LONG-TERM GENERASI IMPACT EVALUATION 27 MAIN RESULTS areas. This analysis is performed for Waves III and IV to FIGURE 23: Distributing PMT at the compare results over time. Posyandu There is some evidence that Generasi is more effective in areas where the needs are greatest (see Appendix Table 15). In particular, for weight checks, immunizations, and vitamin A supplementation, the largest impacts were found in Tercile 1, which is the group of subdistricts predicted to have the lowest outcomes on average (see Figure 24). In Wave III, the program was also found to be most effective at improving elementary school enrollments in the lowest tercile; this effect did not persist in Wave IV. PROGRAM IMPACT ON LONG-TERM OUTCOMES Overall, the improvements made to malnutrition rates This section describes the project’s impact on primary (underweight) in Wave III did not persist in Wave IV and secondary malnutrition outcomes. Primary indicators (Figure 25), and there were no improvements in cognitive of malnutrition included underweight (defined as assessments based on the Raven score.11 ≤2SD weight for age), severe underweight (defined as These outcomes were investigated specifically for NTT ≤3SD weight for age), stunting (defined as ≤2SD height province, where the Wave III improvements in malnutrition for age), severe stunting (defined as ≤3SD height for and stunting were most pronounced. However, Figure 26 age), wasting (defined as ≤2SD weight for height), and shows that the malnutrition indicators appear to have severe wasting (defined as ≤3SD weight for height) all for significantly improved in NTT province in Wave IV; children under three. They also include the Raven score, an if anything, wasting rates appear to have worsened age-adjusted cognitive assessment test. in Generasi programming regions. FIGURE 22: Weighing Infants at the PROGRAM IMPACT ON Posyandu SECONDARY FINAL OUTCOMES AND NONTARGETED OUTCOMES Consistent with the preanalysis plan, some secondary outcomes are also examined. Figure 27 presents results on neonatal (0–28 days) and infant (0–11 months) mortality, as well as home-based test scores (age-adjusted Z-scores). The lack of improvement in these outcomes 11 With respect to Generasi’s impact on Raven scores, the null finding may be a function of low statistical power. Improvements in Raven scores were hypothesized to occur via a reduction in childhood stunting rates, which in turn were thought to affect cognitive outcomes. The estimates for the magnitude of this effect are small and unlikely to be detectable given the sample size. 28 INDONESIA MAIN RESULTS FIGURE 24: Heterogeneity Based on Areas Most in Need (Weight Checks, Immunizations, and Vitamin A Supplements) Wave III Wave IV Weight checks Weight checks Quantile 1 Quantile 2 Quantile 3 –0.1 0 0.1 0.2 0.3 0.4 –0.1 0 0.1 0.2 0.3 0.4 E ect size E ect size Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Wave III Wave IV Immunization uptake Immunization uptake Quantile 1 Quantile 2 Quantile 3 –0.06 –0.04 –0.02 0 0.02 0.04 0.06 0.08 0.1 –0.1 –0.05 0 0.05 0.1 E ect size E ect size Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Wave III Wave IV Vitamin A Vitamin A Quantile 1 Quantile 2 Quantile 3 –0.4 –0.2 0 0.2 0.4 –0.4 –0.2 0 0.2 0.4 E ect size E ect size Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Note: This figure shows that Generasi is having positive effects on weight checks, immunizations, and vitamin A supplementation in the poorest subdistricts. LONG-TERM GENERASI IMPACT EVALUATION 29 FIGURE 25: Impact on Malnutrition Outcomes for Generasi IE Sample Wave III Wave IV Underweight Severe underweight Wasting Severe wasting Stunting Severe stunting Raven score -0.08 -0.04 0 0.04 0.08 -0.08 -0.04 0 0.04 0.08 E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: Generasi’s reduction of underweight that was observed in Wave III is not present in Wave IV. In Wave IV, no effects of Generasi on wasting, stunting, or cognitive ability (as measured by the Raven test) were observed. Alternative visualizations of the trends over time relative to the treatment effects are available in Annex Figure 2. FIGURE 26: Impact on Malnutrition Outcomes in NTT Wave III Wave IV Underweight Severe underweight Wasting Severe wasting Stunting Severe stunting -0.2 -0.1 0 0.1 0.2 -0.2 -0.1 0 0.1 0.2 E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: Generasi’s reduction of underweight and severe stunting in NTT province that was observed in Wave III is no longer present in Wave IV. FIGURE 27: Impact on Secondary Final Outcomes Wave III Wave IV Neonatal mortality Infant mortality Language score Math score Test score -0.15 -0.1 -0.05 0 0.05 0.1 0.15 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 E ect size E ect size Statistically significant at 10 % Error bar shows confidence interval at 95 % Note: Similar to Wave III, Generasi did not appear to have an impact on neonatal or infant mortality or children’s learning outcomes in Wave IV. 30 INDONESIA MAIN RESULTS observed in Wave III is found to continue through Wave IV. quantity of health and education service providers at None of the outcomes appears to improve in Wave IV as a the village level. This analysis reveals that midwives result of Generasi programming effects.12 in Generasi villages tend to work more hours than do those in control areas (Appendix Table 12). In addition, As in the 2011 Wave III report, this IE examines Generasi’s Generasi had a positive impact on the factual health- impact on service delivery outcomes. It explores the related knowledge that mothers receive from the various channels through which Generasi could have posyandu about the proper care of young children. affected basic health and education services using data Generasi locations are also more likely to have a primary from the provider surveys, focusing on changes in the and secondary junior school than are control areas (Appendix Table 10), and secondary junior schools in 12 The analysis also assessed whether there are long-term impacts on Generasi villages tend to be in better condition than stunting, participation in tertiary school, age at first marriage, and wages. There were no significant impacts on any of these indicators. those in control areas. LONG-TERM GENERASI IMPACT EVALUATION 31 UNDERSTANDING CHANGES SINCE 2009 Although Generasi continued to have an impact on growth monitoring, several of the other impacts—most notably the improvements in malnutrition, the reduction in stunting in NTT province, and the improvements in enrollment—do not seem to have persisted through the 2016 evaluation. The lack of an effect on enrollments can be at least partially explained by the change in funding emphasis toward health, but it is less obvious why the improvements in malnutrition did not persist; indeed, a main goal of this IE was to test whether Generasi has produced continued improvements in malnutrition. One likely explanation is that the smaller effects of Generasi can be attributed to the improvements in the overall health and education environment that occurred across Indonesia, affecting both Generasi control and treatment locations. These overall improvements have been particularly large in historically poorly performing districts, areas where the Generasi effects were found to be the strongest in the 2009 IE. As a result of these general improvements in poorly performing areas, there is less room for improvement on many of Generasi’s targets for the Wave IV evaluation. INCREASE IN OTHER HEALTH AND EDUCATION PROGRAMMING Since 2009, Generasi IE districts have experienced overall improvements in access to health and education due to substantial changes in national policy bearing on health and education (Figure 28). The number of social protection programs in control villages has expanded over time, particularly in the areas of health and education.13 13 For example, 92% of control villages report having a Health Indonesia Card, 91% report having PKH, 71% report having the School Operational Assistance Program, 65% report having the Smart Card Indonesia, 44% report having PNPM, and 10% report having the Family Welfare Card. 32 INDONESIA UNDERSTANDING CHANGES SINCE 2009 FIGURE 28:  New Social Protection Programs over Time 0.8 0.7 Average number of new programs in 0.6 0.5 control villages 0.4 0.3 0.2 0.1 0 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Cash subsidy Health Education Community empowerment Clean water Note: Since 2007, the number of social protection programs in control villages (shown in this figure), has grown substantially. Programs related to labor-intensive growth, micro credit growth, or subsidized commodities growth were not graphed because they stayed relatively constant over this time period and are not as relevant to Generasi’s objectives. This time series is restricted to begin in 2007, but the earliest start year on record in the data is 1971. The most significant general policy change is the Congruent with these policy changes, there is evidence enactment of the Village Law in 2014, which drastically of a general trend towards improved access to health increased village budgets and the ability of local services in control areas. Figure 29 depicts changes over governments to fund improvements in access to health time in key health indicators in control areas. Deliveries and education services. The expansion of a subsidized attended by a doctor or midwife, which at baseline were public health insurance program and the launching of estimated at 70%, rose to a high of 92% in Wave IV. an integrated National Health Insurance system in 2014 Prenatal care visits also increased, on average, by an extra markedly increased citizens’ access to health insurance to visit per 24-month period. This general improvement an estimated 70% of the population by 2017; the program trend renders Generasi treatment effects more difficult to aims for full coverage by 2019. identify. A constitutional mandate to allocate 20% of the national There is a similar trend in access to education in control budget to education resulted in a doubling of public areas over time (Figure 30). Participation rates for education spending between 2001 and 2009. In addition, both primary and junior secondary children increased a cash transfer program for poor students was introduced substantially, with junior secondary participation in 2008. By 2014, program coverage had expanded from rates rising to over 70% from a baseline of 59%. More 4.5 million to 11.2 million poor students, and the program importantly, the high baseline level in participation was upgraded to the Smart Indonesia Program to target outcomes, particularly among primary school-aged enrolled students as well as dropouts. These developments children, means that there is little room for improvement in education and health policy may have yielded moving forward. This dynamic partially motivated the improvements across Generasi IE locations that decreased decision to refocus Generasi educational targets away the impact of the Generasi program. from school participation rates in 2014. LONG-TERM GENERASI IMPACT EVALUATION 33 UNDERSTANDING CHANGES SINCE 2009 FIGURE 29:  Evolution of Control Areas over Time, Key Health Indicators Prenatal care Weight checks 12.0 3.0 8.0 2.0 2.3 8.5 2.2 2.2 2.2 7.6 7.5 7.5 4.0 1.0 0.0 0.0 2005 2007 2009 2011 2013 2015 2017 2005 2007 2009 2011 2013 2015 2017 Iron pills Delivery with doctor or midwife 3.0 1.00 2.0 0.92 2.0 1.7 2.2 0.50 0.70 0.75 0.76 1.0 1.6 0.0 0.00 2005 2007 2009 2011 2013 2015 2017 2005 2007 2009 2011 2013 2015 2017 Mean POVERTYACTIONLAB.ORG Survey conducted Note: Since 2007, there have been improvements in key health indicators. Deliveries attended by a doctor or midwife rose from 70% at baseline to 92% in Wave IV. Prenatal care visits also increased, on average, by an extra visit per 24-month period. This general improvement in health indicators may make additional marginal improvements from the Generasi program more difficult. FIGURE 30:  Evolution of Control Areas over Time, Key Education Indicators 7-12 participation rate 13-15 participation rate in SMP 1.00 1.00 0.90 0.98 0.98 0.98 0.90 0.95 0.80 0.80 0.70 0.70 0.71 0.60 0.60 0.63 0.66 0.59 0.50 0.50 0.40 0.40 0.30 0.30 0.20 0.20 0.10 0.10 0.00 0.00 2005 2007 2009 2011 2013 2015 2017 2005 2007 2009 2011 2013 2015 2017 Mean Survey conducted Note: Since 2007, there have also been major improvements in education access. The baseline for the participation of primary school-aged children was high in 2007. Junior secondary participation rates increased to over 70% in 2016 from a baseline of 59%. These shifts, in part, motivated the decision in 2014 to refocus Generasi educational targets away from school participation rates. 34 INDONESIA UNDERSTANDING CHANGES SINCE 2009 Overall, these patterns are consistent with the hypothesis 4. Generasi’s effects on stunting were limited because that the lack of sustained impact is the result of there being the full suite of complementary demand- and less room for improvement in Generasi IE regions over time. supply-side interventions needed to address These areas (particularly control regions) have improved stunting were not fully implemented. their access to health and education, making it difficult for Generasi to continue producing effects in these indicators. Hypothesis 1: General Improvements in Stunting WHY NO CONTINUED PROGRAM There is evidence of a general and substantial decrease IMPACT ON MALNUTRITION in stunting rates across control and treatment areas. This OUTCOMES? decrease is particularly strong in NTT province, which is consistent with Hypothesis 1. Figure 31 presents stunting This section explores four possible hypotheses for why and malnutrition (underweight) rates in control and continued sustained improvements in malnutrition treatment areas over time. Stunting in particular decreases outcomes are not observed in Wave IV: drastically over time, from a high of 40% at the time of 1. The overall substantial improvements in stunting Wave III to a low of 26% during Wave IV. in NTT that occurred in both control and treatment Given how striking the declines were, a second analysis areas may have exhausted the “low-hanging fruit” was conducted to determine whether similar decreases that Generasi was able to address in earlier periods. in stunting trends for infants and young children were 2. Generasi funding produced crowd-in/crowd-out observed in the IE of the PKH program, which samples effects on other program resources that undercut overall poorer households from Generasi but which the efficacy of the intervention. was conducted using the same survey instruments and 3. Implementation issues and delays in the maternal by the same firm. Figure 31 shows similar declines in health and parenting classes may have weakened stunting for both programs over the same period. In any potentially positive impacts this intervention may fact, the overall decrease in stunting is still evident if the have had on behavioral change and malnutrition. comparison between PKH and Generasi areas is restricted FIGURE 31:  Evolution of Control Areas over Time, Health Indicators (children 0–3) Stunting Underweight 60% 60% 50% 50% 44.0% 40% 40% 39.9% 31.9% 30% 30% 27.9% 21.8% 25.9% 25.4% 20% 20% 19.8% 10% 10% 0% 0% 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Generasi PKH Generasi PKH Age-weighted mean Survey conducted Note: There was a substantial decrease in underweight and stunting rates among infants and young children in Generasi IE districts between 2009 and 2016. A similar trend was observed in the PKH program IE that took place in 2013. LONG-TERM GENERASI IMPACT EVALUATION 35 UNDERSTANDING CHANGES SINCE 2009 FIGURE 32:  Generasi Stunting Trends: Compared with PKH, Restricted to Comparable Subdistricts 60% 50% 44.6% Age-weighted percentage 40% 35.7% 40.2% 30% 20% 25.8% 10% 0% 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Year Generasi PKH Note: The trend of declining stunting rates in treatment and control areas in the Generasi and PKH IEs is present when the sample is restricted to comparable subdistricts from both surveys. to comparable subdistricts (Figure 32).14 In PKH IE control Hypothesis 2: Crowd-In/Crowd-Out areas, stunting rates dropped from a high of 45% in Effects 2009 to 36% in 2013, which is similar in magnitude to the This section assesses whether Generasi programming is observed drop in Generasi control areas (Figure 31). This crowding out resources from other programs in treatment evidence is consistent with the general improvement in areas or crowding in resources to control areas, thereby stunting rates described in Hypothesis 1. negating any positive impact on malnutrition or program Declines in stunting were particularly marked in NTT, the targets. To evaluate this possibility, village-level funding lowest-performing province at baseline. Stunting rates patterns were explored from non-Generasi programs in dropped from a high of 50% at baseline to approximately Generasi IE areas. 30% during Wave IV, bringing rates in NTT much closer to that of the other IE provinces. This is consistent with The analysis finds no evidence that control areas received the elimination of “low-hanging fruit” making sustained support from programs that was not also provided to effects less likely, as described in Hypothesis 1. It is also treatment areas. Few statistically significant differences worth noting just how substantial these declines in stunting were found in the revenue that community health are—a decline of over 2.5 percentage points per year in centers and schools receive across Generasi IE areas. the stunting rate is at the upper end of decreases that The differences that were revealed are to be expected have been observed elsewhere (for example, across all given the large number of tests considered. Overall, developing countries, under-five stunting declined from no quantitative evidence was found that Generasi is 44.6% in 1990 to 28.0% in 2011, or about 0.79 percentage crowding in or crowding out resources from other points per year). programs or funding sources. To assess whether there were crowd-in/crowd-out concerns related to the enactment of the Village Law, which resulted 14 Generasi and PKH subdistricts are considered to be comparable if they in a massive increase in village government budgets, Village are both from NTT, West Java, or East Java provinces, are sufficiently rural, and meet other Generasi programming criteria. Law budget data were collected to explore differences in 36 INDONESIA UNDERSTANDING CHANGES SINCE 2009 FIGURE 33:  Generasi Stunting Trends: Control Group by Province (age 0–3) 60% Age-weighted percentage 50% 40% 30% 20% 10% 0% 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Year Jawa Barat Jawa Timur NTT Sulawesi Utara Gorontalo Note: Stunting among infants and children declined in all Generasi IE provinces between 2009 and 2016. Declines in stunting were particularly striking in NTT, the lowest-performing province at baseline. Stunting rates dropped from a high of 50% at baseline to approximately 30% during Wave IV, bringing rates in NTT much closer to that of the other IE provinces. the composition of Village Law expenditures on health Generasi experienced a six-month programming and education across control and treatment areas. No interruption in 2015 as the program transitioned from significant differences were found in how Generasi and MoHA to the newly established MoV. Program funds could control villages are spending Village Law funds. not be withdrawn during this period, which delayed the implementation of Generasi’s program cycle. In particular, To conclude, there is no evidence that Generasi is the program disruption had negative consequences for crowding in or crowding out resources from other the delivery of interventions. To assess the impact of this programs or funding sources. There are no differences interruption, a joint team made up of representatives in how village governments are allocating Village from J-PAL, Kompak, Bappenas, and the World Bank Law funds. conducted a series of qualitative field missions to Generasi Hypothesis 3: Implementation Delays program areas between June and August 2015. The resulting qualitative study revealed that implementation This section examines Hypothesis 3, which suggests of the new indicators was still limited because of a lack of Generasi’s weak long-term impact is a function of understanding by program actors, supply-side problems, implementation delays related to two of the new and the program disruption. Interviews with program indicators: participation of pregnant women and facilitators found that some expressed confusion about male partners in nutrition counseling offered through the new indicators or how to address them, especially the maternal health classes, and participation of parents counseling session indicator. (and/or caregivers) in nutrition counseling offered through classes for infants. With the 2014 revision Figure 34 depicts the progress in training for various staff of program indicators, the Generasi program was in Generasi areas. Programmers intended for training to attempting to fight malnutrition by improving parental begin in treatment locations in Semester I (January to nutrition education. Regardless of whether this strategy June) of 2015; however, because of the disruption training works, it would have had the possibility of working did not begin until at least Semester II (July to December only if the nutrition education was delivered on time 2015). These delays in training and service delivery may and at scale. have rendered Generasi effects less likely. Although there LONG-TERM GENERASI IMPACT EVALUATION 37 UNDERSTANDING CHANGES SINCE 2009 FIGURE 34: Progress in Training: Cumulative Supply in Five Provinces 120% Cumulative percent of target realized 100% 80% 60% 40% 20% 0% 1 2 3 4 5 6 7 8 Community health Village midwife Posyandu cadres Community health center sta Note: In 2014, Generasi introduced targets sta center achieve for participation in maternal health and parenting classes. To help communities these targets, the Generasi program had intended to train posyandu cadres in nutrition counseling that would be offered through these classes. Although the training was supposed to begin in the first half of 2015, it did not begin until the second half of 2015. This delay may have rendered Generasi effects less likely at the time of the Wave IV survey. was an increase in the number of parenting classes held, extent did the disruption of activities affect communities’ they were delayed and therefore were not as effective achievement of the long-term outcomes? as they could have been. To answer this question, the findings from this 2015 small–N The 2015 qualitative study found additional evidence that qualitative study were supplemented with quantitative the disruption interfered with the delivery of other activities analysis of the household survey. To test whether the in Generasi areas beyond the training of health workers and disruption produced detrimental effects in Generasi areas, volunteers. In some villages, posyandu stopped providing maternal, young child, and infant outcomes were examined PMT and experienced drops in attendance. Only a few before, during, and after the disruption. If the disruption villages covered PMT activities during this period, and significantly worsened Generasi service delivery and in general local and district government response to the uptake, worse outcomes than comparable populations disruption was minimal. Further, transportation subsidies before or after the disruption should be expected for for pregnant women were deferred; reimbursements were women who were pregnant or gave birth during the given once funding resumed. disruption, infants aged 0–2 during the disruption, and young children finishing primary school during this time. The joint 2015 qualitative study found that most subdistrict facilitators remained at their posts during the program Figure 35 provides a descriptive representation of this interruption and shifted their attention to activities that analysis. The graph depicts three-month moving averages were less funding dependent until the program resumed. for four different maternal outcomes in one-month Many of the facilitators who did leave accepted other village increments after the disruption. Starting from January posts (for example, in the village administration). Overall, 2015, each number on the horizontal axis represents a the study found that program actors were optimistic one-month interval until December 2016. If the disruption that their village would reach program targets. To what had a noticeable impact on the quality of service delivery, 38 INDONESIA UNDERSTANDING CHANGES SINCE 2009 FIGURE 35:  Maternal Outcomes, Three-Month Moving Averages, January 2015 to December 2016 0.2 1 0.8 0.6 0.4 0.2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Pre-natal dummy Iron pills Good assisted delivery Post-natal dummy Note: This figure shows three-month moving averages for four different maternal outcomes in one-month increments after the implementation disruption. There is no evidence that the disruption produced detrimental effects in Generasi areas for maternal outcomes, or outcomes for young children or infants. a significant change in maternal outcomes should be mothers and infants, who would have been the most likely visible at some point in the time series. However, the four to be negatively affected by the delays. outcomes appear consistent over time. Hypothesis 4: Full Suite of The regression analysis reveals no evidence of differentially Complementary Interventions worse outcomes for the populations most likely to be Needed to Address Stunting negatively affected by the disruption (for example, in Were not Fully Implemented neonatal mortality or height-for-weight indicators). Weight checks at posyandu were also found to have continued This section considers evidence to adjudicate Hypothesis 4, unabated, suggesting that key programming interventions which suggests that Generasi’s effects on stunting were did not stop during the disruption. These results, combined limited because the full suite of complementary demand- and supply-side interventions needed to address stunting with the findings from the qualitative study, show that it is were not fully implemented. In 2014, there were design unlikely the disruption was responsible for the weak effects changes to Generasi, including the addition of targets found in Generasi Wave IV. around participation in prenatal and parenting classes that To conclude, in 2015 there was a program interruption aimed to change behaviors around diet diversity during that delayed the implementation of Generasi’s program pregnancy, exclusive breastfeeding, complementary cycle. In addition, there were delays in the training of feeding, and hygiene. Generasi did not invest directly posyandu cadres to deliver nutrition counseling in prenatal in complementary clean water supply, toilets in houses, and parenting classes. These delays may have rendered and sanitation systems. These types of infrastructure Generasi interventions less effective during this period. investments were not possible because GoI was putting less Yet there is no evidence to support this hypothesis. money into the project from 2010 onward. Within Generasi, Specifically, there is no evidence of worse outcomes for infrastructure investments (including in water) mostly LONG-TERM GENERASI IMPACT EVALUATION 39 UNDERSTANDING CHANGES SINCE 2009 ceased. GoI expected other programs, such as the socioeconomic status index, αtp is a wave-province fixed National Rural Water Supply and Sanitation Project and effect, αk is a subdistrict fixed effect, and αa is an age fixed Community-Led Total Sanitation, to invest in water and effect. Standard errors are clustered at the subdistrict level. sanitation infrastructure and behavioral change in Generasi This approach effectively regresses changes in stunting subdistricts. Yet there is no evidence that these investments rates from 2009 to 2016 on changes in variables related systematically took place in Generasi areas to complement to the child, mother, household, and village environments Generasi. An analysis of the Generasi IE data found no over the same period. This analysis is performed for over differences in sanitation and water programs between 50 different sets of explanatory variables to determine treatment and control areas. which factors are correlated with the observed decline This analysis does not, of course, explain the decline in in stunting between Wave III and Wave IV (2016). This Generasi’s effectiveness from 2009 to 2016. To the extent analysis is not causal, but is meant to provide suggestive that other important drivers of stunting are identified, evidence of which factors are strongly associated with it suggests why the program did not do more overall to stunting. Potential explanatory variables are expressed reduce stunting and what other approaches to stunting as subdistrict averages unless otherwise noted. Table 3 reduction may be effective. summarizes the results of this analysis. To explore what factors are associated with stunting Subdistricts with villages that rely on lake, spring, and declines in the data, a difference-in-differences mineral water over this time period tended to report econometric approach was used, with time, province, higher probabilities of a child being stunted. For example, subdistrict, and age fixed effects, and controls for subdistricts with villages that use lake water for cooking household assets. This regression takes the following form: and drinking were 39 percentage points more likely to y iakt = β 1varkt + γ 1SESkt + α tp + α k + α a + ε iakt report a child as being stunted than were those that do not use lake water (Figure 36). These findings reflect where i is a child, k is a subdistrict, p is a province, t is a those in the literature on the importance of clean water survey wave (2009 or 2016), and a is a three-month age sources for cooking and drinking in reducing stunting group. var is the explanatory variable and yiakt is the outcome (for example, Dillingham and Guerrant 2004). variable, which is a dummy that indicates whether child i is stunted in wave t, subdistrict k, and age group a. Finally, There is also some suggestive evidence that subdistricts β1 is the parameter of interest, SESkt is child i’s predicted that relied on public latrines tended to have lower stunting TABLE 3:  Results of Stunting Difference-in-Differences Analysis Variables Association Variables Association Clean water sources  Maternal knowledge  Clean water programs  Maternal education  Latrine use  Health education  Height measurement  Open garbage disposal  PAUD  Exclusive breastfeeding   lower stunting   no assocation   higher stunting Note: To what extent are changes in stunting rates from 2009 to 2016 associated with changes in variables related to the child, mother, household, and village environments over the same period? Results suggest that changes in clean water and latrine use, height measurement, and Early Childhood Education and Development (PAUD) attendance are associated with reductions in stunting. 40 INDONESIA UNDERSTANDING CHANGES SINCE 2009 FIGURE 36:  Stunting Association with Sources of Water Used for Cooking and Drinking Piped water Pump well water Well water Rain water Lake water Spring water River or stream water Aqua or mineral water -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Association with stunting rates Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Note: Subdistricts with villages that increasingly (decreasingly) rely on lake, spring, and mineral water show increases (decreases) in the probability of a child being stunted from 2009 to 2016. probabilities over this time period compared with those this time period than did those without. Conversely, that have no latrine (Figure 37). On average, subdistricts changes in health education and maternal knowledge with public latrines are about 40 percentage points less levels did not appear to correlate with changes in likely to report stunting than are those with no latrine at all stunting rates. over this time period. Although this analysis does not explain the change from These results suggest the importance of variables 2009 to 2016, it is worth noting that because the Generasi associated with clean water use and access to latrines. program was targeted at improving maternal knowledge On average, villages and subdistricts with access to both rather than infrastructure investments in clean water and tended to experience a steeper decline in stunting over sanitation, its ability to reduce stunting may have been limited. FIGURE 37:  Stunting Association with Latrine Use Wave IV: Latrine use, Wave IV: Latrine use, Wave IV Subdistrict level Individual level Open defecation Own latrine Own latrine subdistrict level Plot Area Shared latrine Shared latrine Open defecation individual level Public latrine Public latrine -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 -0.1 -0.05 0 0.05 0.1 Association with stunting rates Association with stunting rates Association with stunting rates Statistically significant at 10 % Not statistically significant at 10 % Error bar shows confidence interval at 95 % Open defecation means no latrine is used (compared to a private, shared, or public latrine). Latrine responses are relative to no latrine at all. In models with slightly varied controls, open defecation and own latrine use have a statistically significant association with stunting rates. Note: Subdistricts that increasingly (decreasingly) relied on public latrines tended to show decreases (increases) in stunting probabilities compared with subdistricts with no latrines. LONG-TERM GENERASI IMPACT EVALUATION 41 CONCLUSION This document describes the findings of the long-term evaluation carried out in 2016. This evaluation was implemented nine years after program implementation and compares the results from the current survey wave to previous evaluation waves. The main findings of the Generasi IE are as follows. 77 Since 2009, the overall health and education environment in Generasi IE districts has improved dramatically, even in control areas. Vital health indicators, such as deliveries attended by a doctor or midwife, have increased substantially since 2009 and now account for over 92% of births in the sample area. Similarly, school participation rates have risen significantly since 2009: enrollment for school years 7–12 was 98% in 2016. These improvements likely reflect both substantial policy changes and improved household incomes throughout Indonesia. 77 There is now significantly less room for improvement in many Generasi target areas. For example, Generasi’s impact on reduced malnutrition and school enrollments that were present in Wave III are no longer observed in Wave IV. The IE also documents that there have been substantial improvements in precisely those indicators in both treatment and control areas compared with 2009. 77 One of Generasi’s greatest accomplishments is the sustained revitalization of the posyandu, which was accomplished through program facilitation, community participation, and a targets/incentive system. This system has been central to GoI’s efforts to curb infant/child mortality and provide citizens with family planning services since the early 1980s (Leimena 1989). By the late 1990s attendance at posyandu had decreased from 52% to 40% in both urban and rural areas, but with a greater decline in rural locations. 42 INDONESIA CONCLUSION FIGURE 38: Posyandu Activities Are prenatal attendance increased by 0.28 classes on Widely Used average. 77 In the lowest-performing districts, Generasi has continued to be effective at encouraging community members to attend the posyandu and increasing immunizations and vitamin A distribution. Nine years after program implementation, treatment areas in the lowest-performing tercile continue to experience a 0.19 increase in weight check frequency. In the same tercile, immunization rates increased by 3 percentage points (roughly 4% higher than control areas), whereas vitamin A uptake increased by 0.15 supplements (11% increase compared with control areas). 77 Generasi’s initial impact on stunting, concentrated in NTT province, has not been sustained beyond Reasons for the decline include a loss of support the 2009 IE. There are four possible reasons for from nongovernmental organizations and changing this. First, the overall substantial improvements in preferences for private providers in Indonesia stunting in NTT that occurred in both control and (Marks 2007). Despite these setbacks, community treatment areas may have exhausted the “low- participation in posyandu activities continues to hanging fruit” that Generasi was able to address improve nine years after program implementation. This participation has been sustained in part by in earlier periods. Second, Generasi funding communities choosing to allocate portions of their produced crowd-in/crowd-out effects on other Generasi block grants to fund interventions that program resources that undercut the efficacy of incentivize participation at the posyandu, such as the intervention. Third, implementation issues providing nutritional supplements to mothers who and delays in the maternal health and parenting attend, funding subsidies for pre- and postnatal care, classes may have weakened any potentially and remunerating posyandu volunteers. positive impacts this intervention may have had on behavioral change and malnutrition. Fourth, 77 Specifically, Generasi still helps mobilize community Generasi’s effects on stunting were limited because members to attend the posyandu for infant weighing the full suite of complementary demand- and and maternal health and parenting classes. Treatment supply-side interventions needed to address areas experienced 0.13 more weight checks, on stunting were not fully implemented. average, for young children in control areas (a 6% increase compared with control areas), as well as a 73% increase (8.5 percentage points) in attendance POLICY IMPLICATIONS of parenting classes compared with control areas, The evaluation results have three policy implications. particularly among mothers of young children. Prenatal class attendance also increased by 77 Future GoI health-related programming needs to 8 percentage points (25% increase compared with consider how to sustain the posyandu and ensure control areas) in treatment areas. The frequency of that mothers continue to bring their children LONG-TERM GENERASI IMPACT EVALUATION 43 CONCLUSION for weight/height measurement, participation 77 The results show that Generasi is effective at in Early Childhood Education and Development increasing basic service utilization in poor areas, programs, and basic maternal and infant health where baseline service delivery and health indicator services. An implementation disruption in Generasi levels are low but where there are at least some programming that occurred in 2015 when the elements of a functioning supply side. Generasi Generasi program transferred from MoHA to MoV was more effective in 2009, when baseline levels underscores the difficulty of maintaining posyandu of service delivery were much lower; it was most participation without incentives. The disruption effective then in the provinces and districts with meant that funding could not be spent on nutritional the lowest levels of baseline service delivery. Today, supplements, which based on qualitative field Generasi remains most effective at improving reports led to a reduction in posyandu attendance. weight checks, immunizations, and vitamin A in the The future of posyandu success depends on bottom third of districts in terms of predicted levels villages continuing to support participation in of achievement in the absence of the program. This the absence of Generasi. Across Indonesia, village suggests that GoI and other governments worldwide governments could use Village Law funds to that are trying to accelerate the achievement of basic health and education indicators could consider support the posyandu and continue to ensure that applying the Generasi model in contexts where posyandu are sufficiently staffed and compensated baseline levels of health service delivery are low. appropriately. GoI could encourage village governments to use Village Law funds to support 77 As this IE demonstrates, short- and long-term IEs posyandu either by prioritizing the health clinics at are essential to ensuring that government programs the central and district levels and/or incentivizing continue to have an impact as the programs and village governments to allocate resources for this contexts change. IEs can also inform governments purpose. about how to adjust targets appropriately. 44 INDONESIA REFERENCES Abadie, A., M. M. Chingos, and M. R. West. 2013. Endogenous Stratification in Randomized Experiments. Cambridge, MA: National Bureau of Economic Research. Athey, S., and G. W. Imbens. 2017. “The Econometrics of Randomized Exper­ iments.” In Handbook of Economic Field Experiments, Vol. 1, pp. 73–140. Amsterdam, Netherlands: Elsevier. Dillingham, R., and R. L. Guerrant. 2004. “Childhood Stunting: Measuring and Stemming the Staggering Costs of Inadequate Water and Sanitation.” The Lancet 363: 94–95. Duflo, E., R. Glennerster, and M. Kremer. 2007. “Using Randomization in Develop­ ment Economics Research: A Toolkit.” In Handbook of Development Economics, edited by T. P. Schultz and J. Strauss, 3895–62. Amsterdam, Netherlands: Elsevier. Gertler, P. 2004. “Do Conditional Cash Transfers Improve Child Health? Evidence from PROGRESA’s Control Randomized Experiment.” American Economic Review 94 (2): 336–41. Gibbons, R., and K. J. Murphy. 1990. “Relative Performance Evaluation for Chief Executive Officers.” Industrial and Labor Relations Review 43: 30–51. Gordis, L. 2004. Epidemiology. Philadelphia, PA: W. B. Saunders Company. Imbens, G., and J. Angrist. 1994. “Identification and Estimation of Local Average Treatment Effects.” Econometrica 61 (2): 467–76. Kling, J. R., J. B. Liebman, and L. F. Katz. 2007. “Experimental Analysis of Neighborhood Effects.” Econometrica 75 (1): 83–119. Lazear, E. P., and S. Rosen. 1981. “Rank-Order Tournaments as Optimum Labor Contracts.” Journal of Political Economy 89: 841–64. LONG-TERM GENERASI IMPACT EVALUATION 45 REFERENCES Leimena, S. L. 1989. “Posyandu: A Community-Based Vehi­ Skoufias, E. 2005. “PROGRESA and Its Impacts on cle to Improve Child Survival and Development.” Asia the Welfare of Rural Households in Mexico.” IFPRI Pacific Journal of Public Health 3 (4): 264–67. Research Report No. 139. Washington, DC: Inter­ national Food Policy Research Institute. Marks, G. C. 2007. “Changing Nutritional Health in South East Asia.” In Health Change in the Asia-Pacific Region, Weitzman, M. L. 1980. “The ‘Ratchet Principle’ and Per­ edited by Ryutaro Ohtsuka, 64–87. Cambridge: formance Incentives.” Bell Journal of Economics Cambridge University Press. (The RAND Corporation) 11 (1): 302–08. Miguel, E., and M. Kremer. 2004. “Worms: Identifying Wong, S. 2012. What Have Been the Impacts of World Impacts on Education and Health in the Presence of Bank Community-Driven Development Programs? Treatment Externalities.” Econometrica 72: 159–217. CDD Impact Evaluation Review and Operational and Research Implications. Washington, DC: Mookherjee, D. 1984. “Optimal Incentive Schemes with World Bank. Many Agents.” Review of Economic Studies 51: 433–46. World Bank. 2006. Making the New Indonesia Work for Newhouse J. P., and The Insurance Experiment Group. the Poor. Jakarta, Indonesia and Washington, DC: 1993. Free For All? Lessons from the Rand Health World Bank. Insurance Experiment. Cambridge, MA: Harvard University Press. ———. 2008. Investing in Indonesia’s Health: Challenges and Opportunities for Future Public Spending. Olken, B. A. 2007. “Monitoring Corruption: Evidence from Jakarta, Indonesia and Washington, DC: World Bank. a Field Experiment in Indonesia.” Quarterly Journal of Economics 115 (2): 200–49. ———. 2011. Indonesia’s Generasi Program Final Impact Evaluation Report. Washington, DC: World Bank. Olken, B. A., J. Onishi, and S. Wong. 2011. Indonesia PNPM Generasi Program Final Impact Evaluation Report. ———. 2015. An Unfair Start: How Unequal Opportunities Washington, DC: World Bank. Affect Indonesia’s Children. Jakarta, Indonesia and Washington, DC: World Bank. Schultz, T. P. 2004. “School Subsidies for the Poor: Evalu­ ating the Mexican Progresa Poverty Program.” Journal ———. 2017. World Development Indicators. Washington, of Development Economics 74 (1): 199–250. DC: World Bank. 46 INDONESIA APPENDIX TABLES APPENDIX TABLE 1: Questionnaire Modules and Sample Size Panel Sample size Percent from response Module Contents (Wave IV) 2007 panel rate Book 1A: Household Household roster, deaths in previous 24 months, 12,377 50 93.68% core (respondent: household characteristics, migration, water/ female household sanitation, receipt of government poverty head or spouse of programs, participation in informal education, male household consumption, assets, economic shocks, health head) insurance, morbidity, outpatient care use, social capital, community participation, trust in government Book 1B: Married Fertility history, use of health services during 11,264 women age 16–49 pregnancy, opinion of health services, family years planning, status of women, health and education knowledge Book 1C: Children Health of child, school enrollment, attendance, 10,409 age 6–15 years grade repetition, cost of schooling, (respondent: mother scholarships, child labor or guardian of the child) Book 1D: Children Growth monitoring (posyandu), acute 4,604 age < 3 years child morbidity, immunization records, (respondent: mother breastfeeding and nutritional intake, motor or guardian of the development, weight measurement, height child) measurement (table continues on next page) LONG-TERM GENERASI IMPACT EVALUATION 47 APPENDIX TABLES APPENDIX TABLE 1: Questionnaire Modules and Sample Size  continued Panel Sample size Percent from response Module Contents (Wave IV) 2007 panel rate Home-based tests Test of math and reading skills administered at 7,831 home (separate test for ages 6–12 years and ages 13–15 years) Book 1E: Additional Household roster, pregnancy record, child 6,908 33 94.37% households (aged 0–12 years) health measurement: weight of mother and child, height of child, upper arm of child and mother, bacille Calmette-Guerin (BCG) immunization mark Book 2: Village Demography of the village, hamlet 2,323 50 99.74% characteristics information, access to health services and (respondent: village schools, community participation, daily laborer head) wage rate, poverty eradication programs, water and sanitation, transportation, information media Book 3: Community Head of facility background, coverage area, 301 100 100% health center budget, staff roster, time allocation of head doctor and midwife coordinator, service hours, services provided, fee schedule, number of patients per service during the previous month, medical and vaccine stock, data on posyandu, direct observation regarding cleanliness Book 4: Village Personal background, location of duty, 1,197 50 74.83% midwives condition of facility, time allocation, income, services provided, fee schedule (public and private), experiences during past three deliveries, number of patients seen per service during the previous month, equipment and tools, medical supplies and stock, posyandu management, community meetings, supplementary food program 48 INDONESIA APPENDIX TABLES APPENDIX TABLE 1: Questionnaire Modules and Sample Size  continued Panel Sample size Percent from response Module Contents (Wave IV) 2007 panel rate Book 5: Schools Principal background, principal time allocation, 3,316 Junior high Junior high teacher roster, school facilities, teaching schools: 66, schools: hours, enrollment records, attendance records, elementary 96.39%, official test scores, scholarships, fees, budget, schools: 50 elementary direct observation of classrooms, including schools: random check on classroom attendance 95.65% Book 6: Community Respondent characteristics, posyandu 2,401 50 99.41% health post cadre characteristics, service providers, cadre roster, (posyandu) tools and equipment, community meetings, family connections, supplementary food programs Book 7: Subdistrict Respondent characteristics, subdistrict 358 New modules head information, service delivery problems, community development program, data collection, Village Law implementation, list of junior high schools Book 8: Facilitator Respondent characteristics, training, time 1,567 usage, problems in infrastructure, health, and education, case studies, predecessor information, village performance Book US: Child (aged 0–12 years) health measurement: 9,229 Anthropometry weight of mother and child, height of child, upper arm of child and mother, bacille Calmette-Guerin (BCG) immunization mark Note: About 50 percent of married women and children come from panel households, but the married women and children themselves are not necessar- ily panel respondents. LONG-TERM GENERASI IMPACT EVALUATION 49 APPENDIX TABLES APPENDIX TABLE 2: Direct Benefits Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Received scholarship b 0.014** 0.040 7,168 –0.028* 0.311 8,988 SE 0.006 –0.007 0.014 0.463 P 0.061 Received uniform b 0.077*** 0.011 7,168 0.007* 0.013 8,968 SE 0.009 –0.004 0.003 0.113 P 0.053 Received other b 0.061*** 0.011 7,168 0.007 0.022 8,984 school supplies SE 0.008 –0.004 0.006 0.146 P 0.264 Received transport b 0.006*** 0.005 7,168 –0.003** 0.006 8,988 subsidy SE 0.001 –0.001 0.002 0.078 P 0.039 Received other b 0.007** 0.006 7,168 0.002 0.002 8,988 school support SE 0.003 –0.001 0.001 0.044 P 0.103 Received b 0.004 0.006 7,168 0.001 0.001 8,988 supplementary SE 0.004 –0.001 0.001 0.029 feeding at school P 0.530 Received b 0.190*** 0.457 5,847 0.064*** 0.499 4,014 supplementary SE 0.021 –0.017 0.021 0.500 feeding at posyandu P 0.008 Received intensive b 0.022*** 0.046 5,844 0.003 0.055 4,003 supplementary SE 0.008 –0.007 0.010 0.229 feeding P 0.770 Received health b 0.032*** 0.007 4,063 0.033* 0.044 1,391 subsidy for pre- SE 0.005 –0.003 0.015 0.206 and/or postnatal care P 0.059 50 INDONESIA APPENDIX TABLES APPENDIX TABLE 2: Direct Benefits  continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Received health b 0.113*** 0.045 2,511 0.018 0.048 3,177 subsidy for childbirth SE 0.014 –0.008 0.011 0.213 P 0.101 Average standardized b 0.302*** 0.045 effects SE 0.023 0.013 P 0.330 Average standardized b 0.291*** 0.083 effects, health SE 0.026 0.025 P 0.311 Average standardized b 0.313*** 0.012 effects, education SE 0.034 0.015 P 0.478 All outcomes are dummy variables. b, point estimate; SE, standard error; p, randomization inference p-value. *Statistically significant at 10% level; **Statistically significant at 5% level; ***Statistically significant at 1% level (this holds for all tables in the Appendix.) The randomization inference p-values were only calculated for Wave IV (this holds for all tables in the appendix). LONG-TERM GENERASI IMPACT EVALUATION 51 APPENDIX TABLES APPENDIX TABLE 3: Direct Generasi Program Benefits, Provincial Breakdown Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Received scholarship b 0.015** 0.006 0.021 –0.015 –0.046 –0.051 SE 0.006 0.014 0.021 0.019 0.029 0.032 P 0.440 0.203 0.148 Received uniform b 0.042*** 0.123*** 0.136*** 0.005 0.016 0.008 SE 0.007 0.026 0.024 0.004 0.012 0.004 P 0.298 0.322 0.109 Received other school supplies b 0.039*** 0.083*** 0.103*** 0.006 0.003 0.009 SE 0.007 0.021 0.021 0.005 0.011 0.016 P 0.287 0.812 0.646 Received transport subsidy b 0.005*** 0.015** 0.005** –0.000 –0.003 –0.008** SE 0.002 0.006 0.002 0.001 0.006 0.005 P 0.754 0.846 0.013 Received other school support b –0.001 0.001 0.029*** 0.003 0.001 SE 0.001 0.005 0.009 0.001 0.001 P 0.134 0.650 Received supplementary feeding b 0.001 0.002 0.015 0.000 –0.002 0.004 at school SE 0.001 0.002 0.015 0.000 0.002 0.003 P 0.462 0.348 0.387 Received supplementary feeding b 0.163*** 0.200*** 0.258*** 0.065** 0.048 0.071 at posyandu SE 0.027 0.054 0.037 0.027 0.044 0.049 P 0.033 0.370 0.203 Received intensive supplementary b 0.021* 0.025** 0.021 0.002 0.000 0.011 feeding SE 0.011 0.010 0.017 0.013 0.018 0.020 P 0.878 0.982 0.614 52 INDONESIA APPENDIX TABLES APPENDIX TABLE 3: Direct Generasi Program Benefits, Provincial Breakdown continued Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Received health subsidy for pre- b 0.031*** 0.029** 0.038*** 0.006 0.018 0.084*** and/or postnatal care SE 0.006 0.012 0.012 0.025 0.014 0.022 P 0.821 0.383 0.008 Received health subsidy for childbirth b 0.136*** 0.098*** 0.056*** –0.004 0.028* 0.074*** SE 0.020 0.032 0.018 0.015 0.010 0.017 P 0.776 0.062 0.004 Average standardized effects b 0.259*** 0.239*** 0.383*** 0.027 0.062 0.135 SE 0.027 0.038 0.042 0.016 0.035 0.033 P 0.515 0.169 0.587 Average standardized effects, health b 0.269*** 0.267*** 0.318*** 0.033 0.108** 0.268*** SE 0.030 0.052 0.055 0.029 0.047 0.056 P 0.308 0.045 0.001 Average standardized effects, b 0.249*** 0.211*** 0.449*** 0.021 0.003 0.003 education SE 0.042 0.050 0.069 0.017 0.043 0.037 P 0.574 0.955 0.974 All outcomes are dummy variables. b, point estimate; SE, standard error; p, randomization inference p-value Treatment effects represent the net effect of Generasi on outcomes in each province, respectively. LONG-TERM GENERASI IMPACT EVALUATION 53 APPENDIX TABLES APPENDIX TABLE 4. Program Impact on Main Targeted Indicators (With and Without new Indicators) Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Number of prenatal b 0.068 7.631 3,522 0.002 8.537 4,315 visits SE 0.169 4.220 0.159 4.227 P 0.986 Delivery by trained b 0.002 0.777 2,582 0.004 0.925 3,306 midwife SE 0.019 0.416 0.010 0.263 P 0.725 Number of postnatal b –0.026 1.629 2,583 0.020 1.835 3,306 visits SE 0.109 2.453 0.092 2.271 P 0.843 Iron tablet sachets b 0.060 1.739 3,471 –0.009 2.178 4,287 SE 0.053 1.273 0.056 1.411 P 0.873 Percent immunized b 0.002 0.754 2,885 0.003 0.824 3,092 SE 0.014 0.287 0.012 0.257 P 0.801 Number of weight b 0.188*** 2.263 4,390 0.127*** 2.272 3,938 checks SE 0.047 1.120 0.045 1.103 P 0.007 Number vitamin A b 0.041 1.445 2,218 0.063 1.401 2,294 supplements SE 0.044 0.954 0.037 0.971 P 0.116 Percent malnourished b –0.022* 1.445 4,316 0.003 0.174 8,092 SE 0.013 0.954 0.011 0.379 P 0.757 SD (elementary b 0.008** 0.985 5,014 0.002 0.984 10,363 school) enrollment SE 0.004 0.120 0.003 0.125 P 0.552 54 INDONESIA APPENDIX TABLE 4. Program Impact on Main Targeted Indicators (With and Without new Indicators)  continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N SMP (junior high b 0.040* 0.709 2,040 –0.009 0.715 3,371 school) enrollment SE 0.022 0.456 0.017 0.452 P 0.589 Average standardized b 0.045** 0.020 effect SE 0.017 0.014 P 0.202 Average standardized b 0.039** 0.025 effect, health SE 0.020 0.016 P 0.162 Average standardized b 0.070*** –0.004 effect, education SE 0.027 0.021 P 0.856 New indicators Attend parenting b 0.085*** 0.116 4,905 class SE 0.013 0.320 P 0.001 Maternal class b 0.081*** 0.322 4,314 SE 0.019 0.467 P 0.001 Special needs b –0.009 0.970 1,769 enrollment SE 0.009 0.171 P 0.349 Average standardized b 0.045*** effect, including new SE 0.013 indicators P 0.004 Average standardized b 0.064*** effect, including new SE 0.015 indicators, health P 0.001 Average standardized b –0.020*** effect, including new SE 0.024 indicators, education P 0.457 b, point estimate; SE, standard error; p, randomization inference p-value APPENDIX TABLE 5: Program Impact on Main Targeted Indicators, Provincial Breakdown Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Number of prenatal visits b 0.065 0.038 0.077 0.028 –0.011 –0.069 SE 0.195 0.409 0.463 0.179 0.490 0.361 P 0.898 0.987 0.867 Delivery by trained midwife b 0.020 –0.006 –0.048 0.012 0.001 –0.019 SE 0.021 0.039 0.057 0.012 0.020 0.024 P 0.396 0.971 0.549 Number of postnatal visits b –0.130 0.102 0.206 0.061 –0.377* 0.213 SE 0.148 0.200 0.209 0.124 0.181 0.164 P 0.668 0.092 0.269 Iron tablet sachets b 0.082 –0.012 0.035 –0.017 –0.036 0.034 SE 0.067 0.117 0.120 0.072 0.126 0.120 P 0.804 0.802 0.795 Percent immunized b –0.010 0.020 0.019 –0.014 0.003 0.060 SE 0.015 0.038 0.039 0.014 0.030 0.029 P 0.309 0.928 0.108 Number of weight checks b 0.151*** 0.252* 0.240** 0.122** 0.173 0.104 SE 0.058 0.126 0.110 0.059 0.101 0.090 P 0.048 0.175 0.193 Number vitamin A supplements b 0.061 0.083 –0.033 0.055 0.041 0.106 SE 0.057 0.093 0.104 0.048 0.069 0.089 P 0.296 0.641 0.320 Percent malnourished b –0.003 –0.017 –0.090*** 0.003 –0.057* 0.048 SE 0.015 0.037 0.027 0.011 0.025 0.032 P 0.750 0.066 0.153 SD (elementary school) enrollment b –0.004 0.010 0.042*** 0.000 –0.002 0.008* SE 0.004 0.012 0.007 0.002 0.012 0.004 P 0.865 0.895 0.071 SMP (junior high school) enrollment b 0.017 0.022 0.085 –0.013 –0.006 –0.003 SE 0.026 0.059 0.057 0.021 0.044 0.034 P 0.509 0.890 0.922 56 INDONESIA APPENDIX TABLE 5: Program Impact on Main Targeted Indicators, Provincial Breakdown  continued Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Average standardized effect b 0.025 0.051 0.083* 0.014 0.014 0.040 SE 0.020 0.047 0.045 0.016 0.034 0.036 P 0.473 0.703 0.310 Average standardized effect, health b 0.032 0.051 0.059 0.022 0.021 0.042 SE 0.024 0.051 0.050 0.019 0.038 0.044 P 0.337 0.646 0.377 Average standardized effect, b –0.001 0.051 0.181*** –0.017 –0.013 0.029 education SE 0.037 0.061 0.058 0.025 0.077 0.039 P 0.504 0.859 0.498 New indicators (Wave IV only) Attend parenting class b 0.075*** 0.098** 0.106*** SE 0.017 0.034 0.027 P 0.001 0.050 0.003 Maternal class b 0.112*** –0.005 0.048 SE 0.024 0.044 0.042 P 0.001 0.911 0.322 Special needs enrollment b –0.004 –0.006 –0.018 SE 0.009 0.028 0.010 P 0.717 0.856 0.107 Average standardized effect, b 0.045*** 0.032 0.056 including new indicators SE 0.016 0.034 0.033 P 0.006 0.396 0.124 Average standardized effect, b 0.065*** 0.048 0.077* including new indicators, health SE 0.018 0.034 0.042 P 0.001 0.238 0.095 Average standardized effect, b –0.019 –0.021 –0.015 including new indicators, education SE 0.026 0.091 0.032 P 0.535 0.847 0.683 b, point estimate; SE, standard error; p, randomization inference p-value Treatment effects represent the net effect of Generasi on outcomes in each province. LONG-TERM GENERASI IMPACT EVALUATION 57 APPENDIX TABLES APPENDIX TABLE 6: Program Impact on Longer-Term Outcomes Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Malnourished b –0.022* 0.228 4,316 0.003 0.174 8,092.0 SE 0.013 0.42 0.011 0.379 P 0.757 Severely b –0.015 0.069 4,316 0.000 0.044 8,092.0 malnourished SE 0.009 0.253 0.005 0.204 P 0.956 Wasting b –0.001 0.199 3,897 0.018 0.155 7,954.0 SE 0.015 0.400 0.011 0.362 P 0.101 Severe wasting b 0.003 0.089 3,897 0.003 0.042 7,954.0 SE 0.010 0.285 0.005 0.202 P 0.568 Stunting b 0.030* 0.350 3,926 –0.004 0.227 7,956.0 SE 0.017 0.477 0.013 0.419 P 0.773 Severe stunting b 0.006 0.211 3,926 0.004 0.086 7,956.0 SE 0.017 0.409 0.009 0.280 P 0.701 Mortality 0–28 days b –0.001 0.008 2,572 –0.002 0.017 13,373.0 SE 0.004 0.089 0.002 0.130 P 0.428 Mortality b –0.001 0.011 3,301 –0.003 0.027 13,373.0 0–12 months SE 0.004 0.105 0.003 0.164 P 0.286 Language score b –0.023 –0.013 4,308 –0.025 0.000 5,910.0 6 to 12 SE 0.041 1.056 0.030 0.998 P 0.460 58 INDONESIA APPENDIX TABLES APPENDIX TABLE 6: Program Impact on Longer-Term Outcomes continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Math score 6 to 12 b –0.012 –0.060 3,957 –0.027 0.000 5,910.0 SE 0.043 1.045 0.037 0.998 P 0.464 Total score 6 to 12 b –0.015 –0.034 3,943 –0.029 0.000 5,910.0 SE 0.042 1.045 0.034 0.998 P 0.442 Raven score 6 to 12 b –0.010 0.000 6,746.0 SE 0.030 0.999 P 0.749 Average b 0.004 –0.012 standardized effect SE 0.017 0.018 P 0.574 Average b 0.003 –0.013 standardized effect, SE 0.020 0.018 health P 0.467 Average b 0.017 –0.010 standardized effect, SE 0.036 0.030 education P 0.759 b, point estimate; SE, standard error; p, randomization inference p-value LONG-TERM GENERASI IMPACT EVALUATION 59 APPENDIX TABLES APPENDIX TABLE 7: Program Impact on Longer-Term Outcomes, Provincial Breakdown Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Malnourished b –0.003 –0.017 –0.090*** 0.004 –0.057* 0.048 SE 0.015 0.037 0.027 0.011 0.025 0.032 P 0.747 0.063 0.150 Severely malnourished b –0.000 –0.026 –0.053* 0.002 –0.012 0.002 SE 0.009 0.026 0.028 0.005 0.013 0.015 P 0.688 0.478 0.902 Wasting b –0.013 0.048 –0.008 0.004 0.017 0.061* SE 0.017 0.037 0.038 0.012 0.024 0.028 P 0.744 0.552 0.054 Severe wasting b –0.002 0.017 0.007 0.004 –0.001 0.008 SE 0.013 0.022 0.026 0.007 0.009 0.014 P 0.589 0.938 0.578 Stunting b 0.051** 0.011 –0.024 0.001 –0.039 0.010 SE 0.022 0.043 0.031 0.017 0.035 0.021 P 0.961 0.310 0.716 Severe stunting b 0.034 –0.016 –0.061** 0.002 –0.005 0.016 SE 0.021 0.043 0.031 0.011 0.024 0.016 P 0.873 0.877 0.360 Mortality 0–28 days b 0.000 0.002 –0.010 –0.002 –0.003 –0.001 SE 0.004 0.008 0.012 0.003 0.005 0.005 P 0.520 0.629 0.794 Mortality 0–12 months b 0.003 0.004 0.005 –0.004 –0.009 0.000 SE 0.005 0.019 0.013 0.003 0.007 0.007 P 0.309 0.224 0.967 60 INDONESIA APPENDIX TABLES APPENDIX TABLE 7: Program Impact on Longer-Term Outcomes, Provincial Breakdown continued Wave III Wave IV Java Sulawesi NTT Java Sulawesi NTT Generasi Generasi Generasi Generasi Generasi Generasi effect effect effect effect effect effect Language score 6 to 12 b –0.053 –0.032 0.056 –0.018 0.028 –0.079 SE 0.051 0.118 0.094 0.033 0.068 0.080 P 0.625 0.707 0.433 Math score 6 to 12 b –0.011 0.068 –0.086 –0.042 0.034 –0.047 SE 0.052 0.095 0.099 0.042 0.091 0.096 P 0.341 0.733 0.646 Total score 6 to 12 b –0.030 0.033 –0.028 –0.031 0.031 –0.079 SE 0.054 0.086 0.082 0.037 0.077 0.094 P 0.431 0.724 0.491 Raven score 6 to 12 b –0.034 0.040 0.014 SE 0.036 0.067 0.075 P 0.351 0.661 0.853 Average standardized effect b –0.012 –0.012 0.044 –0.022 0.044 –0.029 SE 0.028 0.031 0.031 0.022 0.044 0.046 P 0.337 0.482 0.572 Average standardized effect, health b –0.024 –0.007 0.083** –0.009 0.049 –0.071 SE 0.027 0.037 0.042 0.020 0.044 0.048 P 0.662 0.366 0.126 Average standardized effect, b 0.007 0.008 0.042 –0.035 0.040 0.014 education SE 0.047 0.091 0.072 0.036 0.067 0.075 P 0.358 0.666 0.854 b, point estimate; SE, standard error; p, randomization inference p-value Treatment effects represent the net effect of Generasi on outcomes in each province LONG-TERM GENERASI IMPACT EVALUATION 61 APPENDIX TABLES APPENDIX TABLE 8: Program Impact on Main Targeted Indicators, Interactions With Preperiod Subdistrict-Level Variables, Wave IV Interaction Generasi with preperiod Generasi at effect level 10th percentile Control mean N Number of prenatal visits b –0.258*** 0.035 –0.101 8.537 4,315 SE 0.519 0.065 0.254 4.227 P 0.001 0.624 0.708 Delivery by trained midwife b –0.016 0.030 –0.010 0.925 3,306 SE 0.033 0.040 0.025 0.263 P 0.759 0.567 0.765 Number of postnatal visits b –0.018 0.023 –0.009 1.835 3,306 SE 0.170 0.100 0.137 2.271 P 0.860 0.816 0.947 Iron tablet sachets b 0.096 –0.066 0.029 2.178 4,287 SE 0.190 0.120 0.083 1.411 P 0.447 0.611 0.727 Percent immunized b 0.055 –0.078 0.026 0.824 3,092 SE 0.047 0.063 0.025 0.257 P 0.448 0.279 0.363 Number of weight checks b –0.053 0.085 0.066 2.272 3,938 SE 0.202 0.091 0.084 1.103 P 0.612 0.392 0.458 Number vitamin A supplements b 0.085 –0.014 0.071 1.401 2294 SE 0.135 0.085 0.058 0.971 P 0.382 0.875 0.253 Percent malnourished b –0.004 0.042 0.010 0.174 8,092 SE 0.016 0.096 0.022 0.379 P 0.971 0.678 0.676 SD (elementary school) enrollment b –0.063 0.068 –0.004 0.984 10,363 SE 0.048 0.050 0.004 0.125 P 0.253 0.212 0.503 62 INDONESIA APPENDIX TABLES APPENDIX TABLE 8: Program Impact on Main Targeted Indicators, Interactions With Preperiod Subdistrict-Level Variables, Wave IV continued Interaction Generasi with preperiod Generasi at effect level 10th percentile Control mean N SMP (junior high school) b –0.058 0.084 –0.034 0.715 3,371 enrollment SE 0.044 0.066 0.027 0.452 P 0.434 0.271 0.257 Average standardized effect b 0.046 0.006 SE 0.060 0.024 P 0.479 0.840 Average standardized effect, b –0.033 0.020 health SE 0.055 0.029 P 0.593 0.568 Average standardized effect, b 0.364 –0.052 education SE 0.216 0.036 P 0.122 0.200 b, point estimate; SE, standard error; p, randomization inference p-value APPENDIX TABLE 9: Program Impact on Longer-Term Outcomes, Interactions With Preperiod Subdistrict-Level Variables, Wave IV Generasi Interaction with Generasi at effect preperiod level 10th percentile Control mean N Malnourished b –0.004 0.042 0.010 0.174 8,092 SE 0.016 0.096 0.022 0.379 P 0.971 0.678 0.676 Severely malnourished b –0.003 0.053 0.004 0.044 8,092 SE 0.006 0.080 0.009 0.204 P 0.974 0.593 0.680 Wasting b 0.016 0.013 0.020 0.155 7,954 SE 0.016 0.096 0.017 0.362 P 0.875 0.897 0.296 LONG-TERM GENERASI IMPACT EVALUATION 63 APPENDIX TABLES APPENDIX TABLE 9: Program Impact on Longer-Term Outcomes, Interactions With Preperiod Subdistrict-Level Variables, Wave IV continued Generasi Interaction with Generasi at effect preperiod level 10th percentile Control mean N Severe wasting b 0.005 –0.029 0.001 0.042 7,954 SE 0.007 0.070 0.007 0.202 P 0.948 0.697 0.920 Stunting b –0.039 0.088 0.020 0.227 7,956 SE 0.033 0.074 0.023 0.419 P 0.629 0.296 0.426 Severe stunting b 0.000 0.016 0.008 0.086 7,956 SE 0.013 0.045 0.014 0.280 P 0.992 0.762 0.616 Mortality 0–28 days b 0.001 –0.166** –0.012** 0.017 13,373 SE 0.002 0.062 0.005 0.130 P 0.994 0.046 0.044 Mortality 0–12 months b –0.003 –0.015 –0.004 0.027 13,373 SE 0.003 0.057 0.004 0.164 P 0.978 0.834 0.488 Language score 6 to 12 b –0.025 –0.003 –0.024 0.000 5,910 SE 0.029 0.086 0.061 0.998 P 0.770 0.970 0.734 Math score 6 to 12 b –0.022 0.097 –0.079 0.000 5,910 SE 0.036 0.086 0.063 0.998 P 0.815 0.303 0.219 Total score 6 to 12 b –0.019 0.155* –0.110* 0.000 5,910 SE 0.033 0.080 0.057 0.998 P 0.845 0.099 0.086 b, point estimate; SE, standard error; p, randomization inference p-value 64 INDONESIA APPENDIX TABLES APPENDIX TABLE 10: Results for Service Provider Quantities Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Midwife in village b –0.010 0.828 2,029 –0.009 0.876 2,155 SE 0.015 0.378 0.016 0.330 P 0.622 Number of active b 0.165 4.369 2,029 0.306*** 4.079 2,155 posyandu in village SE 0.159 2.967 0.134 2.765 P 0.008 SD (elementary b –0.002 0.992 2,029 0.007* 0.978 2,155 school) located in SE 0.003 0.088 0.004 0.146 village P 0.091 SMP (junior high b 0.040*** 0.476 2,029 0.033* 0.518 2,155 school) located in SE 0.014 0.500 0.016 0.500 village P 0.078 Number of teachers b 0.058 10.808 1,053 –0.065 10.531 2,125 at SD SE 0.238 2.929 0.197 3.310 P 0.763 Number of teachers b 0.726 22.209 760 0.848 21.359 788 at SMP SE 0.514 10.901 0.596 11.016 P 0.177 Number of full-time b 0.060 7.030 1,053 –0.084 5.973 2,125 teachers at SD SE 0.224 2.826 0.162 2.856 P 0.616 Number of full-time b 0.160 13.854 760 0.547 10.698 788 teachers at SMP SE 0.677 11.751 0.531 11.515 P 0.289 Number of full-time b 1.934* 24.964 264 1.449 34.337 265 health personnel SE 0.967 9.009 1.331 15.629 P 0.307 LONG-TERM GENERASI IMPACT EVALUATION 65 APPENDIX TABLES APPENDIX TABLE 10: Results for Service Provider Quantities   continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Number of full-time b 2.476** 26.241 264 1.152 39.506 265 and part-time health SE 0.901 8.919 1.224 16.331 personnel P 0.390 Number of full-time b 0.354 10.325 264 0.638 14.482 265 midwives SE 0.350 4.340 0.723 7.213 P 0.355 Number of full- b 0.626* 10.711 264 0.285 16.855 265 time and part-time SE 0.315 4.279 0.662 7.681 midwives P 0.670 Total full-time b 0.000 0.000 261 0.000 0.001 262 midwife-to- SE 0.000 0.000 0.000 0.000 population ratio P 0.203 Total full- and part- b 0.000 0.000 261 0.000 0.001 262 time midwife-to- SE 0.000 0.000 0.000 0.000 population ratio P 0.392 b, point estimate; SE, standard error; p, randomization inference p-value 66 INDONESIA APPENDIX TABLES APPENDIX TABLE 11: Results for Service Provider Quality (Health and Education Infrastructure Availability) Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Midwives Has access to water b –0.034*** 0.790 990 –0.044 0.840 983 SE 0.025 0.408 0.028 0.367 P 0.144 Has access to b 0.003 0.968 990 0.009 0.981 983 electricity SE 0.010 0.176 0.008 0.137 P 0.292 Oxytocin in stock b –0.011 0.946 1,034 –0.002 0.840 1,053 SE 0.018 0.227 0.023 0.367 P 0.930 Proportion of last b 0.0933 0.930 1,028 –0.009 0.963 1,039 three deliveries SE 0.2113 0.215 0.009 0.153 using partograph P 0.372 Antenatal care b –0.027* 0.605 1,034 –0.018 0.552 1,052 service items SE 0.016 0.206 0.022 0.302 “always do” P 0.413 (public) Antenatal care b –0.040*** 0.597 1,034 –0.018 0.559 1,052 service items SE 0.013 0.204 0.017 0.259 “always do” P 0.303 (private) Schools Number of b –0.095 6.165 1,053 0.019 6.534 2,125 classrooms (SD) SE 0.113 1.407 0.098 1.792 (elementary P 0.876 school) Number of b 0.042 9.418 761 0.377 10.282 788 classrooms (SMP) SE 0.336 6.217 0.388 6.692 (junior high school) P 0.333 LONG-TERM GENERASI IMPACT EVALUATION 67 APPENDIX TABLES APPENDIX TABLE 11: Results for Service Provider Quality (Health and Education Infrastructure Availability) continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Condition of b –0.019 0.908 1,047 –0.002 0.900 2,116 school building SE 0.012 0.157 0.009 0.153 (SD, scale 0–1) P 0.763 Condition of b 0.003 0.940 752 0.014 0.926 782 school building SE 0.009 0.120 0.010 0.131 (SMP scale 0–1) P 0.135 Has student latrine b 0.006 0.872 1,053 0.001 0.927 2,049 (SD) SE 0.023 0.335 0.013 0.260 P 0.948 Has student latrine b –0.012 0.933 761 0.012 0.942 773 (SMP) SE 0.017 0.250 0.016 0.233 P 0.478 Puskesmas Stock out any b –0.020 0.145 260 –0.001 0.146 264 vaccine last two SE 0.044 0.354 0.046 0.356 months P 0.992 b, point estimate; SE, standard error; p, randomization inference p-value 68 INDONESIA APPENDIX TABLES APPENDIX TABLE 12: Results for Service Provider Level of Effort Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Midwives Hours spent in b 0.067 3.154 1,034 0.804** 2.535 1,053 outreach over past SE 0.353 5.257 0.307 4.295 three days P 0.022 Hours spent b 0.528 13.034 1,034 –0.382 15.456 1,053 providing public SE 0.516 8.036 0.497 7.531 services over past P 0.472 three days Hours spent b 0.785 9.491 1,034 0.924* 9.338 1,053 providing private SE 0.595 8.651 0.491 8.936 services over past P 0.094 three days Total hours spent b 1.316 25.679 1,034 1.351* 27.329 1,053 working over past SE 0.839 12.547 0.704 11.503 three days P 0.059 Number of b –0.039 3.938 1,034 0.357 3.121 1,052 posyandus attended SE 0.200 2.989 0.192 2.648 in past month P 0.144 Number of hours b 0.012 2.977 1,034 0.115 2.556 1,051 midwife spends per SE 0.121 1.935 0.110 1.673 posyandu P 0.341 Teachers Percent present at b 0.004 0.874 1,053 0.004 0.873 2,122 time of interview SE 0.009 0.144 0.009 0.164 (SD) (elementary P 0.641 school) Percent present at b –0.012 0.898 760 0.000 0.884 788 time of interview SE 0.010 0.135 0.011 0.159 (SMP) (junior high P 0.998 school) LONG-TERM GENERASI IMPACT EVALUATION 69 APPENDIX TABLES APPENDIX TABLE 12: Results for Service Provider Level of Effort continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Percent teaching b –0.007 0.649 1,053 –0.005 0.472 2,124 at time of class SE 0.036 0.478 0.032 0.500 observation (SD) P 0.903 Percent teaching b 0.031 0.536 760 –0.004 0.397 786 at time of class SE 0.043 0.500 0.041 0.490 observation (SMP) P 0.934 Puskesmas Minutes wait at b 2.387 28.034 238 3.732 25.719 244 recent health visit SE 3.712 23.111 3.404 21.885 P 0.290 Percent of providers b –0.044* 0.814 264 0.002 0.848 265 present at time of SE 0.026 0.206 0.023 0.189 observation P 0.935 b, point estimate; SE, standard error; p, randomization inference p-value 70 INDONESIA APPENDIX TABLES APPENDIX TABLE 13: Results for Community Efforts in Service Provision, Monitoring, and Participation Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Community effort at direct service provision Number of b 0.162 4.369 2,029 0.306*** 4.079 2,155 posyandus in SE 0.161 2.967 0.134 2.765 village P 0.008 Number of b –0.087 11.812 2,108 0.042 11.917 2,112 posyandu SE 0.089 1.893 0.044 0.686 meetings in past P 0.437 year at selected posyandus Number of b 0.327** 4.794 2,108 0.092 5.194 2,112 volunteers SE 0.132 2.061 0.124 2.041 at selected P 0.528 posyandus Community effort at outreach Number of b –0.390 6.036 2,108 –0.109 5.753 2,111 sweepings SE 0.310 6.483 0.244 4.912 at selected P 0.687 posyandus in last year Number of SD b –0.063 2.426 1,043 0.020 2.422 2,103 (elementary) SE 0.168 3.116 0.075 1.622 school committee P 0.820 meetings with parents in past year Number of SMP b 0.212 2.210 753 0.025 2.387 780 (junior high) SE 0.158 1.431 0.092 1.326 school committee P 0.814 meetings with parents in past year LONG-TERM GENERASI IMPACT EVALUATION 71 APPENDIX TABLES APPENDIX TABLE 13: Results for Community Efforts in Service Provision, Monitoring, and Participation continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Community effort at monitoring Number of SD b 0.098 8.445 1,050 0.196 7.763 2,118 school committee SE 0.328 3.636 0.226 5.313 members P 0.456 Number of SMP b 0.199 7.648 755 0.135 6.611 783 school committee SE 0.299 4.616 0.280 3.369 members P 0.624 Number of SD b –0.108 4.181 1,043 0.091 3.638 2,112 school committee SE 0.288 5.190 0.154 3.519 meetings with P 0.644 teacher in past year Number of SMP b 0.550* 3.555 745 0.166 3.475 781 school committee SE 0.292 3.358 0.289 3.270 meetings with P 0.603 teacher in past year Participation in health/education programs Participation in b 0.033* 0.303 4,441 0.109*** 0.308 3,949 meetings about SE 0.018 0.460 0.021 0.462 health education P 0.001 Proportion of b 0.089*** 0.528 4,422 0.002 0.907 3,948 children under SE 0.020 0.499 0.012 0.291 three with KIA P 0.889 (maternal and child health revolution) Proportion of b 0.042*** 0.608 10,741 0.000 0.603 11,508 households SE 0.013 0.488 0.012 0.489 that think P 0.984 health services improved over last three years 72 INDONESIA APPENDIX TABLES APPENDIX TABLE 13: Results for Community Efforts in Service Provision, Monitoring, and Participation continued Wave III Wave IV Generasi effect Control mean N Generasi effect Control mean N Proportion of b 0.042*** 0.622 10,741 –0.003 0.629 11,508 households that SE 0.013 0.485 0.012 0.483 think education P 0.838 services improved over last two years Spillovers to other types of community activities Participation b 1.977 22.918 10,732 3.189 21.772 11,503 in village labor SE 20.55 53.379 1.842 63.934 (hours worked P 0.158 per household) Women’s b –0.232 4.465 6,334 0.419* 4.404 7,082 participation in SE 0.265 7.574 0.226 6.948 women’s groups P 0.080 (number of meetings) Women’s b –0.009 0.121 6,765 –0.013 0.078 7,770 participation SE 0.036 1.176 0.023 1.200 in government P 0.605 groups (number of meetings) Household b 0.192 10.451 8,070 0.241 9.640 8,653 respondent’s SE 0.429 12.101 0.326 11.632 participation in P 0.483 social groups (number of meetings) Participation in b 0.003 0.969 10,739 –0.010** 0.953 11,503 general election SE 0.004 0.173 0.005 0.212 2009/14 P 0.037 b, point estimate; SE, standard error; p, randomization inference p-value LONG-TERM GENERASI IMPACT EVALUATION 73 APPENDIX TABLES APPENDIX TABLE 14: Service Prices and Supply Wave III Wave IV Generasi Control Control effect mean N Generasi effect mean N Midwife Fee charged for b 16379.890** 346440.400 954 –8670.861 677966.100 775 childbirth at private SE 6717.579 157365.700 16933.950 245800.700 practice P 0.596 Number of childbirths b –0.100 2.833 1,034 0.078 0.420 1,053 at private practice in SE 0.191 3.451 0.067 0.973 last month P 0.314 Fee charged b 19940.050 176162.200 805 –43435.830 352934.900 612 for childbirth at SE 12106.900 168022.200 24703.800 310126.900 government practice P 0.113 Number of childbirths b 1.779** 1.914 1,034 0.008 0.553 1,053 at government practice SE 0.630 4.595 0.151 2.374 in last month P 0.946 Fee charged for b 6766.788 314296.100 877 –68818.760* 565414.500 387 childbirth (average SE 9785.820 163812.700 32081.890 297152.200 of private and P 0.055 government) Total number of b 1.688* 4.747 1,034 0.064 0.973 1,053 childbirths in last month SE 0.661 6.174 0.169 2.669 P 0.700 Fee paid by mother for b 196622.200 1600495.000 309 –325208.700 2257858.000 478 normal childbirth SE 369278.900 2426973.000 328466.900 3017804.000 P 0.341 Fee charged for b 1834.726 14490.230 961 –1635.083 27071.920 926 antenatal care at SE 1267.886 8141.713 1516.174 20958.270 private practice P 0.442 74 INDONESIA APPENDIX TABLES APPENDIX TABLE 14: Service Prices and Supply  continued Wave III Wave IV Generasi Control Control effect mean N Generasi effect mean N Number of ANC visits b 0.011 3.957 1,034 –0.541 3.695 1,053 at private practice last SE 0.342 5.196 0.373 6.214 month P 0.161 Fee charged for ANC at b –72.889 2457.529 820 560.801 4422.897 677 government practice SE 227.355 3289.361 515.799 6812.884 P 0.351 Number of ANC visits b 1.877 5.920 1,034 0.545 3.773 1,053 at government practice SE 1.015 11.659 0.614 7.679 last month P 0.440 Fee charged for ANC b 804.314 8784.846 961 –1947.431* 18639.880 927 visit (average of private SE 739.996 7429.886 1061.654 16111.640 and government) P 0.081 Total number of ANC b 1.861 9.877 1,034 0.004 7.468 1,053 visits last month SE 1.100 13.131 0.731 10.561 P 0.993 Fee paid by mother for b 2301.098 20233.810 1,173 1808.088 32891.140 922 ANC visit SE 2110.550 25949.440 2786.347 34885.480 P 0.573 Fee charged for family b –300.867 14224.920 957 –568.024 23591.060 945 planning visit at private SE 428.997 5749.963 1045.189 16107.590 practice P 0.494 Number of family b 3.099 34.859 898 –0.919 7.750 848 planning visits at SE 3.032 42.517 1.130 16.861 private practice P 0.365 Fee charged for family b –1102.932* 6965.251 792 140.921 7280.660 667 planning visit at SE 528.205 7354.845 909.564 9136.291 government practice P 0.909 LONG-TERM GENERASI IMPACT EVALUATION 75 APPENDIX TABLES APPENDIX TABLE 14: Service Prices and Supply  continued Wave III Wave IV Generasi Control Control effect mean N Generasi effect mean N Number of family b 2.440 18.768 753 –0.191 8.101 614 planning visits at SE 3.830 40.244 2.402 29.179 government practice P 0.953 Fee charged for b –113.114 12144.690 976 –742.552 20136.510 817 family planning visit SE 453.843 6253.920 1314.005 17728.620 (average of private and P 0.478 government) Total number of family b 4.946 45.583 1,016 –1.098 11.798 1,002 planning visits in last SE 4.336 55.440 1.892 28.390 month P 0.559 Fee paid by mother for b 37.197 15820.440 567 –859.915 26607.730 546 family planning visit SE 363.338 4464.001 1206.302 17831.180 P 0.598 Community health center Normal childbirth at b –19228.490 187991.400 197 14912.800 424123.400 242 community health SE 18394.360 133797.100 31395.310 280562.100 center—fee charged by P 0.638 midwife Normal childbirth at b –1.987 39.573 262 14.949 43.843 259 community health SE 5.291 44.054 11.210 66.650 center—quantity P 0.315 by midwife Posyandu Posyandu—fee for visit b –79.335* 282.817 2,073 –0.972 560.152 2,097 SE 45.851 1305.231 64.237 1133.503 P 0.998 Posyandu—quantity of b 11.155*** 40.224 2,075 4.976*** 45.092 2,100 children weighed at last SE 1.693 26.624 1.937 30.580 meeting where service P 0.010 was offered 76 INDONESIA APPENDIX TABLES APPENDIX TABLE 14: Service Prices and Supply  continued Wave III Wave IV Generasi Control Control effect mean N Generasi effect mean N Posyandu—quantity of b 16.330*** 33.372 2,050 4.243** 41.548 2,080 children with nutritional SE 1.819 29.626 1.785 32.275 supplement at last P 0.029 meeting where service was offered Posyandu—quantity b 2.375** 13.071 1,986 –1.136 12.488 2,033 of children immunized SE 1.082 19.735 1.266 25.941 at last meeting where P 0.369 service was offered Posyandu—quantity of b 0.830 5.296 2,044 0.490 4.214 2,077 mothers receiving ANC SE 0.714 15.032 0.424 8.716 visits at last meeting P 0.267 where service was offered Posyandu—quantity of b 1.484** 5.330 2,007 0.754 4.704 2,034 mothers receiving iron SE 0.752 15.516 0.562 10.818 pills at last meeting P 0.196 where service was offered Posyandu—quantity b 11.689*** 42.944 1,954 0.682 51.636 2,007 of children receiving SE 2.517 36.198 2.120 43.198 vitamin A at last P 0.756 meeting where service was offered Posyandu—quantity b 0.228 3.519 1,992 –0.060 2.535 2,026 of mothers receiving SE 0.642 15.288 0.453 9.097 family planning pills P 0.902 at last meeting where service was offered Posyandu—quantity of b –0.010 3.354 2,000 –0.083 2.332 2,025 mothers receiving family SE 0.674 14.748 0.393 8.788 planning injections at P 0.857 last meeting where service was offered LONG-TERM GENERASI IMPACT EVALUATION 77 APPENDIX TABLES APPENDIX TABLE 14: Service Prices and Supply  continued Wave III Wave IV Generasi Control Control effect mean N Generasi effect mean N Schools SD (elementary b –61857.690 119476.100 1,053 38567.920 36180.170 2,124 school)—annual cost SE 73018.360 1607270.000 33116.730 177817.500 of school 2008–09, P 0.470 2015–16 SD—number of students b 19.498 165.317 1,053 1.043 149.146 2,125 enrolled 2008–09, SE 17.038 76.090 4.819 75.270 2015–16 P 0.846 SD—number of students b –1.378 165.902 1,053 0.812 146.269 2,125 enrolled 2009–10, SE 6.041 74.879 4.632 73.687 2016–17 P 0.867 SD—cost of school to b –506.193 16985.570 4,673 –13252.790 59980.800 5,369 parents for previous SE 3833.513 90537.330 12016.200 392562.100 semester P 0.134 SMP (junior high b –1936.810 182102.100 760 22248.350 285914.600 788 school)—annual cost SE 43891.080 822090.900 126163.200 1480152.000 of school 2008–09, P 0.922 2015–16 SMP—number of b 14.453 306.464 760 13.698 288.129 788 students enrolled SE 13.601 248.787 12.818 236.403 2008–09, 2015–16 P 0.263 SMP—number of b 9.067 316.377 760 18.041 285.589 788 students enrolled at SE 11.980 252.919 11.776 227.387 2009–10, 2016–17 P 0.147 SMP–cost of school to b –32695.460* 108210.600 1,774 5165.901 131457.200 1,851 parents for previous SE 17197.480 386334.500 15806.420 277804.700 semester P 0.760 b, point estimate; SE, standard error; p, randomization inference p-value 78 INDONESIA APPENDIX TABLES APPENDIX TABLE 15: Main Targeted Indicators, Heterogeneity Based on Areas Most in Need Group Wave III Wave IV Number of prenatal visits b 1 –0.071 0.004 SE 0.248 0.241 b 2 0.172 0.087 SE 0.244 0.212 b 3 0.216 0.119 SE 0.313 0.269 Delivery by trained midwife b 1 –0.003 –0.002 SE 0.043 0.022 b 2 0.016 0.024* SE 0.031 0.013 b 3 0.032 0.005 SE 0.022 0.006 Number of postnatal visits b 1 0.070 –0.000 SE 0.141 0.115 b 2 0.142 0.290* SE 0.154 0.150 b 3 –0.359 –0.037 SE 0.258 0.261 Iron tablet sachets b 1 0.007 0.023 SE 0.073 0.090 b 2 0.084 0.039 SE 0.076 0.073 b 3 0.174* –0.062 SE 0.098 0.122 Percent immunized b 1 0.012 0.035 SE 0.027 0.023 b 2 0.002 –0.012 SE 0.023 0.018 b 3 –0.007 –0.013 SE 0.020 0.018 LONG-TERM GENERASI IMPACT EVALUATION 79 APPENDIX TABLE 15: Main Targeted Indicators, Heterogeneity Based on Areas Most in Need  continued Group Wave III Wave IV Number of weight checks b 1 0.257*** 0.194** SE 0.096 0.084 b 2 0.184*** 0.130** SE 0.071 0.062 b 3 0.139* 0.083 SE 0.076 0.074 Number vitamin A supplements b 1 0.151* 0.152** SE 0.082 0.077 b 2 0.029 –0.009 SE 0.082 0.053 b 3 0.061 –0.069 SE 0.086 0.079 Percent malnourished b 1 –0.014 –0.003 SE 0.019 0.018 b 2 0.001 0.007 SE 0.018 0.011 b 3 –0.050* 0.009 SE 0.027 0.022 SD (elementary school) b 1 0.014** 0.003 enrollment SE 0.006 0.005 b 2 0.002 0.001 SE 0.003 0.003 b 3 0.001 –0.001 SE 0.001 0.001 SMP (junior high school) b 1 0.036 0.011 enrollment SE 0.040 0.033 b 2 –0.005 0.010 SE 0.036 0.028 b 3 –0.018 –0.035 SE 0.042 0.032 b, point estimate; SE, standard error. Using endogenous stratification, group 1 is defined as those most in need and group 3 is defined as those least in need. 80 INDONESIA APPENDIX TABLES APPENDIX TABLE 16: Stunting Difference-in-Differences Analysis Stunting association N Water sources used for cooking and drinking in the village Piped water b 0.014 8,196 SE 0.058 Pump well water b –0.001 8,196 SE 0.061 Well water b 0.132** 8,196 SE 0.067 Rainwater b 0.081 8,196 SE 0.103 Lake water b 0.385* 8,196 SE 0.211 Spring water b 0.138* 8,196 SE 0.073 River or stream water b –0.031 8,196 SE 0.073 Aqua or mineral water b 0.096* 8,196 SE 0.053 How village members dispose of garbage With a service b –0.138 8,133 SE 0.0998 Burning it b 0.106 8,133 SE 0.117 Into a river or stream b 0.0415 8,133 SE 0.0608 Throwing it into the yard/garden, leaving it to rot b 0.113** 8,133 SE 0.0547 Throwing into a hole in the ground and then covering b 0.0108 8,133 the hole SE 0.058 LONG-TERM GENERASI IMPACT EVALUATION 81 APPENDIX TABLES APPENDIX TABLE 16: Stunting Difference-in-Differences Analysis  continued Stunting association N Open defecationa Open defecation, subdistrict level b 0.040 9,394 SE 0.076 Open defecation, individual level b 0.018 9,394 SE 0.016 Latrine useb Own latrine, subdistrict level b 0.022 9,394 SE 0.079 Shared latrine, subdistrict level b –0.109 9,394 SE 0.120 Public latrine, subdistrict level b –0.396*** 9,394 SE 0.141 Own latrine, individual level b –0.024 9,394 SE 0.016 Shared latrine, individual level b 0.002 9,394 SE 0.022 Public latrine, individual level b –0.003 9,394 SE 0.030 Participated in clean water program Participated in clean water programs b –0.130*** 7,937 SE 0.046 Height measurement The height of the child was measured at the last visit b –0.091** 9,394 to the posyandu, subdistrict level SE 0.043 The height of the child was measured at the last visit b 0.008 8,222 to the posyandu, individual level SE 0.012 82 INDONESIA APPENDIX TABLES APPENDIX TABLE 16: Stunting Difference-in-Differences Analysis  continued Stunting association N Attending PAUD Percent days met over the month, subdistrict level b –0.003*** 3,219 SE 0.001 Percent days met over the month, individual level b –0.006 273 SE 0.007 Mother’s knowledge of how food intake should change with diarrhea When a baby has diarrhea, he/she should be given no food b –0.034 9,070 SE 0.056 When a baby has diarrhea, he/she should be given more b 0.022 9,070 food than normal SE 0.014 When a baby has diarrhea, he/she should be given less b 0.015 9,070 food than normal SE 0.011 When a baby has diarrhea, he/she should be given no b 0.164 9,102 liquid SE 0.238 When a baby has diarrhea, he/she should be given less b 0.016 9,102 liquid than normal SE 0.021 When a baby has diarrhea, he/she should be given the b 0.010 9,102 same liquid as normal SE 0.012 Mother’s level of education Mother’s highest level of education is starting primary b 0.051 9,203 school SE 0.055 Mother’s highest level of education is primary school b 0.018 9,203 SE 0.048 Mother’s highest level of education is junior school b 0.023 9,203 SE 0.048 Mother’s highest level of education is high school b 0.004 9,203 SE 0.049 LONG-TERM GENERASI IMPACT EVALUATION 83 APPENDIX TABLES APPENDIX TABLE 16: Stunting Difference-in-Differences Analysis  continued Stunting association N Mother’s highest level of education is associate degree b 0.070 9,203 SE 0.060 Mother’s highest level of education is bachelor’s degree b –0.004 9,203 SE 0.055 Mother’s highest level of education is master’s/PhD b 0.011 9,203 SE 0.122 Health education Participated in health information outreach activity in last b –0.000 9,194 12 months SE 0.012 Number of health information outreach activities in last b 0.002 2,810 12 months SE 0.002 Exclusive breastfeeding Child is exclusively breastfed, subdistrict level b 0.057 9,394 SE 0.050 Child is exclusively breastfed, individual level b 0.019* 9,375 SE 0.011 b, point estimate; SE, standard error. a Open defecation means no latrine is used (compared to a private, shared, or public latrine). In models with slightly varied controls, open defecation has a statistically significant association with stunting rates. b Latrine responses are relative to no latrine at all. In models with slightly varied controls, own latrine use has a statistically significant association with stunting rates. 84 INDONESIA ANNEX: SUPPLEMENTARY MATERIAL ANNEX TABLE 1: Do Attrition Rates Vary Between Treatment and Control Areas? (1) (2) Variables Book 1A found Book 1E found Treatment 0.00351 –0.000397 (0.00763) (0.00802) Observations 6,045 2,186 Response rate 0.937 0.944 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Dependent variable is a dummy for a baseline household being found in follow-up wave. Same regression specification as for main regressions. LONG-TERM GENERASI IMPACT EVALUATION 85 ANNEX: SUPPLEMENTARY MATERIAL ANNEX TABLE 2: Program Impact on Main Targeted Indicators, Separated Based on 2007–09 Incentive/Nonincentive Randomization Total Generasi Generasi effect Incentive effect incentive effect Control mean N Number of prenatal b 0.102 –0.206 –0.104 8.537 4,315 visits SE 0.195 0.184 0.171 4.227 P 0.537 0.212 0.665 Delivery by trained b 0.005 –0.003 0.003 0.925 3,306 midwife SE 0.011 0.014 0.013 0.263 P 0.623 0.818 0.866 Number of postnatal b 0.039 –0.039 0.000 1.835 3,306 visits SE 0.106 0.110 0.109 2.271 P 0.701 0.688 1.000 Iron tablet sachets b 0.047 –0.115** –0.068 2.178 4,287 SE 0.064 0.065 0.065 1.411 P 0.425 0.048 0.414 Percent immunized b 0.008 –0.010 –0.002 0.824 3,092 SE 0.014 0.014 0.014 0.257 P 0.533 0.436 0.928 Number of weight b 0.144*** –0.037 0.108* 2.272 3,938 checks SE 0.049 0.050 0.054 1.103 P 0.003 0.430 0.099 Number vitamin A b 0.064 –0.001 0.063 1.401 2,294 supplements SE 0.044 0.049 0.045 0.971 P 0.111 0.984 0.289 Percent underweight b –0.009 0.025** 0.016 0.174 8,092 SE 0.012 0.011 0.012 0.379 P 0.423 0.023 0.291 b, point estimate; SE, standard error; p, randomization inference p-value 86 INDONESIA ANNEX: SUPPLEMENTARY MATERIAL ANNEX TABLE 3: Program Impact on Longer-Term Outcomes, Separated Based on 2007–09 Incentive/Nonincentive Randomization Total Generasi Generasi effect Incentive effect incentive effect Control mean N Underweight b –0.009 0.025** 0.016 0.174 8,092 SE 0.012 0.011 0.012 0.379 P 0.423 0.023 0.291 Severely underweight b –0.001 0.001 0.000 0.044 8,092 SE 0.006 0.005 0.006 0.204 P 0.900 0.849 0.991 Wasting b 0.016 0.005 0.020 0.155 7,954 SE 0.012 0.011 0.012 0.362 P 0.153 0.644 0.180 Severe wasting b 0.004 –0.001 0.003 0.042 7,954 SE 0.006 0.006 0.006 0.202 P 0.517 0.883 0.725 Stunting b –0.009 0.010 0.002 0.227 7,956 SE 0.016 0.017 0.015 0.419 P 0.519 0.480 0.939 Severe stunting b 0.002 0.003 0.005 0.086 7,956 SE 0.010 0.011 0.010 0.280 P 0.828 0.766 0.656 b, point estimate; SE, standard error; p, randomization inference p-value LONG-TERM GENERASI IMPACT EVALUATION 87 ANNEX: SUPPLEMENTARY MATERIAL ANNEX TABLE 4: Program Impact on Main Targeted Indicators Limited to Repeated Cross-Sectional Households Generasi effect Control mean N Number of prenatal visits b 0.070 8.564 3,291 SE 0.176 4.110 P 0.717 Delivery by trained midwife b 0.008 0.929 2,494 SE 0.011 0.256 P 0.506 Number of postnatal visits b –0.006 1.875 2,494 SE 0.111 2.287 P 0.961 Iron tablet sachets b –0.039 2.226 3,271 SE 0.061 1.404 P 0.557 Percent immunized b 0.003 0.827 2,334 SE 0.013 0.251 P 0.787 Number of weight checks b 0.141*** 2.245 2,742 SE 0.048 1.105 P 0.009 Number vitamin A supplements b 0.061 1.402 1,698 SE 0.045 1.010 P 0.226 Percent underweight b 0.001 0.170 6,517 SE 0.012 0.376 P 0.936 b, point estimate; SE, standard error; p, randomization inference p-value 88 INDONESIA ANNEX: SUPPLEMENTARY MATERIAL ANNEX TABLE 5: Program Impact on Longer-Term Outcomes Limited to Repeated Cross-Sectional Households Generasi effect Control mean N Underweight b 0.001 0.170 6,517 SE 0.012 0.376 P 0.936 Severely underweight b –0.001 0.042 6,517 SE 0.006 0.200 P 0.939 Wasting b 0.015 0.156 6,413 SE 0.012 0.363 P 0.220 Severe wasting b 0.006 0.042 6,413 SE 0.006 0.200 P 0.385 Stunting b –0.004 0.216 6,410 SE 0.014 0.412 P 0.791 Severe stunting b 0.002 0.083 6,410 SE 0.009 0.276 P 0.885 b, point estimate; SE, standard error; p, randomization inference p-value LONG-TERM GENERASI IMPACT EVALUATION 89 ANNEX: SUPPLEMENTARY MATERIAL ANNEX FIGURE 1: Visualization of Program Impact on Main Targeted Indicators Showing Trends Over Time and Treatment Effects Prenatal visits Delivery 9 1.0 0.9 0.93 8.54 8 0.8 0.76 0.7 0.75 7.60 7.52 7.48 0.69 7 0.6 2007 2010 2013 2016 2007 2010 2013 2016 Control Treatment Control Treatment Postnatal visits Iron pills 2.2 2.50 2.0 1.83 2.25 1.74 1.8 1.72 2.18 1.63 2.00 1.6 1.97 1.75 1.4 1.71 1.2 1.50 1.59 2007 2010 2013 2016 2007 2010 2013 2016 Control Treatment Control Treatment Immunization Times weighed 0.9 2.5 2.4 0.8 0.82 2.3 2.27 0.76 2.2 0.7 2.18 2.18 0.69 2.1 2.14 0.65 0.6 2.0 2007 2010 2013 2016 2007 2010 2013 2016 Control Treatment Control Treatment 90 INDONESIA ANNEX: SUPPLEMENTARY MATERIAL ANNEX FIGURE 1: Visualization of Program Impact on Main Targeted Indicators Showing Trends Over Time and Treatment Effects  continued Vitamin A Underweight 1.7 0.25 0.23 1.6 1.56 0.20 1.5 1.53 0.20 0.17 1.45 1.4 0.17 1.40 1.3 0.15 2007 2010 2013 2016 2007 2010 2013 2016 Control Treatment Control Treatment 7-12 Participation rate 13-15 Participation rate in junior high school 1.00 1.0 0.98 0.9 0.98 0.98 0.98 0.96 0.8 0.94 0.95 0.7 0.68 0.72 0.92 0.6 0.66 0.62 0.90 0.5 2007 2010 2013 2016 2007 2010 2013 2016 Control Treatment Control Treatment LONG-TERM GENERASI IMPACT EVALUATION 91 ANNEX: SUPPLEMENTARY MATERIAL ANNEX FIGURE 2: Visualization of Program Impact on Longer-Term Outcomes Showing Trends Over Time and Treatment Effects Underweight Severely malnourished 0.25 0.23 0.10 0.07 0.20 0.06 0.20 0.05 0.04 0.05 0.17 0.17 0.15 0.00 2007 2010 2013 2016 2005 2010 2015 2020 Control Treatment Control Treatment Stunting Severe stunting 0.5 0.5 0.4 0.4 0.38 0.3 0.36 0.3 0.2 0.2 0.23 0.21 0.21 0.1 0.1 0.09 0.0 0.0 2007 2010 2013 2016 2005 2010 2015 2020 Control Treatment Control Treatment Wasting Severe wasting 0.3 0.3 0.2 0.20 0.2 0.16 0.1 0.12 0.1 0.09 0.05 0.04 0.0 0.0 2005 2010 2015 2020 2005 2010 2015 2020 Control Treatment Control Treatment 92 INDONESIA