Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) Report Number: ICRR0021792 1. Project Data Project ID Project Name P132585 ID-TF GENERASI PROGRAM Country Practice Area(Lead) Indonesia Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-14769 31-Dec-2013 118,401,583.44 Bank Approval Date Closing Date (Actual) 24-Jun-2013 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 31,700,000.00 121,772,086.00 Revised Commitment 121,772,086.00 118,401,583.44 Actual 118,401,583.44 118,401,583.44 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing (Grant) Agreement (FA, p. 5), the Project Development Objective (PDO) was "to empower local communities in poor, rural sub-districts in Project Provinces to increase utilization of health and education services and foster accountability in local service delivery." The 2014 Restructuring revised this objective "to empower local communities in poor, rural sub-districts in Project Provinces to increase utilization of health and education services." This review will assess the following objectives: Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585)  to empower local communities in poor, rural sub-districts in Project Provinces (although this was also a means to achieve objectives 2 and 3 below)  to increase utilization of health and education services  to foster accountability in local service delivery (this objective was dropped during the June 25,2014 level 1 restructuring) The key outcome indicators were:  improved access to and utilization of health services in the target area o percent of pregnant women receiving four prenatal care visits o percent of deliveries assisted by trained professionals o percent of children under 5 weighed monthly  improved access to and utilization of education services in the targeted areas o percent of junior secondary enrolment rate  those involved in planning and decision making meetings o percent of women o percent of poorest community members  percent of sub-districts in which service providers attended inter village meetings (MADs) to discuss the status of health and education services  total number of beneficiaries and share of female beneficiaries b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 25-Jun-2014 c. Will a split evaluation be undertaken? Yes d. Components 1. Community (Kecamatan) Grants (US$30.5 million, of which US$25.5 million was Bank financing at appraisal; US$130.4 million of which US$89.4 million was Bank financing, actual, of which US$63.9 million was Additional Financing or AF). This component financed block grants to project sub-districts to finance village investment activities that improve the use of and access to health and education services. These grants also financed the planning and preparation of subproject proposals, training and capacity building activities for communities, and monthly stipends for community cadres and elected community members of the Sub-district Management Unit (UPK). These grants were about 5 percent of a village's annual block grant allocation. Kecamatan is the Indonesian equivalent of a sub-district. When the National Community Empowerment Program (PNPM) Generasi (Healthy and Bright Generation Program) was discontinued in 2014 and replaced by the Village Law, this component changed its name from the original Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) Kecamatan grants to Community grants (see Section 8 (b) below, Assessment of Quality of Bank Supervision, for further information). 2. Community Empowerment and Facilitation Support (US$4.2 million, of which US$3.7 million was Bank financing at appraisal; US$23.3 million, of which US$20 million was Bank financing, actual, of which US$16.30 was AF). This component financed technical assistance training and salaries of district and sub- district community facilitators to improve skills in diagnosing and overcoming constraints to utilizing health and education services. The first AF in 2014 added the financing of training and technical assistance to strengthen community health volunteer activities (June 25, 2014 letter First AF Amendment). This component also financed improvements in communications and links to local government health and education offices and service providers. Finally, this component also financed the managing of the Management Information System (MIS) database. 3. Implementation Support and Technical Assistance (US$4.0 million, of which US$2.5 million was Bank financing at appraisal; US$27.8 million, of which US$10 million was Bank financing, actual, of which US$7.5 million was AF). This component financed support to strengthen the management and oversight capacity of the Generasi Secretariat, technical assistance for health and education planning, database management, and training for facilitators and Generasi specialists at all levels of program delivery. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost reached US$236.7 million. The project disbursed US$181.5 million. The undisbursed balance resulted from two sources - (i) the US-funded Millennium Challenge Account for Indonesia (MCA-I) (see Financing below) could not fulfill its commitment to finance the implementation of the FY18 program US$3.3 million (ICR, Annex 5, paragraph 7); and (ii) Government's counterpart commitment did not materialize (see Borrower Contribution below). Financing: This Investment Project Financing (IPF) was financed by a grant from the National Community Empowerment Program (PNPM) Support Facility (PSF) Trust Fund. This financing was provided in four installments:  US$31.7 million grant from Australia's Department of Foreign Affairs and Trade (DFAT)  US$81.6 million from the US-funded Millennium Challenge Account - Indonesia (MCA-I)  US$6.0 million from cost savings from PNPM Generasi Scale Up project which closed on December 31, 2014  US$2.4 million from the PSF Multi Donor Trust Fund, provided by DFAT. Borrower Contribution: The Government committed US$114.9 in as counterpart financing and disbursed US$63.1 million actual. Dates: The project was approved on June 24, 2013 and became effective on June 25, 2013. The Mid Term Review was conducted on March 21, 2016. The original closing date was December 31, 2013 but was extended by five years to December 31, 2018, through the following restructurings. The second restructuring of June 25, 2014 was Level 1. All other restructurings were Level 2.  on November 28, 2013 to extend the loan closing date from December 31, 2013 to December 31, 2017, an extension of four years or 48 months. The project was designed as a five year project. Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) However, the funding source was to close on June 30, 2014. The project was approved with the original closing date but with an expected extension once the funding source was extended.  on June 25, 2014 to provide US$86.6 million in additional financing; amend the PDO, drop the element relating to accountability between rural communities and service providers; amend the Results Framework because of additional resources and geographic expansion of the project, updating indicator targets due to revised closing date, add three nutrition related indicators, amend the components and costs, and reallocate funds among disbursement categories.  on May 8, 2015, to provide US$6 million in additional financing from cost savings from a related project that closed on December 31, 2014. At this restructuring, the implementing agency, the Ministry of Home Affairs was transferred to the Ministry of Villages, Disadvantaged Areas, and Transmigration (MOV) as a result of the passing of the new Village Law and the election of a new administration. The Results Framework was adjusted a second time so that targets recognize actual population data from the 2010 census, change the financing plan, reallocate among disbursement categories, change institutional arrangements, financial management, and procurement.  on March 29, 2016 to provide an additional US$2.4 million from the original funding source to support early childhood education and development (ECED) related activities, amend the Results Framework a third time, change components and costs, change the financing plan, and reallocate among disbursement categories.  on September 19, 2017 to change the loan closing date from December 31, 2017 to December 31, 2018 or 12 more months to use cost savings resulting from the integration of Generasi into the Village Law to test new approaches contributing to the Government's 2017 National Strategy to Accelerate Stunting Prevention (StraNas Stunting), and institutionalize lessons from Generasi. 3. Relevance of Objectives Rationale The PDO remained relevant to the country's priorities as set out in the country's Mid-term Development Plan (National Medium‐Term Development Plan or RPJMN 2015-2019). This document was issued in early 2015 to meet the current administration's development challenges. The country's priorities were (i) achieving a more equitable, democratic, and law abiding country; and (ii) improving the quality of life for all Indonesian people. The PDO contributed to achieving the human development dimension encompassing health and education. Generasi’s scale‐up contributed to the RPJMN for 2015‐2019 through its targeted support of delivering health and education services in lagging regions, targeting the poor, and non‐users of services. The PDO was also relevant to the World Bank's Country Partnership Framework (CPF) for FY16-20 and contributed to Objective 9 under Engagement Area 4 “Improved access to quality education and health related services.” That objective would be met by: (i) percentage of pregnant women receiving four prenatal care visits; (ii) percentage of children under 3 weighed monthly; (iii) increase in junior secondary enrollment rate; and (iv) National Stunting Reduction Coordination mechanism launched and operational, and annual anthropometric survey launched. According to the September 16, 2019 email from the Project Team, the prior series of Generasi projects (pilot and scale up) took place under the PNPM Rural Projects. The PDO for this standalone project, Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) separate from the prior series, was ambitious but realistic and achievable because the project used the Generasi platform to reach poor, rural sub-districts in the three project provinces. The problem was clearly defined - the low level of access to health and education services. The project responded to a clearly defined problem of access to health and education services. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective • to empower local communities in poor, rural sub-districts in Project Provinces Rationale Theory of Change: This objective would be met by building the capacity of cadres and frontline service providers at the village level to identify its needs and provide them with a tool to prioritize government assistance (block grants) for health and education services based on these prioritized needs. By providing these village actors with the capacity tools, the project would empower a broad range of beneficiaries, particularly women and the poor, to participate in decisions about allocating resources for improving frontline service delivery in health and education. Evidence for the outputs and outcomes were derived from data generated by the project’s monitoring systems, the Generasi Long-Term Quantitative (Quantitative IE) and Qualitative Impact Evaluation (Qualitative IE), and complementary information collected from Implementation Status Reports (ISRs), field observations, and team assessments. OUTPUTS (cumulative data covering the five year implementation was obtained from the ICR, Annex A, p. 31-42). The Project Team confirmed in an email dated September 16, 2019 that a framework design was used to set targets during implementation, subject to community needs, hence there were no targets for these outputs:  3,863 facilitators (85 percent) at all levels were trained to support community capacity building and empowerment  Health and education sensitization and community awareness raising were delivered to 5,789 villagers in 499 districts  17,367 community members who were part of Village Advisory Teams (TPMDs) were trained in sub- project cycle and community empowerment  61,953 community cadres trained in social mapping, needs assessment, and community organization techniques  236,202 community subproject plans (162,240 plans for health and nutrition. and 66,962 plans for education) were developed Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) OUTCOMES:  61.6 percent of community members who participated in planning and decision making at the village level were women (baseline 65 percent, original target 65 percent, revised target 70 percent, target almost achieved).  45.18 percent of community members who participated in planning and decision making meetings were the poorest in the community (baseline 58 percent, original target 60 percent, target not achieved). The Project Team explained in its September 16, 2019 email that this was an annual indicator that fluctuated between 45 and 57 percent. In 2017, this reached 53 percent. The participation rate declined in the final year because the project facilitators prioritized the introduction of a new score card following the adoption of the national stunting strategy, support for Human Development Worker program, and completing activities in the final year of project implementation.  71.73 percent of subproject beneficiaries were new participants to health and education services (baseline 25 percent, original target 30 percent, revised target 50 percent, target exceeded) These outcomes were part of the cumulative achievements of the project. Evidence provided substantial achievement of empowering local communities. The project was restructured only one year after the project became effective. The Project Team clarified that this restructuring took place too early in the implementation period to assess separate outcomes. The change in PDO did not result in a change in project activities. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective This objective was not revised. Revised Rationale There was no revision to the Theory of Change. The outputs and outcomes achieved under the original objective apply here as well.  The original baseline and target values of 65 percent were revised to target 75 percent women as part of the first AF (June 2014) to encourage a higher achievement because the project closing date was extended. However, as part of the third AF (June 2016), the target value was reduced to 70 percent to accommodate the new focus on increasing the involvement of male caregivers in health and nutrition related decision making processes. OUTPUTS:  There were 1,560,149 direct beneficiaries (original baseline 5.4 million, revised to 3.6 million, original target 6.1 million, revised target 850,000, target exceeded)  Of the total beneficiaries 53.69 percent were female (baseline 55 percent, original target 50 percent, revised target 55 percent, target almost achieved). Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585)  There were 250,398 direct project beneficiaries in 130 sub-districts in the three expansion provinces (baseline, 0, original target 1.3 million, revised target 850,000, target not achieved). According to the Project Team in its September 16, 2019 email, the target value of direct beneficiaries was an annualized target based on the use of block grants. The target until 2017 was 850,000. This was exceeded with 876,832 pregnant women and children benefiting from block grant funds. There was no target for block grants during the one year final extension but village law funds were leveraged by piloting the HDW program to support women and children, reaching a total of 250,398 beneficiaries. The baseline and target values of direct beneficiaries were updated three times. 1. At the first AF in June 2014, the project expanded geographic coverage to include three new provinces - South Sumatra, Central and West Kalimantan. A sub indicator to monitor direct project beneficiaries in the 130 sub districts in these 3 new provinces had original baseline value of 0 and a target of 1.3 million. This increased the target value for the project beneficiaries to 6.7 million and the project reached 1.56 million direct beneficiaries. Target female beneficiaries was increased from 50 percent to 55 percent of beneficiaries and the project achieved 53.69 percent. The target value for the indicator on percent of previous non users that benefited from the sub-projects was revised upwards from 30 to 50 percent to encourage higher target achievement with the project closing extension. The level of achievement reached 71.73 percent. 2. At the second AF in May 2015, the baseline and target values were revised downwards from an original baseline value of 6.7 million to 3.6 million as a result of the 2010 census data. Total target value was therefore reduced from 3.6 million, of which 850,000 were from the three new provinces. The 2010 census data showed low population densities in the three expansion provinces. The level of achievement reached 1.56 million. 3. At the third AF in March 2016, the total beneficiary target was revised downwards to 850,000 total because only the 130 sub-districts would receive grant funding during the 2017 calendar year due to the transition strategy of the new Village Law. The project achieved 1.56 million direct beneficiaries exceeding the revised target. OUTCOMES:  The local communities in poor and rural sub-districts were empowered as evidenced by the 75,153 community members who assumed community leadership roles as Generasi community cadres or as members of the 11 person Village Advisory Team (TPMD). Community members interacted closely with frontline service providers to discuss village proposals. These community members became familiar with basic diagnostic practices that led them to identify demand and supply side constraints. They volunteered in community health posts (posyandu). As a result, the posyandu as a community institution was revived. The activism of the volunteers led to influencing village governments to allocate financial resources to posyandu activities (ICR, paragraph 31). Trained community members also volunteered to become Human Development Workers, serving as village cadres to support the implementation of the Government's National Strategy to Accelerate Stunting Prevention (StraNas Stunting) as frontline nutrition convergence agents. According to the September 16, 2019 email from the Project Team, he HDW program was piloted in 9 provinces, 31 districts, and 3,105 community cadres becoming HDWs (achieving target). The pilot was designed in the last year of project implementation, leveraging funding from the village law to ensure sustainability of results and continuing community engagements after project closing. Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) The cumulative outputs and outcomes provided evidence of substantial contribution to achieving the PDO. Revised Rating Substantial OBJECTIVE 2 Objective • to increase utilization of health and education services Rationale The second objective addressed constraints to community access and utilization of health and education using both demand‐ and supply‐side services. Generasi provided block grants to communities to finance the priorities they determine in the health and education sectors, recognizing specific local constraints and the perceived value‐added to the community. OUTPUTS: (cumulative data covering the five year implementation period was obtained from the ICR, Annex A, p. 31-42). The Project Team confirmed in an email dated September 16, 2019 that a framework design was used to set targets during implementation, subject to community needs, hence there were no targets for these outputs::  All 130 sub-districts achieved 100 percent disbursement rate at the end of the program cycle (baseline 85 percent, original target 90 percent, target exceeded)  291 community health facilities were constructed or rehabilitated (ICR did not provide target values)  53 educational facilities were constructed or rehabilitated (ICR did not provide target values)  15,291 community teachers were trained in Early Childhood Education and Development (baseline 0, original target 15,000, target exceeded)  2,643,208 pregnant or lactating women, adolescent girls and/or children under the age of 5 received basic nutrition services (ICR did not provide target values)  230,864 community health posts were given operational support (the ICR did not provide target values)  258,221 frontline service providers were paid (as incentives) for services delivered (ICR did not provide target values)  766,990 students received financial support (subsidies) for their education (i.e., scholarships, uniform, transportation subsidies) (ICR did not provide target values)  11,965,987 pregnant women, children between the ages of 0 and 5, and school aged children benefited from community health and education services. According to the Project Team in their September 16, 2019 email, a specific target for each group could not be determined beforehand as this depended on overall community needs and priorities. OUTCOMES:  85.68 percent of pregnant women received four prenatal care visits (original baseline was 80 percent, revised baseline 70 percent, original target 85 percent, revised target 80 percent, target exceeded) Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585)  98.79 percent of deliveries were assisted by trained professionals (baseline 75 percent, original target 80 percent, revised target 85 percent, exceeded)  60 percent of children under the age of 3 were weighed monthly (baseline 75 percent, original target 80 percent, revised target 85 percent, target not achieved)  84.62 percent of pregnant women participated in monthly pregnancy and nutrition classes in 3 new provinces (baseline 0, original target 60 percent, target exceeded)  35.65 percent of male caregivers participated in monthly pregnancy and nutrition classes in 3 new provinces (baseline 0, original target 40 percent, target almost achieved)  69.79 percent of parents of 0-2 year olds participated in monthly parenting and nutrition classes in 3 provinces (baseline 0, original target 60 percent, target exceeded)  95.74 percent of junior secondary enrollment rate achieved (baseline 70 percent, original target 75 percent, revised target, 85 percent, target exceeded)  Under the Early Childhood Education and Development (ECED) frontline pilot, community facilitators raised ECED awareness, improved ECED teaching competencies as evidenced by their advocacy with village governments and communities for funding to support their professional development. The Project Team clarified in its September 16, 2019 email that the target was 15,000 teachers and 15,291 teachers were trained. The above outputs and outcomes showed that the residents of the rural areas in the three provinces used the Generasi platform to substantially increase their access and use of health and education services. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective This objective was not revised. Revised Rationale The theory of change was not revised. The outputs and outcomes from the original objective apply here as well. Changes covered updates in baseline and target values as part of AFs (June 2014 and June 2016) and project closing extension.  All the baseline and target values show percentages of beneficiaries receiving the services out of the total target population in the participating villages during the implementation period.  The baseline value for this indicator showing the percent of pregnant women receiving four prenatal care visits was revised downwards from the original 80 to 70 percent. The target value was revised downwards from 85 to 80 percent. The baseline was changed because of improvements in measuring this indicator to account for 4 prenatal care visits during the correct trimester, rather than at any point during the pregnancy. The target value was adjusted because of the updated baseline value and project closing extension. This indicator recorded that 85.68 percent of pregnant women received four prenatal care visits. Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585)  The baseline value for this indicator showing the percent of deliveries assisted by trained professionals did not change but target value was increased from 80 to 85 percent.  The baseline value for this indicator showing the percent of children under 3 weighed monthly was not revised but the target value was increased from 80 to 85 percent. The original indicator was percent of children under 5 weighed monthly. The new target value was introduced consistent with the project's increased focus on preventing chronic malnutrition. Chronic malnutrition had a more severe impact on younger children.  The indicators showing (i) the percent of pregnant women, and (ii) percent of male caregivers, and (iii) percent of parents of 0-2 year olds who participate in monthly parenting, pregnancy, and nutrition classes were updated in the June 2016 restructuring to include "in 3 provinces". The implementation plan for the supply side activities of the project financed under MCA-I was revised to include that the activities were implemented in West Kalimantan, Central Kalimantan, and South Sumatra. The project achieved 84.62 percent of pregnant women, and 35.65 percent of male caregivers, and 69.79 percent of parents of 0-2 year olds participating in pregnancy and nutrition class monthly the 3 provinces.  The baseline value for the indicator showing junior secondary enrollment rate was not changed but the target value was increased from 75 to 85 percent. The project achieved a 95.74 percent of junior secondary enrollment rate. The cumulative achievements of the project provided evidence of the substantial increase in access and use of health and education services. Revised Rating Substantial OBJECTIVE 3 Objective • to foster accountability in local service delivery Rationale Theory of Change: This objective was eventually dropped during the 2014 Restructuring. However, it was maintained as an intermediate outcome indicator. Relevant outputs and outcomes continued to be attributed to this dropped objective during project implementation. The third objective aimed to strengthen the accountability of service providers through vertical and social accountability measures, including monthly inter-village meetings (or MADs), well‐attended public forums that provided a venue for citizen feedback and for sharing information between village cadres and service providers. OUTPUTS: (cumulative data covering the five year implementation period was obtained from the ICR, Annex A, p. 31-42).The Project Team confirmed in an email dated September 16, 2019 that a framework design was used to set targets during implementation, subject to community needs, hence there were no targets for these outputs:  66 districts were trained in MIS data collection and interpretation  66 districts conducted planning and coordination workshops with district level health and education offices Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585)  81 percent of district governments conducted supervision to communities, as planned (baseline 45 percent, original target 454 percent, revised target 50 percent, target exceeded)  All districts conducted planning and coordination workshops to discuss Generasi activities with district level health and education offices (baseline 40 percent, original target 45 percent, revised target 80 percent, target exceeded). The Project Team clarified in its September 16, 2019 email that the target was revised because the Government required participating districts to coordinate with health and education offices to ensure supportive environment for Generasi activities.  96 percent of district level MIS data were completed, verified, entered, and submitted (baseline 80 percent, original target 85 percent, target exceeded)  37.69 percent of sub-districts in which 50 percent of villages conducted cross village audits as planned (baseline 60 percent, original target 70 percent, target not achieved). According to its September 16, 2019 email, the Project Team clarified that the target was not revised. They admitted that it was a difficult target to reach and that they should have revised it in hindsight.  Under the Early Childhood Education and Development (ECED) frontline pilot, more than 15,291 ECED teachers (target was 15,000, target exceeded) were trained to improve their teaching practices and advocacy skills. The DOK funds (earmarked block grant allocations) covered facilitation costs to raise community awareness in ECED, paid for teacher's training, and participation costs. Communities nominated teachers to be trained at the sub-district or district levels. OUTCOMES:  All service providers in all sub-districts (100 percent) attended inter-village meetings (Musyawarah Antar Desa or MADs) to discuss the status of health and education services (baseline 60 percent, original target 65 percent, revised target 80 percent, target exceeded). There were well-attended forums that allowed citizen feedback and sharing of information between village cadres and services providers. Data supporting these claims were obtained from the Long Term Quantitative and Qualitative Impact Evaluations. The 2018 study was funded by the World Bank in collaboration with the Abdul Latif Jameel Poverty Action Lab, the National Development Planning Agency (Bappenas), the Ministry of Villages, Disadvantaged Areas, and Transmigration (MOV), and the Australian Government's Department of Foreign Assistance and Trade (DFAT) (ICR, footnote 13). Two target outputs were not achieved and the outcome indicator was downgraded to an intermediate outcome indicator because of the removal of the PDO objective of "fostering accountability." This justified a modest outcome for this PDO. Rating Modest OBJECTIVE 3 REVISION 1 Revised Objective This objective was dropped. Revised Rationale Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) This objective was dropped during the 2014 restructuring and was reassigned as an intermediate outcome indicator. This objective was considered too ambitious because the Generasi program did not have a direct influence (sanction or reward function) over service providers that indicate accountability. The outcome indicator was not sufficient to show accountability. There were changes made to the baseline and target values of the intermediate outcome indicators to encourage a higher achievement since the project closing date was extended.  The original baseline value for the indicator relating to the percent of district government conducting supervision to communities was increased from 40 to 45 percent. The original target value was increased from 45 to 50 percent.  The original baseline value for the indicator(40 percent of districts) showed the percent of districts that conducted planning and coordination workshops with district level health and education offices to discuss Generasi activities was not changed but the original target value was increased from 45 to 80 percent. The Project Team clarified in its September 16, 2019 email that the target was revised because the Government required participating districts to coordinate with health and education offices to ensure a supportive environment for Generasi activities. Revised Rating Not Rated/Not Applicable OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The three activities reasonably supported the theory of change. The cumulative outcomes provided under each objective were substantial for the first two objectives and modest in the third. With the dropping of the third objective, a split rating of the outcome was applied. However, the third objective, while it was dropped, continued to be monitored and reported on as part of the intermediate outcome indicators in the M&E system. As a result, overall efficacy remained substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: At appraisal, a cost effectiveness analysis was conducted for Generasi. The closest comparator to Generasi was the Government's cash transfer program or Program Keluarga Harapan (or PKH). Costs were calculated as costs of facilitators, transfers to households, real expenditures of blocal grants and marginal cost of public funds (PAD, footnote 5). PKH focused only on the benefits enjoyed by PKH households. If spillover effects from PKH were to include those to non-receipient households in the same sub-districts, the Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) US$8 to US$11 per point cost of Generasi was comparable to that of PKH's cost of 11 per point. The Project Team clarified in its September 16,, 2019 email that a weighted average was used. The weights were set by the government to be approximately proportional to the marginal cost of having an additional individual complete each community indicator (ICR, Annex 4). The analysis concluded that Generasi was more cost effective than PKH (PAD, paragraph 36). Financial analysis was not applicable to the project because the project did not generate revenue (PAD, paragraph 37). At project completion, a benefit cost analysis generated a benefit cost ratio of 1.38 with costs coming from both health and education interventions but benefits only from the health interventions. The benefit cost ratio rose to 1.94 if costs were only those of the health interventions matched by benefits from health interventions. About 70 percent of the costs the interventions were health related, and 30 percent were education related. Even without the costs or benefits from the education interventions, the benefits outweighed the costs (ICR, paragraph 43). The project identified the following economic benefits: (i) benefits of deliveries attended by trained professionals; (ii) benefits of antenatal care; and (iii) benefits of immunization, Vitamin A supplements, and other health services from birth till age 5 years. All benefits were valued at present value against current or future incomes. Monetary values were estimated assuming a minimum wage of US$1,551 per year (source Trading Economics 2019), a 10 percent discount rate, rather than the 3 percent more commonly used for health interventions that include calculations of disability adjusted life years. There was no further justification for why 10 percent and not any other rate was used. The ICR provides an extensive analysis of the project benefits calculation in Annex 4. Operational and Administrative Efficiency: The project was originally approved to close only one year after approval because the funding source was ending at that time. There was an understanding that the original funding source would be extended and that the project would be extended as designed to be implemented over a five year period. In addition, as a result of various AFs, the project covered an expanded geographic area to include three new provinces - South Sumatra, Central, and Western Kalimantan - consisting of 130 sub-districts. The three AFs resulted in five restructurings, including two extensions of closing dates. With the addition of new villages as the project was implemented over a 4.5 year period, June 2014 to December 2018. The newer villages were provided larger block grant allocations to maximize investment impact (ICR, paragraph 44) while the original target participant communities that had benefited from several block grant funding cycles were given smaller allocations to encourage them to improve the quality of existing services. There were no cost overruns even though there were implementation delays arising from the change in implementing agency following the adoption of the new Village Law in 2014. There were also coordination challenges posed by the source of additional funding and complementary supply side interventions from this funding source (i.e., Millennium Challenge Account for Indonesia). These supply side activities included training of service providers, sanitation and hygiene activities, provision of multiple micro nutrient packets) that were financed directly in parallel. However, the providers did not use government implementation modalities and were not directly managed by the project nor overseen by the project, which contributed to negatively affecting operational efficiency. The one year extension of the project closing date did not generate additional costs because the Village Law annual transfers replaced the block grant funding. The project achieved substantial efficiency in both economic terms and administrative operations. There were no cost-overruns, substantial economic benefits and outcomes, and substantial participation from beneficiaries as provided by long-term quantitative and qualitative impact evaluations. There were only minor coordination shortcomings and a reduced scope, (accountability) although efforts in this regard continued to be monitored as part of intermediate outcomes, rather than at the PDO level. These factors justified a substantial rating of project efficiency. Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objective was rated high. The efficacy of objectives 1 and 2 was rated substantial. The efficacy of the original objective 3 was rated modest. However, this third objective was dropped as a result of the June 2014 Level 1 restructuring. A split rating was applied because the PDO was formally changed (reduced scope), one year into project implementation. At that point 25 percent of total project funds were disbursed. All the outcomes of project efficacy for all objectives were cumulative and assessed over the five year implementation period, and after the reduced scope. Efficiency was rated substantial. Applying the disbursement calculation to the assessment, the overall rating is 5, thus justifying an Outcome rating of Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The following posed risks to development outcome:  Government ownership/commitment. There is a risk that the Government may waver in its commitment to deliver health and education services to the community or village level. This risk is mitigated by the Government's continuing commitment to achieve improvements in the quality of life for all, address disparity and inequality as outlined in its Mid Term Development Plan. The human development dimension of this plan focuses on education and health. The Government has also allocated in its 2019 budget IDR60 trillion support for direct fiscal transfers to villages.  Financial Risk. There is risk that the robustness of the financial flows and financial viability of the transfers for health and education services established under the project may not be sustained over time. This risk was mitigated at the national level by the actions of the MOV tasked with implementing Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) the Village Law. The Village Law provided a substantial annual fiscal transfer to all village governments for development and empowerment purposes (ICR, paragraph 55). In 2018, these transfers accounted for approximately 6 percent of the national budget and around 0.5 percent of the country's Gross Domestic Product (GDP). Village administrations received allocations of about US$113,000 per year. The MOV's annual regulation has allowed the use of village funding for expanded types of health and education activities. This financial risk was also mitigated by local governments who issued regulations supporting this expansion. On average, in 2016 a village received IDR900 million a year for health and education services. This allocation increased to IDR1.3 billion in 2017. Village budgets in both years showed that villages allocated 7 percent of their total budgets for health and education activities, exceeding those provided under Generasi (ICR, footnote 25).  Social Risk. There is risk that the capacity of local actors to continue to participate in allocating resources for health and education services may not be sustained. The risk of weakening stakeholder support for these activities was mitigated by the Government's adoption of the National Strategy to Accelerate Stunting Reduction using Human Development Workers that began under Generasi. The Government also adopted a Village Convergence Scorecard with indicators similar to those used under Generasi. The scorecard tracked the delivery of priority interventions. According to the September 16, 2019 email from the Project Team, the community scorecards tracked outputs or intermediate outcomes that focused on use of services, not just access. It was critical that achievements be directly attributed to communities efforts so that they could be easily monitored, transparent, and effectively drive local investments. These clear results were used to determine the portion of incentivized block grants. The communities did not track outcomes such as stunting, education scores because these required different tools and expertise for accurate measurement. By continuing to use HDWs, the Government signaled its support to building the capacity of community cadres in designing and implementing health and education activities in cooperation with both village governments and frontline service providers. The use of the scorecard also mitigates the risk of reduced stakeholder support by creating awareness in the use of its data to inform communities and services providers as they prioritize activities. 8. Assessment of Bank Performance a. Quality-at-Entry The Theory of Change was sound. The PDO was clear. The three components were reasonably designed to achieve the PDO. The results framework had substantial indicators, except for the third element regarding establishing accountability. In 2007, the Healthy and Bright Generation (Generasi) program was piloted as a platform to deliver health and education services to the villages with lagging outcomes in these sectors. That pilot generated three long term impact evaluations. The prior PNPM Rural project also carried out a series of studies. Other community driven development programs financed by the World Bank also generated studies. All these studies informed project design including (i) review of community level target indicators, explicit community focus on out-of-school children and children with disabilities, nutrition counseling, and piloting early childhood education and development series for 3-6 year olds; (ii) strengthened fiduciary controls, Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) such as mobilizing district financial management consultants in all project locations; (iii) improved facilitator pre-service and refresher training; and (iv) strengthened partnerships with sector agencies and organizations active in improving access to basic series to the poor, disabled, and other marginalized groups (ICR, paragraph 50). By the time of effectiveness, the project was ready to be implemented. There was strong government support. Funds were available from the PNPM Funding Source even though the end date of the agreement governing the funding source was being negotiated for extension. The Directorate for Empowerment of Community, Social, and Cultural Institutions in the Directorate General for Village Community Empowerment (PMD), Ministry of Home Affairs (MOHA) was prepared to undertake the project. The Coordinating Minister for People's Welfare, through the PNPM Oversight Working Group working with Bappenas was in place. There were only minor shortcomings in the results framework during appraisal, particularly in the indicators adopted to achieve the objective of fostering accountability in local service delivery. This shortcoming led to dropping this objective during the second restructuring, which coincided with extending the closing date and obtaining additional financing. Quality-at-Entry Rating Satisfactory b. Quality of supervision The World Bank team based in Jakarta, specialist consultants, and DC based technical experts, participated in semiannual supervision missions. Reports and aide memoires were candid assessments of progress and challenges. For example, when there was a change in implementing agency, the task team worked closely with the Government on a transition strategy that included new financial management regulations for Village Law grants. Safeguards and fiduciary compliance were adequately supervised. Safeguards capacity building was embedded in the training that facilitators received prior to deployment, in refresher training, and in mentoring. Safeguard measures were embedded into project activities through facilitation, community participation, and social mapping processes (ICR, paragraph 63). Environmental safeguards were complied with satisfactorily. There were concerns about handover of project financed assets such as donated land plots for small infrastructure and nutrition gardens but these were resolved by project closing. Candor in reports was evident in justifying the four restructurings and additional financing (see Section 10a below). Implementation factors that were subject to government and implementing agency control: In 2015, the Government changed the project implementing agency from the Ministry of Home Affairs (MOHA) to the Ministry of Villages, Disadvantaged Areas, and Transmigration (MOV). This caused a 6 month implementation disruption and affected disbursements. In their September 16, 2019 email, the Project Team clarified that block grants were delayed reaching communities during that period. As a result, communities did not have funds for a short time period to invest in different activities, including nutrition supplements. Nutrition supplements were used as incentives for attendance at the community health posts. While rules did not allow block grants to be used for nutritional supplements, these were still eligible as part Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) of community investments. This led to qualitative field reports of a 15 percent reduction in posyandu attendance in 2016 and 2017 compared to previous years (ICR, footnote 21). The transition to MOV negatively affected safeguards monitoring, fiduciary compliance, and M&E activities. The project team candidly downgraded the implementation rating. Handover caused delays in uploading data on community grant expenditures, beneficiaries, community target indicator performance, and community participation. The National Management Consultant was retained, which stabilized the transition from one ministry to the other. Timely implementation was also negatively affected by the need to coordinate differing governance structures and finance mechanisms introduced by the participation of the US-funded MCA-I in June 2014. According to the Project Team, the MCA-I funded in parallel supply side activities at the same time that their contributions provided AF to the project. These efforts (training of service providers, sanitation and hygiene activities, micronutrient packets) did not use government implementation modalities. These supply side activities were also not directly managed by the project or overseen by the Bank. This affected the timely response to increased demand for village frontline supply side interventions (ICR, paragraphs 53- 54). In addition, there were two pilot activities launched still using the community driven development platform of the project. First, capacity of the community cadres were enhanced to modify their roles as frontline "convergence agents" or Human Development Workers (HDWs) in 31 districts implementing the 2017 National Strategy to Accelerate Stunting Prevention (StraNas). These StraNas districts overlapped with the Generasi districts. Second, the community scorecards were adapted using village and household mapping and data collection to allow HDWs to track the delivery of services. Third, the project technical support for multisector horizontal coordination was expanded in the district, sub-district, and village forums. The favorable experience with these pilot activities countered the minor delays experienced. In sum, there were shortcomings in resolving the difficulties posed by additional funds to deliver multi- sector services to villages and communities.The funding source - the US-funded MCA-I - had a different governance and financing mechanisms that affected the readiness and capacity of frontline supply side providers to respond to increased demand (ICR, paragraph 54). Nevertheless, supervision was satisfactory because of the outcomes achieved. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change was sound. The objectives were clear. The three key activities were expected to generate outputs that would lead to project outcomes as reflected in the results framework. The one shortcoming in the results framework had to do with indicators associated with fostering accountability. There was only one indicator and this was insufficient to directly attribute causality. At the same time, the Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) Generasi was not designed with reward and sanction mechanisms affecting service providers. With this gap, the project did not identify proxy indicators that could be useful to capture accountability. As a result, this aspect of the PDO was relegated to the intermediate outcome indicator level, rather than using the opportunity provided by the restructurings to identify an appropriate proxy indicator at a higher level. However, other intermediate outcome indicators adequately captured the contribution of the activities and outputs toward achieving PDO-level outcomes. All the indicators were specific, measurable, achievable, relevant, and time-bound. Baselines were established as a result of prior impact evaluation studies conducted for PNPM Generasi in general. Also, there were long term impact evaluations conducted, both quantitative and qualitative to ensure that all baselines were covered and project impacts captured. Revised baselines and target indicators were also fully explained. Sampling methods used a control as needed. Comparators were selected for the cost benefit analysis at appraisal but actual costs and benefits were fully hashed out in the Annex on Efficiency (ICR, Annex 4). The M&E framework was designed to be implemented using existing Government systems and therefore well-embedded instituionally. The design also incorporated lessons learned from the scaling up project for Generasi preceding this project. As a result, baselines were established, and targets were realistic. There were occasions were targets were not provided due to the framework design that set targets during project implementation subject to community needs. Targets for performance based grant allocations were aligned with how the Government collected data in the sector. As a result, 8 of the 12 community target indicators were collected by the Government under its posyandu activities (ICR, paragraph 59). b. M&E Implementation The planned baseline data collections were carried out during appraisal as well as during project implementation. Indicators provided in the Results Framework were measured and reported. The Management Information System (MIS) in place reported on achievements and disbursements of the PDO and intermediate outcome indicator targets as provided in the Results Framework. To address the weaknesses of the M&E design, specific baselines and targets were made more realistic during implementation. This was evident in the lowered values for target beneficiaries based on the results of the 2010 census showing lower densities for the three additional new provinces. Beneficiaries were involved in defining target indicators and assessing their achievement as part of the activity focusing on empowerment and facilitation support. The M&E system, including the MIS, was used to monitor implementation progress. There were initial challenges reported after the MIS was introduced. There was a high turnover of district computer operators that affected data gathering. There were also poor quality of training for facilitators noted and weak supervision and on the job support for provincial MIS specialists. During the 2015-2016 transition between implementing agencies as a result of the Village Law and handing over the project to the MOV, the timeliness and quality of data were adversely affected. However, after mitigating measures such as a clear line up of corrective measures embodied in an aide memoire, improved training by provincial specialists and improved national level tracking of human resources. Consequently, the MIS data uploads were reported to have reached 99 percent for all 5,798 villages. The M&E functions and processes were likely to be sustained after project closing. Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) c. M&E Utilization The M&E system was reported to have been used to inform project implementation. The new initiatives piloted under Generasi (ECED training pilot, HDW pilot) and the transition from Generasi to Village Law (VL) were all informed by the outputs generated by the M&E system. Evidence was provided by adjustments of the results framework and block grant investments were influenced by the Government decision to allow villages to use funding for basic social services. An MIS tool tracked annual village budgets to compare expenditures on Generasi activities and VL activities. Using data form 74 percent of the participating villages village allocations for Generasi like activities rose from 6.5 percent in 2016 to 7.2 percent in 2017 and to 9.7 percent in 2018. M&E findings were communicated to local villages (as part of the score card for example) and to provincial and national governments. The project went through five restructurings to accommodate additional financing. M&E target data were utilized and updated through these processes. M&E activities were used to inform the piloting of new initiatives under Generasi and the transition to the VL grants. Formal adjustments were made to the results framework as well as to the MIS in place to accommodate new data as a result of the introduction of these pilot activities. These included the collaboration with the Millennium Challenge Account for Indonesia, the transition and integration with the VL grants, the early childhood education and development pilot training, and the pilot for the Human Development Workers. MIS tracked annual village budgets to monitor how well integration was taking place and compared village spending in health and education compared to previous block grant amounts (ICR, footnote 23). M&E data together with the long term evaluations conducted provided considerable evidence in achieving outcomes. The long term evaluations used focus group discussion, semi structured interviews, observations and descriptions, document collection, mobile information and communication technologies, and videography. The interviews covered the range of project stakeholders at the province, district, sub- district and village levels and local government representatives, service providers, facilitators, and project beneficiaries (ICR, footnote 22). In summary, the M&E system as designed and implemented was generally sufficient to assess the achievement of the objectives and test the links in the results chain. There were moderate weaknesses in a few areas, such as in the choice of indicators for the objective regarding the fostering of accountability in the delivery of services. This resulted in a substantial rating of the quality of M&E. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Environmental and Social Safeguards. The project was classified as a Category B project and triggered OP/BP 4.01 Environmental Assessment and OP/BP 4.10 Indigenous People. Compliance with environmental and social safeguards were rated satisfactory. There was a short period when there were concerns about the handover of project financed assets such as the donated land plots for small Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) infrastructure and nutrition gardens. These concerns were resolved by project closing (ICR, paragraph 63) (see section 9(b) above for further discussion). b. Fiduciary Compliance Financial Management: The project complied with financial management policies of the World Bank. Interim financial reports were completed and submitted mostly on time. Financial transactions were processed adequately. Accounting regulations at all levels were maintained well and followed regulations. External audits were prepared in 2015, 2016, and 2017. There were no unqualified opinions. During the 2014-2017 period, there were 316 audit findings reported. 313 of these findings were resolved. The project's financial management reports noted the following shortcomings. During the transition (from the Ministry of Home Affairs to the Ministry of Villages), delays in payment of consultants and facilitators Procurement: The project complied with the Bank's procurement guidelines. The Government used administrative services firms (ASF) to provide administrative and management support. The World Bank conducted ex post reviews and found no significant issues. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR offered 7 lessons from the project operations that could benefit future similar projects. Three of these lessons are presented below with minor editing:  Both men and women have important roles to play in deciding how to improve maternal and child health and nutrition. In this project, a gender strategy targeted an Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) increase in male participation in maternal and child health and nutrition. This gender strategy included clear target indicators for male and female participation, and was widely disseminated to all stakeholders. Frequent reminders gave stakeholders strong signals that the benefits from improved maternal and child health and nutrition could be achieved sooner or increased by involving both male and female parents in the decision making process. This effort could also be a useful approach to breaking down gender barriers in delivering basic social services.  Community driven development (CDD) may be a useful platform to deliver health and education services by integrating both demand and supply side interventions. In this project, these services were delivered to the village and community levels in which they targeted the remaining non-service users in poor, rural communities. On the demand side, this project empowered community and village residents by giving them training and capacity building tools to increase the effectiveness of service delivery. These included social accountability tools such as social mapping, community scorecards, and social monitoring. These tools improved their ability to influence how the community allocated village funds for basic social services and allowed villagers to target those who had not been served before. On the supply side, incentives were offered to service beneficiaries and providers to increase available services and their utiilzation (ICR, paragraph 36).  Using performance based grants allows communities to set priorities based on local conditions and constraints. In this project, the performance based nature of the grants motivated village cadres to target improvements in enrollment in primary and secondary school. At the time of implementation, non Generasi investments in village education programs increased. The targets were reached but updating the original targets and providing additional incentives for good performance could have convinced the community to increase allocation for the sector, improving outcomes. Periodic review of targets could also be an opportunity to validate the effectiveness of the incentive system of a performance based grant. The experience in this project showed that the link between incentive and performance called for a better understanding by the community participants. 13. Assessment Recommended? No 14. Comments on Quality of ICR This ICR provided a good overview of the project in a concise manner, following OPCS guidelines. The report was internally consistent and the theory of change referenced to better understand how the ratings were reached. The report was results oriented and generally aligned with achieving the project objectives. The report presented robust evidence from long term qualitative and quantitative impact evaluations that were financed separately and conducted over a period of time. The impact evaluation studies were a credible source. The evidence provided by these studies and from various project reports were presented and referenced throughout the report. The annexes completed the evidence base to support the reported outcomes. The quality of analysis was sufficient and concisely summarized important points. There were clear links between the evidence and the reported findings. Lessons, though numerous, were based on evidence provided by the project experience. Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID-TF GENERASI PROGRAM (P132585) The analysis of lessons was for the most part clear and candid. A minor shortcoming was in the completeness of data and information. Particularly in justifying substantial efficacy from disbursing the first 25 percent of the project after year 1 followed by the second restructuring, which dropped the third objective. The Project Team justified in their September 16, 2019 email that there were no outcomes that could be assessed before the level 1 restructuring took effect because it was too early in the implementation stage to assess any outcome. The evidence of outcomes at project closing sufficiently supported a substantial rating. a. Quality of ICR Rating Substantial Page 22 of 22