Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) Report Number: ICRR0021864 1. Project Data Project ID Project Name P115592 HN Social Protection Country Practice Area(Lead) Honduras Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-47740,IDA-52940,IDA-56030 31-Dec-2014 77,116,918.16 Bank Approval Date Closing Date (Actual) 29-Jun-2010 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 0.00 Revised Commitment 77,113,801.93 0.00 Actual 77,116,918.16 0.00 Prepared by Reviewed by ICR Review Coordinator Group Anthony Martin Tyrrell Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project objectives as stated in the Financing Agreement (p. 7) of August 2, 2010 were to: (a) strengthen the institutional capacity of the Ministry of the Presidency (MoP) and the Family Allowance Program (PRAF) to administer and manage the Bono 10.000 Program through the development of transparent mechanisms and instruments for targeting Program beneficiaries, monitoring compliance with Program co-responsibilities, and making payments to Program beneficiaries; and (b) increase: (i) school attendance among students in grades 1 to 6; and (ii) the use of preventive health services among families participating in the Program. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) The Project received two additional financings (AFs) each of which involved changes to the project objectives (July 8, 2013 and February 26, 2015) and further underwent five level-two restructurings (April 22, 2013, November 20, 2014; January 4, 2017; July 10, 2017; and December 29, 2017) that involved various changes including changes to outcome indicators and revisions that were principally associated with scaling up activities, adding greater specificity to targeting, and to reflect changes in the institutional arrangements. The PDO under the second AF in February 2015 was to: (a) improve the institutional capacity of Recipient’s institutions to manage the CCT Program, by strengthening transparent mechanisms and instruments for targeting Conditional Cash Transfer (CCT) Program beneficiaries, monitoring compliance with the CCT Program co-responsibilities, and making payments to the CCT Program beneficiaries; (b) provide income support to eligible beneficiaries; (c) increase the use of preventive health services and school attendance in grades 1 to 9 among CCT Program beneficiaries in rural areas; and (d) improve the Recipient’s capacity to respond promptly and effectively to an eligible emergency. As the revisions did not materially change the objectives, and as the scope of the project and its ambition increased, this ICR Review did not apply a split evaluation because it would not add value in this case. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 26-Feb-2015 c. Will a split evaluation be undertaken? No d. Components Component 1: Institutional strengthening of the CCT Program (Planned US$12.8 million; Actual US$10.4 million). The component was described in the PAD as being aimed at strengthening the efficiency, effectiveness, and management of the CCT Program and contributing to the development of a Social Protection (SP) strategy by building the capacity in the MoP and PRAF through: (a) enhancements in the organizational and technical capacity of the MoP to coordinate, monitor, and evaluate the CCT program including reviewing the targeting mechanism for Program beneficiaries, developing a Monitoring and Evaluation (M&E) system, conducting Program evaluations, piloting of alternative payment mechanisms and implementing a communications campaign, (b) improvements in the operational and administrative capacity of the PRAF agency including updating the beneficiaries registry, developing a Management Information System (MIS) to support beneficiaries payroll management and verification of compliance of co- responsibilities and creating a unit to attend grievances and complaints; and (c) the development of an integrated SP strategy to design, implement, and monitor coherent SP programs and policies and the development of Unique Registry of Beneficiaries. The responsibilities of implementation and financial management were handed to the Technical Coordination Unit, under the MoP for activities (a) and (c), and to PRAF for activity (b). Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) The component underwent significant changes during the first two restructurings (April 2013, November 2014) and under both AFs (July 2013, February 2015) in response to government’s reorganization, notably the introduction of the Ministry of Development and Social Inclusion (Secretaria de Desarrollo e Inclusión Social, SEDIS). As noted in the ICR (p. 11), these changes allowed the project to support the management of the CCT Program both at the Program - refining the targeting mechanism, linking beneficiaries to existing productive inclusion interventions, and strengthening delivery systems including the MIS, payment mechanisms and monitoring of co-responsibilities - and the sector levels - investing in tools like the Unique Registry of Participants (RUP) that could help consolidate social assistance and subsidy programs - and including piloting a productive inclusion strategy. Later changes to Component 1 focused on improving the inter-sectoral coordination among the Ministry of Development and Social Inclusion (SEDIS), Ministry of Health and the Ministry of Education (MoE); and, supporting regional offices through the de-concentration of the program. Component 2: Conditional Cash Transfer Program Grants (Planned US$ 61.3 million; Actual US$61.6 million). The PAD noted this component would finance cash transfers to eligible beneficiaries who complied with co-responsibilities, which included: (a) children between 0 and 59 months attending preventive health controls; (b) pregnant and postpartum mothers completing their pre- and post-natal check-ups; and (c) children from 1st to 6th grades (primary level) being enrolled in school and attending classes. PRAF was responsible for the implementation and financial management of this Component. The ICR (p. 11) noted that activities under Component 2 also widened the program’s coverage of vulnerable households by adjusting the benefit structure and increasing the scope of the program’s co-responsibilities for education. Groups included as a result of these adjustments include those under extreme poverty, communities with large populations of indigenous peoples, and those in vulnerable areas, such as the Dry Region of Honduras (Corredor Seco). The benefit structure was revised in 2015 from an annual lump sum of HNL 10,000 per household (approximately US$ 44 per month) to a smaller lump sum (an average of HNL 5,967, roughly US$ 20 per month) and incremental benefit per child complying with co-responsibilities. Education co-responsibilities were expanded to include school attendance for grades 7-9 to incentivize households to keep their children in school longer. Health co-responsibilities were reviewed to specify the number of health check-ups to be accounted as co-responsibility compliance e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The estimated cost at appraisal was US$40 million. There were two tranches of Additional Funding totaling $37.3 million that brought commitments up to $77.3 million. Actual project cost was $77.1 million. Financing: Planned support for this Specific Investment Loan was with an IDA Credit of US$40 million (IDA-47740). This was supplemented by two tranches of AF in IDA Credits - $12.3 million (IDA-52940) and Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) $25 million (IDA-56030) – for a total commitment of $77.3 million of which $77.1 million was disbursed. There was no planned or actual borrower contribution. Dates: The project was approved on June 29, 2010 and became effective October 29, 2014. The Mid-term Review was completed October 8, 2012. The project was extended on four occasions (8 July 2013; 26 February 2015; 10 July 2017; and 29 December 2017), and closed on December 31, 2018, four years after the planned closing, which was December 31, 2014. Restructuring: There were two AF and five level two restructurings as summarized in Table 1 below. Table 1: Summary of Restructurings and Additional Financing Amount Disbursed (US$ Dates Key Revisions m) Change in Results Framework; Change in Components and Cost; Reallocation between Disbursement Categories; Change in 22 April, 23.01 Disbursements Arrangements; Change in Institutional 2013 Arrangements (PRAF, rather than MoP became the principal executor of Bono 10,000); Change in Financial Management Additional Financing (scaled-up activities under components and new institutional strengthening activities); Change in Project Development Objectives ( added an Immediate Response 8 July, Mechanism (IRM) Contingent Component in the PDO for eligible 30.26 2013 emergency and a corresponding outcome indicator); Change in Results Framework; Change in Components and Cost; Change in Loan Closing Date(s); Change in Legal Covenants; Change in Procurement Change in Implementing Agency (The Vice Ministry of Social Inclusion (SSIS) absorbed PRAF and took over implementation 20 and coordination of the Program and all project activities); November, 42.80 Change in Results Framework; Change in Components and Cost; 2014 Reallocation between Disbursement Categories; Change in Legal Covenants; Change in Institutional Arrangements; Change in Financial Management; Change in Procurement Additional Financing (Grades 7-9 now eligible for grants); Change in Project Development Objectives (expanded 26 objective to also include education grants to lower secondary February, 43.11 school students (7th to 9th grade) with associated outcome 2015 indicator); Change in Results Framework; Change in Components and Cost; Change in Loan Closing Date(s); Change in Procurement 4 January, 55.65 Change in Results Framework 2017 Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) 10 July, 66.30 Change in Loan Closing Date(s) 2017 29 December, 73.87 Change in Loan Closing Date(s) 2017 Under the first AF in July 2013, the PDO was revised to: make explicit that the project would continue to co- finance cash transfers and to focus in the rural areas; change certain wording to broaden reference to “the Recipient’s Institutions” rather than, exclusively, “MoP and PRAF” and to the “Conditional Cash Transfer (CCT) Program” rather than “Bono 10,000” given the likelihood that the Program’s name could be changed (which it subsequently was i.e., to Bono Vida Mejor); and to add an objective, i.e. improve the Recipient’s capacity to respond promptly and effectively to an emergency, consistent with other projects in Honduras where Contingency Emergency Response components were included. Under the second AF in February 2015 the PDO was revised to expand the eligibility of the Project’s target beneficiaries to also cover students in lower secondary education grades (grades 7-9). 3. Relevance of Objectives Rationale At project formation, the government was committed to breaking the intergenerational cycle of poverty - total poverty and extreme poverty were at 58.7 percent and 36.4 percent respectively in 2009. One of the main objectives of the National Plan 2010-2022 (Plan de Nación 2010-2022) was to reduce income inequality and extreme poverty through increased coverage and better quality of basic health, nutrition and education services, and through the creation of an integrated social protection system. To support these objectives, the Government decided to prepare a new CCT program, Bono 10,000, with the aim of encouraging families to increase investments in education and child health and nutrition noting: (i) for children from the poorest quintile, the enrollment rate decreased sharply after 12 years of age to 63 percent for children aged 13 years old and to 50 percent among children aged 14 years old (compared to 90 percent for children aged 13 and 14 years old in the wealthiest quintile); and, (ii) despite certain improvements in health, disparities in the access and use of health services remained e.g., only 35 percent of women in the poorest quintile gave birth in an institutional facility compared to almost a hundred percent in the richest quintile (w2005/06), and almost 10 percent of households in the poorest quintile mentioned supply-side constraints including the distance to the clinic and closure of the clinic (2004). Infant and child malnutrition was particularly difficult as one in every three children under the age of five years suffered from growth retardation, and more than 10 percent of children under the age of five were severely stunted (2008). In addition, Honduras had a large number of social assistance programs, the most important of which was the PRAF cash transfer program - initiated in 1990 - which was designed to mitigate the effect of crises and reduce structural poverty. The new Government of Honduras proposed to strengthen the social protection system, starting with a CCT, Bono 10,000 which would subsume some of the earlier cash transfer programs and which would improve coverage, benefits and accumulation of human capital for families living in extreme poverty. The program was to represent an important advance in the design of Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) social assistance programs in Honduras and was intended to lay the groundwork for a more comprehensive approach to social protection. At project closing the project objectives continued to be consonant with government and World Bank group strategies to include, for example: Honduras’ Country Vision for 2038 which is organized around three strategic pillars - (i) fostering inclusion; (ii) bolstering conditions for growth; (iii) reducing vulnerabilities – and which includes targets such as the reduction of the incidence of poverty to below 15 percent, and, increasing the average number of schooling years to 9 years; and the World Bank Group’s Country Partnership Framework (CPF) for FY16-20 with specific reference to the CPF objective to “Expand coverage of social programs” and aligned to the foster social inclusion pillar by improving conditions for growth and reducing vulnerabilities to enhance resilience. Project activities ultimately supported aspects of the CPF, FY16-FY20, notably under its first pillar for fostering inclusion. The project objective to ‘improve institutional capacity to manage’ the CCT program did not strictly reflect CPF Objective 1 to ‘expand coverage of social programs’, as the capacity to manage can be improved without expanding coverage although we accept that enhanced management capacity would likely ensure higher quality (e.g., more regular, reliable, sustainable) provision over time. Within the government’s strategic plans, reduction of extreme poverty is a priority as is continued commitment to building institutional capacity to strengthen the social protection system. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen the institutional capacity of Recipient’s institutions to manage the CCT Program Rationale The ICR assessed efficacy based on two objectives. For the purposes of this exercise, IEG has assessed project achievement based on five objectives as per the PDO statement in the AF of February 2015, which were to: improve the institutional capacity of Recipient’s institutions to manage the CCT Program; provide income support to eligible beneficiaries; increase the use of preventive health services; increase school attendance in grades 1 to 9 among CCT Program beneficiaries in rural areas; and to improve the Recipient’s capacity to respond promptly and effectively to an eligible emergency. The core assumption underpinning the project was that poverty in Honduras was perpetuated by a lack of financial and technical support needed to overcome social services’ demand-side barriers at the institutional and household levels and that increased institutional capacity in targeting, co-responsibilities compliance verification, and stable payment cycles - i.e., enhanced management - would improve the delivery of the program and facilitate Program scale-up. The results chain envisaged the provision of capacity building technical assistance and training; a review of the targeting mechanism; the development of a monitoring and Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) evaluation system; the development of a Management Information System (MIS); the update of the beneficiary's register; and the formulation of a social protection policy. These inputs/activities would lead to enhanced program coordination, and monitoring and evaluation as well as an improved targeting mechanism together with an MIS, an updated registry, and an approved social protection policy. The combined effect would plausibly lead to strengthened institutional and program management capacity. Outputs  Communication Strategy Implemented; Information on program was disseminated through local media & adapted to the target population by December 2018;  Against a formally revised target of 60 percent of households registered in the program with updated eligibility information, the outturn at project close was 100 percent of households registered with updated eligibility information;  A Management Information System for the program was operational and generating reports at project close;  A grievance redress mechanism was implemented in 17 field offices with grievances and complaints recorded, follow up tracked, and field staff trained on the use of the grievance redress mechanism.  By project close, 2,296 households (against a target of 2,300) in the program were registered with a local productive inclusion program  At project close, 90 programs (against a formally revised target of 18) were using the Unique Registry of Beneficiaries, which was established under the project.  At project close, the number of individuals incorporated in the Unique Registry of Beneficiaries was 4,600,399, against a target of 4,300,000  At project close 89.52 percent of beneficiary households (against a target of 30 percent) in rural areas received their payment through basic accounts although, as noted elsewhere, the contract with the commercial bank operating the accounts was dropped post-program.  Percentage of beneficiaries that received capacity building on the use of basic bank accounts was 89.56 percent against a target of 50 percent.  Number of children registered in Early Childhood Registry (RENPI,) which collected data on vaccinations and established the first individual database for health service use for children in the country, was 755,425 at project close against a target of 570,000. Outcomes For purposes of this ICR Review, the following are considered indicative of transparent and effective targeting: The target of 80% of households in the program receiving CCTs in rural areas that are extreme poor was exceeded – at 99.89 %; the targeting methodology was published in the Official Gazzette and a Grievance Redress Mechanism was established in 17 departmental offices; and the 2016 EPHPM indicated that 94 and 93 percent of rural program beneficiary households were among the extreme poor in 2014 and 2015, respectively, in terms of household income. The project supported enhanced system capacity through investment in the program’s MIS and its interoperability with health and education information systems, which enabled, for example, an automated Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) process for verifying co-responsibilities; however, verification of health co-responsibilities was more challenging in the absence of a health sector MIS. As such, the project financed the establishment of an Early Childhood Registry (Registro Nacional de la Primera Infancia, RENPI), managed by CENISS, which collected data on vaccinations and established the first individual database for health service use for children in the country. Four alternative payment mechanisms were piloted; however, the project closed without the establishment of an alternative payment method. The first three pilots were not scaled up because: (i) providers did not find the business profitable; (ii) there was a lack of a supervisory legal framework; and, (iii) the government opted for the use of basic saving accounts for payments, rolled out as part of its financial inclusion strategy but without buy-in of the banking sector. The project supported a fourth pilot in 2016 that successfully ensured payments to 8,860 beneficiary households in three departments via a commercial bank in 2016 and part of 2017 to 8,860 beneficiary households (using basic bank accounts) in three departments; however, following project close the contract with the commercial bank was not renewed due to more expensive than anticipated costs per transaction. The ICR (p. 14) notes that, while in terms of payment delivery there were no major improvements, substantial progress was made in the processing of payments which included the automated creation of the beneficiary payment payroll, high execution of payments with respect to planed payments and automated verification of education co-responsibilities. Rating Substantial OBJECTIVE 2 Objective Provide income support to eligible beneficiaries Rationale The design of the conditions built into the CCT assumed that insufficient investments by poor households in schooling, health, and nutrition were driven by liquidity constraints, thus resulting in poor human development outcomes leading to low earnings capacity. The CCT would provide the incentive to mitigate those constraints thereby enabling beneficiaries to engage in relevant health and education. The efficient and targeted delivery of the CCT would be dependent on the successful realization of the activities referenced under the Rationale for Objective 1 above. Outputs  Number of households receiving payments in the year in the Program in rural areas was 213,886 against a target of 240,000.  Number of households receiving payments in the year financed through the project in rural areas was 18,807 against a target of 40,000.  Number of indigenous and afro-Honduran households registered in the Program and financed through the Project was 8,503 against a target of 10,000. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) Outcomes The percentage of extremely poor households in rural areas covered by the program in receipt of CCTs was 99.89 percent of households against a targeted 80 percent. Some 218,713 extreme poor households were in receipt of payments, helping to smooth consumption and incentivizing desired behaviors. The CCT transfer represented about 25 percent of the household income of the poorest quintile. Rating Substantial OBJECTIVE 3 Objective Increase the use of preventive health services in rural areas. Rationale The design of the conditions built into the CCT assumed that insufficient investments by poor households in schooling, health, and nutrition were driven by liquidity constraints, thus resulting in poor human development outcomes leading to low earnings capacity. The CCT would serve to incentivize relevant engagement in health and education which would lead, as per this objective, to increased use among rural dwellers (all beneficiaries referenced are rural dwellers) of preventive health services that would ultimately lead to enhanced human development outcomes. Outcomes The PDO indicator, “percent of Program beneficiary children aged 0-23 months with a complete vaccination scheme,” was partially achieved - 48 percent of program beneficiaries aged 0-23 months had a complete vaccination scheme, compared to a target of 70 percent. This is associated with weaknesses in the Ministry of Health’s vaccination information systems (systems used for tracking vaccinations not strong enough to document vaccination levels in program intervention areas in real-time even though IE results indicate that beneficiaries did in fact comply with health co-responsibilities) and inter-sectoral challenges in verifying health co-responsibilities. That said, the 2013 Program Impact Evaluation indicated that infant height and weight monitoring increased by 11.3 percent and 15.9 percent, respectively, as well as increased check-ups for children aged 0-3 years. Rating Modest OBJECTIVE 4 Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) Objective Increase school attendance in grades 1 to 9 among CCT Program beneficiaries in rural areas Rationale The design of the conditions built into the CCT assumed that insufficient investments by poor households in schooling, health, and nutrition were driven by liquidity constraints, thus resulting in poor human development outcomes leading to low earnings capacity. The CCT would serve to incentivize relevant engagement in health and education which would lead, as per this objective, to increased school attendance in rural areas (noting all beneficiaries referenced below are rural dwellers) that would ultimately lead to enhanced human development outcomes, enhanced earnings and reduced levels of poverty. Outputs  98.7 percent of program beneficiaries aged 13-15 years completed primary education (sixth grade), surpassing the 78 percent target;  98.6 percent of female program beneficiaries aged 13-15 years completed primary education (sixth grade), above the 78 percent target;  for lower secondary completion (ninth grade), the project also exceeded its end targets of 60 percent with 94.4 percent of program beneficiaries aged 16-18 completing lower secondary education (ninth grade);  95.7 percent of female program beneficiaries aged 16-18 years completed lower secondary education;  the Impact Evaluation (2017) confirms these results with evidence that school enrollment and attendance of program beneficiaries who were 17 years old at the time of the study had increased by 5.1 percent (male) and 5.2 percent (female). Outcomes 85.5 percent of students in grades 1-6 and 92.7 percent of students in grades 7.9 attended school at least 80 percent of the time, surpassing the respective targets of 85 and 70 percent of students participating in the program who comply with the co-responsibility of 80 percent school attendance. Although the zero baselines reported in in the ICR make this result a bit problematic to interpret, overall this observation, combined with others above, provides support for the claim that payments made to eligible households incentivized the use of education to improve human capital development of children. Rating Substantial OBJECTIVE 5 Objective Improve the Recipient’s capacity to respond promptly and effectively to an eligible emergency. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) Rationale This objective was added as part of the second AF in July 2013. The ICR did not assess the achievement of the objective because, according to the ICR, no eligible emergency occurred over the project cycle. However, IEG is of the view that it should be possible to assess enhanced capacity to respond through improvement in systems, training of personnel, budgetary management, and in relation to the country's Immediate Response Mechanism (IRM) Operations Manual. In that regard the ICR provides information regarding enhanced capacity (as under the first Objective assessed above) regarding, for example, the development of an MIS, more accurate registry and targeting that could be interpreted to indicate enhanced capacity to respond to an emergency situation. On the other hand, the ICR also identifies threats to development sustainability associated with a weak budgetary management and incomplete intuitional arrangements and the failure to further develop key aspects of process such as ensuring regular payment of transfers. In addition, the TTL informed IEG that during project implementation a workshop with the Ministry of Finance was conducted at which it was concluded that the direct cash transfers should be an eligible expenditure to respond to an eligible emergency and would be included in the Operation Manual (OM) for CERC; however, the government agreed with the Bank that before including this expenditure in the OM it would be required to assess if the necessary conditions pertain in Honduras to allow for the use of cash transfers as an eligible expenditure of the CERC. To this end the Bank’s team is currently carrying out a Bank-executed technical assistance funded by a trust fund. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The aggregation of three almost fully achieved objectives and two partly achieved objectives is consistent with a Substantial rating for efficacy. Overall Efficacy Rating Substantial 5. Efficiency A standard economic analysis was not conducted for the ICR as administrative costs associated with the program could not be obtained. Two impact evaluations (IEs) were undertaken (2013 and 2017); however, because of changes during implementation - with particular reference to the target population, coverage, and the size of the cash transfer - the assumptions underpinning the original economic analysis did not hold, and the ICR (pp.15-16) states that estimation on program impact on the basis of the IEs is difficult to compare with those estimated at appraisal. However, certain findings may indicate enhanced efficiency. For example, in the context Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) of reduced benefit size in 2015 of roughly 40 percent in nominal terms, both impact evaluations reported a reduction in the poverty headcount, poverty gap, as well as an increase in school enrollment and school attendance among 6- to 17-year-olds. Targeting efficiency was found to be strong and the ICR suggested that, by project close, the program had become one of the best targeted CCT programs in the Region. About 90 percent of program beneficiaries were extremely poor and 94 percent of beneficiaries were relatively poor in both 2014 and 2015. Following the government’s decision to re-engineer targeting to focus on the extreme poor (rather than both extreme and moderate poor), and subsequent 2014 project restructuring, the project provided Technical Assistance to fine- tune the Proxy Means Test (PMT - including characteristics such as, for example, location, ownership of durable goods, and demographics) to more accurately identify the extreme poor. Subsequently, data was collected on approximately 120,000 new households (including Indigenous People), and the Unique Registry of Participants was updated using the new Unique Socio-economic Form to validate existing beneficiary profiles. On the other hand, the project experienced delays in the implementation of key activities associated with a combination of factors including: high turnover of project personnel, particularly those involved in the Productive Inclusion (PI) strategy; delays in the procurement of key technical advisory services (due to an inadequate pool of consultants, poorly written terms of reference) that, in turn, caused delays in the completion or piloting of key activities including upgrade of the Program MIS, establishment of alternative payment mechanism, de- concentration of program operation and certification of the RUP; and freezing of all international funds as part of tightened measures to meet the fiscal targets agreed with the International Monetary Fund. Other delays were associated with changes in the government’s institutional structure and program design that made it difficult for systems and personnel to adjust mid-stream. The institutional reform (end 2014 to beginning 2015) saw the Family Allowance Program (PRAF) absorbed into the Vice-Ministry of Social Integration (SSIS) to form the Ministry of Development and Social Inclusion (SEDIS). Program changes included the targeting of extremely poor households only and changes in the benefit structure. The required adaptation took close to an entire year for the new targeting parameters to go into effect. Finally, the frequency of payments and associated unpredictability represented an inefficiency of note. During the first three years of implementation, one payment would take as much as three months. The introduction of the automated generation of beneficiaries’ payroll and the establishment of a payment schedule in the program’s operations manual, meant that the process could be reduced to around 45 days by project close; however, payments were still highly unpredictable such that in 2015 and 2018, only two payments were made (against a target of three). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated Substantial given broad alignment with the country CPF at project closing. Efficiency is rated Modest because of significant shortcomings in the efficiency of implementation. Efficacy is rated Substantial, as the project almost fully achieved most objectives. The overall outcome rating is therefore Moderately Satisfactory, consistent with moderate shortcomings in project preparation, implementation and achievement. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR noted various potential risks to development outcome associated with the broader socio-economic context in Honduras notwithstanding ongoing government commitment to reducing poverty and promoting the accumulation of human capital among the extreme poor, as well the government’s ongoing engagement with both the World Bank and IADB towards building on progress achieved by the project. Potential threats to development outcome include the failure to (i) fully institutionalize the Center for Information on Social Sectors (CENISS - which manages the Unique Registry of Participants (RUP) and Registry of Institutional Supply (ROI)) that may undermine its potential, (ii) to include the budget for the program in the ongoing national budgetary; and (ii) the failure to further develop key aspects of process such as regular payment of beneficiaries through an alternative payment mechanism, and the verification of health co-responsibilities. 8. Assessment of Bank Performance a. Quality-at-Entry The project was hastily composed, in part to qualify for the IDA’s Crisis Response Window, but also soon after the global financial crisis and political transition in Honduras (the new government asked the World Bank and IADB for support in defining the parameters of the proposed new program, Bono 10,000). Consequently, aspects of due diligence required to establish the government’s readiness to implement the project were not carried out. As it turned out, readiness and capacity were underestimated and this ultimately required the project to seek data to fulfill reporting requirements from other sources as well as restructurings designed, for example, to refine indicators in line with what were later deemed to be more Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) realistic and relevant objectives and targets. Given a relatively high rate of projected transfer (HNL 10,000 per year/US$ 407 and reaching 600,000 households) the project’s economic analysis assessed that the program would have a transformational impact on poverty but also noted that the risk to fiscal sustainability of the Program would be critical. That level of payment was later found to be unsustainable and beyond available national and external sources and the payment structure was altered to a maximum monthly payment of HNL 833 /US$34 per household and a reduction of the coverage target (noting a more defined targeting on the extreme poor). The ICR notes that, at closing, the Program’s financial sustainability remained one of the greatest risks to the development objectives. On the other hand, the World Bank worked closely with IADB, drawing on their combined knowledge, research and experience with other CCT Programs to help inform program design and taking into account, findings from Bank studies (e.g., Institutional Governance Review, and Public Expenditure Review) and the design also drew on results from local analytical work analyzing the performance of existing Social Protection programs and projects in Honduras. Risk assessment, including (as above) the question of sustainability, was robust. Other risks identified included those related to pressures for the Ministry of Presidency to expand program coverage without the necessary prerequisites, and potential administrative duplication given the still developing institutional arrangements. The ICR noted that a risk not adequately identified was the one related to a lack of resources on the part of the health and education sectors for the verification of co-responsibilities. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision There were 16 supervision missions conducted over the project lifecycle. Implementation progress was rated Moderately Satisfactory for 14 of the 16 Implementation Status Reports (ISRs) conducted over the project cycle (rated Satisfactory for the first ISR, and Moderately Unsatisfactory for the eleventh ISR). Supervision missions were carried out jointly with the IADB. This allowed for a cohesive approach to identifying recommendations and strategies and for discussing these at high level meetings with a view to identifying and resolving threats to the achievement of project’s objectives. Required modifications were well addressed by restructurings. Expanded activities were included in AFs. Despite several changes to the Task Team, transitions were reportedly smooth with the majority of Task Team Leaders (TTLs) having previously participated in the Project as team members and several members participating from start to finish. Overall the team was largely attentive and proactive in its approach, working to resolve issues and make appropriate changes with a view to ensuring the delivery of the project objectives, albeit within an expanded timeframe. That said, it took some time to address procurement delays (including inadequate pool of consultants and poorly written terms of reference) that impacted implementation. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The objectives were clear and the theory of change - the core project assumption was that poverty in Honduras was perpetuated by a lack of financial and technical support needed to overcome social services’ demand-side barriers at the institutional and household levels - was reasonably established for most objectives which focused on improving capacity to ensure payment of regular and reliable transfers and on incentivizing behaviors amongst recipients that would address structural poverty. The original project design had four outcome indicators and 13 intermediate results indicators. The original outcome indicators were overly ambitious in that they were designed to capture information on the CCT Program harvested from data that PRAF was supposed to have already been collecting in its MIS on targeting, payments and verification of co-responsibilities in health and education; however, that data was not available and no baselines were available during the design phase. M&E design also envisaged support from, for example, impact evaluations, process evaluations, beneficiary satisfaction surveys, social audits, and quarterly Project Management Reports. The ICR notes that during the design phase the World Bank adjusted the education sector’s standard for good attendance from 75 percent to 80 percent in line with international standards. Maternal and child health indicators for the project were also in line with international standards, above the norm in Honduras at the time. (e.g., World Health Organization standards envisaged monitoring of vaccinations for children aged 0-5 years compared to the standard – 0-3 years – in Honduras. b. M&E Implementation The original four outcome indicators and 13 intermediate results indicators were later revised to nine outcome indicators and 14 intermediate result indicators. There were challenges to M&E implementation early in the project cycle associated with a lack of data that prevented reporting on some indicators until the fourth ISR in 2012. Problems reporting on indicators resulted, in part, from the fact that the Management Information System (MIS) was being developed and only became fully functional in 2015. While reporting on indicators in the education sector was facilitated by the existence of Administrative System for Education Centers, the MIS used to verify health co-responsibilities, the Early Childhood Registry, took longer to develop. However, it was possible to report on the PDO indicators based on information from other sources such as the first impact evaluation. The ICR reports that the project team was proactive in addressing weaknesses in information systems and the project’s results framework by ramping up efforts to collect data through these other systems, as well as through Restructurings (where indicators were added or dropped to reflect the changing political and policy priorities and to address redundancies between the project objective and intermediate result indicators). In addition, project M&E was supported by two IADB-financed impact evaluations, a beneficiary satisfaction survey, an IADB- financed satisfaction survey on pilot payment mechanisms, a social audit (which involved participation of civil society), and an evaluation of operational processes. c. M&E Utilization The ICR (pp. 22-23) reports use of M&E for feedback purposes resulting in program adjustments such as: revision of the benefits structure in pursuit of health and education impacts for all members of Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) households (rather than individuals, such as a single child within targeted households), and to increase fiscal sustainability; a change in focus to enhance results for the extreme poor; the extension of the education grant based on findings from the IE that found higher enrollment and attendance among children in grades 1-6; and increased emphasis on alternative payment mechanism pilots to counter the unpredictability in payment receipts and long delays at payment centers, which were found to be among the major sources of dissatisfaction for program beneficiaries M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as category C and did not trigger environmental safeguards. Given the presence of Indigenous peoples in the project area, the project triggered OP/BP4.10. An Indigenous and Afro-Honduran Peoples Plan (IAPP) was prepared by the borrower to seek broad community support for the project, and to ensure the services rendered by the project fully respect the dignity, human rights, and culture of indigenous and Afro-Honduran peoples. What was initially referred to as the Indigenous Peoples Plan (IPP) was published in 2010 and later updated (2014) to reflect institutional and program changes. The updated version observed the expansion of coverage into remote areas with a high presence of Indigenous People (IP). Over the course of the project, the ISR ratings for social safeguards were largely Satisfactory and Moderately Satisfactory. The ICR notes that the final MS rating reflects difficulties encountered in the provision of ID cards to IP households and delays in incorporation of mitigation activities – e.g., revision and mainstreaming of indigenous peoples in Operational Manual and Guidelines; inclusion of ethnicity variables in the MIS and FSU; approval of a protocol for the participation of indigenous communities; formulization of inter-institutional coordination with indigenous federations; and adaptation of the Grievance Redress Mechanism’s monitoring of complaints for indigenous groups – as set out under the 2014 IAPP. Prior to closing, a unit was established within the Vice Ministry of Social Integration to apply the IAPP to all relevant activities in that Ministry’s Directorates. b. Fiduciary Compliance The government was overall compliant with the interim financial reporting and the external audit requirements; however, during project implementation, risks were identified associated with deficiencies in the development of automated payment mechanisms, and the Management Information System’s processing payrolls for the cash transfers. For example, during the project cycle the auditors qualified the opinion on the Financial Statements for two Credits (5294-HN & 4774-HN) due to beneficiaries not meeting the eligibility criteria. The ICR reports that six-month external concurrent audits played an important oversight and control role by reviewing eligibility, payments and co-responsibilities for beneficiary households. Shortcomings in financial management included the identification of some ineligible expenditures associated with beneficiaries’ non-compliance with co-responsibilities; however, the government agreed to substitute these expenditures for eligible expenditures. Contracting was largely in Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) line with guidelines although there were some delays in some activities due to delays in the procurement approval processes of the government’s budgetary modifications and government-imposed budget ceilings. Other matters affected the procurement focus including the specification of Terms of Reference and Technical Specifications, lack of bidders, and changes in key personnel. That said, the project closed with a strong procurement unit that successfully managed the completion of several large procurement processes. c. Unintended impacts (Positive or Negative) The ICR (p.18) reported that the program contributed to reducing child labor by 2.6 percent, with a larger reduction (4.5 percent) found among children between 12-14 years old, with attendant implications for school attendance - the second impact evaluation indicated that transfers may offset the need for children to work, allowing them instead to attend school. The second impact evaluation also indicated that beneficiary households made homes with respect to ceilings, batteries, stoves, gutters, and latrines – housing poverty reduced by 18.8 percent relative to the control group. In addition to spending more on food, health, and education, beneficiary households spent 27 percent more than the control group on housing. d. Other The first impact evaluation found that female beneficiaries, relative to female non-beneficiaries, were more likely to make decisions with respect to the health, education, and discipline of their children, as well as regarding daily purchases, the purchase of assets for their homes, and home repairs. To a lesser extent, they were also found to exercise greater control over the decision of whether to work outside of the household and the number of children they and their husbands wished to have (ICR, p.18) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) The ICR (pp. 26-27) offered several lessons and recommendations, including the following lessons restated by IEG: Adopting a collaborative approach with other development partners facilitates the pursuit of common objectives. Under the project, IADB was the main partner, and effective collaboration avoided duplication and ensured focus on this area of policy that was a priority for the government of Honduras. A thorough understanding of the trade-offs to be made promotes the balancing process between swift project preparation and ensuring adequate, in-depth analysis that allows for a realistic assessment of implementation capacity and potential. In this instance the rapid preparation of the project ensured ownership by the government; however, the relatively light analysis of existing capacity overlooked challenges in that regard, which ultimately led to bottlenecks and delays in implementation. Securing support to the project, notably in the context of political change, is facilitated by evidence generated by impact evaluation. In this case, the results of two impact evaluations conducted under the project helped maintain focus and support throughout the project cycle, providing valuable feedback regarding progress, challenges and successes. In addition, IEG identified the following lesson: Adopting a strategic approach that ensures that a number of conditions are in place, such as appropriate legal framework and political commitment, is necessary in support of CCT scale up. In this case, some of these conditions were not in place to support the proposed alternative payment mechanism, and the piloting did not translate to the national level despite its success 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well written, and mostly consistent in its internal logic; however, there are certain aspects that could have been improved upon. Whereas the ICR covered the technical aspects of the project, the narrative or story of the project could have been more clearly elaborated. The ICR lacked clarity on a number of points including, for example, the question of disbursements/costs where disbursements as presented in the summary of financing arrangements (p. 2) are greater than the sum of the costs associated with the two components (noting that the borrower’s comments (p.53), although this matter was subsequently clarified in discussion with the TTL. The ICR did not assess the objective to enhance government capacity to manage an ‘eligible emergency’ because no such emergency occurred. However, it should have assessed readiness even though the emergency was never triggered (e.g., through for example, relevant staff training, technological readiness, trial runs etc.) in the same way as improved readiness for natural disasters can be demonstrated before any Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Social Protection (P115592) such disaster occurs. That said, the ICR is candid about the projects relative strengths and weaknesses and provides a thorough assessment across the various headings, drawing as appropriate on various data sources as well as impact evaluations undertaken during the project cycle. In addition, the ICR presents a number of valuable lessons with reference to, for example, the need to achieve balance between rapid project preparation and assessment of implementation capacity and the need for certain key conditions to be in place to support scaling up on a system-wide basis. a. Quality of ICR Rating Substantial Page 19 of 19