Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004675 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA 4493-GD ON A CREDIT IN THE AMOUNT OF SDR 3.2 MILLION (US$5.0 MILLION EQUIVALENT) TO THE GOVERNMENT OF GRENADA FOR THE SAFETY NET ADVANCEMENT PROJECT May 31, 2019 Social Protection & Jobs Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS Currency Unit = Eastern Caribbean Dollar 0.720331US XCD = US$1 1.388250$ = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Axel van Trotsenburg Country Director: Tahseen Sayed Khan Senior Global Practice Director: Michal Rutkowski Practice Manager: Pablo Gottret Task Team Leader(s): Snjezana Plevko ICR Main Contributor: Suzana N. de Campos Abbott ABBREVIATIONS AND ACRONYMS ACCA Association of Chartered Certified Accounts AGO Accountant General’s Office CBR Central Beneficiary Registry CCT Conditional Cash Transfer CDB Caribbean Development Bank DA Designated Account DLI Disbursement Linked Indicator DPL Development Policy Loan DRO District Revenue Office EU European Union FM Financial Management GDP Gross Domestic Product ICT Information and Communication Technology IEC Information and Education Campaign IO Intermediate Outcome IT Information Technology IUFR Interim Unaudited Financial Report MDG Millennium Development Goals MIS Management Information System MOE Ministry of Education MOF Ministry of Finance MOH Ministry of Health MOSDH Ministry of Social Development and Housing MOU Memorandum of Understanding Non-Lending Technical NLTA NF Assistance Necessitous Fund OECS Organization of Eastern Caribbean States OM Operational Manual ORAF Operational Risk Assessment Framework PA Public Assistance PCU Project Coordination Unit PDO Project Development Objective PMT Proxy Means Test PS Permanent Secretary SEED Support for Education, Empowerment and Development Program SIL Specific Investment Loan SNAP Safety Net Advancement Project STA School Transportation Allowance TOR Terms of Reference RBL Results-Based Lending UNICEF United Nations Children Fund UNIFEM United Nations Development Fund for Women UN Women United Nations Entity for Gender Equality and the Empowerment of Women TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs ............................................................................................................11 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................12 C. EFFICIENCY ...........................................................................................................................18 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................20 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ...................................................................................22 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................23 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 26 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................26 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................27 C. BANK PERFORMANCE ...........................................................................................................28 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................29 V. LESSONS AND RECOMMENDATIONS ............................................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 39 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 41 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 42 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 43 ANNEX 6. DISBURSEMENT LINKED INDICATORS ................................................................... 44 ANNEX 7. ORIGINAL RESULT INDICATORS ............................................................................ 47 The World Bank Grenada Safety Net Advancement Project (P123128) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P123128 Grenada Safety Net Advancement Project Country Financing Instrument Grenada Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Government of Grenada Ministry of Social Development and Housing Project Development Objective (PDO) Original PDO The Development Objectives of the Project are to: (i) strengthen the basic architecture of the consolidated Conditional CashTransfer (CCT) Program and the capacity of the Ministry of Social Development to implement it; (ii) improve coverage of poorhouseholds receiving cash transfers; and (iii) improve education outcomes of poor children and health monitoring of vulnerablehouseholds. Revised PDO The revised Development Objectives of the Project are to strengthen the basic architecture of the SEED Program and the capacity of the Ministry of Social Development and Housing (MOSDH) to implement it. Page 1 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 5,000,000 4,631,940 4,258,814 IDA-49930 Total 5,000,000 4,631,940 4,258,814 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 18,200,000 0 Total 0 18,200,000 0 Total Project Cost 5,000,000 22,831,940 4,258,814 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 05-Jul-2011 05-Dec-2011 05-May-2014 31-Dec-2015 30-Sep-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 24-Nov-2015 2.57 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Implementation Schedule 20-Dec-2017 3.65 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial Page 2 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 12-Feb-2012 Satisfactory Satisfactory .74 02 21-Aug-2012 Satisfactory Satisfactory .86 03 10-Feb-2013 Moderately Satisfactory Moderately Satisfactory 1.36 Moderately 04 01-Sep-2013 Moderately Unsatisfactory 1.36 Unsatisfactory Moderately 05 26-Mar-2014 Unsatisfactory 1.95 Unsatisfactory 06 29-Nov-2014 Unsatisfactory Unsatisfactory 1.99 07 15-Jun-2015 Unsatisfactory Unsatisfactory 2.57 08 14-Dec-2015 Unsatisfactory Unsatisfactory 2.57 09 05-May-2016 Moderately Satisfactory Moderately Satisfactory 2.63 10 04-Nov-2016 Moderately Satisfactory Moderately Satisfactory 2.63 Moderately 11 23-Jun-2017 Moderately Satisfactory 3.42 Unsatisfactory 12 19-Jan-2018 Moderately Satisfactory Moderately Satisfactory 3.65 13 17-Oct-2018 Satisfactory Moderately Satisfactory 4.01 SECTORS AND THEMES Sectors Major Sector/Sector (%) Social Protection 100 Social Protection 90 Public Administration - Social Protection 10 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Page 3 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Social Development and Protection 100 Social Protection 100 Social Safety Nets 90 Social protection delivery systems 10 ADM STAFF Role At Approval At ICR Regional Vice President: Pamela Cox Axel van Trotsenburg Country Director: Francoise Clottes Tahseen Sayed Khan Senior Global Practice Director: Keith E. Hansen Michal J. Rutkowski Practice Manager: Helena G. Ribe Pablo Gottret Task Team Leader(s): Ludovic Subran, Karla J. McEvoy Snjezana Plevko Suzana Nagele de Campos ICR Contributing Author: Abbott Page 4 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Country Context 1. Strong growth in the 1980s and early 1990s had resulted in important human development milestones for Grenada, but the country’s vulnerability to shocks threatened further progress. Grenada and other Member States of the Organisation of Eastern Caribbean States (OECS) had made good progress towards the achievement of several 2015 Millennium Development Goals (MDGs), including universal primary education, gender equality, improved maternal health, and reduction of extreme poverty and hunger. Still, in subsequent years, substantial economic volatility endangered poverty reduction efforts and human capital accumulation. High vulnerability to external economic shocks (including changes in terms of trade and declines in tourism flow), and natural disasters led to high variation in growth, from 12.3 percent in 2005 to -7.7 percent in 2009. These, in turn, had an impact on poverty, which rose from 32.1 percent in 1998 to 37.7 percent in 2008. The Government’s ability to effectively address poverty and unemployment was being challenged by low school enrollment rates for children over 15 years of age (55 percent); absenteeism and tardiness of poor children in both primary and secondary education (only 83 percent of children in the first quintile attended school five days a week); high repetition rates at the secondary level and low university pass rates (31 percent); high malnutrition (9 percent measured by low birth rates); and a trend towards a higher death rate caused by Non— Communicable Diseases (NCD). 2. A wave of crises during 2007-2010 further affected poor households, and Grenada’s tight fiscal space and high debt levels prevented the Government from implementing sound countercyclical social policies. Although data was not available, it was assumed that lower levels of tourism revenues and transfers from abroad—including foreign direct investment and remittances, combined with large negative commodity shocks (food and fuel) and natural disasters, had further reduced the purchasing power among the poor, and raised unemployment levels. Grenada’s then existing social protection instruments were not entirely appropriate to mitigate the impact of these shocks, and constrained resources hampered investments in stronger programs and systems before the crises hit. As a result, Grenada’s safety net was unable to respond fully to emergencies or promote effectively productive investments among the most vulnerable. Sectoral Context 3. Grenada’s spending on social assistance was highly variable and dependent mostly on the availability of funding. Year-on-year budget allocations varied with revenues, including external budget support, and responses to shocks (relief programs). As a result, there were limited investments in building systems to deliver better targeted, cost- effective programs, such as direct transfers to households. Intermittent funding also had a low impact on poverty reduction and in promoting positive behavioral change (accumulation of human capital and investments in productive assets) among poor households. Page 5 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 4. A Social Safety Net Assessment (SSNA) carried out in 2009 found that Grenada’s safety net needed improvements in several areas. 1 First, there were a myriad of social assistance programs with overlapping objectives spread across several Ministries, which resulted in inefficient parallel systems. Second, household 2 data and social workers’ assessments pointed to under-coverage among the poor and inclusion of non-poor households. Third, none of the programs had operational manuals to document their policies and procedures, or reliable management information systems to track beneficiaries and payments. Also, some programs were only partially linked to promoting positive behavioral changes such as increased school attendance. Fourth, there was little public information about the programs. And, finally, monitoring and evaluation against clear objectives was scarce. 5. Following the recommendation of the SSNA and with support from the World Bank’s non-lending technical assistance, the Government of Grenada (the GOG), consolidated its three main cash transfer programs: The Public Assistance Program implemented by the Ministry of Social Development (MOSD), the Necessitous Fund, and the School Transportation Allowance Scheme, both managed by the Ministry of Education (MOE). This effort was reinforced by the prior actions established in a US$8 million Economic and Social Development Policy Loan and Credit (P117000), in March 2010. The prior actions under the DPL also supported the GOG’s establishment of an Inter-Sectoral Steering Committee to guide the safety net reform effort. The Steering Committee prepared an Action Plan defining the Government’s priorities in this effort. The first priorities were the consolidation of the three main cash transfer programs, and the development of an enhanced and unified targeting mechanism to transfer cash to poor families. The Action Plan also included: (i) the establishment of a unit in the MOSD to administer the newly consolidated program; (ii) the creation of a Central Beneficiary Registry, and (iii) the formulation of a Social Protection Policy. The Cabinet also established an Inter-Ministerial Technical Committee (TC) comprised of representatives from the MOE, the Ministry of Health (MOH), the MOSD and the Ministry of Finance (MOF) to guide the day-to-day work for the consolidation of the three programs. On April 11, 2011, the GOG issued its Cabinet Conclusion for the establishment of the unified cash transfer program, under the new Support for Education, Empowerment and Development (SEED) conditional cash transfer (CCT) Program. The GOG had committed EC$9.6 million for Public Assistance, EC$1.7 million for the Necessitous Fund, and EC$1.3 million for the School Transportation Allowance—roughly in line with previous years’ allocations—and the MOF prepared a Special Warrant to create a budget line for the program under the MOSD and a Reallocation Warrant to merge the three programs into one budget line under the MOSD. The Reallocation Warrant was approved by Parliament consolidating the three safety net programs under the new SEED Program. Theory of Change (Results Chain) 6. The Project’s original Theory of Change, was based on the assumption that by consolidating cash transfer programs, improving their targeting, and adding conditionalities in health and education tied to their delivery, these would result in : (i) a strengthened Conditional Cash Transfer Program and the MOSD’s capacity to implement it; (ii) improved coverage of cash transfers among poor households, and (iii) improved education outcomes among poor children and health monitoring of vulnerable households. Ultimately, under this original PDO, the Project’s long-term impact was expected to result in a reduction in poverty, both systemic and transient (in response to different crises). 1Social Safety Net Assessment for Grenada, L. Blank, 2008. 2 The most important of the targeted cash transfer programs were categorically targeted to the elderly poor and poor children in schools. Page 6 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Figure 1: Results Chain for Original Project Design Project Development Objectives (PDOs) 7. The original Project Development Objectives (PDOs) were to: (i) strengthen the basic architecture of the consolidated Conditional Cash Transfer (CCT) Program and the capacity of the MOSD to implement it; (ii) improve coverage of poor households receiving cash transfers, and (iii) improve education outcomes of poor children and health monitoring of vulnerable households. Key Expected Outcomes and Outcome Indicators 8. Key Outcome Indicators defined to track progress towards the PDOs were: • Increased percentage of households in the poorest quintile that receive benefit; • Increased percentage of the program transfers reaching the poorest quintile; • Increased school attendance rates of children in beneficiary households, and 3 • Increased percentage of beneficiaries receiving adequate preventive primary health services. The Project also had 25 Intermediate Results Indicators (Annex 1). 3The Ministry has protocols for pregnant and lactating women and children 0-5 years of age. It was to develop protocols for children 6-18 years old, and for the elderly prior to the introduction of co-responsibilities for beneficiaries. Protocols for the elderly would be based on whether they suffered from a chronic condition. Page 7 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Components 9. The Project was financed by an SDR 3.2 million (US$5.0 million equivalent) IDA Credit designed to disburse against actual eligible expenditures and against the achievement of performance milestones, reflected through Disbursement-Linked Indicators, under five results areas. The five results areas, designed to ensure the adoption of an overhauled business model, were: (i) Institutions, Budget and Coverage; (ii) Beneficiary Outreach and Targeting; (iii) Management Information System and Making Payments against Co- responsibilities; (iv) Communication, and (v) Monitoring and Evaluation. The Project comprised two Components, described below. 10. Component 1: Improving Conditional Cash Transfers (US$22.2 million estimated project cost; US$4.5 million IDA financing). This Component aimed to improve the delivery of cash transfers by financing payments of cash benefits under a consolidated cash transfer program characterized by improved coverage, and the adoption of co-responsibilities by beneficiaries to facilitate their human capital development. Education co-responsibilities required 90 percent attendance rates for children of beneficiary households between ages 6-18 who were enrolled in schools 4. For health, co-responsibilities included preventive health visits according to established protocols each one for particular categories of beneficiary household members including pregnant and lactating mothers, the elderly, the disabled and the chronically ill. Credit disbursements were made against the payment of benefits (considered eligible expenditures under the Credit) and the achievement of selected performance milestones under the five results areas. 11. Disbursements under this Component included both a baseline amount of conditional cash transfers and an additional amount of conditional cash transfers made to support and encourage the gradual introduction of system improvements over four years. The baseline amount, not linked to performance milestones, was to disburse US$250,000 per semester for the entire project duration to finance cash transfers made. The additional amount would also finance cash transfers made, but these would be conditioned and limited to the adoption of critical processes in target years for appropriate phasing to advance improvements of the social safety net. These additional amounts would be associated with eight Disbursement-Linked Indicators (DLIs) under the five results areas, and the amounts allocated to each of the DLIs would be disbursed upon verification of the respective achieved milestones on each November during the project implementation period. Verification and attainment of milestones would also be reviewed as part of the Project’s annual audit by an external auditor. The timing of expected compliance with DLIs was indicative but linked to the agreed phasing of the critical roadmap for improvements in the social safety net. Each DLI was assigned a value between US$250,000 and US$500,000. A Table summarizing the milestones, DLIs and proposed Credit disbursements is presented in Annex 8. 12. Component 2: Improving Implementation Capacity of the MOSD (US$1.0 million estimated project cost; US$0.5 million IDA financing). This Component aimed to bring the external knowledge necessary to improve the processes involved in the overhauled cash transfer program. This Component was to finance 4The Missing Generation: A Situational Analysis of Adolescents (10-14) in the Caribbean Community (UNICEF, 2008), found that the proportion of students who complete 3 or more years of education progressively declines until reaching eight grade where only 53 percent of the 10 to 19 year-old cohort finishes the grade. Page 8 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) technical assistance to the MOSD for strengthening its management, coordination, supervision, monitoring and evaluation of the CCT program at the central and local levels. The amount allocated to this Component was small compared to the project cost allocated to cash transfers and focused on supporting the quality design of the intermediate instruments and processes to meet the DLIs under Component 1. The activities to be financed under this Component were organized around the five results areas supported by Component 1, and included consultants’ services for the: (i) preparation of a contingency plan for crisis response through the CCT; (ii) funding for external audits; (iii) design and piloting of a Proxy Means Test (PMT); (iv) development of a Management Information System (MIS); (v) design of a Central Beneficiary Registry (CBR); (vi) design of materials and protocols for the educational workshop and training for beneficiaries; (vii) design of the training plan for MOSD and partner Ministries’ staff, and (viii) design of an impact evaluation and an enhanced process evaluation. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 13. The PDO was revised in a Project Restructuring dated November 11, 2015 to read: “The Project Development Objectives are to strengthen the basic architecture of the SEED Program and the capacity of the Ministry of Social Development and Housing (MOSDH) to implement it”. Given the simplified scope of the project, the Project’s Overall Outcome Rating in Section II below is assessed following a split rating, as required by the ICR Guidelines dated July 1, 2017. Revised PDO Indicators 14. PDO indicators were revised to reflect the new, simplified PDO. The original PDO Indicators were dropped, and the following two PDO Indicators were introduced: • Increased percentage of poor households receiving the SEED cash transfer, and • Increased percentage of the program transfers that reach poor households with children 5. Baseline data for both of these indicators was to be determined once the Proxy Means Test (PMT) was applied. The revised Results Framework also reduced the number of Intermediate Results Indicators to nine (Annex 1). Revised Components 15. The Project’s components were not revised, per se, but there were several adjustments to the DLIs and disbursement arrangements. Specifically, the Project originally included eight DLIs that were to be met each November during the implementation period. These were revised to no longer be dated, i.e., the 5 The Program Document does not specify the age of “children” for purposes of measuring progress towards this indicator. The write- up does, however, mention that “The change would ensure that the focus on children and youth (ages 0-24) who, according to the data presented in the 2008 Country Poverty Assessment, represent 66.4 percent of Grenada’s poor”. Page 9 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) disbursements related to the indicators would occur whenever the DLIs were met. Also, the number of DLIs was reduced to six (Annex 6). Further, the restructuring discontinued the unlinked disbursements of US$250,000, scheduled for each May and November during project implementation period as these disbursements were initially intended to support the financing of SEED benefits, which, by the time of restructuring, were fully financed from the country’s own budget. Other Changes 16. The Project Restructuring also modified the DLIs’ compliance verification arrangements. Based on the request by the Government, instead of the external audit, detailed verification protocols were developed for each DLI as a part of the restructuring, and the verification was done independently by Bank experts that were not part of the Project team. The restructuring also revised the allocation of financing across the Project’s categories and components to increase the share of financing allocated to MOSDH’s capacity building and institutional strengthening component. Finally, the restructuring extended the Credit’s Closing Date by two years, to December 31, 2017. Through a Restructuring approved on December 20, 2017, the Closing Date was further extended by nine months to September 30, 2018. Rationale for Changes and Their Implication on the Original Theory of Change 17. Following effectiveness, Project implementation progressed at a slower pace than anticipated. This was principally been due to: (i) an ambitious Project design that did not anticipate implementation capacity constraints; (ii) challenges in maintaining inter-sectoral coordination needed for the success of the conditional cash transfer program designed under the Project - “Support for Education, Empowerment, and Development” (SEED); (iii) a change in Government following the 2013 elections; and (iv) frequent turnover and limited capacity of MOSDH staff, particularly within the SEED Unit. Therefore, this resulted in the delayed contracting of critical consultancies necessary for the roll out of the SEED Program. 18. Despite the slower pace of Project implementation than the initially envisaged, there was progress in implementing the safety net reform. The Government consolidated three of Grenada’s social protection programs and created the “Support for Education, Empowerment, and Development” (SEED) program. In addition, the GoG began developing a robust targeting mechanism, the Grenadian Living Conditions Indicator (GLCI), to re-certify existing beneficiary households and identify new ones. In July 2013, as required by a Prior Action under the Programmatic Resilience Building Development Policy Credit 1, the Government adopted a policy framework for the strengthening of the design and programming of its social safety nets that included, inter alia, increasing the poverty reduction focus through social safety net policy design and programming, and ensuring efficient, effective, transparent, participative and accountable social protection service delivery (modernizing systems and building capacity). Some of the instruments under the Project were necessary for the operationalization of the framework, and the first Restructuring, informed by the mid-term review (MTR) of the Project in May 2014, was initiated following the MTR, incorporating the necessary Project adjustments to reflect the strong focus on this agenda by the Government, as well as facilitate Project implementation. Page 10 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Figure 2: Results Chain for Revised Project Design II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 19. The Project’s original PDOs were aligned with the Regional Partnership Strategy for the Organization of Eastern Caribbean States (OECS, of which Grenada is a member) for the period 2010-2014 and to the current Regional Partnership Strategy (RPS) for the OECS for the period FY15-19. 6 The most recent RPS notes that for years, the OECS had been trapped in a spiral of low growth, high debt, limited fiscal space and growing social problems, exacerbated by a number of external shocks, including the effects of the global economic and financial crisis and natural disasters. To address these, the RPS prioritized three areas of engagement, all ranked as high priorities by governments and stakeholders, these included: competitiveness, public sector modernization, and resilience. The first two of the original Project’s PDOs (i.e., to (i) strengthen the basic architecture of the consolidated Conditional Cash Transfer (CCT) Program and the capacity of the MOSD to implement it; and (ii) improve coverage of poor households receiving cash transfers) directly addressed Outcome 6 under the resilience area of engagement related to: Improved targeting and reduced fragmentation of social protection systems. The 6World Bank, Regional Partnership Strategy for the Organization of Eastern Caribbean States (OECS) for the Period FY15-19, Report No. 85156-LAC dated October 17, 2014. Page 11 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) key indicators to measure progress towards these objectives were to measure the percentage of households in the poorest quintile receiving the benefit, and percentage of transfers reaching the poorest quintile. Despite the implementation of the new targeting tool, Grenadian Living Conditions Index (GLCI), that was designed based on data from the 2011 Census and 2008 CPA, the MOSDH decided not to undertake an impact evaluation, focusing instead on ensuring the smooth roll-out of the GLCI, and effective implementation of the CCT Program. As a result, the baseline data needed to measure the original indicators was not collected. However, the revised indicators under the first restructuring focused on targeting efficiency and measuring the percentage of poor households, as identified by the PMT, receiving the SEED cash transfer. 20. The original Project’s third PDO (I.e., improve education outcomes of poor children and health monitoring of vulnerable households), and associated Key Indicators to measure progress were not aligned with the RPS’ strategy focus for both the education and health sectors that emphasized the establishment of quality education standards and technical and vocational education and training, and the enhancement of data to support evidence-based health policy to address non-communicable diseases, respectively. The Key Indicators to measure progress towards this third objective were to measure (and hence were to act as an incentive to) increased school attendance rates of children in beneficiary households, and increased percentage of beneficiaries receiving adequate preventive primary health services. Although both of these issues were raised in the Project Appraisal Document’s Background section, the RPS did not highlight either of these as prioritized issues requiring focused support. 21. During implementation, the Project’s relevance improved as the restructuring brought its PDO more in line with the RPS. With the restructuring, the third PDO was eliminated; and the indicator related to measurement of impact on human capital was replaced, while maintaining the focus on children and youth (ages 0-24) who, according to the data presented in the Country Poverty Assessment, represented 66.4 percent of Grenada’s poor. However, analysis carried out for the Mid Term Review (MTR) revealed that most of SEED’s beneficiaries were elderly, accounting for 3,263 out of the 5,249 total SEED beneficiaries as of April, 2014, receiving 78 percent of the total transfers. MOSDH anticipated that over time, using the new targeting tool, SEED beneficiary households would increasingly be comprised of households with children. Thus, Inclusion of this indicator ensured steady monitoring of progress towards this objective. The second component of the PDO was also eliminated but remained as an outcome. As a result, the Project’s PDO and its overall focus after restructuring were fully in line with the RPS, and especially in its focus to strengthening the resilience of vulnerable groups to shocks (an Intermediate Outcome under the Project). As a result, overall relevance of the Project’s PDO is rated Substantial. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 22. The ICR’s assessment considers the achievement of the original and revised PDOs through the entire implementation period following the indicators in the Project’s original and revised Results Framework. However, as of the date of the restructuring, it was not possible to measure progress towards targets of any of the Key Outcome Indicators because the instruments to measure and monitor these were being developed; therefore, progress towards Intermediate Outcome Indicators is used to evaluate achievements during the period of implementation before the Restructuring. Details on project outputs are presented in Annex 1. Following are the original Key Outcome indicators and the revised ones following the project restructuring: Page 12 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Original Key Outcome Indicators: Increased percentage of households in the poorest quintile that receive benefit; Increased percentage of the program transfers reaching the poorest quintile; Increased school attendance rates of children in beneficiary households, and Increased percentage of beneficiaries receiving adequate preventive primary health services. Revised Key Outcome Indicators: Increased percentage of poor households receiving the SEED cash transfer Increased percentage of the program transfers that reach poor households with children 23. The Project was successful in strengthening the basic architecture of the SEED Program and the capacity of the Ministry of Social Development and Housing (MOSDH) to implement it. The Project was able to identify key technical interventions and provide the legal and institutional framework and instruments that the SEED consolidated cash transfer program required to meet the revised PDO objectives and exceed the targets of its revised Key Outcome Indicators. The design of the restructured Project relied on the links between the Intermediate Outcome Indicators to the DLIs and to the Project development objectives. This allowed for continued focus and achievement of the PDOs. The percentage of poor households receiving the SEED cash transfer increased from 68 percent in 2016 to 82 percent by the Project’s completion in September 2018, surpassing the target of 80 percent (Revised Outcome Indicator #1). While in 2015, 64 percent of SEED beneficiaries were elderly, accounting for over seventy percent of the total transfers, the percentage of SEED transfers reaching poor households with children increased from 47 percent in 2016 to 75 percent in 2018 (Revised Outcome Indicator #2). 24. At the time of the Project’s Restructuring, its implementation had made substantial progress overall, and thus its progress towards meeting its original PDOs is rated Substantial. As described in the following paragraphs, the Government had taken important steps both towards strengthening the architecture of the SEED program, and towards strengthening the capacity of MOSD to implement it. As a result, progress towards both of these objectives is considered High at the time of restructuring. A rating of High is justified since without the initial, important progress that had been made, even the restructured project would not have been able to meet its revised objectives. The important policy and other decisions that were adopted and activities carried out to support implementation of the SEED Program were essential to the success of the restructured project. With these basic building blocks in place, the Project was also well on its way towards meeting its second objective of improving the coverage of poor households with cash transfers), and, as described in paragraph 30 below, progress towards this objective is rated Substantial at the time of restructuring. There had been no progress towards improving the educational outcomes of poor children and improving health monitoring of vulnerable households as per the original project’s third objective. Hence, the rating of progress towards this objective is considered Negligible at the time of restructuring. Page 13 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Objective 1a 7: Strengthen the Basic Architecture of the SEED (Consolidated Conditional Cash Transfer (CCT) Program) 8 25. The Project was successful in strengthening the basic architecture of the SEED Program 9. Progress towards this objective was Satisfactory before the Project Restructuring as several instruments for the SEED Program’s architecture were being developed, tested and approved at the time, and remained as Satisfactory after the Restructuring when these instruments were finalized and began to be utilized. As previously noted, the Government created a separate budget line in the Central Government budget dedicated to the SEED Program, and in 2011 consolidated three previously existing cash transfer programs (OII 10 #1), 11 including the Necessitous Program 12, School Transportation Allowance and Public Assistance Programs to establish the SEED Program under the Ministry of Social Development and Housing (MOSDH). 13 During 2014, the Government began developing a robust targeting mechanism, the Grenadian Living Conditions Indicator (GLCI), to re-certify existing beneficiary households and identify new ones. The use of the GLCI was approved by the Cabinet in late-2015 (OII #6 and RII #6), and the Government began, after a pilot in one parish (OII #7), assessing all existing and new households using the new targeting instrument. By the time of the Project restructuring in 2015, Phase I of the Beneficiary Management Information System (BMIS), a modernized MIS generating one beneficiary list and using one payment method (OII #3), had been deployed to MOSDH for testing and soon after became operational. The GLCI began being applied to all existing SEED households by the end of 2015. By project completion, 99 percent of beneficiary households in the registry were assessed using the new targeting tool (RII #3). By project completion, the SEED Program architecture relied on a populated, tested and fully functional BMIS which contained a comprehensive reporting module that enabled the detailed monitoring and evaluation of the SEED Program, and created a tool for analysis of the poor and vulnerable population included in the social registry and, more importantly, for further development of the social protection system in Grenada. 26. Following the improvements in the targeting mechanism, the Government prepared a Phase-Out Plan to remove ineligible households. A phased approach was considered essential given the political and financial implications of applying the improved targeting instrument. The Phase-Out Plan was approved by the Government in August 2015, this included an action plan for managing the phase out process for ineligible households and setting a road map for complementary enrollment of new households that had not previously been receiving SEED transfers but were now eligible based on new targeting criteria. As a Prior Action for the third operation under the World Bank’s three-phase Programmatic Resilience-Building Development Policy 7 The original PDO had three objectives, of which two of them were composite objectives: objective (i) strengthen the basic architecture of the consolidated CCT program and the capacity to the MOSD to implement it, and (iii) improve education outcomes of poor children and health monitoring of vulnerable households. For purposes of this section, evaluation of both of these have been broken down as objectives 1a and 1b, and objectives 3a and 3b. 8 Indicators in Italics were deleted in the November 2015 project restructuring; indicators in Bold were introduced in the project restructuring; indicators in normal font were in place throughout implementation. 9 This objective was not substantively modified in the Project’s restructuring. The only change was that the term “Consolidated Cash Transfer (CCT) Program” was changed to “SEED Program”. This is because, by the time of the restructuring, the SEED Program had already been established. 10 This section refers to several indicators in the Projects’ original and revised Results Framework. The following acronyms are use, followed the number of the indicator in the Results Framework: Original Outcome Indicator (OOI), Original Intermediate Indicator (OII), Revised Outcome Indicator (ROI), and Revised Intermediate Indicators (RII). 11 References to Intermediate Indicators are defined as Original Intermediate Indicators (OII) and Restructured Intermediate Indictors (RII), both followed by the number of the indicator in the original and restructured Results Framework. 12 Education based cash transfer program. 13 The Public Assistance program, managed by MOSDH, was one of the largest cash transfer programs targeted to the elderly, disabled and indigent poor; the Necessitous Fund and the School Transportation Allowance, both managed by the Ministry of Education, benefitted students at the primary and secondary school levels. Page 14 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Program (PRBDP) 14, the Government institutionalized the implementation of the Phase-Out Plan for the SEED program, by notifying all ineligible beneficiary households of their status and completing the transitioning out of the program for the first two of three groups (at least 70 percent) of beneficiary households identified as ineligible and non-vulnerable (para. 56). All the households falling in the ‘Non-vulnerable’ category, totaling 434, were notified and phased-out; only 34 appealed the decision to the Program’s Appeals Committee, an eleven-member committee established by Cabinet Resolution in 2014 to address complaints by existing and potential SEED beneficiaries (RII #10). In addition, the Government started using the new targeting tool exclusively for new enrollments in SEED since 2016 and plans to apply the tool to all its social programs. The Government has also recently finalized the installation of the new BMIS, which is a critical element in the development of a central beneficiary registry. The removal of ineligible beneficiaries allowed the Government to increase the coverage of the poorest households in the SEED Program 15, a complementary aspect to the objective of strengthening social safety nets. Thus, Social safety net coverage was also improved. Objective 1b: Strengthen the Capacity of the MOSD to Implement it (the consolidated CCT Program) 27. The Project was also successful in strengthening the capacity of the MOSDH to implement the SEED Program. Progress towards this objective was High before the Project restructuring as several decisions and instruments to strengthen the MOSDH’s capacity were being developed, tested and approved at the time, and remained High afterwards. In October 2011, the Government established and staffed the SEED Unit within the MOSDH to manage the roll-out and implementation of the SEED Program (OII #2). The Unit was staffed with a Manager, a Safety Net Officer, a Data Entry Officer and six SEED Officers. In November 2012, a Monitoring and Evaluation Officer was incorporated, and in 2013 a Field Supervisor, 3 Clerical Officers and 17 SEED Field Officers joined the Unit. The main shortfall was the absence of a Management Information System (MIS) Officer; a qualified technical specialist from the Ministry of Education provided support to the Unit until a MIS Officer was contracted in 2017. The BMIS described above greatly improved the Ministry’s capacity to implement the Program, as did the training of MOSDH, Ministry of Education, and Ministry of Health staff on administration and implementation of social safety nets (OII #10, OII#11) 16. Five members of the MOSDH attended a World Bank-organized Social Safety Net Core Training Course (OII #11). Several other actions cemented the MOSDH’s ability to implement the Program, including: (i) formalization of a Memorandum of Understanding between MOSDH and the Ministries of Education, Health and Finance to guide cooperation for the Program’s implementation; (ii) completion of an Operational Manual outlining the Program’s implementation arrangements and operational rules; (iii) Cabinet 14 World Bank, Programmatic Resilience-Building Development Policy Program, Phase 1, Phase 2 and Phase 3, Report Nos. 87023, 92951, and 106238, dated June 11, 2014, September 29, 2015, and November 15, 2016, respectively. 15 This had been the PDO indicator under the original Project, but it was deleted during the restructuring. 16 A three-day training on Social Protection for over 25 Government staff from MOSD, MOE, and MOH 25 was organized in November 2013. The training included modules on CCTs, outreach and communications for SP; targeting; MIS systems; enrollment; appeals, case management, payments, institutional coordination and the political economy of safety nets. The training was delivered by SP experts and also included two sessions delivered by a senior Government official from Belize’s Ministry of Human Development who provided Belize’s experience on their roll-out of the BOOST CCT, its targeting and management information systems. Through the NLTA, the Bank was also able to facilitate a regional workshop on management information systems in Grenada in October 2011 with participants from nine OECS countries; donor representatives from the UN, UN Women, and the Bank; members of the OECS Secretariat; and technical experts from Jamaica’s PATH CCT program, came together to learn about international experience and best practice in establishing and maintaining information systems and Central Beneficiary Registries. As the event was hosted in Grenada, it helped provide in-depth training to the recently established SEED Unit and other MOSD staff at the time. The NLTA also financed a regional workshop on targeting during preparation which included participants from MOSD. A South South TF also financed participation in a regional M&E workshop in the DR; south-south exchange between Jamaica and Grenada on MIS systems and between Mexico and Grenada on PMTs in 2012. These supported capacity building aimed at readying the Ministry to implement the project. Page 15 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) approval of a three-year budget plan for the SEED Program, issued in late 2014; and (iv) training of eleven members of the Appeals Committee to carry out their duties in the Program’s normal operations, and especially during implementation of the Phase Out Plan. 28. The SEED Unit in MOSDH also carried out a wide-reaching dissemination campaign directed at the Program’s beneficiaries (RII #8). Supported by a Communications Officer, the Unit designed a communications strategy and a related dissemination campaign designed to familiarize the public with the SEED. Also, beneficiaries’ information meetings were included as part of the enrollment process to ensure that all the beneficiaries were well informed about the key aspects of the Program, including: (i) its goals and objectives; (ii) the application process, including how and where to apply for benefits; (iii) the co-responsibilities to be met by beneficiary families in both education and health; (iv) the objective of targeting; (v) the termination of benefits; (vi) the payment process; (vii) where and when to collect benefits; (viii) the rights of beneficiaries, and (ix) the Program’s complaints and appeals mechanism. The beneficiaries’ information meetings were organized in each of Grenada’s parishes through informal settings, such as parent teacher meetings, at health centers and at District revenue offices on payment days, and Beneficiary Handbooks were provided to attendees. In those cases where beneficiaries were not able to attend general meetings, SEED Officers arranged visits to the beneficiaries’ households to present information to them. A total of 80 percent of household beneficiaries participated in dissemination meetings and received related information and Handbooks (OII #9). The SEED Unit also conducted a communications campaign to the general public with promotional materials, to familiarize the entire population with the Program’s objectives. 29. The Process Evaluation (RII #9) for post-disaster social protection was carried out with funding under a World Bank-executed Rapid Social Response Grant. The evaluation focused on social protection and labor programs’, including the SEED Program’s, readiness to support disaster preparedness and response. The Evaluation reviewed written policies and procedures, and compared these to the existing implementation ones, with a view to identifying gaps in the provision of social protection suggesting improvements in the program’s processes and procedures. With respect to the SEED Program, the Process Evaluation recommended that a new Operations Manual for social protection and labor integrating support for disaster risk management should: (i) train SEED Officers to perform damage and needs assessments for households; generate lists of SEED beneficiaries living in dwellings and geographic areas vulnerable to hazards; (iii) generate lists of SEED beneficiary households with members highly vulnerable to hazards, such as the elderly, the disabled and the chronically ill; (iv) develop protocols for proactive and regular information sharing with key disaster risk management stakeholders; (v) provide regular basic disaster preparedness information to beneficiaries during payment deliveries, such as the location of shelters and protocols for evacuation; (vi) establish rapid payment mechanisms for potential victims of disaster; (vii) assist in the implementation of damage and needs assessments for households through SEED Officers; (viii) establish mechanisms to carry out rapid payments of disaster funds to affected populations; and (ix) develop protocols for referrals to other services, such as psychosocial support. Based on the above evidence, the efficacy for both the original and revised Objectives 1a and 1b are both rated as High. Objective 2: Improve Coverage of Poor Households Receiving Cash Transfers The Outcome indicators below supported the original Objective 2. Page 16 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Original Outcome Indicators: Increased percentage of households in the poorest quintile that receive benefit; Increased percentage of the program transfers reaching the poorest quintile; Revised Outcome Indicators 1718: Increased percentage of poor households receiving the SEED cash transfer Increased percentage of the program transfers that reach poor households with children 30. This objective was eliminated from the PDO as part of the Project Restructuring. However, even though the objective was eliminated in the restructured Project’s Development Objective statement, the restructured Project achieved this objective as measured by the Key Outcome Indicator defined as, ”the percentage of poor households receiving the SEED cash transfers” which increased from 68 percent in 2016 to 82 percent by the Project’s completion in September 2018, surpassing the target of 80 percent (Revised Outcome Indicator #1). While in 2015, 64 percent of SEED beneficiaries were elderly (accounting for over seventy percent of the total transfers), through the Project’s focus on poor households with children, the percentage of SEED transfers reaching this group increased from 47 percent in 2016 to 75 percent in 2018 (Revised Outcome Indicator #2). However, while the Project provided technical support for the institutional aspects of these results, they are not entirely attributable to the Project as they were supported also by prior actions under the PRBDP, as described also in paragraph 25 above. Based on the above, the efficacy of this original objective is rated as Substantial. Objective 3a: Improve Education Outcomes of Poor Children Key Indicator that supported original Objective 3a: Increased school attendance rates of children in beneficiary households 31. This objective was eliminated from the PDO in the Project Restructuring, and progress before then was negligible. The MOSDH did formalize a Memorandum of Understanding with the Ministry of Education to guide cooperation for the Program’s implementation. However, baseline data was not available at appraisal, and no progress had been reported nor had monitoring of conditionalities taken place. It was not possible to measure progress towards this objective before the restructuring for the reasons explained in paragraph 20 above (i.e., the instruments to achieve these expected results were only being developed at the time of the restructuring). Nevertheless, the Borrower indicates in its completion report that disaggregated data collected by the SEED Unit point to increased access to higher level (tertiary) education by children of SEED beneficiaries, as well as exceptional academic performance by those students. The SEED program increased the number of poor children 17 Although the restructured Project eliminated this objective, the revised Key Indicators included here were maintained as Key Outcome Indicators for the Project as a whole. They are referenced here for purposes of illustration, to show that even though the objective was eliminated from the PDO statement, the Project was successful in achieving this objective. 18 The technical definition of these indicators changed following restructuring: rather than depending on a household survey to measure program coverage in the poorest quintile, the Proxy Means Test that had been developed with support under the Project was used for identifying the poor. Page 17 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) receiving financial support, which may have helped retain some students in secondary school (primary school enrollment is nearly universal), but available data are insufficient to assess the impact of SEED on school enrollment. Objective 3b: Improve Health Monitoring of Vulnerable Households Key Indicator that supported original Objective 3b: Increased percentage of beneficiaries receiving adequate preventive primary health services. 32. This objective was eliminated from the PDO in the Project Restructuring and progress before then was negligible. The MOSDH did formalize a Memorandum of Understanding with the Ministry of Health to guide cooperation for the Program’s implementation. However, baseline data was not available at appraisal, and no mechanisms to monitor conditionalities had been put in place. The instruments to achieve these expected results were only being developed at the time of the restructuring. Although it is possible that SEED transfers may have helped increase the use of health services, there is no data to substantiate this. Justification of Overall Efficacy Rating 33. Overall efficacy is rated Substantial before the Project Restructuring given the progress achieved in the original IOIs (based on sub-ratings of High for Objective 1, Substantial for Objective 2, and Negligible for Objective 3), and High afterwards, based on the full, and in some cases exceeded, achievement of Key PDOs and IOIs. C. EFFICIENCY 34. The Project’s efficiency is based on the: (i) expected economic impact of the SEED Program given its poverty impact due to improved targeting and more efficient program delivery, and (ii) efficiency with which the Project was implemented. It is not possible to calculate an ex-post economic analysis comparable to the ex-ante economic analysis prepared at appraisal. The economic analysis in the PAD estimated the Project’s benefits based on the: (i) income effect for families receiving cash transfers; (ii) increase in future earnings associated with higher educational attainment; and (iii) more efficient program delivery (though this was not quantified). 35. The new GLCI instrument helped improve the targeting of the SEED beneficiaries and focus the Program on the most vulnerable households with children. The economic analysis in the PAD estimated that, with a 20 percent inclusion error, and under different assumptions of transfer modalities, the Project interventions would lead to a reduction in poverty of at least 5.9 percentage points for a coverage of 8,000 households, with at least 6.6 percentage points with coverage expanded to 10,000 households. While the full economic analysis was not prepared following the closing of the Project, the Project undoubtedly had a positive impact on poverty reduction through improvements in the program’s targeting efficiency, and expansion of the program’s focus to households with children. While it is not possible to measure the impact of increased Page 18 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) earnings associated with higher educational achievement as the activities and indicators related to the original objective 3 were eliminated from the Project at restructuring, again the Project’s focus on households with children undoubtedly will have an impact on this measure. Extensive research shows that well designed CCT programs have significant impacts on poverty and human development, with significant effects on schooling and health of beneficiary families. Further, as reported in the PAD, expenditure patterns for these show that in addition to spending on basic necessities, households also spent on health and education. 36. The consolidation of three transfer programs under a single new program, with a unified beneficiary roster contributed to its efficiency. The three legacy programs were implemented by three separate units in two ministries (MOSDH and MOE). While there are no data available to document the costs of designing, instrumenting and implementing these programs, their consolidation into the SEED Program and the modernization of the implementation framework for the SEED reduced the overall costs of administration and enabled the provision of a better targeted program. 37. The Project was implemented efficiently. Despite initial delays due to an ambitious Project design that did not anticipate implementation capacity constraints, challenges in maintaining inter-sectoral coordination needed for the success of the SEED, and frequent turnover and limited capacity of MOSDH staff, project implementation accelerated following the change in administration in 2013, together with the priority it assigned to protecting the vulnerable from the effects of its fiscal stabilization program. By the mid-term review in May 2014, the foundations for the SEED Program had been launched (staffing of the Unit, political decisions to merge programs, agreement on PMT formula, etc.), and with the subsequent restructuring Project’s implementation was back on track. The Project was completed with the Credit’s cumulative Closing Date extension of two years and nine months, substantially meeting all of its targets. Further, associated trust funds were mobilized for several project activities (mapping of social programs, the process analysis, and the review of cash transfer payment system) thereby leveraging project funding thus improving the Credit’s efficiency. Finally, despite institutional capacity constraints, the Project’s results-based disbursement mechanism promoted efficiency by facilitating the disbursement process and providing an incentive for the Government to keep implementation on track and to focus on achieving the PDO. Table 1 below provides a summary of estimated and actual project costs, by both category of expenditure and project component, for both the IDA Credit and the Government’s counterpart contribution. While the information on actual Government contributions to the Project has not been received from the Project Implementation Unit by the completion of this document, the Government has covered all the costs related to the public outreach on the SEED. Also, from the very beginning of Project implementation the Government has been paying the SEED transfers, ensuring the sustainability of the program beyond the life od the Project. During initial implementation, the Project was developing the instruments to improve the Program, and implementation was slow at the beginning. Thereafter, the efficiency with which the Project was implemented improved significantly, and the SEED Program became more efficient and its transfers better targeted once the new framework and instruments were put in place. Page 19 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Table 1. Estimated and Actual Project Costs, by Category of Expenditure and Project Component (US$, millions) 19 IDA Credit Counterpart Total Project Costs Estimated Actual Estimated Actual Estimated Actual Consultants’ services 0.3 0.3 0.4 0.7 Equipment and materials 0.15 0.21 0.1 0.25 Training 0.05 0.12 0 0.17 Transfers 4.5 3.6 17.7 22.2 Component 1 4.5 3.6 17.7 22.2 Component 2 0.5 0.63 0.5 1.0 Total Project Cost 5.0 4.23 18.2 23.2 Assessment of Efficiency and Rating 38. Efficiency is rated Modest, mostly due to slow implementation before restructuring. D. JUSTIFICATION OF OVERALL OUTCOME RATING 39. The combined Outcome rating is determined by weighting the percentage of the Credit that was disbursed before and after the Project Restructuring. As a restructured project, the Project’s Outcome Rating has been guided by the analysis in Table 2 below. Table 2: Project Outcome Rating Weighted by Disbursements Relevance of Efficacy by sub- Efficiency Outcome Numeric % Weighted Final PDOs objective Ratings Value Disbursed Value Outcome Before (1a) High Restructuring (1b) High (2) Substantial (3a) Negligible 4= Substantial (3b) Negligible Modest MS=4 2,568,617.71 64% 2.56 Moderately Overall: Satisfactory Substantial After (1a) High Restructuring (1b) High MS=4 1, 444,166.35 36% 1.44 Overall: High 19The information has been requested from the Project Implementation Unit, but it was not received by the time of completion of the document. Page 20 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 40. The Project’s overall outcome is rated Moderately Satisfactory. Outcome under the original objectives is rated Moderated Satisfactory (a value of 4), based on Substantial relevance of original objectives, Substantial efficacy, and Modest efficiency. Outcome under the revised objectives is rated Moderately Satisfactory as well (a value of 4), based on Substantial relevance of objectives, High efficacy in the achievement of project objectives, and Modest efficiency. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 41. Given that the overall focus on the SEED program is on the beneficiary household, the Project indicators did not track specific data on the number of female beneficiaries or female headed households. However, the BMIS allows analysis of households across different dimensions including gender, and the available data points out to an equal participation of female beneficiaries in the SEED. Institutional Strengthening 42. The Project had a strong impact on the institutional establishment, development and strengthening of its SEED Program, as described in detail in Section II B. Mobilizing Private Sector Financing 43. N/A Poverty Reduction and Shared Prosperity 44. The SEED programme succeeded in improving the coverage among the poorest and most vulnerable citizens in Grenada and providing them with a cash transfer tied to co-responsibilities that promote human capital development. At the time of the introduction of the new targeting tool, the GLCI, there were a total of 2,853 households in the SEED, 33 percent of which were classified as Least Deprived, Vulnerable and Non- Vulnerable.11 45. An assessment of living conditions in the overall population was done using the by applying the GLCI instrument (and by extension determining potential eligibility for SEED) to the effective sample of households (29,065 households) from the 2011 Census. The results revealed that there were 4,770 households in Grenada which fell within the category of Most-Deprived, while there are 4,900 households that were Mid-Deprived. These findings confirmed the need to improve targeting among the existing beneficiaries of the SEED and expand the program to ensure the inclusion of additional households belonging to these two categories. 46. By the end of the Project the overall number of SEED beneficiary households increased to 5,849, out of which 4,665 (80 percent) are classified as Most- and Mid-Deprived, compared with 67 percent before the Project. Page 21 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Other Unintended Outcomes and Impacts 47. N/A III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 48. The Project responded to the Government’s request to establish a social safety net focused on mitigating the effects of the recent crises on poor households, including economic shocks and natural disasters. This was especially required in view of the Government’s tight fiscal space and high debt levels that prevented it from implementing sound countercyclical social policies. 49. The Project’s design was grounded in and supported by a strong program of analytical work. A Social Safety Net Assessment carried out in 2009 found that Grenada’s safety net required improvements in several areas and recommended the consolidation of the three then existing cash transfer programs into a consolidated CCT program. On the basis of this work, the Government began the preparatory work to facilitate the consolidation of these three programs with the support of a World Bank OECS Social Safety Net Non-Lending Technical Assistance. 50. Preparation relied on the Government’s full commitment. As part of the preparatory work mentioned above, the Government formed an Inter-Sectoral Steering Committee to guide the social safety net reform effort, prepared an Action Plan outlining its priorities, including consolidation and development of an enhanced and unified targeting mechanism, and formed an Inter-Ministerial Technical Committee comprising representatives from the Ministries of Education, Health, Social Development and Finance. By April 2011, the Grenada’s Cabinet issued a Conclusion to allow for the formal establishment of the Support for Education, Empowerment, and Development (SEED) Program, for which it had requested the World Bank’s technical and financial support. 51. The Project’s social protection objectives and design were aligned with the World Bank Group’s Regional Partnership Strategy (RPS) for the period 2010-2014, especially its second pillar related to “Protecting and Improving Human Capital”. The Project was directly responsive to the RPS’ Objective 5, on “Rationalized social safety net systems”. Nevertheless, the third sub-objective of the Project’s PDO, “improve education outcomes of poor children and health monitoring of vulnerable households” was not identified as one of the RPS’ priorities on education and health, which included: (i) an increase in the proportion of qualified teachers; (ii) a better-skilled post-secondary labor force, and (iii) improved data and understanding about the region’s chronic noncommunicable diseases. More importantly, however, it is not clear that either the co- responsibilities responded to an identified priority, nor that the activities under the Project would have allowed it to meet the objective to achieve improved educational outcomes and health monitoring. In fact, improved educational outcomes and health monitoring would have required much more than simply recording attendance at school or preventative health visits. Page 22 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 52. The Project was ambitious, especially when education and health co-responsibilities were introduced under Component 1. Not only did the Project’s education and health objective not respond to the prevailing RPS, but the introduction of co-responsibilities brought unnecessary complexity into the Project’s design. It required coordination with other ministries and agencies that would monitor compliance with co- responsibilities thereby adding to institutional complexities and these would have also required sound information systems to monitor compliance, that were only being developed under the Project. The process was expected to be simpler in education since the School Transport Allowance (which was consolidated to form SEED) was already conditional as it relied on School Attendance Officers to monitor school attendance at schools in Grenada (truancy officers), and the SEED aimed at building on these existing arrangements for school attendance monitoring and improving coordination between the MOSD and MOE. The MOE and school attendance officers were very involved in establishing the arrangements outlined in the SEED manual. The new arrangements were required for the health co-responsibilities, where MOH had established protocols for health visits but there were no processes in place for monitoring these visits or sharing the data with another ministry. There was an issue with sequencing of activities, consolidating the existing programs, developing targeting mechanisms, establishing sound information management systems, etc., that would not have allowed for proper monitoring of co-responsibilities during this early development of the SEED Program. All of these issues became more relevant in an environment with limited institutional capacity, and where the very foundations of the institutions that would be responsible were only being established under the Project. 53. The Project’s design took into account lessons learned from other safety net improvement operations and identified several risks and mitigation measures that were built into its design. Overall, the Project’s risk analysis following the Operational Risk Assessment Framework (ORAF) was considered Medium, driven by Impact. Although the ORAF addressed Implementing Agency, Project Design and Project Delivery Quality risks (and rated all M-1 as well), it did not specifically address the risk related to the monitoring of co- responsibilities. B. KEY FACTORS DURING IMPLEMENTATION 54. The SDR3.2 million (US$5.0 million equivalent) IDA Credit for the Project was approved by the World Bank’s Board of Directors on July 5, 2011, was signed on September 20, 2011, and became effective on December 5, 2011. Several factors during implementation affected the Project’s performance, as described below. 55. The Project was implemented in a very difficult macroeconomic environment. Between 2009 and 2012 Grenada’s suffered one of the worst recessions in decades. Tourism revenues and foreign direct investment (FDI) flows had collapsed in the aftermath of the global financial crisis, and GDP contracted by a cumulative 8.2 percent during the period (para. 55). The current account deficit remained large, and with the drying up of FDI, financing relied more heavily on debt- related flows. The economy began to recover in 2013, with GDP increasing by 1.5 percent, but the fiscal situation that had deteriorated sharply in 2012 continued to worsen. The overall public sector deficit increased to 7.1 percent of GDP in 2013. With negative external transfers and shortfalls in revenues, the Government issued significant amounts of short- term government securities to meet its financing needs. Grenada’s public debt had increased sharply in 2012 and continued to worsen in 2013, culminating in the announcement of a public debt restructuring in March 2013 by a newly elected administration. To address these challenges, the Government announced its New Economy Plan (NEP) Page 23 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) outlined in its 2013 budget presentation. The NEP, supported by the IMF under a 36-month Extended Credit Facility, by the World Bank’s three-phase Programmatic Resilience-Building Development Policy Program (PRBDP)12, and by a programmatic lending program by the Caribbean Development Bank (CDB) closely aligned with the World Bank’s operation--was a long- term development strategy based on strategies for economic growth, fiscal sustainability, social development and debt management. 56. While at the time of appraisal in 2011 the Government had considered the SEED Program a central part of its earlier economic recovery agenda, the new post-2013 economic reality greatly increased its commitment to strengthening the flagship social safety net program. At the time of appraisal, poverty estimates (from 2008, the latest available at that time) indicated that poverty and indigence levels had increased, and with declines in GDP since 2008 it would have been unlikely that the incidence of poverty would have improved. The SEED Program was established to improve the efficiency of the then-existing social protection system and to protect the human capital of the most vulnerable. Recognizing that its short-term fiscal consolidation program supported by the IMF would likely impact the poorest negatively, the Government made social development an integral part of its NEP, and among other social strategies, addressed greater social inclusion through a further strengthening of the SEED Program. Nevertheless, the Project was affected by the lack of inter-sectoral coordination needed required for the SEED Program, especially with respect to its education and health conditionalities related to its third objective. 57. With this strengthened Government commitment, the SEED Program became an integral strategy in its NEP, and of the international financial institutions’ support for the Government’s Plan. In response, the World Bank- supported a Programmatic Resilience-Building Development Policy Program and complementary assistance provided by the CDB included a policy pillar “Supporting Fiscal Consolidation through reducing fiscal, financial and social vulnerability”, and a Program Objective for “Strengthening Social Safety Nets”. Prior Actions under the three phases of the PRBDP, supported by complementary hands-on technical assistance, financing and implementation support under the Project provided a powerful combination of instruments to help achieve the Project’s PDO. The Prior Actions included: (i) for PRBDP1 that the Government had adopted a policy framework for the strengthening of the design and programming of its social safety nets, that included: (a) strengthening the systematic production and use of evidence on poverty and vulnerability for better policy design and programming (research, monitoring and evaluation); (b) increasing the poverty reduction focus through social safety net policy design and programming; (c) carrying out social and legal reforms to improve access to services for the poor and vulnerable; (d) helping families achieve economic independence, and (e) ensuring an efficient, effective, transparent, participative and accountable social protection service delivery (modernizing systems and building capacity); (ii) for PRBDP2, that the Government had established and implemented the eligibility criteria for the application of the targeting instrument and started its implementation, and that the Government had prepared a Phase-Out Plan to remove ineligible households, and (iii) that the Government institutionalized the implementation of the Phase-Out Plan for the SEED program, notifying all ineligible beneficiary households of their status and completing the transitioning out of the program for the first two of three groups (at least 70 percent) of beneficiary households identified as ineligible and non- vulnerable. Prior Actions under the three phases of the PRBDP, supported by complementary hands-on technical assistance, financing and implementation support under the Project provided a powerful combination of instruments to help achieve the Project’s PDO. 58. A change in Government in February 2013, and in project management in April 2013 exacerbated delays in project implementation. The Project had already suffered delays in hiring consultancies that were Page 24 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) mostly the result of limited institutional and technical capacity. While the institutional arrangements for implementation were sound, the sequencing and timing of project activities that not only supported the establishment of the SEED unit, but also the financing of many of the activities required to strengthen SEEDs implementation capacity were not adequate. As a result, in the early stages of implementation there were several delays in contracting and handling routine administrative processes required. Although the initial project management team had been trained on the new, innovative disbursement mechanism, it was often late on complying with the requirements that would result in disbursement of funds. Also, frequent changes in project management and routine turnover in the SEED Unit proved a challenge for implementation as the new team needed time to get familiar with the innovated results-based disbursement mechanism. As a result, project ratings during this early implementation were mostly less than satisfactory. 59. The Mid-Term Review in May 2014 was instrumental in addressing the issues that were affecting the Project, and identifying the changes needed to align it better with the Government’s implementation experience and the country’s institutional capacity. The Government requested a simplification of the Project and its objectives. The review identified changes to the PDO, the Results Framework, disbursement indicators, and others to scale down the Project, and agreed upon an Action Plan for carrying out the analyses and other information required for the Project’s restructuring. Still, the Project remained in Unsatisfactory status for over one year during the time it took for the restructuring to be approved, little, if any, progress was made in implementation during this period, and the Implementation Status and Results Reports (ISRs) show that no funds were disbursed. 60. Implementation improved significantly with the restructuring approved, and the Project’s alignment with the priorities of the Government’s programs and their implementation capacity. The Project accomplished in two years what it had not been able to achieve in its first four years of implementation. By early 2017, the ISR reported continued implementation progress, including the completion of the SEED Program’s legal and operational framework, its Operations Manual, and the signing of Memorandum of Understanding with the Ministers of Education, Health and Finance. The registry of SEED beneficiary households was developed and populated with information on all beneficiary households, all of which were reassessed using the new targeting tool. Elderly beneficiary households were transferred to a new SEED Elderly program, a grievance system was in place and the Appeals Committee was established and made operational. The MOSDH had conducted a beneficiary education campaign and over 80 percent of the beneficiary households had signed the contracts for participation in the program. Further, the use of electronic payments through credit unions and commercial banks had been piloted to a small share of beneficiaries. The main activity that continued to suffer delays was the finalization of the Beneficiary Management Information System that was required to support targeting, eventually co-responsibilities’ monitoring effectively, and support the SEED Program’s further implementation. The BMIS was completed in August 2018, tested in September 2018, and deployed immediately thereafter. Nevertheless, its development was long and faced numerous obstacles, including a period of complete halt in the process, a change of the software development consultant mid-way, and a long search for a developer that could finalize the task. With the remaining activity satisfactorily completed, all DLIs met, the Project’s Development Objectives rating in the ISR was upgraded to Satisfactory. 61. The Rapid Social Response Trust Fund that aimed to establish the building blocks necessary for disaster responsive social protection and labor systems in Dominica and Grenada supported some of the technical assistance activities that were originally included in the Project. 20 First, it financed the mapping and 20 Rapid Social Response Non-Lending Advisory Services and Analytics: Building Block for Disaster-Responsive Social Protection Systems Page 25 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) inventory of social protection programs and services that aimed at supporting individuals, households and communities in the management of risks and volatility and protecting them from poverty and destitution. Also, it financed the development of linkages to disaster risk management objectives. Second, it financed an assessment of payment mechanisms for cash transfer payments. Finally, it financed the process evaluation of post-disaster relevant social protection programs, including the SEED Program, that provided recommendations to improve these programs’ roles in responding to disasters. This final activity was reflected in one of the Project’s Intermediate Outcome Indicators. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 62. The Project’s original Results Framework was complex, mirroring the Project’s original objectives (para. 7). It contained four Key Outcome Indicators (KOIs) and 25 Intermediate Outcome Indicators (IOIs). Ten of the IOIs related to the objectives linked to education and health conditionalities alone. As previously mentioned, it is doubtful even if the Project were to have been able to monitor co-responsibilities that it would have achieved several of the IOIs designed to measure progress. For example, for education co-responsibilities, the IOIs related to repetition rates in public primary and secondary schools, an outcome that would undoubtedly have required multiple parallel activities to show improvement. Similarly, for health co-responsibilities there were eight IOIs that in effect would have relied on the supply side interventions to show improvements. The Project’s M&E also provided input to achieve credit disbursements through the compliance with disbursement-linked indicators that were selected from among the IOIs in the Results Framework (para. 12). 63. The Project’s M&E aimed to monitor the progress and achievement of the PDOs through: (a) regular/routine monitoring to track progress of the project activities against plans, and trigger credit disbursements; (b) a Mid Term Review to assess early results/effectiveness of project activities, and (c) an impact evaluation of project outcomes at project closing to assess the Project’s effects and the extent to which its final development objectives have been achieved, and to identify lessons learned on interventions that either contributed to or hindered their achievement. M&E activities, including training for M&E, were part of the Project’s design under Subcomponent 2.4. 64. The Project’s design was also very responsive to the need for effective monitoring of the SEED Program’s activities through the design, development and implementation of a comprehensive Beneficiary Management Information System. The consolidated beneficiary registry, the BMIS, and related staff training on the use of the new systems, are all factors that strengthened the Government’s capacity to monitor program activities and results and supported the evaluation of outcomes over the longer term, and on the future development of its broader system of social protection. in Dominica and Grenada, (P159285). Page 26 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) M&E Implementation 65. The Project’s Results Framework, including its PDOs, was streamlined significantly in the Project restructuring. The revised PDO was limited to the first sub-objective of the original PDO (and represented in effect two objectives). The Key Outcome Indicators and Intermediate Outcome indicators (IOIs) were also adjusted to reflect the restructured project that eliminated co-responsibilities, and also those IOIs for which targets had already been achieved. Although the restructuring eliminated the Project’s second objective (to improve coverage of poor households receiving cash transfers), for some reason the restructuring maintained the KOI related to this objective in its Results Framework (although with slightly different wording to reflect the establishment of the SEED Program), i.e., Increased percentage of poor households receiving the SEED cash transfer, and Increased percentage of the program transfers that reach poor households with children. 66. Initially sporadic, Project monitoring information, including updates on progress towards results indicators and compliance with disbursement-linked indicators gradually improved during the implementation of the Project. Late finalization of the BMIS limited the M&E utilization during the Project, but now the fully functional BMIS generates key data which enables the Government to use it to inform further strengthening and development of the social safety nets. It is also now strengthening the focus on measurement of education co- responsibilities. M&E Utilization 67. See paragraphs 62 and 63. Justification of Overall Rating of Quality of M&E 68. Quality of M&E is rated Substantial. Although there were problems with the design and definition of the Project’s original results framework, these were addressed satisfactorily under the restructuring. More importantly, the Project has put in place a beneficiary registry and a solid, comprehensive BMIS for the SEED Program that allows not only for routine monitoring, but also serves to inform further development of the SSN and improve the Program’s efficiency through evaluation and eventual linkages with other programs, such as those addressing disaster response. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 69. Environmental and social. None of the World Bank’s environmental or social safeguards were triggered by the Project. 70. Fiduciary. Financial Management, including disbursements, and procurement were handled by the Project Coordination Unit in the Ministry of Finance, in coordination with the MOSD. To help strengthen fiduciary compliance, a consultant was hired to help carry out procurement and financial management responsibilities under the project. Disbursements were triggered by compliance with pre-defined performance milestones (supported by documentation certifying eligible expenditures), and, initially, by periodic disbursements of US$250,000 in each May and November. The DLIs were adjusted and the system of periodic disbursements was eliminated in the project restructuring (para. 18). With credit disbursements supported by cash transfers as Page 27 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) eligible expenditures, procurement processes were limited, as only activities under Component 2 (mostly Consultants’ Services and Training) were subject to the World Bank’s Procurement Guidelines. The Project faced no major issues with respect to either financial management and procurement processes, but it did face significant delays in contracting consultants for important activities such as the Proxy Means Test, for Phase II of the BMIS, and for the Information, Education and Communications activities. C. BANK PERFORMANCE Quality at Entry 71. The World Bank’s performance in ensuring quality at entry is rated Moderately Satisfactory. As described above, the Project’s focus on social protection built upon earlier technical work and responded to a clear Government’s priorities. The Project’s design was well-grounded and evidence-based. However, the Project was overly complex and ambitious, especially given the Government’s limited capacity and frequent turnovers, unfamiliarity with a new type of instrument, and the fact that the processes for monitoring of co- responsibilities in education needed strengthening, while those in health were only being established. Further, it is not clear that coverage of education and primary health were priorities for Grenada, nor, if they were, that the Project alone would have been able to support the achievement of this aspect of the PDOs, as measured by the related Intermediate Outcome Indicators. Quality of Supervision 72. The World Bank’s quality of supervision is rated Satisfactory. The World Bank’s implementation support was strong and timely, and as such was able to help improve implementation and prospects for meeting the restructured PDOs. The implementation support team worked closely with the team preparing the Programmatic Resilience Building Development Policy Credit series to ensure coordination of the social protection aspects of those operations and further support the achievement of the PDOs. Following the mid-term review, the team also expeditiously initiated the restructuring of the Project to bring the PDOs and the Project’s design more in line not only with what was achievable under an initial project with capacity building limitations but also to allow for the development of a new, efficient and effective social protection system, necessary within Grenada’s reality. Implementation support was intensive—the team held weekly audio or video conferences to ensure that project implementation remained on track. The realism and dedication with which the team carried out its responsibilities responded to what was required to bring the Project to a satisfactory conclusion. Today, Grenada counts with a more efficient, targeted, and monitored social protection system, with qualified staff and established processes and the Government is analyzing options to expand the system to address the needs of other vulnerable population groups. The Government recognizes the World Bank’s strong partnership and technical support in this effort. Justification of Overall Rating of Bank Performance 73. Overall World Bank performance is rated Moderately Satisfactory, based on the Moderately Satisfactory rating for the Quality at Entry and a Satisfactory rating for Quality of Supervision. Page 28 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) D. RISK TO DEVELOPMENT OUTCOME 74. Risk to development outcome is considered Modest. This is based on the Government’s strong commitment to social protection, as witnessed by the established legal and institutional framework for the SEED Program. The Government that took office in 2013 has recently been reelected, and its second term will last through 2023. Therefore, the Government’s financial commitment to the Program is strong: the 2019 budget for the SEED Program totals US$18.0 million, an increase over the previous year. This amount also includes resources for the Program’s continued strengthening. Further, the MOSDH has signed Memorandum of Understanding to progress on the monitoring of conditionalities, and the Government is considering linking the SEED Program and its registry to help respond to disasters. V. LESSONS AND RECOMMENDATIONS 75. The Project’s design and implementation experience offered the following lessons: 76. Project design should be kept in line with the capacity of the implementing institution. The Project’s design was unnecessarily complex, its Results Framework extensive and detailed, and the planned scheduling of activities, with some that needed to be completed before others began, made it difficult for the project as originally designed within the planned five year implementation period. One of the key lessons was that in an environment where institutional capacity is extremely limited, the slightest change of human resources in the MOSDH has a huge impact on the overall Project. Despite strong Government’s commitment to the Project, its implementation picked up substantially only after its restructuring. 77. Using a DLI-based instrument. The Project was the first DLI-based project in the OECS, thus there were a number of important lessons to be learned and applied in designing similar projects in the future. First, it is critical to ensure good understanding of the instrument among the counterparts and verifiers. Second, the DLIs should be simple, clear, and the verification strategy should be agreed upon at the time of their development. It is also important to include an investment support under the project to ensure that all the activities critical to achieve the DLIs can be accomplished. Finally, a strong implementation support facilitates the progress towards the achievement of the DLIs. 78. When adopting innovative approaches to project design, it is important to ensure that the client understands how those approaches will be applied. The Project followed a results-based financing approach, in which disbursements were triggered in part by compliance with identified milestones that were deemed important to keep implementation on track. While the initial counterpart team that participated in the preparation of the Project had a solid understanding of the Project and the DLI mechanism, that understanding diminished with changes in counterpart’s team. While the Bank’s team anticipated this and ensured support, reaching the strong level of understanding of the complex disbursement mechanism required more time than the standard investment lending. 79. When establishing co-responsibilities for a cash transfer program, it is important to be very selective in choosing the indicators. It is important to ensure that co-responsibilities promote compliance with actions that are relevant to the country’s priorities, that supply of services exists to allow compliance with the conditions, and that the mechanism are in place to allow for monitoring of co-responsibilities. Page 29 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 80. Substantial investment in development of a management information system should include the requirement for in-house MIS support. While the support of a local MIS consultant was planned under the Project, the actual engagement of such consultant was delayed, and the development of the MIS was well advanced before the local MIS support was in place. The first MIS developer was initially very keen to understand the program and work with the MOSDH, but the relationship soured over time leaving the MOSDH with very limited options when the consultant became unresponsive. The MIS was eventually completed by the consultant that was engaged to provide in-house MIS support. Based on this experience it is vital, especially in small countries like Grenada, not only to ensure that the Project envisages financing of critical positions in implementing agencies, but also require that these positions are filled early on during the Project implementation. 81. The mix of various financing instruments, especially mix of policy credits and technical investment support, facilitates the implementation. The support in ensuring the critical policy developments and implementation of the phasing out plan was a vital facilitator for the successful project implementation. . Page 30 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Strengthen the basic architecture of the SEED Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increased percentage of poor Percentage 0.00 80.00 82.00 households receiving the SEED cash transfer 31-May-2015 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Target Exceeded. Indicator introduced in November 2015 Project Restructuring; Baseline was to be determined once the PMT was applied; Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increased percentage of the Percentage 47.00 75.00 75.00 program transfers that reach poor households with 31-May-2015 31-Dec-2017 30-Sep-2018 Page 31 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) children Comments (achievements against targets): Target Achieved. This indicator was introduced in November 2015 Project Restructuring; Baseline was to be determined once the PMT was applied; A.2 Intermediate Results Indicators Component: Component 1: Improving Conditional Cash Transfers Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion The three cash transfer Text No merger of Programs merged and Three programs programs are merged programs, separate under a single budget merged under the institutionally and placed budgets MOSD with a single under MOSD budget budget 27-May-2011 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Target Achieved. This indicator was DLI A.1; The target had been met at the time of the Project Restructuring in November 2015 Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 32 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) MOUs between MOSDH and Text Agreement does not 1 MOUs between 3 agreements signed MOF, MOH and MOE on exist MOSDH and MOF, SEED implementation signed MOH and MOE on SEED implementation signed 27-May-2011 31-Dec-2017 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Target Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of SEED Percentage 0.00 95.00 82.00 beneficiaries identified as eligible through the new 27-May-2011 31-Dec-2017 30-Sep-2018 targeting tool Comments (achievements against targets): This indicator was DLI B.2 and it was achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of beneficiaries Percentage 0.00 95.00 95.00 Page 33 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) who received information 27-May-2011 31-Dec-2017 30-Sep-2018 about rules and procedures of the SEED program Comments (achievements against targets): Target Achieved. This indicator was DLI D.2; At the time of the Project Restructuring, the IEC Strategy had been established by beneficiary training for eligible households (identified through the PMT) had not been completed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of households Percentage 0.00 82.00 82.00 identified as eligible for the CCT based on the PMT 27-May-2011 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): This indicator was DLI B.2 and it was achieved. Component: Component 2: Improving Implementation Capacity of the Ministry of Social Development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Cabinet approval of the use Text Use of the PMT is not No Cabinet has approved Cabinet has approved of the PMT yet approved by the use of PMT use of the PMT Cabinet Page 34 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 27-May-2011 31-Dec-2017 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Target Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Installation and Text MIS does not exist 1 The MIS fully The MIS fully implementation of a operational operational. modernized MIS generating one beneficiary list and using 27-May-2011 31-Dec-2017 31-Dec-2017 30-Sep-2018 one payment method Comments (achievements against targets): Target Achieved. This indicator was DLI C.1; The MIS had been deployed to MOSDH for testing at the time of the Project Restructuring; Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion The Appeals Committee Yes/No N N Y operational 27-May-2011 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Page 35 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Target Achieved. This indicator was DLI D.3; This indicator was revised to read “The Appeals Committee operational” in the Project Restructuring, with a target of 90 completion by 31-Dec-2017 Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Results from Program Text No evaluation of 1 Evaluation results Evaluation results Evaluation disseminated program published and published and disseminated disseminated 27-May-2011 31-Dec-2017 31-Dec-2017 30-Sep-2018 Comments (achievements against targets): Target Achieved. This indicator was DLI E.4; This indicator was revised to read “Results from Process Evaluation disseminated” in the Project Restructuring, and the wording of the target was revised, as highlighted above; Page 36 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 To strengthen the basic architecture of the SEED Program and the capacity of the Ministry of Social Development and Housing (MOSDH) to implement it. 1. Increased percentage of poor households receiving the SEED cash transfer Outcome Indicators 2. Increased percentage of the program transfers that reach poor households with children 1. The three cash transfers are merged institutionally and placed under MOSD budget 2. Creation of a SEED unit in MOSD appropriately staffed 3. Installation and implementation of a modernized MIS generating one beneficiary list and using one payment method 4. Percentage of beneficiary households in the MIS for whom co- responsibilities are verified and reported 5. Agreement between MOF, MOSD, MOH, and MOE on formula for Intermediate Results Indicators PMT 6. Pilot of PMT carried out in one Parish 7. Percentage of households identified as eligible for the CCT based on the PMT 8. Percentage of beneficiaries who received information about rules and procedures of the SEED program 9. The Appeals Committee operational 10. Results from Program Evaluation disseminated 11. Cabinet approval of PMT 1. Institution, Budget and Coverage, Technical Assistance and Key Outputs by Component Transfers (linked to the achievement of the Objective/Outcome 1) 2. Beneficiary Outreach and Targeting, Consultants’ Services Page 37 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) 3. Management Information Systems, Developing a Central Beneficiary Registry, Consultants’ Services 4. Information, Education and Communication Campaign, Consultants’ Services 5. Monitoring and Evaluation of Disbursement Linked Indicators, improved Management Information System, Training 6. Contingency Plan for Crisis Response, Technical Assistance 7. External Audit. Technical Assistance 8. Design and Pilot of PMT, Consultants’ Services 9. Design of MIS, Consultants’ Services 10. Design of Central Beneficiary Registry, Consultants’ Services 11. Design of Materials for Protocols for Educational Training and Training for Beneficiaries, Training Design of a Training Plan for an Impact Evaluation and an Enhanced Process, Consultants’ Services Page 38 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Ludovik Subran Co-Task Team Leader Karla McEvoy Co-Task Team Leader Emmanuel Njomo Senior Financial Management Specialist Evarist Baimu Senior Counsel Maria Beatriz Orlando Senior Economist Amer Hasan Economist Erik Alda Social Development Specialist Asha Williams Social Protection Specialist Victor Ordonnez Financial Management Specialist Supervision/ICR Snjezana Plevko Task Team Leader(s) Sonia Cristina Rodrigues Da Fonseca Procurement Specialist(s) Shonell Jodian Robinson Financial Management Specialist Gibwa A. Kajubi Social Specialist Mirta G. Sanchez Team Member Patricia M. Bernedo Team Member Clarisse Torrens Borges Dall Acqua Environmental Specialist Asha M. Williams Team Member B. STAFF TIME AND COST Page 39 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 32.727 150,276.09 FY12 0 0.00 Total 32.73 150,276.09 Supervision/ICR FY11 0 0.00 FY12 16.600 83,930.79 FY13 23.227 63,454.22 FY14 37.696 168,183.29 FY15 8.018 54,755.56 FY16 8.490 61,564.95 FY17 11.101 60,634.64 FY18 11.100 72,726.65 FY19 11.158 78,409.82 Total 127.39 643,659.92 Page 40 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (%) Component 1: Improving 4.5 3.60 80 Conditional Cash Transfers Component 2: Improving Implementation Capacity of 0.5 .63 126 the Ministry of Social Development Total 5.00 4.23 84.6 Page 41 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 4. EFFICIENCY ANALYSIS Please see the paragraphs 32-36 in the main text. Page 42 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS N/A Page 43 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 6. DISBURSEMENT LINKED INDICATORS DLI No., Original DLI Revised DLI Revised DLI at Rationale for Change Amount No. and Restructuring and Year of Amount at Expected Restructuring Compliance The recipient has taken No Change – DLI was met at DLI A.1 steps to establish the the time of Restructuring. ($250,000) consolidated CCT Program through: (a) approval of a Year 1 special warrant creating a budget line dedicated to the CCT Program; (b) approval of a reallocation warrant to transfer funds to the CCT Program from two existing social safety net programs, and (c) establishment of the Cast Transfer Unit within the MOSD with structure, functions and responsibilities as well as resources acceptable to the Association Unified management Beneficiary DLI C.1 was met at DLI C.1 information system is DLI2 Management Restructuring. A new DLI2 ($250,000) established, and a single ($250,000) System is was introduced to reflect payment cycle is used for operational the critical milestone to Year 1 effecting Cash Transfers operationalize the BMIS. At to beneficiaries under the the time of the CCT Program restructuring, there had been delays in completing the design and roll-out of the BMIS. Meeting this DLI was to ensure that critical processes related to administration, management and monitoring of SEED could be carried out. Page 44 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) DLI No., Original DLI Revised DLI Revised DLI at Rationale for Change Amount No. and Restructuring and Year of Amount at Expected Restructuring Compliance 800 individuals, who This DLI was removed, as DLI A.2 hitherto had never the beneficiaries of the SEED ($500,000) received Cash Transfers, program are households. receive Cash Transfers Year 2 after their eligibility to become beneficiaries has been established through the proxy means test 70 percent of 30 percent of DLI B.2 was replaced with DLI B.2 beneficiaries are DLI4 beneficiaries are three consecutive DLIs that ($250,000) identified as eligible to ($200,000) identified as eligible would support the transition receive Cash Transfers to receive SEED to the new system. Year 2 through the proxy means through the new test (Baseline=0 in April targeting tool 2011) 60 percent of DLI5 beneficiaries are ($250,000) identified as eligible to receive SEED through the new targeting tool 80 percent of DLI6 beneficiaries are ($300,000) identified as eligible to receive SEED through the new targeting tool 70 percent of beneficiaries 70 percent of This DLI was unchanged. DLI D.2 have received critical DLI 1 beneficiaries have ($250,000) information about the CCT ($200,000) received critical Program including information about Year 2 eligibility criteria, the SEED Program enrollment procedures, including eligibility conditions for payments, criteria, enrollment appeals and complaints procedures, procedures, and exit from conditions for the CCT Program payments, appeals (Baseline=0 in April 2011) and complaints procedures, and exit from the Program Page 45 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) DLI No., Original DLI Revised DLI Revised DLI at Rationale for Change Amount and No. and Restructuring Year of Amount at Expected Restructuring Compliance DLI C.3 70 percent of beneficiaries DLI C.3 was incorporated in ($500,000) are paid based on the new DLI 2 (above). verification of compliance Year 3 with applicable education and health conditions set forth in the Operations Manual (Baseline-0 in April 2011) DLI D.3 70 percent of complaints The Appeals This DLI was revised to reflect ($250,000) and appeals in relation to DLI3 Committee is the establishment, training the CCT Program are ($300,000) operational and functioning of the SEED Year 3 attended to and resolved Appeals Committee. pursuant to standards and procedures of the CCT Program’s appeals and governance mechanism (Baseline=0 in April 2011) DLI E.4 Evaluation report on the This DLI was removed. Given ($250,000) CCT Program, focusing on several methodological the impact of the program issues, an impact evaluation Year 4 and its processes, is would not be carried out for published on the MOSD the CCT within the project website period. However, a process evaluation would be prioritized, and its results used to inform necessary improvements to support meeting the objectives of the CCT. Page 46 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) ANNEX 7. ORIGINAL RESULT INDICATORS Page 47 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 48 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 49 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 50 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 51 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 52 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 53 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 54 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 55 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 56 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 57 of 58 The World Bank Grenada Safety Net Advancement Project (P123128) Page 58 of 58