Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004878 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF017709 ON A SMALL GRANT IN THE AMOUNT OF USD 2.75 MILLION TO THE ChildFund International - Guatemala FOR Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) December 27, 2019 Social Protection & Jobs Global Practice Latin America And Caribbean Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Acting Regional Vice President: J. Humberto Lopez Country Director: Yaye Seynabou Sakho Regional Director: Luis Benveniste Practice Manager: Pablo Gottret Task Team Leader(s): Hugo Martin Brousset Chaman ICR Main Contributors: Karla J. McEvoy / Claudia P. Rodriguez Alas Director: Luis Benveniste Senior Global Practice Director: Michal J. Rutkowski Practice Manager: Pablo Gottret Task Team Leader(s): Hugo Martin Brousset Chaman ABBREVIATIONS AND ACRONYMS ACODIHUE Association of Integral Development Cooperation of Huehuetenango (Asociación de Cooperación al Desarrollo Integral de Huehuetenango) ASQ-I Ages and Stages Questionnaires-Inventories CEDRO Cooperation for Western Rural Development (Cooperación para el Desarrollo Rural de Occidente) CEO Evaluation and Orientation Houses (Casas de Evaluación y Orientación) COCODE Community Development Council (Consejos Comunitarios de Desarrollo) CODEDE Provincial Development Council (Consejos Departamentales de Desarrollo) COMUDE Municipality Development Council (Consejos Municipales de Desarrollo) COMUSAN Municipal Council for Food and Nutritional Security (Comisión Municipal de Seguridad Alimentaria y Nutricional) CF ChildFund CPS Country Partnership Strategy CSO Civil Society Organizations DMS Municipal Health Districts (Distritos Municipales de Salud) ECD Early Childhood Development IE Impact Evaluation (Evaluación de Impacto) ENSMI National Survey of Maternal and Child Health (Encuesta Nacional de Salud Materno Infantil) FM Financial Management ICR Implementation Completion and Results Report IPP Indigenous Peoples’ Plan IRI Intermediate Results Indicator ISR Implementation Status and Results ITT Intention to Treat JSDF Japan Social Development Fund MAGA Ministry of Agriculture, Livestock and Food Security (Ministerio de Agricultura, Ganadería y Alimentación) MDAT The Malawi Developmental Assessment Tool MIDES Ministry of Social Development (Ministerio de Desarrollo Social) MINEDUC Ministry of Education (Ministerio de Educación) MSPAS Ministry of Public Health and Social Assistance (Ministerio de Salud Pública y Asistencia Social) NGO Non-governmental Organization NNLS Our Smart and Healthy Children (Nuestros Niños Listos y Sanos) PDO Project Development Objective PS Procurement Specialist SESAN Secretariat of Food Security and Nutrition (Secretaría de Seguridad Alimentaria y Nutricional) s.d. standard deviation TOT Treatment on the Treated SDG Sustainable Development Goals WASH Water, Sanitation and Hygiene TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 15 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 27 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 33 ANNEX 2. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 45 ANNEX 3. IMPACT EVALUATION METHODOLOGY ................................................................. 46 ANNEX 4. DESCRIPTION OF THE NUESTROS NIÑOS SANOS Y LISTOS (NNSL) PILOT PROJECT .. 49 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Pilot to Improve the Development and Nutrition of P145410 Young Children in Poor Rural Areas in Guatemala Country Financing Instrument Guatemala Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency ChildFund International - Guatemala ChildFund Guatemala Project Development Objective (PDO) Original PDO The Development Objective is to strengthen the capacity of parents and communities to improve and monitor child development outcomes (physical, cognitive, socio-emotional and linguistic skills) for children under two in project intervention areas. Page 1 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-17709 2,750,935 2,704,667 2,704,667 Total 2,750,935 2,704,667 2,704,667 Total Project Cost 2,750,935 2,704,667 2,704,667 KEY DATES Approval Effectiveness Original Closing Actual Closing 07-Nov-2014 18-Mar-2015 07-Nov-2018 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 21-May-2018 2.01 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Procurement Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory High RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 08-Dec-2015 Satisfactory Moderately Satisfactory 0.30 Page 2 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Moderately 02 22-Dec-2016 Moderately Satisfactory 0.56 Unsatisfactory Moderately 03 23-Oct-2017 Moderately Satisfactory 1.55 Unsatisfactory 04 21-Sep-2018 Moderately Satisfactory Satisfactory 2.38 05 25-Jun-2019 Moderately Satisfactory Satisfactory 2.71 ADM STAFF Role At Approval At ICR Regional Vice President: Jorge Familiar Calderon J. Humberto Lopez Country Director: J. Humberto Lopez Yaye Seynabou Sakho Director: Luis Benveniste Luis Benveniste Practice Manager: Mansoora Rashid Pablo Gottret Task Team Leader(s): Lucy Katherine Bassett Hugo Martin Brousset Chaman ICR Contributing Author: Karla J. McEvoy Page 3 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context at Appraisal 1. At the time of project appraisal, Guatemala faced one of the highest levels of poverty and inequality in Latin America. With a GNI per capita of US$3,130, Guatemala was considered a lower middle-income country in 2012; however, its social indicators remained low compared to other middle-income countries within and outside the region. Poverty had increased from 51 to 53.7 percent in 2011 due to the global food, fuel, and financial (FFF) crisis. Child malnutrition was one of the highest in the world: stunting and anemia rates were 49.8 and 47.7 percent, respectively, affecting nearly half of all children under five. Under-five mortality was 30 per 1,000 live births, higher than the regional Latin American average of 19 per 1000 live births. Within Guatemala, there were deep regional and ethnic disparities in child outcomes. Children in rural areas suffered disproportionally from malnutrition; about 59 percent of children 3-59 months in rural areas were stunted, compared to 34.4 percent in urban areas. The northwestern departments were the most affected by chronic malnutrition, with rates as high as 70 percent of children 3-59 months. As indigenous children mostly live in rural areas, their stunting and underweight rates were almost twice that of non-indigenous children according to the Encuesta Nacional de Salud Materno Infantil (ENSMI) 2008-09. 2. Socioeconomic status, social and political exclusion, geographic isolation and cultural practices are key determinants of nutrition outcomes in Guatemala. As previously stated, indigenous people live mostly in rural areas, many in isolated locations that lack public services. Moreover, studies show that nutrition and hygiene habits also determine malnutrition in the first months of life. According to the ENSMI 2008-09, only 49.6 percent of children 0 to 6 months were exclusively breastfed, and the average month of breastfeeding for infants was only 3.4 months. Coffee, on the other hand, is introduced early among the poorest and indigenous populations. The poor hygiene and sanitary conditions in which the poor live cause childhood diseases such as diarrhea and respiratory infections, and these conditions are exacerbated by malnutrition. Even though malnutrition decreased from 54.5 percent in 2002 to 49.8 percent in 2008-09 (ENSMI 2008-09), at that rate, it was estimated that it would take four decades to eradicate chronic malnutrition in Guatemala. 3. Early childhood stimulation, in particular, is a critical input to maximize the impacts of nutritional interventions. Children who are stunted or otherwise malnourished will benefit from effective nutritional interventions, especially before the age of two, but they cannot catch up with well-nourished children in overall human development (including growth, cognitive, language, social, and motor development) if they do not receive proper stimulation in the early years. In some cases, children who have access to adequate nutrition fail to eat and to grow properly because of the lack of stimulation and attention they experience at an early age. This condition may be called "failure to thrive" when children refuse to eat and, as a result, do not grow properly. Or it may be called "deprivation dwarfism" when children eat but still fail to grow because the emotional and stimulation deprivation that they face depresses the endocrine system and inhibits the production of pituitary growth hormones, a substance known to be essential for the normal growth and development of body cells (World Bank, 2009). 4. The cumulative benefits of the combined approach of early childhood nutrition and early childhood stimulation interventions have been empirically proven in several contexts. The leading general medical Page 4 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) journal, The Lancet, found that poor caring and parenting practices during the first 1,000 days of life, or the “window of opportunity,� also have an important impact on malnutrition and child development. Apathy, lower levels of play, and more insecure attachment are associated with underweight and stunting in young children and with more problems with conduct, poorer attention, and poorer social relationships at school age. Evidence from Bangladesh and Uganda (Lancet, 2011), demonstrates that community based-psychosocial stimulation, with or without food supplementation, can improve child-rearing practices of mothers of severely malnourished children and the quality of their home environment. Evidence from Jamaica (World Bank, 2009) shows that stunted children ages 9 to 24 months who received the benefits from a combination of nutrition supplements and stimulation were, in the short term, able to catch up to non-stunted children faster than those only receiving nutrition supplements, and in the long term, were able to show cognition benefits at ages 11 years and 17 years, while those only receiving nutrition supplements were not. Theory of Change and Results Chain 5. Given the above, achieving improvement in childhood wellbeing in indigenous areas in Guatemala required an innovative approach that combined nutrition and stimulation. The Nuestros Niños Sanos y Listos (NNSL) pilot project was a culturally relevant project that placed mothers and caregivers at the center of their children’s development by giving them the tools to stimulate and monitor their children’s physical, cognitive, socio-emotional, and linguistic skills. The project was designed to complement ongoing interventions by the Government1 and other partners by piloting a set of community-based nutrition and parenting/early childhood stimulation interventions aimed at: (i) improving caretakers’ interactions with young children to enhance their physical, cognitive and emotional development outcomes; (ii) encouraging caretakers to adopt optimal caring and feeding practices for young children; and (iii) increasing community participation in monitoring child development indicators. The Grant was made to ChildFund (CF) (the implementing agency), who in turn hired two local NGOs (the Association of Integral Development Cooperation of Huehuetenango – ACODIHUE; and the Cooperation for Western Rural Development – CEDRO) to carry out the work in the communities. The two NGOs hired social workers, who, in cooperation with indigenous community governing structures (COCODES), identified volunteer females they could train in early stimulation concepts and exercises. These volunteers were known as Madres Guías, and they worked directly with parents and caregivers to educate them on the importance of early stimulation and activities to engage their children. The use of volunteer mothers from the communities to deliver these services (home visits) is part of similar Government-funded programs, such as the Cuna Más program in Peru. 1The project complemented the then- current public interventions to improve child nutrition, especially the Hambre Cero (Zero Hunger) Program. This program was the central initiative of the Government’s social policy. It aimed to reduce the prevalence of chronic malnutrition in children under-five and child mortality through the coordination of multisectoral interventions. Page 5 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Activities Outputs Outcomes Long-term Outcomes Develop and integrate Diagnostic of early child stimulation Households with practices/beliefs enhanced early activities into community nutrition activities around early childhood stimulation practices stimulation Cultural adaptation of the Increased community childhood stimulation Implementation of awareness on programs, and monitoring Ventana de la Vida parenting/early tools program early childhood stimulation stimulation and and development Validate early stimulation parenting sessions messages through group Parents/caretakers’ Improved physical, Train social workers, modality) caring and feeding cognitive and Madres Guías, and health practices for young emotional staff on parenting/early Implementation of children is improved development of stimulation Reach Up program beneficiary children (early stimulation and Increased community Train parents & parenting sessions communities on parenting outreach by through home visits) COCODES, health and & early stimulation nutrition institutions Sensitize the COCODES Monitoring Tool to track through community and other government and child development fairs civil society organizations on the importance of early Community Fairs childhood stimulation Parents, caretakers, and communities Train midwives in Baseline and endline monitoring child exclusive breastfeeding data development and early childhood stimulation Impact Evaluation Develop messages to promote positive child care & feeding practices Establish impact evaluation baseline and endline Evaluation of pilot/dissemination of findings Project Development Objectives (PDOs) 6. The Project Development Objective (PDO) was to strengthen the capacity of parents and communities to improve and monitor child development outcomes (physical, cognitive, socio-emotional, and linguistic skills) for children under two in Project intervention areas. Page 6 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Key Expected Outcomes and Outcome Indicators 7. The PDO level results indicators were the following: (a) Percentage of children achieving the expected development level for their age (b) Percentage of participating households with adequate early stimulation in the home (c) Percentage of participating parents that recognize and value key activities for infant and child nutrition and development (d) Percentage of communities regularly monitoring child development outcomes Components 8. Component 1: Promotion of physical, cognitive, linguistic and socio-emotional development through a parenting and early stimulation intervention (US$1,865,111.62). This component was delivered through three subcomponents: 1.1 Mapping early stimulation practices in (and applicable to) the participating communities; 1.2 Developing and integrating a parenting/early child stimulation component into community nutrition activities that were already being carried out (e.g., the Hambre Cero program); and 1.3 Sensitization of state, municipal, and local staff, health facilities staff, and civil society organizations (CSOs) on the importance of the integrated approach to Early Child Development (ECD) and the roles of each entity to promote it through, inter alia, the carrying out of training, seminars, workshops, and the provision of technical assistance. 9. Component 2. Enhanced social and behavior change communication to achieve improved child nutrition and development (US$129,359.37). This component had three subcomponents: 2.1 Investigation of current child feeding and caring behaviors; 2.2 Development of an enhanced social and behavior change mode; and 2.3 Roll out of social and behavior change approach. It is important to note that this type of nutrition-related behavior change activity was also to be carried out under the World Bank’s Crecer Sano project (P159213), approved on March 24, 2017, which was being prepared at the same time as the JSDF Grant. The Crecer Sano’s PDO is to improve selected practices, services and behaviors known to be key determinants of chronic malnutrition (with an emphasis on the first 1,000 days of life) in the intervention areas. Despite this, the Grant team decided to proceed with financing the nutrition activities, since it was likely that Crecer Sano would not be implemented in time to overlap with the JSDF Grant. 10. Component 3. Project management and administration. Monitoring and evaluation, and knowledge dissemination (US$756,464.01). This component included two subcomponents: 3.1 Project management and administration; and 3.2 Monitoring and evaluation and knowledge dissemination. Significant Changes during Implementation 11. The Grant was restructured twice2: • In January 2015: to add an expenditure category, and to re-allocate Grant proceeds among expenditure categories; and 2Only the second restructuring appears in the ICR datasheet, because the information platform for trust funds was only established after January 2015 – so the first restructuring was processed by email and does not appear in the trust funds platform. Page 7 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) • In May 2018: to (1) eliminate some nutrition communication and behavior change activities for families; (2) add nutrition-related communication and behavior change activities at community fairs, to promote nutrition sensitive activities;3 (3) reallocate resources across components and across disbursement categories; (4) extend the Grant’s closing date from November 7, 2018 to June 30, 2019, to allow implementation of the activities that were delayed, to ensure a more robust impact evaluation and to allow more children to receive the project benefits for a longer period; (5) change in procurement arrangements to allow remaining procurement to be carried out under the new World Bank’s Procurement Regulations for Borrowers under Investment Policy Financing, approved in July 2016 and revised in November 2017; and (6) change the Results Framework. 12. While the PDO was never modified, some new indicators were added to measure activities that directly contributed to the achievement of the PDO, while others were dropped to reflect the elimination of several nutrition activities under Component 2. In addition, some outcome indicator targets were revised downward to reflect the slow start of Grant activities, and to be in line with what the improved measurement tools indicated was more realistic to achieve in terms of outcomes. Tables 1 and 2 summarize the changes made to PDO and intermediate indicators. Table 1. Summary of Original and Revised PDO Indicators Indicators Revisions Justification PDO Indicator: Percentage of New end target: 60% The end target was adjusted children achieving the expected because, once the Barrilete and development level for their age. other tools to measure early stimulation practices in families End Target: 70 % had been adapted to the Guatemalan indigenous context and applied to the pilot, it was clear that the original target was not in line with international evidence from other similar interventions, and that it could likely not be reached even within the extended Grant implementation period. 3Agencies such as Secretariat of Food Security and Nutrition (SESAN) and the Ministry of Public Health and Social Assistance (MSPAS) provided training on several issues including WASH, minimum nutrition packages, etc., during the community fairs. Page 8 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) PDO Indicator: Percentage of Percentage of The new articulation of the participating households with participant households indicator (‘enhanced early adequate stimulation at home. with enhanced early stimulation practices’) more stimulation practices accurately reflects the comprehensive set of skills that End Target: 70% New End Target: 40% are supported under the Grant. The downward revision of the target was due to the fact that: (1) many families migrated during certain seasons for work and so did not remain consistently within the program; and (2) attrition, given that program attendance was based purely on motivation and not by any monetary compensation. Page 9 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) PDO Indicator: Percentage of Dropped The activities this indicator was participating parents that meant to measure were scaled recognize and value key activities down under the Grant’s second for infant and child nutrition and restructuring. Activities were development. dropped because it had become clear that ChildFund had more End Target: 70% expertise in delivering parenting practices at the community level, whereas it would have to develop nutrition activities from scratch (including adaptation of measurement tools), and that would have entailed collaborating with MSPAS and NGOs, which would delay implementation even more. The Bank and ChildFund teams decided therefore to drop most of the nutrition activities, to focus on early stimulation, since ChildFund had a comparative advantage and could make an impact, and leave it to the Crecer Sano project to carry out the bulk of nutrition activities. Doing this ensured that ChildFund could carry out quality work while still maintaining an integrated approach to combating childhood malnutrition. PDO Indicator: Percentage of Dropped Under the second restructuring, communities monitoring child Grant activities focused on development outcomes. working with parents instead of community monitoring. End Target: 70% Number of children that are direct New indicator This core indicator was added to beneficiaries of the Project. monitor the final number of children receiving benefits End Target: 5,500 children through the Project. Page 10 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Table 2. Summary of Original and Revised Intermediate Result Indicators Component Original Indicators Revised indicators Justification IRI_1: Diagnostic of early No changes childhood development beliefs and practices in beneficiary communities is completed IRI_2: Monitoring tool to No changes track child development is developed IRI_3: Early stimulation Dropped Other early stimulation Component 1: package of activities packages in the results Promotion of developed framework – specifically, physical, cognitive, participation in Ventana linguistic and socio- End Target: Yes de la Vida and Reach Up emotional pilot programs – are developments better able to measure through a parenting progress on early and early stimulation stimulation activities, and intervention were already developed. IRI_4: Percentage of project Number of project The indicator was re- communities implementing communities worded to precisely activities with caretakers implementing articulate the type of Level 1 (Ventana activities that were being End Target: 100% de la Vida) and measured. Level 2 (Reach Up) methodologies End Target: 100 % Page 11 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) IRI_5: Percentage of local Percentage of Indicator was revised to and municipal leaders and COCODES more accurately reflect health facility staff (Consejos project activities. The end sensitized on the Comunitarios de target was revised importance of ECD Desarrollo) downward, however, implementing because (despite multiple End Target: 100% activities in favor sensitization efforts) of childhood in some COCODES would their communities not allow the pilot to be implemented in their End Target: 70% communities, nor start their own initiatives. IRI_1: Summary of key Dropped Grant was restructured constraints to optimal child to focus on early feeding and caring stimulation activities (see Component 2: behaviors is completed explanation in Table 1, Enhance behavior above); therefore, the change End Target: Yes indicator was no longer communication to relevant. achieve improved child nutrition and IRI_2: Behavior change Dropped Ibid development approach defined End Target: Yes Page 12 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) IRI_3: Percentage of Number of The indicator was municipalities community fairs changed to reflect the implementing social and implemented with inclusion of training on behavior change an ECD approach early stimulation and communication with target good nutrition practices groups End Target: 188 during community fairs. community fairs This ensured that some End Target: 100% nutrition activities remained in the Grant, while others originally intended to be financed by the Grant, were to be carried out under the Bank’s Crecer Sano project. IRI_1: Baseline survey No changes conducted IRI_2: Quarterly progress Revised end Target increased to align reports completed target:16 reports with extension of closing date. End Target: 14 reports IRI_3: Mass events hosted Revised end As previously stated, the Component 3: for departmental health target: 2 events mass events were Project management systems dropped under the Grant and administration, and would instead be monitoring and End Target: 16 events carried out by the Crecer evaluation, and Sano Project. The revised knowledge end target reflects what dissemination had already been carried out before the second restructuring of the Grant. IRI_4: Final survey Final Impact Revised indicator conducted Evaluation survey references the type of conducted survey (Impact End Target: Yes Evaluation). End Target: Yes Page 13 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) IRI_5: Final Report Dropped Child Fund reported Completed by ChildFund progress quarterly; therefore, a final report End Target: Yes was no longer relevant. IRI_6: Number of Madres New New indicator is relevant Guías (MG) trained to apply to monitor early level 1 methodology stimulation activities of (Ventana de la Vida) Ventana de la Vida. End Target: 595 MG IRI_7: Number of Madres New New indicator is relevant Guías trained to apply level to monitor early 2 methodology (Reach Up) stimulation activities of Reach Up. End Target: 450 MG IRI_8: Number of children New New indicator is relevant participating in level 1 to monitor early activities (Ventana de la stimulation activities of Vida) within “Casas de Ventana de la Vida. Evaluación y Orientación�- CEO's strategy End Target: 4,480 children IRI_9: Number of children New New indicator is relevant participating in level 2 to monitor early activities (Reach Up) stimulation activities of Reach Up. End Target: 1,920 children IRI_10: Percentage of New New indicator is relevant Madres Guías with quality to monitor early implementation of level 1 stimulation activities. and level 2 activities ChildFund had developed a tool to assess quality of End Target: 70 MG implementation. Page 14 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) IRI_11: Number of CEOs New New indicator is relevant implemented to monitor early stimulation activities End Target: 80 CEOs under Ventana de la Vida. IRI_12: Number of New New indicator is relevant midwives in project to monitor activities to catchment areas trained in be provided to midwives exclusive breastfeeding and as they are the main opportune stimulation by maternal health care the social workers providers in the communities of End Target: 900 midwives intervention. Other changes 13. The Grant’s legal agreement was modified to reflect the above changes, as well as the payment of social workers’ salaries for the period between April 2016 and June 2019. Rationale for changes and their implication for the original theory of change. 14. The changes to Grant activities and the subsequent reallocation of resources reflected the realization that ChildFund did not have capacity to deliver both the stimulation and the nutrition activities, and was in fact better equipped to provide the former. The Bank and implementing teams added training in early stimulation and good nutrition practices (the latter provided by the Ministry of Public Health and Social Assistance - MSPAS) at community fairs, to ensure that some limited nutrition training for the community and particularly for midwives, would take place. The decision to scale down nutrition activities was also justified by the finalization of the preparation of the IBRD Project Crecer Sano, which included nutrition-related activities that the Grant team believed could be provided to Reach Up and Ventana de la Vida beneficiaries. The restructuring also allowed the financing of social workers’ salaries, which had to be approved by the donor country, which was key to achieving the Grant’s objectives. The revisions to Grant activities did not have implications for the theory of change. II. OUTCOME A. Relevance of PDOs Rating: High 15. The NNSL project was in line with Guatemala’s Country Partnership Strategy (CPS) FY13-16. One of that CPS’s pillars was to provide support for improved results in social sectors. This included ensuring the sustainability of achievements of the ongoing health, nutrition and education projects that came to an end during the CPS period. Page 15 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) 16. The Grant objectives continue to be relevant, in line with Guatemala Country Partnership Framework for FY17-204 and with the Guatemalan government’s national priorities. The Grant contributes to Pillar 1 (Fostering Inclusion of Vulnerable Groups) of the Country Partnership Framework, and the administration that took office in January 2016 placed health, education, stronger economic growth, and increased transparency at the center of its policy agenda. Recognizing the important role that human capital development plays in contributing to economic growth (and vice versa), the Government’s 2016-2020 Plan emphasizes the need for improvements in health and education, and reduction of chronic malnutrition. In February 2016, the Government established the Commission to Reduce Chronic Malnutrition, and in March 2016, the President officially launched the National Strategy to Prevent Chronic Malnutrition 2016-2020. The Government also recently updated its Primary Health Care (PHC) Model, adopting a multidimensional (individual-family- community) approach and integrating complementary aspects of traditional Indigenous health beliefs and practices. 17. The Project is aligned with the WB’s twin goals to reduce poverty and promote shared prosperity, and the WB Human Development Strategy. Pillar 2 of the Human Development Strategy calls for investments to promote “a strong healthy start for all.� Likewise, the intervention is well aligned with multisectoral interventions promoted by the WBG group to invest in the early years, and which call for a comprehensive approach to invest in the first 1,000 days. The recently launched Human Capital project also refers to the potential gains that countries can secure by investing in key education and health and nutrition services at an early age, contributing to maximize the potential productivity of children when they reach adulthood, if Governments secure appropriate service delivery networks, including investments such as the ones promoted under the NNSL pilot. 18. The Project comes at a critical juncture in global efforts to reduce malnutrition. The Project’s objectives are in line with the Sustainable Development Goals (SDG) to end malnutrition by 2030, including achieving internationally agreed targets on stunting and wasting in children under five years of age. The Project also supports SDG 3, to ensure healthy lives and promote well-being for all, and SDG 10, to reduce inequalities within countries. Guatemala is also participating in the Scaling Up Nutrition Movement.5 4 Report No. 103738-GT. 5 The Scaling Up Nutrition (SUN) Movement was launched in September 2010 by individuals from governments, civil society, donors, United Nations, business and researchers. It is an effort to eliminate all forms of malnutrition under the principle that everyone has a right to food and good nutrition. From https://scalingupnutrition.org/ Page 16 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) ACHIEVEMENT OF PDOS (EFFICACY) Rating: Substantial 19. A rigourous impact evaluation to measure the effectiveness of two delivery modalities was carried out and its preliminary results show that both modalities were successful at improving parental practices on child stimulation, increasing the variety of play material and the play activities conducted by the caregiver. In terms of the PDO indicators, one of the three revised PDO indicators was achieved, and the other two revised ones were partially reached. 20. A clustered randomized control trial was implemented in the departments of Huehuetenango, San Marcos, Quiche and Totonicapán where 113 communities were randomly assigned to one of the three groups: (T1) home visits, (T2) group meetings or (C) control group. Baseline data was collected between October 2016 and January 2017 on 2,022 households and 2,022 children up to 24 months. Households were revisited two years later between showing a low attrition rate of 13 percent. The program reached approximately 50 percent of the children in each treatment groups (T1 and T2). The impact of the program is measured based on intention- to-treat (ITT) effects to avoid a selection bias in the comparison between treatment and control groups. Treatment on the treated has also been estimated to reflect the impact on those that effectively received the treatment (TOT). • Family Care Indicators 21. Preliminary analysis using ITT method shows that the size of the impact on play activities between the caregiver and the child was of 0.12 standard deviations (s.d.); this increases to 0.37 s.d. under the TOT method. When measured as a percentage change under the ITT method, Ventana de la Vida participants reported a 6.27 percent increase in play activities compared to the control group, while Reach Up reported an increase of 3.6 percent. Under the TOT method, families in Ventana de la Vida report a 11.7 percent increase in play activities while Reach Up shows a 7.48 percentage increase. 22. Parental practices on child stimulation also improved. The magnitude of the impact is 0.16 s.d. (ITT method) and 0.37 s.d. (TOT method). Compared to the control group, Ventana de la Vida, with the ITT method, led to slightly higher results than Reach Up (3.10 and 2.72 percent improvement in parent practices, respectively). Both program modalities show about a 5.6 percent increase in improved parental practices under the TOT method. • Child Cognitive Indicators 23. Preliminary estimations show that both program modalities had an impact on childhood cognitive skills. For Ventana de la Vida, the size of the impact was 0.27 s.d. (ITT) and 0.6 s.d. (TOT). For Reach Up, this impact was 0.20 s.d. (ITT) and 0.46 s.d. (TOT). This confirms that the improvement in parental care, including play, translates into improved children’s cognitive fine motor and language skills. 24. Children in Ventana de la Vida show a 1.65 percent increase in fine motor skills compared to the control group while Reach Up children show a 1.38 percent increase (ITT method). This impact is higher under the TOT Page 17 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) method: cognitive and fine motor skills for children in Ventana de la Vida increased by 3.01 percent compared to the control group, while those skills increased by 2.82 percent for children in Reach Up. 25. In terms of language skills, children in Ventana de la Vida show a 2.06 and 3.75 percent improvement when measured by the ITT and TOT methods, respectively. For children in Reach Up, language skills improved by 1.25 percent (ITT) and by 2.55 percent (TOT), compared to the control group. 26. As shown in table 3, the magnitude of the changes is statistically significant for all domains. Most effects for Ventana de la Vida are highly significant at the 1 percent level, independently of the method. The size of the effects is medium, according to international standards.6 The intervention had no impact on nutritional outcomes, in line with results from international evidence.7 27. In conclusion, the IE shows that, despite the continued high level of stunting of children (60 percent at baseline, 73 percent endline), the Guatemala pilot was successful in improving cognitive fine motor and language skills, thereby ensuring children will be better prepared to learn once they start school. By preparing them to learn better in school, the NNSL intervention is helping young kids to catch up on cognitive skills that should help them to change their life path. Table 3. Size of the Effects of NNSL Program as Percentage Change (preliminary results) Intent-to-Treat Effects Treatment on the Treated (ITT) Effects (TOT) Ventana de la Ventana de la Reach Up Reach Up Vida Vida Family Care Indicator FCI Play Activities with Adults Scale 6.27% *** 3.66% * 11.47% *** 7.48% * Family Care Indicators Overall Scale 3.10% ** 2.72% ** 5.66% ** 5.61% ** Cognitive Outcomes, MDAT Fine motor, MDAT 1.65% *** 1.38% ** 3.01% *** 2.82% ** Language, MDAT 2.06% *** 1.25% * 3.75% *** 2.55% * Note: * p<0.10, **p<0.05, ***p<0.01 Source: Trias, Julieta and Irma Arteaga (2020) “Home visitation or group intervention? Effects of early stimulation on child wellbeing in rural Guatemala: A cluster randomize control trial� World Bank mimeo . 6Effect sizes of 0.3 -0.6 s.d. are considered in the literature as medium-size effects, while effects above 0.6 s.d. are considered high. 7 The international evidence on parenting programs for child stimulation is mixed. These programs, when successful, have shown improvements in parental practices for child stimulation and child development, notably cognitive and language skills. Effects on cognitive and language skills range between 0.35 to 0.47 standard deviations (Rao et al. 2014, Aboud et al. 2015, Britto et al. 2017). For parental practices, while the evidence is thinner, the reported effects range between 0.40�0.55 standard deviation (Aboud and Akhter, 2011; Nahar et al. 2012, Aboud et al. 2013 for Bangladesh, Walker et al. 2004 for Jamaica, and Attanasio et al. 2014 for Colombia). In addition, these programs have been less successful in improving nutritional outcomes. Page 18 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Efficiency (economic analysis, aspects of design and implementation) Rating: Substantial 28. An economic analysis was not carried out during Grant preparation. An economic evaluation was, however, conducted as part of the overall assessment of the two early stimulation models, with the objective of estimating the total cost over the entire four-year program, the monthly ongoing cost while the program was fully operational in all targeted areas, and the cost-efficiency of each modality measured as the average cost per child beneficiary. This study took a provider perspective of the financial costs, meaning that only the costs to the program implementers were included rather than all direct and indirect costs to the wider society. However, a basic estimation of the imputed cost of the volunteer Madres Guías was done as this is an important element in the sustainability of the implementation methodology. Financial costs were estimated using accounting data provided by the implementing organizations in the form of detailed account ledgers. 29. At a cost of US$267 for Ventana de la Vida and US$173 for Reach Up, the cost per registered child beneficiary per year is relatively similar between the two methodologies. However, when the cost-efficiency metric of cost per child is calculated based on number of active children in the program the difference in cost per child is much larger, with Ventana de la Vida at US$268 per child per year and Reach Up at US$420. Ventana de la Vida has a very low rate of attrition of children beneficiaries (1 percent) meanwhile the rate is much higher in Reach Up (59 percent). The attrition rate will be an important factor when interpreting the impact. When bringing together the impact and cost per impact, it may be more useful to evaluate the program based on cost per active child rather than cost per registered child. Based on the preliminary results and the costing exercise, group meetings are the modality most cost-effective for scaling up. 30. While strict cost comparisons between countries are difficult, Colombia’s implementation of the Reach Up program cost US$500 per child. In any case, cost-efficiency is only one factor that should be considered when evaluating a program to make recommendations for the future. Differentials in impact, sustainability, acceptability, and issues of equity should also take into consideration, alongside cost differentials. 31. Other dimensions of efficiency. The Grant was extended one time, for approximately 8 months, to account for a delay in starting activities because of the time it took to adapt the Reach Up program’s materials and assessment and monitoring tools to the Guatemalan indigenous context. The cumulative supervision costs for this Grant represented 9 percent of the total Grant amount. 32. Given that the cost of both models is within international parameters, and that administrative costs, excluding social workers’ salaries, was in line with appraisal estimates, efficiency is rated Substantial. Page 19 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Overall Outcome Rating: Moderately Satisfactory 33. Because the Grant was restructured to revise some PDO targets, the outcome rating for the project was assessed against the original and revised project development indicator targets. Tables 4 demonstrates the calculation of PDO before and after the restructuring. Table 4: Calculation of Overall Outcome Rating Original Restructured Relevance8 High Efficacy Modest Substantial Efficiency9 Substantial #1: Outcome ratings Moderately Unsatisfactory Satisfactory #2: Numerical value of 3 5 outcome ratings #3: Disbursement 2.01 million .74 million #4: Share of disbursement 74% 26% Weighted value of the 2.22 1.30 outcome rating (# 2 x # 4) Final outcome rating 3.52(rounded up to 4.0) = Moderately Satisfactory 34. The Moderately Satisfactory (MS) rating is further supported by the preliminary results of the impact evaluation carried out under the Grant which, as previously indicated, shows that NNSL improved parental practices for early childhood stimulation and increased children’s cognitive fine motor and language skills. These impacts are expected to help children learn better in school and increase their human capital development. Other outcomes Gender 35. Beneficiary women in both programs reported a change in their approach to motherhood. Women learned the development milestones they should expect their children to achieve, how to identify their children’s needs by the way they cry; how to teach their children concepts of space, forms and texture; to play, sing and dance with their children; to practice good hygiene; and to understand the objectives behind the toys and to create toys from materials around the house. Because of the program, mothers report that they are, in general, more involved with all their children, including their older ones. Female and male social workers also reported having learned to be better parents. 8 One rating for relevance in line with the ICR guidelines. 9 One rating for efficiency in line with the ICR guidelines. Page 20 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) 36. Women’s perceptions of their children’s development. Mothers reported that because of the program their children are less fearful, less timid, socialize better with other children, learn to walk and talk faster than their older siblings; they play, sing and dance, and are in general smarter. 37. Women were empowered by the program and gained respect in the community. Women in these communities face multi-layers of marginalization. They are poor, they live in rural areas - many in remote and isolated places-, and they are indigenous. Most of them have little or no education. The program gave them the opportunity to learn and to have a voice. Madres Guías became leaders, and the ones who implemented Ventana de la Vida and led the CEOs learned to facilitate group meetings. According to their perceptions, both Madres Guías and beneficiary mothers in both programs learned to socialize, became less timid, increased their self-esteem, learned to express themselves, and started to have a voice in other areas of society, such as their children’s school and during community meetings. They became role models for other women in the community, and they claim that the program transformed their lives.10 38. The NNLS created new spaces where women could come together to share experiences in parenting. The CEOs, in particular, became spaces for women to gather and learn from each other. In many cases these spaces -exclusively for women and mothers- had been non-existent in the communities prior to the pilot.11 Some fathers also participated in CEOs and gained interest in participating in their child’s development. Communities (COCODES) in general gained interest and some of them decided to continue with the activities even when the project closed. Community 39. The program generated changes in the household and in the community. Children older than two also benefited from the programs. They learned the same songs and activities and benefited from their mother’s new way of relating to her children. Mothers reported that these children became proud of their mothers and started to help them in the program activities. For example, older children of Madres Guías sometimes assisted their mothers during household visits or CEOs. They would help them read the guidelines when their mothers were illiterate. Mothers reported that while playing with their children, they forgot about their own problems and were happier. Husbands were also happier when they noticed the changes in their wives and children. Beneficiary mothers shared their new knowledge with other women who were not part of the program. Even women without children asked to be Madres Guías because they recognized the importance of children’s development. COCODES also learned to respect the work of Madres Guías and to recognize the rights of children. Community leaders realized that they had often prioritized infrastructure projects and did not think of children as important stakeholders. In the aftermath of the program, at least one COCODE requested funds from COMUDES for children’s projects. Institutional strengthening 40. The community fairs helped to create synergies among government institutions and the NNLS program to sensitize the broader community. Community fairs were implemented as an opportunity to sensitize and train community members that were not participating in the NNLS program, including parents, community 10 Systematization document, August 2019. 11 Ibid. Page 21 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) leaders, or other community members in general. The community fairs also facilitated interinstitutional collaboration, as different institutions came together to staff the fair. Some of the institutions that collaborated with the program are the Municipal Council for Food and Nutritional Security (COMUSAN), Municipal Health Districts (DMS), Ministry of Agriculture, Livestock and Food Security (MAGA), Ministry of Public Health and Social Assistance (MSPAS), and the Secretariat of Food Security and Nutrition (SESAN). The collaboration extended outside the fairs and became helpful resources. For example, during CEOs, the DMS would refer families to the NNLS to start the ECD program. Or if a Madre Guía identified during the Barrilete assessment12 that a child needed medical assistance, she would refer the child to the Health Center. Through collaboration with MSPAS, some communities that initially had rejected the program, opened up to it. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME Preparation 41. Initial objectives and targets for beneficiary population were too ambitious. The original project document stated that the project was expected to reach 12,200 poor indigenous children. However, that number proved to be too high in practice. A list of eligible children (age 2 and under) had been drawn from birth records and community referrals. As implementation began, however, it became clear that there was not such a high number of children fulfilling age eligibility in the target communities. In addition, from the existing eligible children, some had to be part of the control group for the impact evaluation and could not receive the program; other families with eligible children simply did not want to participate in the program. Unlike typical cash transfer programs, where there is usually an over demand for the program, NNLS did not provide a transfer but a learning program. Therefore, community leaders had to be persuaded about the benefits of the program to let it start, then families had to be persuaded to participate. As a result, target numbers were adjusted to 5,500 children in the mid-term restructuring. Implementation Factors under implementing agency’s control 42. At the beginning of the project there were significant delays due to ChildFund’s inexperience with World Bank procurement and financial procedures. This caused lengthy and complex hiring processes. There was a lack of annual planning documents to assist with monitoring, there were weaknesses in internal control, including non-compliance with payment policies and procedures, inaccuracies in the auxiliary records, and incomplete documentation of expenditures. There were also deficiencies in the preparation of SOEs and financial reports. The World Bank therefore provided training on financial management (FM) and procurement 12 The Barrilete is an assessment tool to monitor the development of children aged 0-36 months. The tool consists on a questionnaire aimed to measure 5 areas of development: i) gross motor skills, ii) fine motor skills, iii) language and communication, iv) socio-emotional skills, and v) cognitive development. The tool is divided in 8 phases of 3 months each: 0-3 months, 3-6 months, and so on up to 24-36 months. Depending on the child’s age the social workers and Madres Guías evaluated the level of development of the child and assigned a score based on a traffic light scale: red if the child could not do any of the activities in the 5 areas assessed; yellow if the child could do some of the activities; and green if the child was able to complete all the activities. Depending on the score, the social workers and Madres Guías provided recommendations to the parents or caretakers. This evaluation was conducted to each beneficiary child every 6 months (Systematization document). Page 22 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) issues and carried out close supervision of both. In addition, ChildFund hired a FM person with prior experience in Bank’s FM and procurement systems, which helped improve its performance in this area. 43. Staff changes in implementing agency caused delays. There were a few staff changes in ChildFund during program implementation. The program coordinator was dismissed in August 2015 without a handover of records and plans, causing delays during a critical implementation period. During the program life, the program coordinator changed three times. The M&E and financial positions also changed once. The technical staff (social workers) had on average 1.5 changes during the life of the project (although by the end of the implementation, both CDRO and ACODIHUE had consolidated strong and permanent teams of social workers, which helped strengthen project performance during the final stage of implementation, when more communities joined, more children were reached, and results started to materialize). These changes created a lack of continuity in community activities, diminished supervision to Madres Guías, and placed more work load on social workers who had to cover gaps until a new person was hired. 44. Difficulties with the survey firm hired to conduct the baseline. There were several quality issues with the firm hired to conduct the baseline evaluation. Specifically, there were delays with the delivery of products, incomplete information on numerators’ credentials, and failure to provide access to a data dashboard for World Bank staff to monitor progress. The quality issues were exacerbated by the lack of direct communication between the baseline firm and World Bank staff (ChildFund, given the Grant’s design, had contracted the firm and the firm therefore wanted to work only with ChildFund staff). With a lot of supervision and pressure from the World Bank, the baseline was delivered. To avoid further problems, a different firm was hired to conduct the endline. The firm was hired directly by the World Bank so that Bank staff could supervise the work. 45. Initially it was hard to find staff that met the technical profile required for the program . Both NGOs, CEDRO and ACODIHUE, had issues filling the positions of program coordinators. There was a shortage of psychometrists that could understand and measure the indicators in the indigenous context, as well as a dearth of early childhood educators that could implement the two curricula. These capacities had to be created among social workers and numerators through intense and continuous training which took considerable time not originally anticipated in the project design. 46. The NNLS pilot project was appropriately adapted to the Guatemala Mayan indigenous rural context. More than 90 percent of NNLS beneficiaries were indigenous, and the following tasks were carried out to ensure the cultural appropriateness of Grant activities: o An Indigenous Peoples’ Plan (IPP) was prepared in 2014 and updated in 2018. It provided: i) a review of positive and negative impacts of the pilot and mitigating measures; ii) a plan and framework to respect local indigenous governance, language preference and traditions; iii) a protocol for consultation during implementation; and iv) detailed Grievance and Redress Mechanisms (GRM). o Program implementation respected indigenous governance. The NNLS was introduced to communities only after getting acceptance from the local indigenous authorities. The program was first presented through all levels of government structures: the Provincial Development Council (CODEDE), the Municipality Development Council (COMUDE) and finally to the Community Development Council (COCODE). Social workers introduced and explained the program to the COCODES during their traditional community assemblies with the participation of all community members. The COCODEs had the authority Page 23 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) to grant or deny approval for the introduction of the program in the community. If the COCODE agreed to introduce the program, volunteer Madres Guías were recruited or appointed by indigenous leaders during the assemblies. o Program implementers were from the local communities. The implementing agency, ChildFund Guatemala, partnered with two local non-governmental organizations (NGOs) to run the program on the ground: the Association of Integral Development Cooperation of Huehuetenango (ACODIHUE) and the Cooperation for Western Rural Development (CEDRO). These two NGOs had presence and extensive work in the program communities. ACODIHUE and CEDRO hired field workers that were from the communities and spoke the local languages Mam and K’iche’ to interact with the mothers and children. Madres Guías were at the heart of program implementation, as they were in charge of teaching and executing the two ECD program modalities. The Madres Guías were trained in the two ECD modalities by the social workers, so they could then implement the program with the mothers. The trainings and implementation of the curricula were done in Mam and K’iche’. o ECD curricula and training materials were socio-culturally adapted. The Reach Up methodology, created by the University of West Indies for an urban and mestizo context, was adapted to the Guatemalan rural indigenous setting. The curricula were edited to incorporate illustrations depicting Mayan people, local songs and native animals. Toys were also recreated to emulate familiar objects made of local recycled materials. The guidelines were simplified and included more illustrations, rather than text so they could be more easily understood by Madres Guías with low education levels13, including some who were illiterate. One problem was that the guidelines were only in Spanish, which meant that the Madres Guías had to simultaneously translate from Spanish to Mam or K’iche’ when conducting the activities. In addition to the curricula, the Barrilete tool for monitoring child development had to be adapted so Madres Guías and mothers would understand the goals they were working toward. o Monitoring and evaluation tools were also adjusted with support from experts from the University of Missouri. Tools for the impact evaluation also had to be adapted to the indigenous rural context. During the data collection for the baseline, the evaluation team initially tried to adapt the Ages and Stages Questionnaires-Inventories (ASQ-I) as a technique to collect information on children’s developmental stage. Focus groups were conducted to adapt and validate the questionnaires. However, after much testing the team concluded that the ASQ-I was not a cultural relevant tool to collect information from a rural, indigenous population, with high incidence of poverty, malnutrition and low levels of education. A quest for a non-western tool that would work in such settings was found in The Malawi Developmental Assessment Tool (MDAT). The MDAT was developed as a culturally appropriate tool for rural Malawi to reliable identify children with delayed development and neurodisabilities. The evaluation team conducted two pilots to culturally and linguistically adapt the MDAT tool to the indigenous Guatemalan setting (including translation to Man and K’iche’). After intense training and piloting of the evaluation firms, the MDAT was used to conduct the baseline and endline of the impact evaluation. 47. Continuous training was paramount to the implementation of NNLS and generation of local capacities. The social workers were trained directly from the experts in the two ECD modalities being implemented. 13 According to the systematization document, 17 percent of Madres Guías never attended school, 34 percent did not finish elementary school and only 22 percent finished Elementary school. Page 24 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) ChildFund trained and certified social workers in the Ventana de la Vida technique, and staff from the Jamaican University of West Indies, trained and certified them in the Reach Up program. The training consisted of four 10-day workshops and the grade point average to get certified was 80 out of 100. The training included the curricula but also on the monitoring of child development using the adapted Barrilete tool. The trainings included interactive sessions, demonstrations and dramatizations so they could replicate them in the trainings for Madres Guías. Subsequently, social workers trained Madres Guías using andragogical techniques, including demonstrations of the concepts and encouraging active participation. Concepts and activities were practiced over and over through interactive exercises and plays until the skills were mastered. Madres Guías were also trained to facilitate workshops, since they were the ones to apply the curricula directly with mothers and children. Madres Guías continued to receive feedback from social workers during the implementation of the program. Trainings were repeated as some Madres Guías quit and new ones came onboard. 48. The commitment of program staff (ChildFund, NGOs, and social workers) and volunteers was at the heart of program success. ChildFund has a long trajectory implementing Ventana de la Vida which gives them the necessary experience to work in the Guatemalan indigenous context. Its staff is knowledgeable, committed and show passion for the work. The two NGOs, CEDRO and ACODIHUE, are grassroots local organizations with ample knowledge of the field. Their staff, including the social workers were members of the communities and spoke the indigenous languages. This gave them an advantage to introduce the program in a cultural appropriate manner with a truly understanding of the ground. It is also important to understand that this program was mostly run by volunteers. At the center of program implementation were the Madres Guías, who voluntarily donated their time and effort to learn the ECD program curricula and taught it to other mothers. They were self-selected or appointed by leaders. They worked out of conviction that they were contributing to their communities, and thus were responsible and reliable in their duties.14 Even though they did not receive a monetary remuneration, program managers provided them with other incentives such as public recognition at community events, materials to facilitate their activities and recreational field trips twice a year as an opportunity to get together with other Madres Guías. 49. The perseverance and commitment of social workers were also key for success. Social workers played a central role since they were in charge of training and supervising Madres Guías as well as monitoring children’s development. They had to overcome many challenges such as sensitizing the community leaders from the COCODEs to obtain approval for the project and building Madres Guías’ capacities. They struggled with invisible barriers, with existing traditions and paradigms for child bearing, as well as machismo at the household and community levels. They had to manage frustration, solve problems and be creative to help Madres Guías to learn despite their low schooling. They needed to be consistent and constant with the methodology and know how to provide appropriate feedback to the Madres Guías. They had to be dynamic and inspiring to motivate and bring the application of the curricula to life. One legacy of the NNLS program, therefore, is the skills and capacities generated in social workers and Madres Guías. 50. Respecting the governance of the indigenous communities helped legitimize program activities in the communities. The COCODEs played a key role during the life of the project, as they granted approval to initiate 14When women were asked about their motivation to be a Madre Guía, they responded: they enjoy seeing children’s development, they gain the respect of the community, other mothers and children, and their own children. They like to support first time mothers, they like to share information in the CEOs, it makes them happy to work with children and to teach other women. Page 25 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) program activities, mediated grievances, facilitated communications during community fairs, provided recognition to Madres Guías and participated in closing activities. Factors outside implementing agency’s control 51. The time needed to adapt curricula and impact evaluation tools was underestimated. During early program stages, a second treatment arm, the Reach Up Early Childhood Parenting Program, was introduced. Reach Up is an internationally recognized and well-evaluated program from Jamaica. ChildFund decided to pilot it, as it would be a good opportunity to test the impact and cost efficiency of both programs. However, introducing Reach Up involved training the implementation agency in the new methodology, and adapting its materials to Guatemala’s indigenous context. In addition, the surveys and monitoring tools had to be translated, contextualized and tested, all of which took longer than expected. 52. To gain acceptance program staff had to overcome barriers including changing the community “mindset.� The expectation of beneficiary communities was to receive a transfer or infrastructure project, as is the case with traditional government programs. People were not familiar with programs that required them to organize and learn without any apparent economic compensation. In addition, the program had to compete with domestic and economic activities that mothers may have considered as “more productive� than playing with their children. Many women were skeptical at the beginning until they started to notice the changes the program was generating in their children. 53. The existence of a macho culture and patriarchal traditions was an obstacle in some communities. Program participants expressed that this issue was more prevalent in the Mam communities, where community leaders and husbands decided if the women could participate in the program. Some COCODEs would not let the program start in their communities. Even if the program was available in the communities, some husbands would not let their wives participate. 54. There was attrition of program participants due to agricultural cycles and migration. In Huehuetenango in particular, families temporarily migrate each year for about six months to participate in agricultural activities in other regions. There is also a lot of permanent migration to the United States. Even if families did not migrate, the number of absences in the CEOs increased during the harvest season. Factors under Bank’s control 55. The ability to be flexible, incorporate feedback and learn from mistakes kept the program operating smoothly. Day to day operations benefited from constant feedback and adaptation. Processes were adjusted along the way to improve implementation. Examples of changes introduced are: o Hiring social workers to focus exclusively on interinstitutional relations, as the additional task of cultivating relations with different Ministries and local authorities, which had not been envisaged during the project’s design but the need for which became evident during implementation, proved to be too much for the social workers focusing on the training and supervision of Madres Guías.15 15The social workers for institutional relations coordinated all the inter-institutional activities. They created and nurtured relations at the departmental (provincial) and municipal level with community leaders, municipalities and central government agencies and ministries such as SESAN and MSPAS. They also provided support to the social workers in the field working with Madres Guías and produced educational materials. Page 26 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) o Adapting the numbers and distribution of families assigned to Madres Guías. Instead of assigning equivalent numbers per person, ranges were established to allow flexibility to assign families based of geographical dispersion. o A package of non-monetary incentive was provided to Madres Guías o Given that the Reach Up modality did not include a community meeting, it was decided to organize meetings every three months to gather groups of three Madres Guías and their beneficiary mothers and children to share experiences. o For the Ventana de la Vida modality, another volunteer mother was assigned to the Madre Guía to entertain the older siblings that also attended the CEOs. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION Rating: High 56. Design. The project design included monitoring and evaluation and knowledge dissemination through: (i) the carrying out of monthly monitoring and analysis of Project progress and status of child development and behaviors; (ii) financing process evaluations; (iii) the elaboration of an experimental impact evaluation design and the carrying out of baseline and follow-up surveys using a representative sample of young children in treatment and control communities; and (iv) the dissemination of lessons learned from the Project process evaluation and evidence of impact among stakeholders and government decision-makers to encourage sustainability and scale-up.16 57. Implementation. The Bank team carried out regular field visits to Guatemala City and to the communities where the two programs were being implemented. Regular missions, which always included field visits, were critical to providing timely technical feedback to ChildFund, which allowed them to implement corrective measures throughout implementation. Examples of how feedback affected implementation include how both modalities were implemented, adjustment of M&E tools, implementation of the community fairs and changes to their content (messages to be included, adding Wash, Sanitation and Hygiene (WASH), etc.), community validation and implementation of grievance mechanisms, among others. 58. The findings from these missions led to two restructurings. In addition, an ICR mission was included in the last supervision mission before the Grant’s closing. Most critically, the impact evaluation included in the Project’s design was completed as envisaged, and its preliminary results are shared in this report. 59. Utilization. As a result of implementation support missions, process evaluations, and ChildFund’s progress reports, adjustments were made to program design during implementation. Examples of such adjustments include the introduction of the Reach Up modality; the revision of Reach Up’s curriculum to adapt to the 16Through LEGO, the Reach Up and Ventana de la Vida modalities are being reviewed, and one intervention being considered merges both (home visits plus group sessions). There are also discussions with MINEDUC to utilize the model, as this has already been tested for these areas in Guatemala. Page 27 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Guatemalan context; the adaptation of the IPP; the introduction of checklists for social workers to measure how well Madres Guías were performing; increasing the frequency of encounters between social workers and Madres Guías; standardizing a supervision mechanism for social workers; and changes to the Barrilete, among others. 60. Justification of overall M&E rating. The M&E rating is High, given that the design was appropriate, the work was carried out and is of good quality, and results of the work are being used to inform other ECD initiatives. B. ENVIRONMENTAL, SOCIAL, FIDUCIARY COMPLIANCE 61. Because most beneficiaries were indigenous, the Indigenous Peoples (OP 4.10) safeguard should not have been triggered, but instead the project should have been considered an indigenous project. However, project documents indicate that the preparation team did trigger it and therefore an IPP was prepared. The IPP was supervised and monitored by the Bank team throughout implementation. The IPP helped ensure that indigenous beneficiaries’ cultural practices and societal structures were carefully considered at every step. As previously stated, program material was adapted to fit the Guatemalan context, and two local non-profit organizations were hired to help gain access to these traditional communities. There were no environmental impacts resulting from the project. Regarding fiduciary compliance, while the implementing agency initially struggled to follow Bank policies and practices, by the end performance was satisfactory. C. BANK PERFORMANCE Quality at entry: Moderately Satisfactory 62. The Grant design was based on international best practices, and carefully considered how it would fit within other Guatemalan initiatives to improve nutrition and overall development of poor children. However, as noted earlier, PDO indicator targets were too ambitious, which was evident once the adapted Barrilete instrument and other measurement tools were introduced. In addition, the time to adapt the ECD curricula and assessments tools to the indigenous Guatemalan context was underestimated, leading to implementation delays. A well-respected NGO that had deep experience in indigenous communities, and that knew and respected local customs and practices, was selected to implement the grant. However, the fact that ChildFund had never implemented a World Bank project should have signaled the need for intense training and supervision during preparation and initial stages of Grant execution. Instead, implementation took some time to take off as ChildFund climbed the learning curve. In addition, because the team had not learned about Guatemala’s labor laws, it took more time to hire the Program Coordinator, also affecting the initial implementation phase. Had the Bank team learned of the particularities of Guatemala’s labor laws, it may have also avoided the halt to implementation that took place while the issue of social workers’ payment was resolved. The Bank team, however, successfully obtained a waiver to pay for social workers’ salaries on an exceptional basis, readjusted the Grant through a comprehensive restructure, and provided needed training and support to get ChildFund up to speed and the pilot project on track. Because of these initial issues at entry, quality is rated as MS. Page 28 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Quality during implementation: Satisfactory 63. The Bank team carried out regular supervision missions and worked diligently, in between those missions, to resolve implementation bottlenecks identified during those visits . One example is the restructuring of the Grant to drop nutrition activities under Component 2: when it became clear that ChildFund did not have the capacity to deliver high quality instruction in this area in addition to early childhood stimulation activities within a limited timeframe, the Bank team added a nutritional component at the community fairs, and counted on Crecer Sano activities to salvage the nutrition part of the community interventions. Another example was the issue of ineligible expenditures caused by the hiring of social workers as regular employees, instead of consultants. In 2017, the FM specialist noted that the social workers hired by the local organizations for carrying out subprojects were appointed under a permanent employment contract instead of under consultant services. Initially this was considered to be against the operational manual and in violation of the sub-grant agreement signed with Child Fund. ChildFund explained that this arrangement was made to comply with Guatemala’s Labor Law, which stated that when a person is hired for more than a year, is provided with a physical work space, tools and equipment to perform a job, and reports to a supervisor, that person is automatically considered an employee (Implementation Status and Results, December 2017). The team sought approval from the donor (Government of Japan) for these exceptional expenditures, based on the Bank team’s argument that social workers were key to the achievement of the PDO and did not contradict overall Bank guidelines on eligibility of expenditures. The Japanese government approved this request in January 2018. The ability to make practical, rational, and timely decisions in these two cases, showcases the good judgment and flexibility of the Bank team which was key to the successful outcome of the Grant. 64. Most importantly, the success of the two modalities helped ChildFund to win a grant from the Lego Foundation to continue work on early stimulation interventions. The Lego grant is considering merging the two modalities (home visits and group sessions) and has required a Memorandum of Understanding (MOU) to be signed between ChildFund and the Ministry of Education, which should strengthen ties between the NGO and central government interventions. The Crecer Sano project and MIDES are also potential spaces to try to secure continuity and sustainability, although when new authorities are in place will there be progress in this area. V. LESSONS LEARNED AND RECOMMENDATIONS 65. By generating local capacities, the NNLS empowered people, particularly women, to be the managers of their own development, which bodes well for the sustainability of the initiative. Traditionally indigenous communities are presented with top down models, reminiscent of colonial dominance and that are referred by indigenous people as mere assistance, where they are only passive recipients of social handouts. NNLS was not an extension program either, where an external organizer come to the communities and implements without generating capacities. The NNLS had mothers carrying out the program and teaching their peers, which at the end resulted in increased capacities, self-esteem and determination, increasing the social capital of these communities, using as a common narrative the wellbeing of their children. Page 29 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) 66. To work with indigenous communities, it is critical to respect their societal structures and to hire social workers from the community to ensure smooth implementation. Because these communities have a long history of being marginalized and because of years of societal unrest, the only way to gain entry to them is to work with organizations who the indigenous trust. The two NGOs hired by ChildFund met these criteria, and each understood the indigenous governance structure. This was key to knowing with whom to meet and establish agreements, and even with that, there were a few communities that refused to participate in the pilot. In addition, it was crucial to hire local social workers that spoke the Mayan languages Mam or K’iche’ because they had to train the Madres Guías and being able to interact with beneficiary mothers and children to monitor child development. Some children or mothers did not speak Spanish, and some concepts do not translate directly between Spanish and the Mayan languages, so the social workers needed to interpret and explain the concepts more accurately in the Mayan language. There were also variations between the same Mam language from different locations, so the social workers needed to be familiar with these differences. Without this language mediation of the social workers, the program would not have had the same degree of success. 67. Stipends to cover costs incurred by the Madres Guías should be considered, as having their performance based on altruism and personal commitment may not be sustainable in the long run. Madres Guías spent time training, preparing the program and walking long distances to do house visits, particularly in the Reach Up modality. Any change in their life may alter their availability to volunteer. This issue came up in all interviews conducted with Madres Guías during the ICR research. In each interviewed group, there was at least one Madre Guía who expressed that they should receive an in-kind or monetary compensation. To this, program managers responded that if Madres Guías were to be paid, that would change the incentives for which the work is done. However, Madres Guías and their families live in such poverty and have competing responsibilities that it would make sense to provide some amount of compensation for their volunteer work. 68. Implementing pilots, and incorporating feedback mechanisms is critical to incorporating lessons learned in a timely manner. Doing so helped NNLS adjust and improve implementation during the pilot. Since the ECD program curricula was new, program implementors had to pilot and test not only the contextualization of methodologies and monitoring tools, but also had to learn and adjust during the implementation process. Program managers therefore included feedback mechanisms into implementation. Periodic revision meetings were conducted, and social workers and Madres Guías were given voice to suggest and make changes. A consultation process for beneficiaries was established at the community fairs and a grievances redress mechanism was established. In terms of the impact evaluation, the piloting of the ASQ-I helped determine that this was not an appropriate tool to be implemented in rural indigenous contexts, which to the adoption of the MDAT assessment instead. Feedback and flexibility resulted in dynamic operational processes that improved Grant execution. 69. When JSDF grants are executed by NGOs/CSOs, particular attention should be given to building procurement and financial capacity within these organizations. ChildFund was inexperienced with World Bank financial systems as most NGOs/CSOs would be since they are not the traditional implementing agencies of World Bank projects. The Bank should implement an upfront fiduciary assessment and develop a formal FM/procurement immersion program to build capacity JSDF grants’ recipients. Some hand-holding and accompaniment should be provided until the grant implementors are comfortable with the procedures. In the Page 30 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) case of ChildFund, it made a difference to hire a procurement specialist with significant experience on World Bank’s FM policies and procedures. 70. It is important that World Bank teams review national labor regulations before any project to guarantee that Bank FM policies do not supersede them. The Guatemalan Law did not allow contracting the social workers as consultants. By not realizing this before going into implementation, the Bank spent time and energy resolving the issue, and requesting an exception from the Grant donor. 71. The Bank should standardize FM and procurement policies for all its trust funds, so they do not conflict. Under the WB’s Investment Project Financing (IPF), contracting social workers as permanent staff would not had been an issue. The issue was with JSDF financial policies. Putting these FM policies in line would have avoided the violation of grant policies. 72. In low capacity environments, the Bank should take responsibility for impact evaluations to ensure independence of assessments and also quality. In the case of Guatemala, ChildFund contracted the survey firm for the baseline, but when there were problems with the quality of the firm’s work, it was difficult for the Bank to work directly with the firm to resolve those problems. The Bank could not assume a supervisory role and had to rely on ChildFund to interact with the firm. The Bank team addressed this issue by directly hiring a different firm to collect the endline data, once the baseline firm had completed its work. Fortunately, the Bank had piloted and adapted the assessment tools before the firms were contracted, so the quality of the data was not compromised. 73. How the survey firm reaches out to the households could affect the future engagement for the program. In a few cases, the survey firm collecting the baseline told households that the pilot program would bring other services to them, which was never part of the project design. This led to a rejection of the program from some families/communities, and it could have ramifications if future work is to be carried out in those areas. 74. Close collaboration between the operational and research teams is key for allowing flexibility in the design of the evaluation to respond to realities in the field. For example, the adaptation of the instruments and the preparation of the survey firm were taking longer than expected, but the program needed to start the intervention. The research team therefore reviewed the list of communities and separated those that could be used for the first phase, and those that were going to be part of the impact evaluation. This allowed the operational team to start the implementation in areas outside the study without the need to wait for the baseline completion. 75. Given the high rates of stunting and malnutrition in the country, the World Bank should take advantage of the pilot experience to disseminate the lessons learned and to show case it internationally as a reference for other Latin American countries with indigenous populations. The Grant generated capacities in psychometrics and in the implementation of culturally appropriate ECD curricula by people from the same indigenous communities. With NNLS, Guatemala became a pioneer in the socio-cultural adaptation of the internationally recognized Reach Up and MDAT tools to a rural indigenous context. The program spent significant time and resources adapting Reach Up, the respected early childhood parenting program from Jamaica, and MDAT, the alternative assessment tool for childhood development in rural Africa. Other World Page 31 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Bank-funded projects such a Crecer Sano currently under first stages of implementation in Guatemala, should absorb the majority of these lessons and material produced, to also ensure sustainability of this experience. . Page 32 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase community participation in monitoring child development indicators Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of children Percentage 0.00 70.00 60.00 64.00 achieving the expected development level for their age 18-Mar-2015 07-Nov-2018 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): Baseline: No data available at that time. Objective/Outcome: Improve caretakers' interactions with young children to enhance their physical, cognitive and emotional development outcomes Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of participating Percentage 0.00 70.00 40.00 32.00 households with enhanced early stimulation practices 18-Mar-2015 07-Nov-2018 30-Jun-2019 30-Mar-2019 Page 33 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Comments (achievements against targets): A new measurement tool was added to better reflect this outcome; target updated. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of children that are Number 0.00 5500.00 5500.00 4506.00 direct beneficiaries of the Project 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): This is a new indicator included as part of the restructuring. A.2 Intermediate Results Indicators Component: Component 2. Enhanced social and behavior change communication to achieve improved child nutrition and development Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of community fairs Number 0.00 188.00 188.00 230.00 implemented with an ECD approach 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): Page 34 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Component: Component 3. Project management and administration. Monitoring and evaluation, and knowledge dissemination Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Baseline survey conducted Text No Yes Yes Yes 20-Oct-2015 01-Jun-2016 01-Jun-2016 01-Mar-2017 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Quarterly progress reports Number 0.00 16.00 16.00 16.00 completed 18-Mar-2015 07-Nov-2018 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Mass events hosted for Number 0.00 16.00 2.00 2.00 departmental health systems 18-Mar-2015 07-Nov-2018 30-Jun-2019 30-Mar-2019 Page 35 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Comments (achievements against targets): activities were scaled down as part of restructuring Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Final Impact Evaluation survey Text No Yes Yes No conducted 18-Mar-2015 07-Nov-2018 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): Component: Component 1: Promotion of physical, cognitive, linguistic and socio-emotional development through a parenting and early stimulation intervention Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Diagnostic of early childhood Text No Yes Yes Yes development beliefs and practices in beneficiary 18-Mar-2015 30-Nov-2015 30-Nov-2015 15-Jun-2018 communities is completed Comments (achievements against targets): Indicator Name Unit of Baseline Original Target Formally Revised Actual Achieved at Page 36 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Measure Target Completion Monitoring tool to track child Text No Yes Yes Yes development is developed 18-Mar-2015 01-Apr-2016 01-Apr-2016 15-Jun-2018 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of project Number 0.00 100.00 100.00 94.00 communities implementing Level 1 (Ventana de la Vida) 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 and Level 2 (Reach Up) activities Comments (achievements against targets): This is a new intermediate indicator included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of COCODES Percentage 0.00 70.00 70.00 76.00 (Consejos comunitarios de Desarrollo) implementing 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 activities in favor of childhood Page 37 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) in their communities Comments (achievements against targets): Baseline: Data not available at that time. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of Madres Guias Number 0.00 595.00 595.00 371.00 trained to apply level 1 methodology (Ventana de la 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Vida) Comments (achievements against targets): This new intermediate results indicator was included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of Madres Guias Number 0.00 450.00 450.00 523.00 trained to apply level 2 methodology (Reach Up) 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): This new intermediate results indicator was included in the restructuring. Page 38 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of children Number 0.00 4480.00 4480.00 2833.00 participating in level 1 activities (Ventana de la Vida) within 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 CEO's strategy Comments (achievements against targets): This new intermediate result indicator was included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of children Number 0.00 1920.00 1920.00 1673.00 participating in level 2 activities (Reach Up) 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): This new intermediate results indicator was included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 39 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Percentage of Madres Guias Percentage 0.00 70.00 70.00 47.00 with quality implementation of level 1 and level 2 activities 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): This new intermediate results indicator was included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of CEOs implemented Number 0.00 80.00 80.00 79.00 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 Comments (achievements against targets): Unit was changed to Percentage at final ISR dated June 25, 2019. This new intermediate results indicator was included in the restructuring. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of midwives in the Number 0.00 900.00 900.00 1473.00 catchment areas of the project, trained in exclusive 18-Mar-2015 30-Jun-2019 30-Jun-2019 30-Mar-2019 breastfeeding and oportune stimulation by the social workers Page 40 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Comments (achievements against targets): This new intermediate results indicator was included in the restructuring. Page 41 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1: To strengthen the capacity of parents and communities to improve and monitor child development outcomes (physical, cognitive, socio-emotional, and linguistic skills) for children under two in Project intervention areas. 1. Percentage of children achieving the expected development level for their age. End Target: 60 % (revised from original target of 70%) 2. Percentage of participant households with enhanced early Outcome Indicators stimulation practices. End Target: 40% (revised from original target of 70%) 3. Number of children that are direct beneficiaries of the Project. End Target: 5,500 children Component 1: Promotion of physical, cognitive, linguistic and socio-emotional development through a parenting and early stimulation intervention IRI_1: Diagnostic of early childhood development beliefs and practices in beneficiary communities is completed IRI_2: Monitoring tool to track child development is developed Intermediate Results Indicators IRI_4: Number of project communities implementing Level 1 (Ventana de la Vida) and Level 2 (Reach Up) methodologies. End Target: 100 % IRI_5: Percentage of COCODES (Consejos Comunitarios de Desarrollo) implementing activities in favor of childhood in their communities End Target: 70% Page 42 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Component 2. Enhanced social and behavior change communication to achieve improved child nutrition and development IRI_3: Number of community fairs implemented with an ECD approach End Target: 188 community fairs Component 3. Project management and administration. Monitoring and evaluation, and knowledge dissemination IRI_1: Baseline survey conducted IRI_2: Quarterly progress reports completed. End Target: 16 reports IRI_3: Mass events hosted for departmental health systems. End Target: 2 events IRI_4: Final Impact Evaluation survey conducted IRI_6: Number of Madres Guías (MG) trained to apply level 1 methodology (Ventana de la Vida). End Target: 595 MG IRI_7: Number of Madres Guías trained to apply level 2 methodology (Reach Up). End Target: 450 MG IRI_8: Number of children participating in level 1 activities (Ventana de la Vida) within ‘Casas de Evaluación y Orientación- CEO's strategy. End Target: 4,480 children IRI_9: Number of children participating in level 2 activities (Reach Up). End Target: 1,920 children Page 43 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) IRI_10: Percentage of Madres Guías with quality implementation of level 1 and level 2 activities. End Target: 70 MG IRI_11: Number of CEOs implemented. End Target: 80 CEOs IRI_12: Number of midwives in project catchment areas trained in exclusive breastfeeding and opportune stimulation by the social workers. End Target: 900 midwives 1. Key Outputs by Component 2. (linked to the achievement of the Objective/Outcome 1) 3. 4. Page 44 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) . ANNEX 2. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 1. ChildFund provided comments to the draft ICR in December 2019. These comments were fully incorporated into the final report. Page 45 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) ANNEX 3. IMPACT EVALUATION METHODOLOGY 1. The objective of the evaluation is to measure the impact of each of the implementation modalities (home visits and group meetings) on children's development and parents' behavior. More specifically, the evaluation seeks to answer the following questions: 1. Does the program improve children's cognitive development (problem solving, communication, and fine motor skills)? Does the program improve nutritional indicators? 2. Does the program help to promote changes in perceptions, behaviors in parents that lead to improved early childhood stimulation in the home? Does the program contribute to reducing mothers' levels of depression and strengthening their empowerment? 3. Which modality of the program (home visits or group sessions) is most cost-effective? 2. The evaluation seeks to capture the causal effects of the intervention through an experimental design based on a randomized controlled trial (RCT). 115 communities located in the departments of Huehuetenango, San Marcos, Quiche and Totonicapán in which the program had the capacity to operate, were randomized into three groups: - Treatment 1: Home visits (38 communities) - Treatment 2: Group meetings (38 communities) - Control: No child stimulation program (39 communities) Baseline Survey 3. Within each community, approximately 20 children under 12 months, pregnant women plus a maximum of 5 replacements were selected to participate in the baseline survey using an ad hoc census collected prior to the baseline survey and also information from the community. 4. The baseline was collected between October 2016 and January 2017. The final sample collected includes 2,022 households and 2,022 children up to 24 months in 113 communities. End-line survey 5. The endline survey was collected between February and May 2019 on 113 communities and 2,022 households. Analysis 6. Given that not all households in the treatment group received NNSL, the impact of the program is measured based on intention-to-treat (ITT) effects to avoid a selection bias in the comparison between treatment and control groups. Treatment on the treated was estimated to reflect the impact on those that effectively received the treatment (TOT). 7. The study will estimate the ITT and the TOT effects. Page 46 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Intent-to-Treat (ITT) Analysis Ordinary least squares (OLS) models were ran to examine the effects of random assignment to Reach Up or home visits (HV) and Ventana de la Vida or group meetings (GM). 𝑦𝑖� =𝛼0 + 𝛼1 𝐻𝑉𝑖� + 𝛼2 𝐺𝑀𝑖� + 𝛼3 𝑋𝑖� + 𝛼4 𝛾� + 𝜖𝑖� Treatment on the Treated (ToT) Analysis To understand the effects of “participation in HV� and “participation in GM�, the lottery assignment was used as an instrument �𝑎𝑟𝑡𝐻𝑉𝑖� = 𝛽0 + 𝛽1 𝐻𝑉𝑖� + 𝛽2 𝑋𝑖� + 𝛽4 𝛾� + 𝜇𝑖� �𝑎𝑟𝑡𝐺𝑀𝑖� = 𝛿0 + 𝛿1 𝐺𝑀𝑖� + 𝛿2 𝑋𝑖� + 𝛿4 𝛾� + 𝜋𝑖� ̂ 𝑖� + 𝛼3 𝑋𝑖� + 𝛼4 𝛾� + 𝜖𝑖� ̂ 𝑖� + 𝛼2 �𝑎𝑟𝑡𝐺𝑀 𝑦𝑖� =𝛼0 + 𝛼1 �𝑎𝑟𝑡𝐻𝑉 8. The team also explored the distance to the center of the community as that could have affected the probability of participating in the program. Selected Instruments 1) Child development: Malawi Developmental Assessment Tool (endline), Age and Stages Inventory (baseline). 2) Instruments for measuring early childhood stimulation in the home. a. Short version of "Home Observation for the Measurement of the Environment (HOME-SF). b. Family Care Indicator (FCI) 3) Maternal Knowledge on child development: KIDI 4) Maternal Depression: Center for Epidemiologic Studies Depression Scale-Revised (CESD-R). Validation of the methodology 9. Balance among groups were exhaustively tested to validate the randomization process (106 variables), see figure 5.1. The team tested 22 variables for family characteristics, 13 child characteristics (age, gender, language, premature, birthweight, breastfeeding), 20 economic indicators (employment, hours worked, salary), 17 variables for nutrition (intake of proteins, dairy, fruits, cereals, vegetables, sugars, etc.), 21 intermediate outcomes (all subtests of home, fci, maternal depression, and time spent with children), 13 outcome variables (weight for age, height for age, stunting, extreme stunting, underweight, fine motor, problem solving, communications). Page 47 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Figure 5.1. Baseline Balance 100% 80% 60% 40% 20% 0% Child is low Maternal Percentage in Father is Caregiver can Protein Stunting birth weight depression poverty employed read intake Control Group Ventana de la Vida Reach Up 60 50 40 30 20 10 0 Communications Fine Motor Problem solving Control Group Ventana de la Vida Reach Up Note: Outcome variables are reported as raw scores. Source: Trias, Julieta and Irma Arteaga (2020) “Home visitation or group intervention? Effects of early stimulation on child wellbeing in rural Guatemala: A cluster randomize control trial� World Bank mimeo Page 48 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) ANNEX 4. DESCRIPTION OF THE NUESTROS NIÑOS SANOS Y LISTOS (NNSL) PILOT PROJECT 1. Nuestros Niños Sanos y Listos (NNSL) is a pilot project implemented by ChildFund Guatemala together with CEDRO and ACODIHUE, two local NGOs with extensive work experience in the areas of intervention. The project’s main objective was to strengthen the capacity of parents and communities to improve and monitor child development outcomes (physical, cognitive, socio-emotional, and linguistic skills) for children under two in intervention areas. The Project had two intervention modalities: 2. Ventana de la Vida (Group intervention) The methodology has been developed and implemented by ChildFund for about a decade. It consists of group meetings held twice a month in which a Madre Guía would train between 10 to 15 mothers in early childhood stimulation. The meetings took place at a community center or at the house of the Madre Guía and lasted for about an hour. The meetings began with a welcoming message and a song, introduction of the topic by the Madre Guía, demonstration of the activities, games and exercises. The mothers practiced the activities with the children and the meeting ended with a group discussion summarizing a couple of key messages. The mothers committed to practice, at home, the activities they learned at the group meetings. Educational materials and toys were made available at each session. The meetings venues were called Casas de Evaluación and Orientación (Houses for assessment and orientation), or CEOs. 3. Reach Up (Individual home visits) Reach Up is an internationally recognized early childhood parenting program developed by Jamaica’s University of the West Indies and backed by over 30 years of research. The program consists of forty-five minute home visits, in which a Madre Guía shows mothers simple ways of interacting with their children using recycled toys, books, songs and personal interactions. At each visit the Madre Guía introduces a topic, demonstrates the activities and explains the objectives of the toys and how to use them. The Madre Guía observed the caregiver practicing with the child and provided feedback, the visit typically ended with a song. The toys were left at home until they were replaced with different toys at the next session, since they were topic- and age-specific. In order to have a group experience, this modality combined the home visits with a group session every 3 months so mothers could share and discuss what they had learned in the program. 4. Both modalities include agendas or guides for the Madres Guías to deliver the topics at each session. The content and presentation of these materials were adapted to the indigenous Guatemalan context. Assessment Tool 5. To monitor the development of the children, both modalities used an assessment tool called “ El Barrilete� (the Kite). The tool is designed to monitor the development of children 0-36 months in the following five areas: gross motor skills, fine motor skills, language and communication skills, socio-emotional skills and cognitive skills (see figure 6.1). Each of these areas represented a side of the kite, and an analogy was made that if one of the sides of the kite was not functioning properly, the kite would not fly well. Similarly, a child needed to develop all the five areas in order to thrive. The Barrilete has eight phases of evaluation according to the age of the child: 0-3 months; 3-6 months; 6-9 months; 9-12 months; 12-16 months; 16-20 months; 20- 24 months and 24-36 months. Page 49 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) 6. The Barrilete used a traffic light color code to assess the child: red if the child could not do any of the activities in the five areas assessed; yellow if the child could do one or two activities in each of the areas; and green if the child could do all the activities in the five areas being evaluated. The Barrilete was implemented by the Madres Guías and social workers. Based on the results, they would offer the parents feedback to strengthen the areas that needed more development. In addition to the Barrilete, the Madres Guías and social workers also measured weight and height of the child and checked the vaccination schedule. Figure 6.1. Barrilete Assessment Tool (0-3 Months) Targeting and Eligibility Criteria 7. NNLS was implemented in 100 communities in the Departments of Huehuetenango, Quiché, San Marcos and Totonicapán (see figure 6.2). These Departments have the highest levels of malnutrition (more than 70 percent) and poverty (80 percent) in the country according to the ENSI. Within these Departments, municipalities with the highest level of chronic malnutrition were chosen, which were also prioritized by the Hambre Cero (Zero Hunger) Program, a government flagship program. 8. About five to six communities were selected in each municipality considering the incidence of children under two years, geographical feasibility to facilitate supervision and absence of other nutrition and childhood development interventions. One hundred and fifty communities were selected in total. Page 50 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Figure 6.2. Geographical Distribution of the NNSL Pilot Project NNSL COVERAGE BY DEPARTMENTS Quiche Totonicapán San Marcos Huehuetenango 11% 15% 49% 25% Source: NNSL Infographics, ChildFund 2019 Beneficiary profile 9. In total 4,161 children benefited from the project (51 percent boys and 49 percent girls). The caretakers that participated in the program were almost exclusively women (99 percent), with their average age ranging from 28 to 31 years old depending on the Department. About 95 percent of the beneficiaries were Mayan from the K´iché (62 percent) and Mam (33 percent) ethnic groups. The level of education of the caretakers was low. About 20 percent of the beneficiary caretakers did not have any schooling, while 43 percent did not finish elementary school. About 77 percent of the caretakers were stay-at-home mothers while 9 percent worked in the informal sector. Page 51 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) 10. The program trained 965 Madres Guías but only 849 were certified. About 98 percent of them were Mayan also from the K´iché and Mam ethnic groups. The average age of the Madres Guías was about 32 years. About 17 percent of them were illiterate and 34 percent did not finish elementary school. Most Madres Guías were stay-at-home mothers and about 90 percent did not receive any other social program. To get certified, Madres Guías received six days of four-hour sessions. The trainings were conducted in the local language and were adapted to the context and low level of education of the Madres Guías, using andragogic techniques. Other Program Areas Community Fairs 11. The community fairs were implemented to sensitize and train the broader community on issues of childhood development. During the fairs, community members and leaders who were not receiving any of the two modalities of the NNSL program could participate. The fairs were conducted every two to three months and were structured in “Centers�, or stands, featuring four main areas: 1. My world of toys. This center focused on sensitizing about the importance of playing to stimulate children’s brain development. This center usually included an activity to create a toy out of recycled materials. 2. Food of love -breastfeed, give life. This center emphasized the importance of exclusive breast feeding during the first six months of life, including the benefits of the colostrum. This center would train mothers on how to effectively breastfeed. 3. Water, sanitation and adequate food management. At this center, participants are sensitized about adequate sanitation to reduce the incidence of infectious gastrointestinal and respiratory diseases that affect children’s development. Participants were trained on sanitation best practices and effective management of waste. 4. My favorite menu. The objective of this center was to learn the nutritional value of healthy food, many of it already planted and harvested by the families. Participants learned about healthy recipes using local produce. 12. Project staff coordinated with government institutions for some of the specialized content delivery at the community fairs. Some topics were prepared in coordination with MAGA, SESAN, DMS, MSPAS, MIDES as well as local and municipal authorities. 13. Community Fairs also combined cross-cutting issues of interest for the community such as children’s rights, parenting with love and disaster risk management. The activities at the centers followed the Decroly method, a pedagogical approach developed by Belgian Ovile Decroly (1871–1932). The Decroly method organizes educational content based on centers of interest and educative games. The centers delivered the content based on three activities: 1) observation: the participant needs to have direct contact with object and situations; 2) association: the participant should relate the space and content with his/her own situations. S/he should be able to identify a cause and effect; and 3) expression: the participant should execute an activity such as reading, writing, drawing, estimating or working on a craft. Page 52 of 53 The World Bank Pilot to Improve the Development and Nutrition of Young Children in Poor Rural Areas in Guatemala (P145410) Training to Midwives 14. The NNLS also coordinated with health districts to train midwives as they are the first line of maternal health care providers in the communities of intervention. Every three months, one-hour trainings were provided to about 50 midwifes on pre-natal and early childhood stimulation. The project trained 611 in total midwives. Consultation process and grievance mechanisms 15. As established in the IPP, the NNSL developed a community consultation process, on program performance. At the community fairs, along with the traditional four centers, a consultation center was added in which community members were asked to provide feedback about the program. The consultations were conducted in the local language and asked three specific questions: what has been good about the NNLS project? What has been bad? How can the project be improved? The questions had to be answered from the perspective of the children, the families and the community. Madres Guías also held their own consultation process during their evaluation or “strengthening� meetings. 16. A grievance mechanism was also established and communicated to the community during the consultation process. This mechanism included phone numbers of technical staff who could be contacted to submit complaints. There were also channels to scale complaints to NNSL leadership and community authorities if necessary. Page 53 of 53