Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) Report Number: ICRR0021885 1. Project Data Project ID Project Name P143915 Safety Net and Skills Development Country Practice Area(Lead) South Sudan Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-52840 31-Dec-2017 19,353,760.64 Bank Approval Date Closing Date (Actual) 21-Jun-2013 28-Feb-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 21,000,000.00 0.00 Revised Commitment 21,000,000.00 0.00 Actual 19,353,760.64 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (page 5), the project objective was to provide access to income opportunities and temporary employment to the poor and vulnerable and put in place building blocks for a social protection system. The Project Appraisal Document (PAD, page 5) and the Implementation Completion and Results Report (ICR, page 6) stated the same project development objectives (PDO). Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) As a key project activity (job skills development for youth) and corresponding key indicator were dropped, reflecting a reduction in the scope of the project, the project is subject to a split ratings assessment. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components 1. Social Protection System and Project Management (Appraisal: US$ 5.5 million; Actual: US$ 2.7 million): This component aimed to develop basic features of a social protection system and build Government capacity to implement the proposed project. Activities included: technical assistance with the drafting of a social protection policy; development of basic operational tools including a targeting mechanism to identify potential beneficiaries, beneficiaries registry, an integrated Management Information System (MIS) to monitor progress made by programs at the beneficiary level, a payment transfer and accountability system intended to ensure secure, predictable and timely payments to beneficiaries, and development of M&E tools including regular assessments and Community Score Cards. Capacity building activities included: establishing a Technical Support Team (TST) at the federal, state and county levels; contracting NGO and private sector partners to implement the Public Works and Skills Development components; designing a communications campaign. 2. Public Works (Appraisal: US$ 11.0 million; Restructuring: US$ 15.5 million; Actual: US$ 16.7 million): This component aimed to provide income for youth and women from poor households through temporary/seasonal wages linked to participation in public works. The temporary work would consist of ten days/month for three months in rural counties and for four months in urban counties. These paid labor intensive public works would help build, repair, or maintain local infrastructure that would benefit the community at large. 3. Skills Development (Appraisal: US$ 3.5 million; Restructuring and Actual: US$ 0): This component aimed to to provide youth from poor and vulnerable households with access to livelihood opportunities through comprehensive skills development (technical, entrepreneurship and life skills) training. A market assessment/skills gap study and youth needs assessment study would be conducted in the six targeted counties. At the end of the training period, implementing partners would also provide support to beneficiaries to start or engage in income generating activities. An unallocated amount of US$ 1.0 million was also included at project appraisal for unexpected costs. At the time of project restructuring, Component 3 (Skills Development) was dropped, in order to rapidly scale up income-generating activities, in response to the outbreak of violence in July 2016. Funds from Component 3 and also the unallocated portion of US$1.0 million were reallocated to Component 2 (Public Works). Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The project cost at appraisal was US$ 21.0 million; the actual cost was US$ 19.4 million. Financing  The project was financed entirely by an IDA Credit of US$ 21.0 million, of which US$ 19.4 million were disbursed. The ICR did not report on whether the remaining balance was cancelled. Borrower contribution  There was no planned Borrower contribution. Dates  June 2013: The project was approved by the Bank Board.  November 2014: The project became effective.  September 2017: The project was restructured, due to changing circumstances on the ground arising from the outbreak of violence in July 2016. Component 3 (Skills Development) was dropped and funds from this dropped component and the US$1.0 million unallocated component were reallocated to Component 2 (Public Works). The target for the number of beneficiaries was increased, while the key project indicator on developing a targeting mechanism to identify youth for the skills development training was dropped. Lastly, the project closing date was extended from December 2017 to December 2018.  January 2018: The Mid-Term Review was conducted.  December 2018: The project closing date was extended from December 2018 to February 2019 to allow for completion of activities.  February 2019: The project closed. 3. Relevance of Objectives Rationale The country of South Sudan achieved its independence in 2011; since then, the post-independence period has been marked by high volatility and significant barriers to economic growth and human development. Incidences of violence and tension have continued with neighboring Sudan and amongst militia groups, driven in part by competition over natural resources. Disparities in income and access to resources, climactic shocks, food shortages, poor quality of social services and infrastructure, and lack of employment opportunities have also been significant causes of instability. Given the poverty and vulnerability of the South Sudanese population, and the continuing social, political, and economic instability faced by the country, there was an immediate need for interventions to address short-, medium- and long- Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) term challenges. There was a need to bridge the gap from the limited livelihood opportunities (both in the agricultural sector and in the private sector) through the transition period of re-establishing basic social services and infrastructure and, ultimately, to an expanded economy with more employment and livelihood opportunities. Therefore, this project aimed to provide labor-intensive public works to generate temporary employment, while at the same providing valuable infrastructure development. These aims remained highly relevant as the country faced macroeconomic collapse, high inflation, and increasing food prices over the project period. The project also aimed to increase the government's capacity to deliver social protection programs in the longer term. No social protection sector program was established in South Sudan during the early period of its independence, therefore the project aimed to support the development of a policy. The South Sudan Development Plan for 2011-2013 identified the importance of social protection in reducing risk, vulnerability, poverty and exclusion in the newly independent country. The Bank's strategy paper for South Sudan at appraisal - the Interim Strategy Note for FY2013-14 - highlighted the creation of jobs and livelihoods, under the umbrella of a social safety nets approach, as a focus area, although no specific outcomes were identified. At completion, the PDO remained well aligned with the strategies outlined in the World Bank’s Country Engagement Note (CEN) FY18-19 for South Sudan, namely: (a) support to basic services for vulnerable populations, and (b) support to food security, livelihoods and basic economic recovery. The CEN acknowledged that social protection interventions were critical to supporting livelihoods and generating early economic recovery. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Provide access to income opportunities and temporary employment to the poor and vulnerable (ORIGINAL indicators/targets) Rationale As indicated in the theory of change, the project's outputs included development of public works projects in order to create short-term employment opportunities for the poor. Another output was to establish a skills development program, including a market assessment and support for training providers, to increase employability of the poor and vulnerable. A participatory, inclusive and transparent community-based targeting process at the Boma level would identify beneficiaries for the public works and skills development programs. These initiatives were envisioned to contribute to increasing access to income opportunities and temporary employment to the poor and vulnerable, particularly women and youth. Outputs Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915)  Creation of 4,032,670 days of work (original target: 2,190,000; revised target: 3,600,000) through 517 public works projects (original target: not set; revised target: 150). The public works nature of the projects focused on community assets (roads, clearing of drainage channels, general cleaning of public assets, trash collection) and innovations in group farming in rural areas. Beneficiaries reported that the works increased mobility and access to markets, improved delivery of social services, and reduced incidents of communicable diseases.  Capacity building support to local structures through training, workshops, and coaching (as reported by the project team), including in the following areas: vulnerability assessment and mapping, teamwork, reporting, conflict resolution, community mobilization, business cycles, payment processing, and record-keeping. However,  The skills training assessments/program were not implemented as planned.  The skills development evaluative studies were not conducted as planned.  Community-level communication campaigns were ad hoc, rather than standardized and systematic (as reported by the project team), leading to implementation challenges in localities with the inadequate communications. Outcomes  53,163 participants were employed in public works projects and received income, far surpassing the original target of 25,000. Of these, 74.0% were female. According to the ICR (page 18), meaningful participation of women in the public works activities was achieved through: (a) participation of women in the selection of public works activities; (b) work activities and payment sites in close proximity to participants’ houses; (c) assignment of work tasks based on physical ability (the ICR expresses this as "light and appropriate work for women"), including consideration for pregnant and lactating mothers included in the work norms; (d) selection of work hours taking into account women’s household obligations; (e) provision of work site shelter for children during work hours; and (f) zero tolerance towards violence.  The percentage of beneficiaries selected in accordance with eligibility criteria was not available (target: 85%). Instead, the ICR reported that 80% of participants in an independent qualitative beneficiary satisfaction survey (conducted in November 2018) were "satisfied with the categories of households selected as these were deemed to be the most vulnerable." The ICR also noted that poverty was an eligibility criterion (eight of the nine targeting criterion contained poverty) and poverty was near universal (approximately 80 percent between 2016 and 2018).  63.6% of workers (participants, also referred to in the ICR as beneficiaries) were paid on time (defined as cash paid fifteen days after the end of a work cycle), falling short of the target of 80.0%. According to the ICR (page 14), the shortfall in achievement was in part due to MIS glitches, internet connectivity issues, limited access due to weak infrastructure, insecurity and/or the rainy season, and low staffing. The Beneficiary Assessment revealed that the majority of respondents (78 percent) were "unsatisfied" with the payment process, namely the long time it took to receive money, noting that delays in payment disrupted their cash flows, caused challenges with their creditors, and forced them to constantly borrow. However, challenges such as the rainy season, absent participants, difficult terrain, connectivity, and verification issues often required additional time to deliver cash into the Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) hands of the workers (participants). Further, the implementing partners did not always the payment schedule properly and participants incorrectly expected to receive payment immediately upon completion of work  52.7% of beneficiaries (participants) ranked the program as "fair," "good," or "very good" in the areas of targeting mechanism and selection, payment mechanism, timeliness, and functioning of the oversight and coordination structures. This fell short of the target of 80.0%.  100% of selected Bomas posted the list of approved beneficiaries in public places (or announced the list at a public meeting), achieving the original target of 90.0%. Also,  The independent project evaluation carried out in Juba indicated positive impact of the cash transfers. 36.0% of respondents bought food for their households with the cash, while 26% and 22% paid school fees and medical bills, respectively. 89% indicated that their household income increased, and of these, 82% reported that the increased income led to greater frequency and better quality of meals. A number of beneficiaries also reported to the ICR mission that they used the cash transfer to save and start small scale income generating activities. However,  The key output to design, test, and finalize a targeting mechanism to select poor youth for skills development training was not completed. Consequently, the actual skills training programs and assessments were also not completed. Achievement is rated Modest due to significant shortcomings in the activity to develop a skills development training program for youth, including failing to achieve the key project indicator to establish a targeting mechanism for youth. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Provide access to income opportunities and temporary employment to the poor and vulnerable (REVISED indicators/targets) Revised Rationale Outputs See outputs reported above, for Objective 1. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) Outcomes  53,163 participants were employed in public works projects and received income, far surpassing the revised target of 34,000.  100% of selected Bomas posted the list of approved beneficiaries in public places (or announced the list at a public meeting), achieving the revised target of 100.0%.  See other outcomes listed above in Original Objective 1 on compliance with eligibility criteria, timeliness of payments, and beneficiary satisfaction. Achievement is rated Substantial due to surpassing of the target for providing income opportunities and employment to the poor and vulnerable, although there were moderate shortcomings in the quality of the process such as timeliness of payments and compliance with eligibility criteria. Revised Rating Substantial OBJECTIVE 2 Objective Put in place building blocks for a social protection system. Rationale According to the theory of change, the project's outputs included institutional support to the Ministry of Gender, Children, and Social Welfare (MGCSW) to develop a social protection policy and the operational tools to implement the policy. These outputs were envisioned to help the Government of South Sudan put in place building blocks for a social protection system in the country. According to the Project Appraisal Document (page 5), these "building blocks" were understood to be a social protection policy, and operational tools to improve beneficiary identification and registration, monitor the program, and ensure that cash benefits were transferred to eligible beneficiaries in a secure, transparent and timely manner. Outputs  Technical assistance in developing a social protection policy.  Technical assistance in developing operational tools including a Management Information System (MIS), payment system, and targeting tool.  Development/testing of a targeting mechanism to select poor youth for skills development training.  Establishment of an appeals committee for grievances, a community-based structure that aimed to address grievances related to the project, as well as to mitigate against social risks, tensions, disputes, and in extreme cases, violence. This community-based grievance redress mechanism was functional and deemed largely capable of addressing complaints raised by beneficiaries, including incidents of violence in one project area. Nonetheless, shortcomings in the mechanism existed, such as limited capacity to identify, report on, and mitigate emerging social risks. Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) Outcomes  A national social protection policy framework was approved by the Council of Ministers and launched in May 2016. A Social Protection Unit was established, staffed, and equipped within the MGCSW. A National Social Protection Technical Working Group, with support from the World Bank and the United Nations International Children’s Fund (UNICEF), was also established to bring together the Government, humanitarian and development actors for strategic coordination and alignment. Discussions with the World Food Programme (WFP) and UNICEF representatives during the ICR mission affirmed their appreciation of MGCSW’s leadership and commitment to sustain the Technical Working Group.  Basic operational tools in support of the implementation of the policy framework, such as the MIS, grievance redress mechanism, targeting mechanism, and payment mechanism were launched. Data collection, storage and processing by the MIS was developed by project closing; however, the ICR (page 34) reported that the MIS still required "significant improvements in order to be considered as fully functional." The MIS independent assessment report (January 2019) noted issues on data privacy and security, internal data quality checks, synchronization with the biometric system, restricted access and audit trails, and lack of hardware, software, and trained staff to operate the system. The project introduced a three-step targeting process to select project beneficiaries (geographical targeting, followed by community-based targeting to select households). The payment mechanism was established, in partnership with a commercial bank, based on two-factor authentication. The payment procedure, involving three parties and the World Bank, allowed for rigorous checks and balances. The biometric component strengthened the registration and payment verification system, which aimed to reduce potential risks of fraud, thereby greatly enhancing transparency and accountability in the utilization of resources. However, as noted previously, there were some shortcomings in the process regarding timeliness of beneficiary payments and level of satisfaction of beneficiaries.  Local level oversight and coordination structures, including State Technical Committees, County/Municipality Core Teams, Payam/Block and Boma/Quarter Council Development Committees, Appeals Committees, and Community Supervision Teams were established. The various structures played a role in mitigating risks, ensuring smooth implementation, serving as a liaison between the project and the communities, and promoting a culture of accountability and transparency. More specific information on the effectiveness of these local entities was reported by the project team as follows: on average, 67% of participants were "satisfied" with selection of beneficiaries, coordination between the structures and the beneficiaries, supervision of work and encouraging beneficiaries to work, and leaders offering a sense of direction and order; anecdotally: local structures served as effective entry points on the ground and helped generate strong commitment and ownership of project activities by local communities, thereby supporting smooth implementation and management of grievances, strengthening legitimacy of the project, and improving trust by communities of local institutions (in contrast, in locations were community mobilization and capacity building were not carried out effectively, there were higher levels of implementation challenges). Achievement is rated Substantial due to establishment of the social protection framework and several operational tools, although there were some shortcomings in the actual functionality of the tools. Rating Substantial Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall Efficacy is rated Modest due to the first objective only partly achieved, while the second objective was fully achieved. Overall Efficacy Rating Primary Reason Modest External shock OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Overall Efficacy under revised indicators/targets is rated Substantial due to both objectives fully achieved. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency The PAD (page 17 and Annex 7) estimated the cost-effectiveness of the proposed wage transfer in the public works component. The public works activities were expected to reach 26,300 households and 173,580 individuals with income transfers equivalent to US$3 per household for 30 days per year in rural areas and 40 days per year in urban areas. The cost-effectiveness ratio was estimated at 0.48, meaning it was expected that it would cost US$2.07 project cost to transfer US$1.00 in wage benefits to the intended recipients. The PAD also provided an indicative illustration of potential economic returns for the skills development component based on a similar initiative (Adolescent Girls Initiative) being implemented in another part of the country. The ICR (Annex 4) revisited the estimate of the cost-effectiveness of the wage transfer and came up with a cost- effectiveness ratio of 0.42. The ICR suggested that due to the program's high labor intensity and the precision of the targeting (community-based targeting through house to house visits), the cost-effectiveness is favorable and comparable to earlier and similar programs in the Africa region and the world (for example, 0.40 cost- effectiveness ratio in Liberia and 0.47 in Ethiopia). Also, the opportunity cost for participating in the program (foregone income) was assumed to be very low due to very low formal labor force participation rates. Indirect benefits, which were not included in the appraisal calculation, were also likely to be significant due to the creation of public works assets intended to benefit the community. No internal rate of return was estimated at either appraisal or completion. Efficiency in implementation had some shortcomings, due to significant external shocks which led to early implementation delays. It took 17 months for the project to become effective after approval by the Bank Board, Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) due to lengthy government approval processes and government reshuffling, departure of a major implementing partner in the middle of the project period, and initial inefficiencies in the beneficiary payment mechanism due to manual verification. Modifications in project design and implementation arrangements helped to ensure eventual completion of project activities within 15 months of original project closing date. Project management was shared among three Bank projects. Efficiency is rated Modest based on significant shortcomings in efficiency of implementation (including dropping of an entire project component) due to external shocks. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objectives is rated High due to strong alignment with government strategy and Bank strategy. Efficacy is rated Modest under the original objectives due to significant shortcomings in achieving outcomes for the first objective. Efficacy is rated Substantial under the revised objectives due to both objectives fully achieved. Efficiency is rated Modest based on significant shortcomings in efficiency of implementation (including dropping of an entire project component) due to external shocks. Project under original objectives/indicators Overall outcome is rated Moderately Unsatisfactory due to High Relevance of Objectives, Modest Efficacy, and Modest Efficiency. Project under revised objectives/indicators Overall outcome is rated Moderately Satisfactory due High Relevance of Objectives, Substantial Efficacy, and Modest Efficiency. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) According to harmonized OPCS/IEG guidelines on restructured projects, the overall outcome is weighted according to the amount of the Loan that disbursed before and after the restructuring. At the time of restructuring in 2017, US$2.75 million, or 14.2%, of the Credit had disbursed. The original project numerical rating is 0.426 (0.142 x 3 (Moderately Unsatisfactory)) and the revised project numerical rating is 3.43 (0.858 x 4 (Moderately Satisfactory)); therefore the combined rating is 3.856, which rounds up to 4, or Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The national social protection policy was approved and launched, although effective implementation is at risk due to several factors, including continued volatility in security and economic conditions. The operational tools to ensure implementation are also not yet fully functional and effective. A follow up operation, in the form of a $40 million IDA Credit for a South Sudan Safety Net Project, is currently under preparation and will ensure continued financing for the project interventions. Community-level capacity, through the community structures established by the project, has been improved and ownership of community-level infrastructure is high, but the sustainability of such assets is tenuous as operations and maintenance plans (and funding) were not clearly defined. 8. Assessment of Bank Performance a. Quality-at-Entry The results framework was logical and straightforward, with the project design kept relatively simple to take into account very weak capacity and the challenging macro environment. Drawing from prior experience, the technical assistance for implementing agencies was clearly defined and relied on twinning arrangements with government staff and international consultants (in the project implementation unit) to strengthen capacity. Another lesson incorporated into the project design was the use of inclusive and transparent community-based targeting, given that poverty data was not available. The risk assessment appropriately rated the overall project risk as High given the fragile environment and low institutional capacity. The M&E design included clear data collection arrangements. Implementation arrangements were flexible in order to adapt to challenges as they arose (for example, difficulty in accessing some project areas, departure of some implementing partners due to security risks); however, the institutional arrangements were complex given the low capacity environment, with a multitude of local entities being responsible for targeting, implementation, verification, and grievance redress. Quality-at-Entry Rating Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) Moderately Satisfactory b. Quality of supervision The project implementation period was marked by volatile conditions on the ground and hence the need to make multiple adjustments to the project design and implementation arrangements. Security issues prevented field supervision in several project areas outside of Juba. Field supervision was also affected by temporary relocation of Bank staff from the Country Office from December 2013 to May 2014 and from July 2016 to April 2017. However, overall, there were a higher than usual number of Bank supervision missions and bi-weekly video conferences with implementing agencies to support implementation, anchored by task team leaders and core members who remained the same until project closing. There was appropriate candor in ISR ratings, as Development Objective and Implementation Progress ratings were both in the Unsatisfactory range during the months of stalled activity. Project restructuring was effectively used to respond to the changes in circumstances on the ground, including a change in scope of the public works activities to reach a higher number of beneficiaries in a shortened period of time; dropping of the skills training component (and key indicator); and the use of UNOPS to implement public works activities outside of the capital city of Juba. Challenges to fiduciary management were also effectively addressed by strengthening risk mitigation measures, such as requiring prior review for all procurement. Project monitoring was effectively carried out, although there were some shortcomings in the conducting of evaluative studies. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change was clear and the results framework accordingly robust. Each of the indicator definitions also identified a data source, targets, and baseline values. Project monitoring arrangements were also clear, encompassing a management information system, beneficiary satisfaction surveys (with a particular focus on targeting and transfer mechanisms), community score cards, and independent evaluations of the implementing partners. The evaluative aspects of the M&E design were largely focused on measuring the quality of the process (compliance with eligibility criteria, timeliness of payments, beneficiary satisfaction) and only anecdotally capturing the actual impact on beneficiaries livelihoods. b. M&E Implementation Regular data collection and reporting were mainly coordinated and conducted by the project implementation unit, which included an M&E Specialist and a private sector firm to design the MIS. Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) However, there were some data quality issues in the beneficiary payment process, as there were data mismatches between the biometric system and the MIS due to lack of full integration of the two systems. An independent evaluation for public works activities in Juba was commissioned by Action Africa Help International, although the same assessment could not be carried out in the other project areas. An independent beneficiary satisfaction verification exercise, with particular focus on targeting and transfer mechanisms, was conducted using a community-scorecard method; however, it was conducted only once towards the end of the project period project implementation instead of being an annual exercise, as per the M&E design; thus an assessment over time could not be made. c. M&E Utilization Project progress as measured through the monitoring system was regularly discussed with the project implementation unit, identifying areas requiring attention. The MIS was functional and provided critical data but due to some remaining quality shortcomings in the system, there was a need to triangulate MIS data with data from other sources, such as the progress report from the implementing partners. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to small-scale civil works. The safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered. The project initially relied on a portfolio-wide Environmental and Social Screening and Assessment Framework containing broad parameters, rather than a project-specific instrument. Safeguard policies on Pest Management (OP/BP 4.09), Indigenous People (OP/BP 4.09) and Projects on International Waterways (OP/BP 7.50) were also triggered (for the latter safeguard International Waterways, the project was subsequently granted an exception to riparian notification because the proposed activities would be limited to the improvement or rehabilitation of existing water delivery schemes). A project-specific Environmental and Social Management Framework was eventually prepared which included an Integrated Pest Management Plan, and Environmental and Social Management Plans for specific public works activities. An international safeguards consultancy firm was hired by the project implementation unit to implement and monitor safeguards measures for three World Bank projects in the country. The firm seconded two international safeguards consultants to the project implementation unit, who supervised safeguards implementation, provided training, and supported safeguards officers at the various implementing agencies. The project team provided regular supervision of safeguards policies and the Government produced a safeguards closure report. According to the ICR, the report outlined challenges faced during implementation, particularly on social safeguards issues, including: (i) lack of a community mobilization strategy and tailored communication materials; (ii) no specific training and sensitization on gender-based violence prior to implementation; (iii) limited resources to ensure regular M&E of safeguards across project Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) locations; and (iv) unsystematic management of voluntary land donation. The project team confirmed that the project was in compliance with all safeguard policies. b. Fiduciary Compliance The project implementation unit included a Finance Specialist and a Procurement Officer to support the three ongoing Bank operations in the country. Following the outbreak of violence in 2016, additional fiduciary risk mitigation measures were adopted to mitigate against heightened fiduciary risks, ensure accountable and effective delivery mechanisms, and enhance monitoring and evaluation of implementation progress. These measures included: (a) direct payments to service providers; (b) switching from report- based disbursement system to the transaction-based Statement of Expenditure disbursement system; and (c) prior review for all project procurement requests regardless of the amount involved. Also, given the need to verify tens of thousands of payments transactions in a high-risk environment, the project team was proactive in rolling-out biometric registration; however, the process continued to experience challenges of data mismatch between the biometric system and the MIS due to lack of full integration of the two systems. No ineligible expenditures were presented during the project period, and interim unaudited financial reports were presented in a timely fashion, with the final annual audit report providing an unqualified/clean opinion. There were some minor issues noted in management letters, including weaknesses in the process of handing over assets, delayed implementation of previous audit recommendations, failure to maintain a comprehensive risk management strategy, and continued reliance on manual accounting systems for project financial reporting purposes. c. Unintended impacts (Positive or Negative) Community-level trainings on water, sanitation and hygiene and also financial literacy were not part of the original project design but were undertaken in Juba City at the initiative of the implementing partner. As reported in the ICR, training participants reported improved awareness of hygiene practices (i.e. ensuring cleanliness when breastfeeding), as well as of the importance of saving. Some participants reported engaging in small-scale income generating activities. Also, engagement in public works was reported to provide beneficiaries with a platform for greater interaction and dialogue among the diverse ethnic groups, contributing to strengthened community cohesion and social unity. Nine out of ten of the respondents of the independent evaluation in Juba felt that the Project united people of different ethnic groups. d. Other --- 11. Ratings Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons Lessons from the ICR (page 29-30), adapted by IEG:  The use of third parties as implementing partners can be very effective in fragile and conflict- affected settings, especially in situations where government capacity and systems are nascent and access to project locations is limited. In the case of this project, the engagement of UNOPS as implementing partner -- supported by a commercial bank as the payment service provider -- helped mitigate the capacity gap, enabled access to remote and insecure areas, and mitigated high fiduciary risks.  A project design that addresses the specific daily needs and challenges of women is an effective way to promote gender equity and inclusion. Discussions with implementing partners and beneficiaries revealed that the targeting process usually took place in the morning, which was the time of day that women were usually available at home or in the neighborhood. Meaningful participation of women in public works activities was then achieved through: (a) participation of women in the selection of public works activities and in the community structures; (b) work activities and payment sites were in close proximity to beneficiaries’ houses; (c) assignment of work tasks based on physical ability (the ICR expressed this as "light and appropriate work for women"), including consideration for pregnant and lactating mothers included in the work norms; (d) selection of work hours taking into account women’s household obligations; (e) provision of work site shelter for children during work hours; and (f) zero tolerance towards violence. An additional lesson drawn by IEG:  Capacity building measures to expose national institutions to international expertise can be effective in developing nascent capacity. In the case of this project, the twinning arrangements with the Technical Support Team with international experts, knowledge exchange, and study tours, contributed to the project implementation unit's transition from one staffed fully by international consultants to one with a majority of national staff. In addition, working closely with international NGOs, a United Nations agency and a commercial bank also opened opportunities to learn new skills and build capacity. Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Safety Net and Skills Development (P143915) 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, well organized, and results-oriented. It provided a candid narrative on the context (outbreak of violence) and its impact on the project design and experience. The theory of change was overall sound, albeit focused on the short-term with output-oriented indicators given the emergency context. The results chain was well-documented - with satisfactory quality of evidence - and supported the main conclusions. Lessons were clearly derived from project experience, as outlined in the ICR. The ICR was overall consistent with guidelines, although it did not explicitly state whether all safeguards were in compliance with policies (the TTL subsequently provided confirmation on 12/18/19). a. Quality of ICR Rating Substantial Page 16 of 16