Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Report Number : ICRR0021604 1. Project Data Project ID Project Name P113586 S. Sudan Emerg. Food Crisis Res. P Country Practice Area(Lead) Additional Financing South Sudan Agriculture P125381,P130305,P149700 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H9220,TF-12069,TF- 31-Dec-2017 29,043,380.91 15934,TF-15935,TF-93011,TF- 99943,TF-99946,TF-A1813 Bank Approval Date Closing Date (Actual) 03-Oct-2008 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 20,430,000.00 20,430,000.00 Revised Commitment 20,401,783.50 20,401,783.50 Actual 20,401,783.50 20,401,783.50 Prepared by Reviewed by ICR Review Coordinator Group Maria Vanessa J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Corlazzoli Pellekaan PHPROJECTDATATBL Project ID Project Name S. Sudan Emerg. Food Crisis Res. P ( P151031 P151031 ) Page 1 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0.00 Bank Approval Date Closing Date (Actual) 12-Apr-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 PHPROJECTDATATBL Project ID Project Name South Sudan Emergency Food Crisis P149700 AF IV ( P149700 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0.00 Bank Approval Date Closing Date (Actual) 13-Mar-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 PHPROJECTDATATBL Project ID Project Name Southern Sudan Emergency Food P145339 Crisis Re ( P145339 ) Page 2 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0.00 Bank Approval Date Closing Date (Actual) 13-Mar-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 PHPROJECTDATATBL Project ID Project Name SS: Additional Financing - Food P130396 Crisis ( P130396 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0.00 Bank Approval Date Closing Date (Actual) 15-Jun-2012 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components Page 3 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) a. Objectives According to the legal agreement of 2008, the objective of the Southern Sudan Emergency Food Crisis Response project was “to increase access to food for consumption by the food insecure households and groups in six counties in Southern Sudan” (Legal Agreement TF 93011 (2008), pg. 8). There were variations of how the PDO was presented in the Project Appraisal Document and the legal agreement (PAD, para 17). However, the differences were minor and primarily relate to shifts in language related to the location of project implementation. The PDO was revised three times during the life of the project. In 2011, the original objective was changed to place emphasis on three key changes “(i) to support adoption of improved technologies for food production by eligible beneficiaries, (ii) increase storage capacities for staples, and (iii) provide cash or food to eligible people participating in public works programs in six countries in Southern Sudan.” (Additional Financing 2011, pg. v). In many ways, the 2011 changes to the PDO moved away from emphasizing access to food consumptions towards tackling food production challenges. Grant agreements drafted between 2011 and 2013 contained the three main project objectives listed above. (TF 093011, TF099946, TF099943 2011 pg. 6; and TF015934, TF015935 (2013), pg. 8). In 2012, minor changes to the objectives were made to align the geographical reach of the project and recognize the change in the name of the country (ICR, para 10). There was no material change in the PDO. In 2014, the PDO was once again changed to decrease the relative importance of the objectives related to the storage of capacities for staples. The new language of the objectives included “(i) to support adoption of improved technologies for food production and storage, and (ii) provide cash or food to eligible beneficiaries in the territory of the Recipient.” (Additional Financing 2014, pg. v). Legal agreements from Trust Funds in 2014 and 2016 contained the same objectives described in the 2014 Additional Financing documents (TF H922-SS 2014, pg. 4; Legal Agreement TFA1813 (2016), pg 4). IEG notes that the original PDO focused on "increased access to food for consumption". The PDO revised in the course of the Additional Financing in 2011 was framed in terms of three elements focused on improved technologies for food production, storage capacity for staples, and cash or food for public works programs which replaced the original focus on "access" to food for consumption. While technically the subsequent third revision of the PDO in 2014 amended the previous PDO, its elements were exactly the same but with different relative importance. Hence, for this review IEG has adopted a spit evaluation of the operation based on two periods, namely 2008 to 2011 and 2012 to 2017, because the ambition of the PDO changed materially in 2011 (compared with the original PDO in 2008) and remained basically the same thereafter. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Page 4 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Date of Board Approval 29-Jun-2011 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components Component 1: Support to Agricultural Productivity (Appraisal cost was estimated at US$3.0 million. At closing the cost was US$11.43 million). The component sought to target food insecure households to double their own food production and improve post-harvest management of selected staple food crops all year around (PAD, para. 19). It also aimed to promote and increase the acreages of selected staple food crops for seed bulking in cooperative societies in surplus and food deficit areas. It aimed at increasing consumption through trade (PAD, para. 19). Key activities that supported this component included community grant mechanism, finance productivity enhancement, procurement of oxen and oxen farming technology, seeds, hand tools, and other agricultural equipment. The component sought to work with selected NGOs to construct low cost food banks and food granaries at strategic locations, including at the farm level. The ultimate aim was to reduce food price vulnerabilities at the community level through improved domestic production of selected staple food crops. Component 2: Strengthening Community Safety Nets (Appraisal cost was estimated at US$1.5 million. At closing the cost was US$13.2 million). This component aimed at mitigating shortfalls in food availability for consumption to the food insecure households through public works. The goal was for food insecure households to manage shocks resulting from escalating food prices arising from low food supply associated with seasonal floods and inaccessibility of some of the areas. The focus was to establish public assets that would enhance agricultural productivity. Key activities that supported this component included repairs to rural roads, river crossings, and construction of village storage (PAD, para. 20). Component 3: Support to Project Implementation Arrangements (Appraisal cost was estimated at US$0.50 million. At closing the cost was US$2.5 million). This component sought to finance administrative, operational and technical assistance costs. It included trainings, purchasing of goods and equipment for the project implementation unit. It aimed at strengthening the public structures for coordination and account for project management costs (PAD, para. 21). In 2015, an additional financing project paper introduced a new component. Component 4: Support for Improved Nutritional Outcomes. (Appraisal cost was estimated at US$2.30 million. At closing the cost was US$2.3 million). This component (introduced as an additional component at part of the Additional Financing in 2011) sought to increase the production and consumption of foods of high nutritional value and build institutional capacity to support nutrition-sensitive agriculture. The component had three sub-objectives. First, it aimed at strengthening farmer groups to increase the diversification of locally produced foods and build the capacity of smallholder farmers. Second, it also sought to improve household knowledge and practices of food and nutrition to improve food consumption patterns. Activities to support this sub-objective included promoting new technologies and incentives to Page 5 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) reduce post-harvest losses, test labor, and introduce energy-saving technologies. The third sub-objective was to strengthen institutional capacity to improve integration of nutrition in agriculture through the promotion of a multi-sectoral approach to address malnutrition and explore bio-fortification to enhance nutrition (ICR, para.13). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. At appraisal, the total cost of the project was estimated at US$5.0 million, of which US$5.0 million would be allocated by the World Bank (PAD, pg. 1). At closing, the total project cost was US$29.43 million (ICR, pg. 11 and pg. 51). Financing. A total of US$29.43 million was financed through eight different legal agreements. Funds came from the Food Crisis Response Trust Fund, the Global Food Crisis Response Trust Fund, and the Policy and Human Resources Development Fund (TF-93011, TF-99943, TF-99946, TF-12069, TF-15935, TF-15934, IDA-H9220, TF-A1913). The actual disbursement at the end of the project was US$29.043 million (ICR, pg. 3). Borrower Contribution. The PAD did not require the Borrower to contribute towards the project. Grant Agreements reviewed did not stipulate that the Borrower had to contribute towards the cost of the project. Dates. The project was approved on October 3rd, 2008 and became effective on October 27th, 2008. It underwent a midterm review on February 16th, 2011. As already described in detail above the project underwent four additional financing restructurings. In June 29th, 2011 and on June 15th of 2012, the project received additional financing and changes were made to the PDO, results framework, and loan closing dates. In October 1st, 2013, the project received additional financing and there were changes to the loan closing dates. In February 28th, 2014, the project received additional financing and changes were made to the PDO and loan closing dates. In April 11th, 2016, the project received additional financing and changes were made to the results framework, components and costs, and loan closing dates. The project was implemented over a period of 9.5 years. Its original closing date was December 31st, 2017. There were no project closing date extensions and the project closed on the original date of December 31st, 2017 (ICR, pg. 3). Page 6 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) 3. Relevance of Objectives Rationale Country Context: At the time of appraisal, Southern Sudan was emerging from a 25-year protracted violent conflict with the Government of Sudan that ended with the signing of the Comprehensive Peace Agreement (CPA) in July 2005. By 2008, following a plebiscite, the Government of Southern Sudan (GOSS) had been established but its capacity was very weak (PAD, para. 9). The country faced continuous residual violence from the previous conflict, returning refugees, and country- wide structural food insecurity. According to the PAD, key factors that affected high food prices that caused food insecurity included: (i) low agricultural productivity, (ii) almost complete dependence on rain-fed agriculture, (iii) slow recovery of the food production and marketing sub-sectors, and (v) very poor infrastructure (PAD, para. 8). Given the difficulties in addressing these challenges Southern Sudan was heavily dependent on food aid. The land-locked country relied on food imports which led to high food prices due to expensive fuel and transportation costs. At the time of appraisal, the global commodity prices were increasing and food prices in local markets had increased by 30% in a three-month period (ICR, para. 2, and PAD paras. 5 and 6). As a result of these conditions, the project sought to increase food production to protect the targeted beneficiaries from rising food prices and make local food markets accessible through improved rural roads (PAD, para. 8). Given the context, the 2008 objective of the project were substantially relevant. By 2011, the situation in South Sudan had not drastically improved. While secession had been peaceful, violence continued in the northern border areas. In August 2011, the UN Food and Agricultural Organization (FAO) stated that it expected that only half of the food required would be produced domestically. The ICR stated that press reports indicated that 1.2 million people would be “severely food-insecure” in 2011-2012 (The Guardian, September 2011). Key factors affecting food insecurity included erratic rain, loss of farmers, and inter-communal conflict. Given the conditions in 2011, the revised 2011 PDO, though no longer focused on the original substantially relevant objective of an “increase in access to food for consumption” (the classic requirement for food security) the three elements of the revised PDO namely “improved technologies for food production, storage capacity for staples, and cash or food for public works programs” were certainly also substantially relevant to the Bank and Government development goals and to the welfare of the people of South Sudan. Page 7 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) A weakness of both objectives, however, was the lack of clarity on how to verify the achievement of either the original objective “to increase access to food for consumption” or the revised objective to “increase improved technologies for food production, storage capacity for staples, and cash or food for public works programs." Previous Sector Experience: At the time of appraisal, two Bank-assisted projects were being implemented that focused on enhancing rural livelihoods, including increasing food productivity and institutional development. In fact, these projects focused on implementing activities in similar geographical locations to the Emergency Food Crisis Response Project (EFCPR). The ICR reflected that given the geographic alignment some of the activities in this project could have been implemented under “The Support to Agriculture and Forestry Development Project (SAFDP)” project. In fact, attaching the grant to the SAFDP could have facilitated better synergies, simplified monitoring, and reassessed the continuation of EFCPR when SAFDP closed in 2012 (ICR, para. 20). Alignment with World Bank Assistance Strategy: The 2008 and 2011 PDOs of the project were aligned with the World Bank’s Interim Strategy Note (ISN), 2007. The Interim Strategy Note outlined the need to support rural development. That said, this request is one of ten themes the World Bank would support. The Interim Strategy Note (ISN) of 2013-2014 described in more detail the level of food insecure. It fact, the 2008 and 2011 PDOs contributed towards the ISN’s main objective to “support the most vulnerable, through funding for food security and protection of basic services.” (ISN 2013-204, para. 65). The 2008 and 2011 PDOs also supported the second objective of the 2017 Country Engagement Note (CEN), which was to “support the livelihoods, food security, and basic economic recovery.” The CEN highlighted that 50% of the population is food insecure (CEN, para. 29). While the project was designed as part of the Bank’s strategy for emergency support to South Sudan. The development goal it sought to address were an urgent need for financial assistance to alleviate the food crisis, as well as put in place infrastructure in the short-term to improve food production (PAD, para. 14). Alignment with Government Development Strategy: The PDOs established in 2008 and 2011 also aligned with the Government development strategy in particularly its medium-to long term approach. As described in the PAD, the Government’s strategy included a multi-sector approach steered through the Natural Resource and Rural Development working group (PAD, para. 12). The aim was to increase domestic food production, improve rural infrastructure, and strengthen existing agricultural production systems (PAD, para. 12). Conclusion: Food insecurity was one of the many challenges that South Sudan faced in the post-conflict period. High levels of food insecurity continued throughout the period of project implementation and the priority remained constant as seen through the emphasis in the different ISNs and the CEN in 2017. This review concludes that the original PDO defined in 2008 and the revised PDO defined in 2011 were both substantially relevant since they addressed food security and both food supply and demand issues to a varying extent, and were aligned with Government and Bank development strategies for South Sudan. The Page 8 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) objectives remained relevant throughout the project’s implementation period. The relevance of the PDOs defined in 2008 and 2011 are therefore both rated substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To increase access to food for consumption by the food insecure households and groups in six counties in South Sudan. Rationale This project was implemented through the course of 9.5 years. The project’s original aim was to increase access to food, by addressing some of the root causes of food insecurity in South Sudan, such as lack of access or affordability of food which were in turn caused by low food production, and poor agricultural infrastructure. Simultaneously it aimed at providing cash or food to vulnerable persons. The theory of change (defined retroactively in the ICR) stated that if there was an increase in production of crops and livestock due to improved technologies and support for post-harvest management, then there would be increased food availability, agricultural labor opportunities, and increased income among farmers. Moreover, the project’s theory of change also stated that cash for work or food programs would increase consumption due to increased household income, and increased food availability due to increased storage capacity, improved access to food markets, and improved facilities for sellers. Outputs: • 56,277 hectares were planted with cereals in the project area between 2008 and 2011 (neither baseline nor target provided, ICR, para. 28a). • Crop production was supported through maladoas, toria, ox plows, power tillers, tractors, and hoes (targets not provided, ICR, para 23 and 24). • 5 women’s groups were trained in improved vegetable production techniques and installing vegetable gardens in Tonj North (target not provided, ICR para. 24). • 25 women received sheep and goats in Tonj North (target not provided, ICR para. 24). • Hand tools were provided, including hoes, rakes, spades, watering cans, and plastic buckets (ICR, para. 36) • 82,000 farmers were supported with post-harvest technology between 2008 and 2011 (target not provided, ICR, para 26). • 24,462 people participated in the cash for work program (Target exceeded, Target: 20,000, ICR, para. 29). Outcomes: Page 9 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) By 2011, 189,000 farmers had adopted an improved technology (the original target of 130,000 out of a planned number of beneficiaries of 150,000 was exceeded, ICR, footnote 1 and para. 22). The impact of technology on the yields was positive, with yields of sorghum increasing by 100 percent in Panyikang, 71% in Morobo and 43% in Yambio. Farmers supported by the project cultivated an area ranging between 0.43 to 1.3 hectares. The project contributed towards an increase in food production. The farmer beneficiaries cultivated on average about 1.0 feddan (0.425 hectares) equaling 80,750 hectares (ICR, para 28a). It was estimated that in Yambio and Morobo, 83% of cropping was used for cereals (ICR, para. 28a). With a more conservative estimate of 70% of cropping area used for cereals, it means that 56,227 hectares were planted to cereals (ICR, para. 28a). Data for 2011 showed that in the six counties where the project operated, the total area under cereal production was 66,268 hectares. The ICR states that “85% of the cereal production in the project (EFCRP) counties in 2011 were attributable to the project” (ICR, para. 28a). An implementing partner reported a 80% increase in households with diversified income source and a 40% increase in target households with vegetable gardens producing green leafy vegetables (ICR, para. 24). With respect to the cash for work program the ICR stated that implementation partners (IPs) were not familiar with the design of these programs. Nevertheless, the ICR noted that the World Bank task team, the Ministry of Agriculture and Food Security and the IPs collaborated to implement the program and “by the end of phase 1 in June 2011, the project reached 24,462 people with cash for work, exceeding the PDO target of 20,000” (para 29). Conclusion: The ICR does not provide clear measurements on improved consumption of food by the project’s target group. Pre-restructuring in 2011, the project was able to support farmers with post-harvest technology. It is also evident that there was an increase in food production in the six counties where the project was operating. Unfortunately, the ICR did not provide data that showed that food insecure households increased their access to the increased food production or that these households increased food consumption. Therefore, it was impossible to assess the impact that increased food production had on an “increase in access to food for consumption”. While it can be assumed that those who worked on cash for work programs gained additional money to purchase food, no specific information for the first phase of the program was available. On this basis, the efficacy of the achievement of the project's original objective is rated modest. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Page 10 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Objective 1 Revision 1 Revised Objective (i) To support adoption of improved technologies for food production by eligible beneficiaries, (ii) increase storage capacities for staples, and (iii) provide cash or food to eligible people participating in public works programs in six countries in South Sudan. The scope of the project increased from 150,000 farmers for the original objective to 215,500 beneficiaries in of which 190,000 were farmers and 25,500 were individuals benefitting from participation in public works (Additional Financing Paper 2011, footnote 1). The 2014 additional financing and restructuring increased the estimated number of beneficiaries to 384,500 (ICR, pg. 38) . Revised Rationale The revised objective changed the ambition of the project from food security for the target group to elements aimed at increasing the domestic production and availability of food, and “to increase the ability of targeted food insecure households to buy food by providing cash-for-work or obtain it directly through food for work (Additional Financing Paper 2011, para11). The project’s theory of change had therefore changed accordingly. Key performance indicators were revised and improved to better measure results in the areas of increased storage capacity for staple foods, and the impact of cash/food for work programs. (i) Adoption of Improved Technology for Food Production Outputs: • Hand tools were provided, including hoes, rakes, spades, watering cans, and plastic buckets (ICR, para. 36) • An additional, 83,620 farmers adopted improved technology post-2011 (target not provided, ICR, para 28b) • 4.8 tonnes of vegetable and legume seeds were distributed, including amaranth, collard, cowpea, eggplant, okra, onion, tomatoes, and watermelon (target not provided, ICR, para. 36). • Demonstration plots covering 12.6 ha provided training on agronomic practices and animal husbandry (target not provided, ICR, para. 36). • Small kitchen gardens were established for 2,300 beneficiaries (79% of women) (target not provided, ICR, para. 37). • 5,000 vulnerable households were provided with investment vouchers to acquire 2,000 chickens and 18,000 sheep or goats from local markets (target not provided, ICR, para. 36). • 400 beneficiaries (38% of women) were trained on fishnet braiding, processing, and preservation (target not provided, ICR, para. 37) • Fishing kits comprised of gill nets, fishing hooks, and twine were distributed in Jur River (target not provided, ICR, para. 36). • Vouchers to 6,000 mothers of children under-five were provided to access nutritious foods in local markets (target not provided, ICR, para. 37). • 18 mother-to-mother support groups, with 450 women, were provided with business or income generation skills (target not provided, ICR, para. 37). Page 11 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) • 2,500 beneficiaries (82% of women) were trained on food processing and storage (target not provided, ICR, para. 37). • 8 joint activities and meetings involving agriculture/food security and nutrition coordination mechanisms took place (target met, target: 5, ICR, pg. 44). • The project also aimed to raise awareness and draft action plans to mainstream nutrition in agriculture. • By the end of the project, 40 agricultural professionals reported an improved understanding of the role of nutrition (target not met, target: 120, ICR, pg. 43). Outcomes: As noted by the outputs described above, the project provided vegetable seeds, small farm equipment, and training to women and vulnerable households. Consequently, by the end of the project, a total of 272,620 farmers had adopted improved technologies (target met, target: 266,000, ICR, para. 22). As a result of the introduction of new technology, 125,211 farmers adopted practices and initiatives for post-harvest handling and management (target of 100,700 met, ICR, para. 26). The project contributed towards an increase in food production. According to the project’s Additional Financing Project Paper, 2011, average yields of sorghum had increased 100% in Panyikang, 71% in Morobo, and 43% in Yambio (ICR, para. 23). Moreover, in the last phase of the project, 4,500 households increased the number of foods they produced (target of 1,800 exceeded, ICR, para. 36). While data collection on cereal production only began in 2010, almost 18 months after the project began, it was estimated that 83,620 farmers had adopted improved technologies on an additional 106,616 hectares of land (ICR, para 28b). Cereal production continued to increase every year until 2015 when it was negatively affected by the ongoing conflict. In 2010, total cereal production was produced on 69,606 hectares. In 2014 it was 159,984 hectares, but by 2016 it was down to 131,111 hectares (ICR, pg. 16). The project also brought back into production 104,741 hectares of idle land (target of 38,800 hectares was exceeded, ICR, pg. 41). Conclusion: The ICR demonstrates that there was increased food production in the project's target area, although the extent to which increases were attributable to the project after 2011 was not clear. It also indicated that 6,000 households increased their dietary diversity after the project was restructured (target of 1,900 exceeded, ICR, para 37). From 2016 until project closing, a total of 2,781 households adopted healthier eating habits (target of 1,650 exceeded, ICR, para. 38). (ii) Storage: Increase storage capacities for staples, Page 12 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) • The ICR stated that a total of 11,092 metric tons of storage capacity was built at community and household level (target of 8,700 exceeded, ICR, para. 27). • The ICR did not provide information on the location or the number of storage facilities for staple foods were built. • By 2011, 74% of targeted food insecure households in project areas had food reserves (target of 64% exceeded, ICR, pg. 41). Unfortunately, subsequent progress could not be assessed because this indicator was dropped in 2012, due to difficulty in terms of time and cost of obtaining accurate and reliable data, as well as potential problems with attribution, and the lack of a reliable baseline (ICR, pg. 42). Outcomes: Beginning in 2016 until project end, a total of “2,500 households were using improved food processing and storage methods and facilities” (target of 1,500 exceeded, ICR, para. 38). (iii) Cash for work and unconditional food distribution: to provide cash or food to eligible people participating in public works programs in six countries in South Sudan Outputs: • Between 2011 and 2016, the project supported 110,000 beneficiaries with cash for public works. • A total of 47,595 people participated in the public works program of which 62% were youth (target of 37,300 exceeded, target for youth of 70% net met, ICR pg 42). Outcomes: A total of 58,680 vulnerable persons benefited from unconditional food or cash transfers (target of 138,000 not met, ICR, pg. 33). The project did not achieve its target due to an error in the contract between the Ministry of Agriculture and Food Security (MAFS) and the World Food Program (ICR, pg. 38). While the project reached fewer individuals, it nevertheless provided a greater amount of food than originally planned. The cash for work program repaired 6.5 km of the Morobo-Iraga road, and 7km of Aboroto-Yuguge road (ICR, para 32). Key activities included clearance of stumping, road formation, drainage, murram digging, and spot murraming. In Morobo, 23.6 km of feeder road was rehabilitated by bush clearing, excavating, and spreading murram to selected sections. The Karua and the Wuli bridges were also renovated. In Tali, 11 km of feeder road was improved between the Tali center to the Pagara Boma. The ICR did not provide evidence that improved infrastructure such as roads had an impact on food prices or on improving access to local markets. Similarly, the ICR did not provide any data or estimates on income increases from cash for work programs or how the cash-for-work programs supported purchasing or access to food in the target area. Page 13 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Conclusion: The project reached a total of 339,433 direct beneficiaries of which 47% were women (revised target: of 384,500 not met, ICR, pg. 38). The project distributed seeds, supported crop production, and provided training for women and vulnerable households. It also distributed sheep or goats to local markets. Through unconditional food or cash transfers, it reached 58,800 people. The ICR demonstrated that the project supported the adoption of improved technologies and it increased food production. During the life of the project, it had made substantial progress in providing increased storage capacity for staple foods. The project also provided cash or food to eligible people participating in public works programs. For the most part, it met or exceeded its revised objectives, even in the midst of a difficult fragile and conflict affected operating environment. On balancing the objectives achieved and not achieved and the positive responses from the project’s beneficiaries, this review concluded that the efficacy of the revised objective was substantial. Revised Rating Substantial PHOVRLEFFRATTBL Rationale Prior to restructuring, the project aimed at increasing access to food for consumption by the food insecure households. The underlying theory of change of the project aimed to tackle both supply side and demand side obstacles. The project was able to increase food production and implement cash for work programs. It also distributed post-harvest technology to farmers. Unfortunately, the project did not present significant data to show that the activities led to improved access for the consumption of food. The project did not make sufficient attempts to measure increased access or consumption of beneficiaries. Therefore, this review rates the efficacy of the original objective as modest. After the restructuring of the objective, the project continued to be successful at introducing improved production technology, providing training, preparing farmland, and increasing staple food storage capacity. The cash for work programs provided additional income to project participants, as well as improved infrastructure that was identified as a contributor to increased food prices. Food production continued to increase until the civil war. Based on these achievements this review rates the efficacy of the revised objective as substantial. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Scope of Analysis: No financial or economic analysis was carried out prior to project implementation in 2008. The project papers for the additional financing from 2011, 2012, 2013 also did not include any economic or financial analysis. While the 2013 project paper committed to collecting data for ex-post computation, the civil war that erupted in 2013 delayed implementation planning and limited field data collection (ICR, para. 42). Page 14 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Financial Analysis: Financial gross margins in the project area were estimated in preparation for the 2014 project paper based on representative crop models for sorghum and maize. The ICR noted that based on data from the earlier Emergency Food and Nutrition Security Project (EFSNP) it was estimated that the average gross margin for six months before the project was US$257 per ha for sorghum and US$150 per ha for maize (ICR, para. 43 and pg. 53). However, these two crops did not represent 100% of the cropped area for this project (EFCRP). Other components of this project such as seeds and cuttings for groundnuts, cassava, pulses, vegetables and sesame, as well as mechanization, were important but their gross margins were not analyzed (ICR, para. 44). Moreover, both prices and financial returns were volatile throughout the life of the project (ICR, para. 98). With respect to the cash for work program, according to the ICR comparative data from other similar projects suggested that it would cost US$2.07 to transfer US$1.00 in wage benefits to beneficiaries. The ICR provided no reason why this efficiency measure could not have been calculated for this project. Nevertheless, the ICR noted that EFNSP’s cash for work program reached 47,595 people with an investment of US$8.2 million, roughly US$172 invested per beneficiary and hence using the 2:1 ratio between costs and wage benefits may have transferred a wage benefit of US$86 to each beneficiary for that project. On the other hand, the nutrition component’s cash for public works program in this project generated about US$4.6 million in wages. With 110,000 benefiting from these public works, the project paid about US$42 wage benefit per person annually (ICR, para 46) which was 11% of the estimated poverty line (Annex 4, para 95). There is no comment in the ICR on the reasons for these different results in terms of wage benefits per beneficiary. At the same time the ICR noted that introducing cash wages would probably have increased purchasing power of consumers and hence stimulated demand in local markets, although no evidence was provided (ICR, para. 45). The project also invested an average of US$54 per person for 6,000 beneficiaries to increase production and dietary diversity (ICR, para. 46). In comparison, total food distribution costs were approximately US$5 million or an average of US$85 each for the 58,680 beneficiaries of the program (ICR, para. 46) but it is not clear from the ICR whether or not the average cost per beneficiary included the cash for work program. The high costs of food distribution were due to expensive “land transportation, storage and handling” which comprised 63% of the contracted cost compared with 17% for the cost of food in high conflict areas in South Sudan (ICR, paras 33 and.46). The economic benefit-cost ratio of increased community grain storage was estimated at 1.3 with an economic NPV of US$1million and an ERR of 20%. The data and assumptions used to arrive at these figures was not available to the ICR team, as a result, follow-up analysis was not conducted (ICR, Annex 4. pg. 52). Operational Efficiency: The project’s management costs were increased at a consistent rate as additional financing was provided for the project. Originally the PAD allocated US$500,000 for project administration of a US$5 million project (about 10%). By the end of the project, it had spent US$2.5 million in administration from US$29.43 million – about 8.5%. This was an acceptable result in the FCV environment. The efficiency of this project as reported in the ICR is rated as modest due to the lack of clarity of the analysis of the efficiency of food production, food/cash distribution programs and food storage investments. It is Page 15 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) acknowledged that the project was able to maintain acceptable operational efficiency, despite a civil war and the fragile security environment. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome This project was implemented in a fragile, conflict, and violence context, and during a civil war. The original PDO defined in 2008 and the revised PDO defined in 2011 addressed food security, food supply and food demand issues to a varying extent. These objectives were aligned with Government and World Bank development strategies and were therefore substantially relevant. The objectives remained relevant during the project’s implementation. Despite increases in food production attributable to the project and cash for work for almost 224,500 people there was no evidence that the project’s original objective to increase access to food for consumption by the food insecure households was achieved. Post restructuring of the objective and under very difficult operating circumstances, the project was successful at introducing technology, providing training, preparing farmland, and increasing food storage capacity. The cash for work programs provided additional income to project participants, as well as improved infrastructure that was identified as a contributor to increased food prices. Food production continued to increase until the civil war. Consequently, the less ambitious objective of increasing food production, increasing the storage capacity of staple foods and providing food/cash for work programs was achieved. Hence the efficacy of the project’s revised objective was substantial. Despite taking account of the fragile and insecure environment in which the project operated, its efficiency is rated as modest due to the lack of evidence. The project was able, however, to maintain an acceptable level of operational efficiency, despite a civil war and a fragile environment. Page 16 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Before Restructuring After Restructuring Relevance of Objective Substantial Substantial Efficacy (PDO) Modest Substantial Efficiency Modest Modest Outcome Rating Moderately Unsatisfactory Moderately Satisfactory Numerical Value of Outcome Rating 3 4 Disbursement US$5 million US$24.43 million Share of Disbursement 17% 83% Weighted Numerical Value of the Outcome 0.51 3.32 Rating Overall Outcome Rating Moderately Satisfactory In summary the project’s performance had moderate shortcomings and its outcome is therefore rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The risk to development outcomes remains high. Insecurity and Conflict: South Sudan has continued to experience conflict and insecurity, which has led to internal displacements. The ICR reflects that the progress made by the project is likely to have already been lost due to the displacement of people, and damage or destruction of infrastructure in the conflict-affected areas (ICR, para. 92). For example, it is unknown whether the cropping and public works activities survived conflict in Maban county (ICR, para. 92). In Yambio, some infrastructure remained but farmers have fled the area (ICR, para. 92). Nagero and Morobo counties remain inaccessible (ICR, para. 92). Continuous Donor support: Given the current context, donor support is required to continue to focuse on humanitarian aid, including food insecurity. The World Food Program and FAO, continue to support production for food security and nutrition programs (ICR, para. 93). 8. Assessment of Bank Performance Page 17 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) a. Quality-at-Entry According to the ICR, the project’s design reflected the lessons that had already been learned through the “Support to Agriculture and Forestry Development Project” (SAFDP). It had a comparatively simple results framework with a few indicators that were directly related to the project activities instead of broad sector monitoring indicators (paragraph 83). Indeed, many programs in South Sudan were also addressing complementary issues of food production and safety nets (ICR, para 80). However, by creating a separate project, it created a management burden on the Ministry of Agriculture and Food Security (MAFS) which was implementing several other projects with low staff capacity (ICR, para. 54 and 55). Implementation arrangements were complex with several different committees responsible for implementation, coordination, and linking national level and local level implementing partners (ICR, para. 81). The government official responsible for this project had other responsibilities within the government, making it sub-optimal given the fiduciary, safeguards, and technical issues that needed to be attended to (ICR, para. 55). The project was implemented by experienced NGOs which supplemented weak Government capacity (ICR, para. 82). The implementing NGOs included Action Africa Help, Norwegian Peoples Aid, World Vision, Mercy Corps, HARD-SSS, and ACTED. These NGOs were good implementing partners ensuring safeguard management and that the implementation of activities were completed in the conflict affected areas. The project also provided technical assistance to the PIU (ICR, para. 82). The selection of far flung implementation sites should have been avoided given the poor transportation infrastructure in the country, according to the ICR (para. 84). The design team lacked important information with regards to weather, physical condition, social dynamics of the proposed project area, and the nature of local markets (ICR, para. 25). The PAD, and as a result the project, did not incorporate a conflict-sensitive lens. For example, the PADs proposed beneficiary selection criteria did not take into account the conflict dynamics between actors (PAD, para. 18). The ICR concluded that, despite workshops on M&E with implementing agencies, more capacity building and a basic workable monitoring systems could have been established (ICR, para. 83). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision a.Quality of supervision The World Bank team continued to be engaged despite the fragile operating environment. The emergency nature of the project combined with delays in contracting with NGOs, led to confusion on Page 18 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) how to implement parts of the project and how to divide the agriculture production activities between subsistence and more market-oriented farmers (ICR, para. 59). The team was based out of the Juba office and worked closely with the government and implementing NGOs to monitor progress. It hired additional consultants to provide support. It undertook regular field visits and joined project coordination meetings (ICR, para. 86). The team also provided leadership on project implementation due to the lack of experience in the Government and NGOs (ICR, para. 86). During the civil war in December 2013, the Bank office was evacuated twice to Nairobi making supervision impossible (ICR, para. 87). Despite these challenges, the team prepared the 2014 additional financing. Insecurity ultimately precluded field visits in some areas and impeded project implementation in several areas (ICR, para. 88). The World Bank Team and MAFS adopted a flexible approach and relied on the judgment of implementing agencies to determine the safe areas to implementing activities (ICR, para. 88). The project was also affected by economic turbulence and the decline of world oil prices, which accounted for 90% of South Sudan’s national revenue (ICR, 64). The country’s difficult economic situation was exacerbated by the return of refugees and returnees that put pressure on overall food demand and services. Growing insecurity, low government revenue, and move to monetize the debt led to high inflation, including food (ICR, para. 64). The project also experienced weather shocks, including a drought that destroyed all the planted crops in Upper Nile State in 2009 (ICR, para 67). Despite working under challenging conditions, the ICR acknowledges that there were oversights and missed opportunities. For example, as noted already the project could have focused on capacity building for government staff at the national level that could have had a longer-term impact, instead of leaning on hiring technical assistance. The reporting and recordkeeping were also not optimal. According to the ICR, the team’s candor about the implementation issues was weak (ICR, para. 89). There were also errors that were made during the additional financing of 2014 and in 2016. For example, changes to the PDO made in 2014 were probably not necessary (ICR, para. 89). Moreover, in the 2016 additional financing, the PDO should have been revised to include and add a more nuanced focused into the nutritional programming (ICR, para. 89). The many restructurings to add financing enabled programming to continue but added administrative burden (ICR, para. 65). The small and short-duration grants prevented NGOs from establishing long-term relationships and made it difficult to plan long-term activities (ICR, para. 65). Page 19 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Conclusion: The team showed commitment to South Sudan and prepared several additional financings to bolster funding and support restructuring, even in the midst of a civil war. That said, there were weaknesses in the project monitoring system, including poor indicator selection and lack of baselines. The growing insecurity ultimately made field level supervision nearly-impossible (ICR, para. 90). The security situation and the effects that it had on the program were outside the control of the Bank. That said, there were errors made in the additional financing requests and monitoring systems that could have been avoided. Overall, Bank Performance is rated as Moderately Satisfactory Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design This project’s theory of change aimed to increase access to consumption of food by increasing production of food by tackling key root causes. It aimed at increasing food production through the introduction of seeds and technology but also address some of the issues that led to high prices of food, such as poor infrastructure. By implementing cash for work programs, food insecure households would have the means to acquire food at local markets and through the cash-for work programs contribute towards improving infrastructure and roads. Through several additional financings, the project was tailored to add new activities to existing components, as well as one additional component. The additional financing supported the original theory of change. The project adequately adapted to the ever-evolving security, humanitarian, and conflict situation in South Sudan. For example, the project added unconditional food distribution and a nutrition program. The objective was refined and adjusted to reflect the changes in the activities and project. The project indicators were revised several times during the course of the project. Acceleration of food production was measured through the proxy indicators of (a) percent of targeted farmers adopting improved technologies for food production. The safety net was measured through (b) the percent of Page 20 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) target population receiving cash transfers (ICR, para. 8). As noted by the ICR, these indicators “aligned with the goals of the PDO, but they stopped at measuring inputs rather than whether the amount of food accessible to households for consumption increased (ICR, para. 8). Overall the indicators for this project were weak, particularly at the PDO level. For example, they failed to adequately measure access to food for consumption. Other potential indicators that could have been more useful could have been crop and livestock production increases, changes in local market prices, or actual food purchased by program participants, changes in consumption patterns, and improved access local markets (ICR, para. 8). The project also added, modified, and removed indicators. The initial results framework only had 5 indicators. The initial indicators did not measure access to food or increased consumption. Over the course of the project, all of the original project’s indicators were either modified or dropped. The final indicators provided an overview of the project’s achievements, but some gaps remained particularly related to the impact of the project on food security. That said, given the operating environment, it would have been difficult to adequately measure impact level outcomes. The monitoring system was also not adequately set up to collect key information on outputs and outcomes. The system as outlined in the PAD 2008 was vague. It focused more on outlining the responsibilities for the implementation of the monitoring system than the details on how information was to be collected . Although a monitoring agency, under the Multi-Donor Trust Fund for Southern Sudan, was supposed to be responsible for the day to day monitoring of the performance of the project (PAD, pg. 46), implementing agencies were expected to be responsible for M&E and collecting data in line with the results framework (PAD, pg. 46). Bank task teams were expected to conduct frequent missions and ex-post reviews, as well as four comprehensive supervision missions per year. The PAD did not mention the need to establish baselines for monitoring and evaluation. The project team tried to overcome some of the weaknesses of the PAD by holding two workshops with implementing agencies to reach agreement on the indicators and their measurements (ICR, para. 70). The ICR stated that there was also confusion over the indicators, including a lack of common definitions, and difficulty in measurement (ICR, para. 70). The challenges of a poorly designed monitoring and evaluation system were also compounded by the difficulty of collecting data in active and remote conflict areas as well as the overall paucity of readily available data in 2008. b. M&E Implementation Monitoring and evaluation systems remained weak throughout the life of the project (ICR, para. 71). By the end of the project, there was no M&E specialist on staff and the PIU did not have staff or budget to prepare a borrower’s completion report (ICR, para 71). Page 21 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) The PIU assembled progress reports relying on data collected by NGO implementers (ICR, para. 71). The project did not set up a consolidated management information system database. There was also no defined methodology to guide the gathering of data (ICR, para. 72). The Government was not enthusiastic about outside technical assistance to bolster M&E activities (ICR, para. 71). The monitoring system lacked the capacity to track basic output activity data, such as types of the technology used, amount and types of seeds distributed, the production level of crops, process earned by producers, kilometers of roads rehabilitated (ICR, para. 72). In the early years of the project World Bank team could conduct regular field missions and verify data provided by NGOs. The outbreak of the civil war in December 2013, meant that field missions were significantly reduced (ICR, para. 73). While implementation resumed in March 2014, field visits only began in early 2016. c. M&E Utilization Data were collected by NGO implementing agencies. This data was consolidated and used to enrich the project knowledge base (ICR, para. 29). While the World Bank team consistently reported problems with data verification, it was able to conduct field verification missions until 2013. That said, qualitative data would have been useful to fully understand the scope and impact of the project (ICR, para 74). Conclusion: The project had a solid and clear theory of change. Through several additional financing, the theory of change continued to be refined and improved as the project was implemented. Indicators were modified, added, and dropped with the aim to collect data that could speak to the contribution of the project towards the intended outcomes and the revised objective. The monitoring system was weak throughout the life of the project. It lacked basic infrastructure to measure, track, and collect consistent data, including output data. While field missions were conducted until 2013, the World Bank Team reported problems with data verification. Overall, this review rates M&E performance as modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Environmental Category “B” Partial Assessment. At appraisal, it triggered the following safeguards: Environmental Assessment (OP 4.01) and Pest Management (OP 4.09). In 2011, the additional financing added the following safeguards: Indigenous Peoples (OP/BP 4.10) and Involuntary Page 22 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) Resettlement (OP/BP 4.12). The 2012 additional financing added the Natural Habitat (OP4.04) and Forest (OP4.36). The project prepared the following documents Environmental and Social Management Framework (ESMF), Pest Management Plan, Indigenous Peoples Planning Framework, and Resettlement Policy Framework. An Environmental and Social Screening and Assessment Framework (ESSAF) was prepared for the whole of South Sudan in 2012. This study concluded that most people in South Sudan met the criteria in the Indigenous People’s OP 4.10 and as a result, an IPP was not necessary (ICR, para. 77). The project was in compliance with safeguards throughout the life of the project. There were delays in preparing the two following management plans, the Environmental and Social Management Plans (ESMPs) and the Resettlement Action Plans (RAPs). An Environmental and social audit conducted by the Bank in 2013 found that safeguards continued to be implemented satisfactorily at the NGO level. A consulting firm was hired by the PIU to manage all safeguards issues. This firm continued until the end of the project and assured safeguard compliance. According to project documents, project interventions remained small in scale and resulted in minimal safeguard risks that were mitigated by the NGO implementers. As reported in the ICR, “the collaboration between NGOs and local authorities was particularly helpful in managing localized conflict between groups” (ICR, para. 77). b. Fiduciary Compliance Financial management performance was rated moderately satisfactory for most of the project. The period between June 2016 and July 2017 it was rated moderately unsatisfactory due to disruptions related to the July 2016 conflict (ICR, para. 78). A consultant financial management specialist supported the project (ICR, para. 78). To mitigate the heightened fiduciary risk following the conflict, the implementation of key project components was contracted to FAO. All interim financial reports and audit reports, with the exception of the audit report of the financial year June 30, 2017, were submitted within the timelines stipulated in the legal agreement (ICR, para. 78). All the audit reports expressed unqualified opinions with no major financial management issue except for the audit report for the financial year that ended on June 30, 2017 (ICR, para. 78). The auditor expressed a disclaimer of opinion based on the inability to obtain documentary evidence related to an expenditure of US$2.5 million incurred through FAO (ICR, para. 78). Withdrawal applications were submitted to the Bank on a regular basis with no major delays in funds flow. Procurement rating remained high throughout the project implementation. Challenges included institutional weakness to implement project procurement activities and geographical and political factors (ICR, para. 79). With a new government in place, regulations had no legal framework and to date, no Procurement Act Page 23 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) has been passed by the government (ICR, para. 79). Project procurement negatively impacted the project as it often took more than six months to hire a firm (ICR, para. 79). As a result, the project was unable to adhere to the cropping calendar (ICR, para. 79). Conflict also affected the procurement and contract management, which was supported by experienced procurement consultants. Project supervision was limited to Juba town as staff were unable to move beyond Juba town. Procurement objectives were largely met with no mis-procurement activities recorded (ICR, para. 79). c. Unintended impacts (Positive or Negative) None d. Other None 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR contained an excellent set of lessons. The following three lessons were chosen from those in the ICR with some amendments in language. 1. In fragile and conflict environments consider implementing the project with reputable NGOs that are familiar with the context. For example, in this project, working with NGOs that knew the context and had relationships with local leaders was critical to gaining access and implement activities on a timely basis. The project was effective at working with NGOs that had experience implementing agricultural activities and could hit the ground running rapidly. That said, the World Bank and the implementing agencies must also create the time to ensure that there is a common vision, including an understanding of goals, methodologies, and monitoring priorities (ICR, para 94). Page 24 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) 2. In fragile and conflict environments, project design needs to be flexible, adaptive, and with longer implementation timelines. A flexible and adaptive management approach by the government and NGOs partners ensured that the project could continue to operate even as local circumstances changed. For example, in Northern Bahr el Ghazal, government and an NGO partner shifted activities once the initial plan of a donkey- powered traction did not work due to a tse-tse fly infestation (ICR, para. 94). That said, small grants lasting a short period of time prevented planning for longer engagement and created trust issues with communities (ICR, para. 94). 3. Fragile and conflict environments also provide space to innovate. For example, NGO partners, working with government scientists, were able to introduce good quality seeds with traits beneficial to the South Sudan context. For example, they introduced Red Beauty groundnut, which was more tolerant of rosette and leaf spots. They also introduced NERICA rice which is more resistant to environmental stress and contains more protein than traditional varieties (ICR, para. 94). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, well written and interesting with a logical outline which complied with OPCS guidelines. The ICR also provided a useful theory of change that helped frame the report and the findings. The efficacy section was (given the implementation difficulties encountered) quite clear. In contrast the efficiency section, which understandably sought to use data from other similar projects, needed more clarity and substance. The ICR correctly pointed out key weaknesses in the project’s monitoring and evaluation system that prevented the project from fully articulating its results. The ICR also acknowledges the impact of the operating environment while simultaneously providing a candid reflection of missed opportunities. The lessons learned were excellent and are potentially beneficial to those designing projects operating in fragile and conflict environments. There were several minor inconsistencies between results in the ICR's narrative and those reported in the Annex’s results framework. Two serious and confusing inconsistencies in the reporting on ratings were, however, noticed in the ICR. The first was the rating of the project's overall outcome which the ICR narrative argued in paragraph 48 "simply did not have the data to prove efficacy and efficiency to meet the moderately satisfactory threshold". On the other hand the summary of key ratings in the ICR's Data Sheet indicated a rating of "moderately satisfactory" for the project's outcome. The second was the rating of the project's quality of monitoring and the evaluation which was stated as “negligible” in paragraph 75 in the ICR but recorded as “modest " in the summary of key ratings in the Data Sheet. Page 25 of 26 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review S. Sudan Emerg. Food Crisis Res. P (P113586) This review salutes the candor of the ICR's main text and acknowledges the considerable difficulties of proving a project's efficacy and efficiency as well as meeting the challenges of M&E in an FCV environment such as South Sudan. Nevertheless, with respect to the project's efficacy this review considered that the information diligently put together in the ICR supported a rating of "substantial" after restructuring. Consequently, in conjunction with a rating of "substantial" for relevance and "modest" for efficiency this review rated the project's overall outcome as "moderately satisfactory". Based on the data that were accumulated (despite the challenges), this review rated the quality of M&E as "modest". While there were important shortcomings, the review rates the quality of the ICR as substantial. a. Quality of ICR Rating Substantial Page 26 of 26