Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) Report Number: ICRR0021593 1. Project Data Project ID Project Name P109224 UG-ATAAS Country Practice Area(Lead) Uganda Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) COFN-C1300,IDA-47690 30-Jun-2015 122,379,981.59 Bank Approval Date Closing Date (Actual) 22-Jun-2010 25-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 120,000,000.00 0.00 Revised Commitment 119,997,788.82 0.00 Actual 115,179,981.59 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ebru Karamete Christopher David Christopher David Nelson IEGSD (Unit 4) Nelson P108886_TBL Project ID Project Name P108886 UG-SLM Country Program GEF (SIP) ( P108886 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-97184 7200000.00 Bank Approval Date Closing Date (Actual) Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) 22-Jun-2010 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 7,200,000.00 Revised Commitment 0.00 7,200,000.00 Actual 0.00 7,200,000.00 2. Project Objectives and Components DEVOBJ_TBL a. Objectives This is a fully blended IDA/GEF financed project that has both project development objectives and global environment objectives. The formulations of the project development objectives were identical in the Amendment to the Financing Agreement (p. 3) and the Project Appraisal Document (p. 7), which were: “to increase agricultural productivity and incomes of participating households by improving the performance of agricultural research and advisory service systems in the Republic of Uganda”. Global Environment Objective (GEO) is: “is to enhance the environmental sustainability and resilience of agricultural production to land degradation and climate risks”. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The project originally had five components: 1. Developing Agricultural Technologies and Strengthening the National Agricultural Research System (Appraisal: US$137.8 million, Actual: US$ 115.5 million). Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) In order to strengthen research capacity, this component provided support to the National Agricultural Research Organization (NARO) for technology development and to National Agricultural Research System (NARS) for institutional strengthening. 2. Enhancing Partnerships between Agricultural Research and Other Stakeholders (Appraisal: US$72.4 million, Actual: US$67.1 million) The component planned to finance activities that would facilitate close links between public and private research agencies and agricultural service providers, processors, marketing agents and farmers. Main activities included technology scale‐up through nine Zonal Agricultural Research and Development Institutes and district adaptive research support teams covering all Ugandan districts, and multi-stakeholder innovation platforms (MSIPs) established from community to zone levels; scale‐up of SLM practices through farm trials, ZARDI demonstrations, and farmer field days. After the restructuring in 2015, Component 2 was modified to be jointly implemented by NARO Secretariat and the MAAIF, with extension activities scaled up to fill the vacuum left by the exit of NAADS advisory provision. 3. Strengthening the National Agricultural Advisory Services (Appraisal: US$317.8 million, Actual: US$ 151.5 million) This component planned to finance the delivery of demand‐driven and market‐oriented advisory services through performance‐based contracts with professional and certified AASPs at the district level. However, the component was dropped after the restructuring in 2015. 4. Supporting Agribusiness Services and Market Linkages (Appraisal: US$63.0 million, Actual: US$6.7 million) The component aimed to support smallholders to integrate in value chains via provision of agribusiness development services and matching grants to farmer groups and emerging farmers for promising marketing, value addition, and agro‐processing activities. However, the component was dropped after the restructuring in 2015. 5. Project Management (Appraisal: US$74.5 million, Actual: US$53.6 million) The component financed coordination and management of project activities. Another component was included via Restructuring in 2015. 6. Strengthening Agricultural Support Services (Appraisal: US$0 million, Actual: US$26.9 million) Activities under the new Component 6 facilitated the development of sustainable channels for market‐oriented technology uptake. All activities under the original Component 4 were included except (a) the provision of market information services and (b) the Commercialization Challenge Fund (Subcomponent 4.2), which was then moved, to be implemented under the Agriculture Cluster Development Project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) Project Cost: Total project costs estimated at appraisal was US$665.5 million, and at project closing 63 percent (US$421.4 million) was spent due to a major policy and institutional shift by the government that led to reducing project scope significantly. Financing: The project was financed by an IDA credit (IDA-47690) of US$120 million, a GEF grant (TF- 97184) US$7.2 million; and contributions from other development partners (IFAD loan of US$14 million, EU and DANIDA funds of US$26.3 million) who planned to provide US$41 million. At project closing disbursement under the IDA contribution was US$115.2 million (96 percent of the original amount); the GEF grant was fulfilled and development partners contributed only US$4.6 million (11 percent). EU and DONAIDA funds did not materialize due to policy changes in their aid programs and IFAD decided to cancel its advisory services support commitments after disbursement of US$4.6 million, after the significant policy reforms and implementation changes. Borrower Contribution: The borrower planned to provide US$497 million but ended up providing USD$299 million (45 percent) as the government decided through a reform, to reorient technology uptake grants delivered through the previous extension system that the project relied upon, into a national mechanism for distribution of agricultural inputs outside of the project. Dates: The project was prepared and presented to the Board for approval within seven months, and it was approved on June 22, 2010; but project effectiveness took almost 1.5 years (December 20, 2011) due to a lengthy parliamentary ratification process, general elections in Uganda that took place in February 2011 and staff replacements made in the National Advisory Services Secretariat. The project financing agreement for the GEF grant was signed on July 19, 2011. The project planned to close on June 30, 2016 but closed about 2 years later on June 25, 2018. The reason for the extension was mainly to facilitate large scale policy and institutional changes that interrupted project implementation and that required new implementation arrangements and the revision of components. Restructurings: Two level II restructurings that didn’t require board approval were made under the project. The first restructuring was on April 14, 2015 that included: (i) modification of project components after the government’s transfer of extension services from NAADS to the newly created Directorate of Extension Services in Ministry of Agriculture Animal Industry and Fisheries (MAAIF) and reorienting technology uptake grants for farmer group enterprises delivered through NAADS extension system under Component 3 into a national mechanism for distribution of agricultural inputs outside the project. In this regard, Components 3 and 4 were dropped and a new component was created that included a subset of NAADs activities; (ii) Reduction of project size from US$665.5 to US$421.3 million; (iii) Revision of the Results Framework to match the scope, which included minor changes to the PDO indicators; (iv) Modification of institutional implementation arrangements via dropping of NAADS; (v) Extension of closing date by 1.5 years to December 31, 2017. A second restructuring took place on August 17, 2017, for the re‐allocation of project resources to address a Fall Army Worm outbreak that threatened project gains and for an extension of the closing date by six months to June 25, 2018 to address the effects of a prolonged drought. 3. Relevance of Objectives Rationale Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) The Project development objectives were susbtantially relevant to the World Bank, and to the Uganda government’s country strategies. The agriculture sector has been a key sector for the structural transformation of the economy through value addition, export growth, and employment (during the time of appraisal agriculture accounted for 20 percent of gross domestic product, exports of primary agriculture commodities accounted for over 50 percent of the country’s exports) (PAD p. 1). Although the yields for most products were growing at the time of appraisal, there was significant untapped potential for agricultural growth, particularly for smallholders. Factors that inhibited productivity and commercialization for small farmers included the limited use of improved technologies and inputs and poor integration of smallholders into markets and value chains. In addition, an increasingly variable climate, as well as soil erosion and deteriorating soil quality that was intensified by unfavorable farming practices including poor nutrient management and lack of sustainable land management practices further constrained productivity and production growth. The World Bank and other donors had invested heavily in and provided technical support for the institutional development of agricultural research and extension services, i.e. National Agricultural Research Organization (NARO) and National Agricultural Advisory Services (NAADS). At the time of project appraisal, NAADS had developed into a decentralized service delivery model with farmer institutions playing a central role. The potential of NARO and NAADS was planned to be enhanced by strengthening linkages between the two institutions, and other stakeholders. The project development objectives were and continued to be relevant with the country strategies and priorities. The agriculture sector was prominent as a priority investment area in Uganda’s National Development Plan (NDP I- 2010-2014), Uganda’s new five-year strategic framework for economic development. Raising agricultural incomes was the centerpiece of the Prosperity for All Program, the vision driving the NDP. The Development Strategy and Investment Plan (DSIP) of the Ministry of Agriculture, Animal Industry, and Fisheries (MAAIF) was designed within this framework. The DSIP identified agricultural research and advisory services as two of the core mandates of the Ministry of Agriculture, Animal Industry, and Fisheries (MAAIF). At project closing, Uganda’s Vision 2040 and NDP ‐ II ( 215-2020) continued to prioritize strategic investments in agricultural productivity and commercialization and demonstrated continuity with the project approach by targeting investments in selected commodities, with a strong emphasis on strengthening agricultural research, implementing the Single Spine extension system and strengthening agricultural institutions for service delivery. The Agriculture Sector Strategic Plan (2015- 2020) reviewed progress on DSIP programs and maintained course with its investment objectives of increasing agricultural production and productivity, providing access to critical farm inputs, improving agricultural markets and value addition, and ensuring service delivery. The PDO was aligned with the World Bank strategies at appraisal and closing. The project was a successor to two World Bank projects that had made foundational investments in the institutional development of the main implementing agencies. IDA had supported institution building for core public services, particularly on agricultural R&E, which was reflected in investment projects and Pillar 2 of the Country Assistance Strategy (CAS- FY 2005–2009). Enhancing Competitiveness, Production and Incomes remained relevant and aligned with the FY16–FY20 Country Partnership Framework (CPF). The World Bank’s CPF overlapped with the government priorities, with ‘raising incomes in rural areas and increased agricultural commercialization. Importantly, the project sought to respond to an identified development problem in better linking research, extension services and the ability of farmers to better integrate and align their output to existing and emerging supply chains. The theory of change outlined in the project’s ICR provides an overview of these Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) links, illustrating the need to better combine changed farm practices to new technologies and support an enabling environment for accessing agriculture markets. However, the way these essential constraints interact is not well articulated in the design and the significant structural changes made to the various ministries shows how difficult it is to hold the different strands of the theory together. The development problem is broad, but the PAD does not sufficiently articulate how each of the responses will ultimately lead to the expected changes. Thus, while the project is aligned to country and bank strategies, the relevance of the PDO to specific constraints is not well articulated and thus is rated Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase agricultural productivity and incomes of participating households. Rationale For the purpose of this review we take the objective statement as, "to increase agricultural productivity and incomes of participating households" and the second part, "improving the performance of agricultural research and advisory services" as an intermediate outcome. There are three objectives: (i) to increase agricultural productivity of participating households; (ii) to increase incomes of participating households; (iii) although not rated the GEO objective ‘to enhance the environmental sustainability and resilience of agricultural production to land degradation and climate risks’. Project’s outputs and outcomes serve to achieve all three objectives. One of the main activities under the project’s Theory of Change (TOC) was the demand‐driven development of agricultural technologies for marketable commodities, coupled with delivery of market‐oriented advisory services through farmer groups (FGs), which would enhance the adoption of productivity‐enhancing technologies, innovations, and improved management practices (TIMPs), as well as Sustainable Land Management Activities (SLM). In turn, this was to lead to higher yields, a growth in a high‐quality marketable surplus, larger revenues, incomes, and a shift towards a more profitable enterprise mix over time. Activities that would reinforce these outcomes also included the establishment and strengthening of farmer groups to establish a better bargaining position and increased marketed share of agricultural production and provision of matching grants for building linkages with value chains and agribusinesses. In the long term, agricultural productivity and income increases would be protected against climate risks through SLM activities, leading to higher and sustainable agricultural growth and poverty reduction. While the TOC was sound, the assumptions were overly complex and somewhat unrealistic. For example, two of the assumptions were, the frequency, duration, and quality of demonstration, training, and advisory services could induce behavioral change and adoption of improved practices by farmers; and farmers in their groups would demand high‐quality and improved technologies, even in the presence of imperfectly Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) substitutable inputs varying on the price and quality dimensions. It was not clear what specific incentives would promote the adoption of the improved technologies and practices. Also, it was not clear how sustainability of SLM activities were ensured, as another assumption was that the SLM beneficiaries would have the organizational and financial capacity to provide maintenance on the infrastructure beyond the lifetime of the project. Another weakness with the results framework was that, the SLM practices were monitored only in terms of outputs (ha of land area with improved practices) and for the adoption of improved technologies, "adoption" was not clearly defined. After the project restructuring two project components were dropped, which impacted the ToC. Specifically, the dropping of technology uptake and CCF matching grants for farmer group enterprises meant that farmer groups were still able to generate more marketable surplus due to productivity gains but no longer had direct access to financial support for integrating with established value chains, or commercializing their production, thus lowering the magnitude of income gains that could have been achieved. In response to the project’s theory of change, the following achievements were recorded: Outputs:  91 innovative research projects were supported under the Competitive Grant Scheme; the recipients were 16 agricultural research institutes, 4 universities and 18 private sector enterprises.  81 new technologies were developed for five tracked market-oriented commodities (varieties of maize, rice, beans, cassava plant, and dairy).  216 on‐station adaptive trials, 11,771 on‐farm demonstrations, and 186 SLM community‐level interventions were conducted nationwide by the Zonal Agricultural Research Development Institute (ZARDIs) District Adaptive Research Support Teams (DARSTs), and district‐level extension officers, covering 12 locally prioritized commodities (including bananas, coffee, and fish). Through these activities, 327,059 new direct beneficiaries were reached.  Foundation seeds and multiplication training to 14 private seed companies, several entrepreneurs, and three community producers were provided.  78 Multi-Stakeholder Innovation Platforms (MSIPs) covering all nine Agro-Ecological Zones were given training and financial support to operationalize, and engage in diverse commercial activities such as rice seed multiplication, maize flour processing, banana wine production, honey production, ghee making, fish farming, and other productive undertakings.  The project overall covered 1.68 million households (25 percent of all rural households) through different means (slightly less than the target of 1.7 million). Intermediate Outcomes:  Agricultural technologies generated by the National Agricultural Research Organization (NARO) system grew from 600 to 888 between appraisal and closing, a 48 percent increase, exceeding the target of 20 percent.  In terms of access to extension services and adoption of technologies, the ICR reported that (para 34), during the earlier implementation design of the project (between 2011–2013), the share of beneficiaries with direct access to extension services grew from 47.8 percent to 59.0 percent; the Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) share of beneficiaries trained by NAADS grew from 46.8 percent to 67.5 percent; and the adoption of technologies disseminated by NAADS grew from 36.1 percent to 49.2 percent.  According to the ICR, (para. 35), after the reforms, extension outreach coordinated through NARO, Ministry of Agriculture Animal Industry and Fisheries (MAAIF), and the Local Governments (LGs) led to greater availability and adoption of technologies. The adoption rate at the end of project by project beneficiaries was 78 percent, (specifically 95.3 percent by crop enterprises, 63.3 percent by livestock enterprises, and 31.7 percent by SLM farmers, respectively). However, the ICR did not provide evidence as to how adoption was defined, covered which technologies, and how it was measured under the project. The project covered 20,930 ha of land with SLM practices, significantly exceeding the target of 11,000 ha. Highlands terracing and rehabilitation of degraded watersheds was adopted by a large share of communities, with their final coverage of 3,391 ha and 3,337 ha (771 percent and 556 percent of respective targets). The technologies selected for promotion and scaling‐up were terraces, contour bunds, grass bunds, conservation agriculture (low‐till), rehabilitation/reclamation of degraded watersheds, agroforestry woodlots, agronomic/vegetative SLM practices (mulching, intercropping, rotations, integrated nutrient management, grassland improvement, and so on), small‐scale irrigation, and water harvesting. The ICR also noted that (para. 44) SLM interventions benefited not only the tracked commodities, but others such as coffee and banana plantations for which project interventions helped to control soil erosion that helped to improve yields and income. Outcomes: Even though yield data presented was not completely robust, there is evidence that yield increases compared to the baseline and compared to the non-beneficiaries were substantial and reflected the project’s efforts. At closing, yield growth for maize (80.5 percent), rice (180.4 percent), cassava (126.3 percent), beans (47.6 percent), and milk (120 percent) exceeded the targeted increases of 15 percent and 20 percent for crops and milk, among project beneficiaries as opposed to the baseline (2013). However, the ICR noted that (para. 36), the baseline figures used country-level yields and did not reflect the project beneficiaries’ actual yields per se. According to the Impact Evaluation results, at project closing project beneficiary yields were measured for two seasons: (i) for season A, based on crop-cut method project beneficiary yields in 2017 were higher than the non-beneficiaries’ (125 percent higher for maize, 154 percent higher for rice, 96 percent higher for cassava and 200 percent higher for beans); (ii) for season B, yield differences were lower but less reliable as the data was based on beneficiary recall of the figures (yield difference between beneficiaries and non-beneficiaries were 13 percent for maize, 38 percent for rice, 2.5 percent for cassava, 129 percent for beans) (ICR page 64). Nevertheless, due to an absence of an M&E system systematically tracking beneficiary and non- beneficiary data from the beginning of the project, yield assessment was not robust. Yet the ICR argued that even though full army worm disease impacted particularly maize and beans in 2016-2017 season, yield increases reflected full recovery among project beneficiaries. It is worth noting that cassava yields were particularly attributable to promotion and dissemination of improved seed varieties under the project. Rating Substantial OBJECTIVE 2 Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) Objective To increase incomes of participating households. Rationale The outputs and intermediate outcomes outlined above for Objective 1 apply to the achievements of Objective 2. Outcomes: Net agricultural incomes of project beneficiaries grew about 200 percent over the duration of the project significantly exceeding the targets (20 percent for men and 15 percent for women). At project closing, project beneficiaries had higher incomes compared to non-project beneficiaries. The impact evaluation (2018) shows that net agricultural incomes for beneficiaries were higher than non‐beneficiaries operating maize, rice, cassava, beans, and dairy enterprises with results of 57.8 percent, 39.5 percent, 27.8 percent, 400.5 percent, and 1,387.5 percent, respectively (ICR page 21). Rating Substantial OBJECTIVE 3 Objective To enhance the environmental sustainability and resilience of agricultural production to land degradation and climate risks. Rationale Outputs: The project covered 20,930 ha of land with SLM practices, significantly exceeding the target of 11,000 ha. Highlands terracing and rehabilitation of degraded watersheds was adopted by a large share of communities, with their final coverage of 3,391 ha and 3,337 ha (771 percent and 556 percent of respective targets). The technologies selected for promotion and scaling‐up were terraces, contour bunds, grass bunds, conservation agriculture (low‐till), rehabilitation/reclamation of degraded watersheds, agroforestry woodlots, agronomic/vegetative SLM practices (mulching, intercropping, rotations, integrated nutrient management, grassland improvement, and so on), small‐scale irrigation, and water harvesting. Outcomes: The ICR noted that (p. 23) using the Ex Ante Carbon‐Balance Tool, it was calculated that in aggregate, SLM interventions have sequestered 1,964,831 tons of carbon over the lifetime of the project (US$151.3 million in 2018 prices. The ICR presented output level evidence on the achievement of the global environment objectives (GEO) of enhancing the environmental sustainability and resilience of agricultural production to land degradation and climate risks, but as per ICR Review guidelines, the GEO is not rated. Although it is hard to capture impacts of such activities over the lifetime of investment projects, it is important to establish systems to monitor Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) impacts for the long term on for example: soil quality, fertility and salinity, erosion, vegetation cover, water quality and water levels. Rating Not Rated/Not Applicable OVERALL EFF TBL OLD Rationale Although the project changed course through substantial policy revisions and several components were scaled back, it ended up achieving its development objectives of increasing agricultural productivity and incomes. While not rated, this review considered the GEO as the third objective. However, the ICR presented only output level evidence on the achievement GEO objectives. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The ex-ante economic and financial analysis covered all project activities and assumed three main benefit streams: increase in yields as a result of technologies developed and disseminated; diversification to more profitable crops as a result of stronger market links and increased farm gate prices as a result of higher quality product; and enhanced bargaining power of farmers. The commodities included in the analysis were sorghum, maize, cassava, Irish and sweet potatoes, millet, groundnut, beans, bananas, coffee, and cotton, but did not include rice and investments in livestock production. Assumptions were: (i) yield increase between 4 percent (coffee) and 50 percent (groundnut), (ii) income increase by 20 percent by the end of the project for project beneficiaries. This resulted in an economic internal rate of return (EIRR) estimated to be 33.9 percent and a positive net present value (NPV) amounted to US$1.3 billion. Ex-post economic analysis readjusted certain key parameters: number of beneficiaries, yield increases, adoption rate, and project cost streams. The ex-post analysis additionally included a project carbon balance as a benefit stream (it was estimated that GHG emitted or sequestered by the project measured as CO2 equivalent was 1.2 tons per hectare per year). However, the ICR did not report on the monetary value of CO2 emissions used in the analysis. In addition, rice was also included. Some other assumptions were: (i) the adoption rate of 49.2 percent was used for SLM practices and technologies, which was the lower estimate from 2013; (ii) A lag of 5 years for crop diversification/switching was used to account for slow implementation; (iii) 20 years duration and 2018 prices in 2010 constant terms were applied. Farm budgets (by hectare) were prepared for farms using existing traditional technologies and beneficiary farms using improved new technologies released by NARO. The gross revenue was calculated based on yields per hectare and farm‐gate sale prices. The ICR did not present financial rates of returns coming from the farm budget analysis. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) Based on these assumptions, the analysis yielded an EIRR of 37.5 percent, a NPV of US$700 million, and a NPV per beneficiary of US$309. The analysis showed higher returns than estimated at appraisal, mainly due to larger yield growth than anticipated and accounting for carbon benefits. However, the ICR did not provide monetary amounts for the annual carbon benefit streams included in the analysis. Sensitivity analysis was used to test for impacts of elite capture (reaching only a small portion of total number of beneficiaries), adverse weather shock, increase in fuel prices, and unexpected drops in agricultural prices. All things being equal, in the case of extreme elite capture that created the largest negative impact the project would still yield an EIRR of 13 percent. Operational and Administrative Efficiency: The project had significant delays, closing date was extended for 3 years due to elections, and government policy and institutional changes. Right after the project’s effectiveness, the elections led to complete staff changes in NAADS, which delayed implementation. After the GoU’s introduction of a new extension policy in 2013 and the subsequent transfer of the extension mandate to the MAAIF from NAADS, the lead agency responsible for implementing more than 60 percent of the project (by cost), meant that the project underwent a major realignment midway through implementation. At restructuring, significant modifications were made to components and activities. This major change created substantial disruptions in human resources and institutional capacity. Due to administrative and operational inefficiencies project efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  34.00  Not Applicable 100.00 ICR Estimate  38.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The Project development objectives were susbtantially relevant to the global environment, World Bank, and Uganda country strategies in general. Efficacy is rated substantial as project development objectives to increase agricultural productivity and incomes of participating households, was substantially achieved. Due to administrative and operational inefficiencies of the project, overall efficiency is rated modest. These ratings result in a moderately satisfactory outcome rating. Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome There are three types of risks to development outcome: Institutional Sustainability Risk. Government’s recent policy measures, agriculture sector strategy and national agriculture policy, have provisions to help continue project activities going forward. In addition, the government recently increased funding for the new agricultural extension system established under the project, thus ensuring that resources will remain available for the research and extension agencies. Regarding SLM activities, interventions have been mainstreamed into local government plans and budgets to ensure the maintenance of community level structures. Moreover, through project support, NARO and the MAAIF has improved or built capacity (systems, human resources and budgets) on planning, budgeting, coordination, M&E, and reporting processes to operate a demand‐driven technology development and dissemination system with close collaborations between farmer groups, Zonal Agriculture Research Development Institutes, and local governments. NARO is exploring the option of commercializing its technology outputs to become a financially sustainable entity. Risk to Sustainability of Technology Adoption. There is an ongoing public program called ‘Operation Wealth Creation’ (OWC) that competes with and has the potential to crowd out NARO’s improved seeds and planting materials. The program aims to support several strategic agricultural interventions including farming inputs free of charge, which are not complemented with advisory services. The continued provision of free, and often, lower‐quality inputs under OWC could lead to the dis‐adoption and potential reversal of gains in agricultural productivity achieved under the project. Risks from Pests and Disease. Uganda experienced a FAW outbreak in 2016 that had caused yield losses of 15–20 percent, the economic and social damage in several African countries was significant. Project support helped containing the threat from ‘fall army warm’ through surveillance, monitoring, mass communication and pest control. MAAIF is now committed to provide continued support to districts with pesticides and surveillance for pests and diseases in priority commodities. The government is in dialogue with the Brazilian government to explore collaboration mechanisms and the establishment of a biological control facility in the country. A newly established early warning system will help contain the outbreaks as well. 8. Assessment of Bank Performance a. Quality-at-Entry The project was designed to be the government’s flagship project in the agriculture sector, with a large financial scale and national coverage. It drew lessons from the two preceding IDA‐financed projects, Agricultural Research and Training Project (closed in 2009) and National Agricultural Advisory Services Project (closed in 2009), as well as program areas identified by the Development Strategy and Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) Investment Plan’s formulation process. Some of the key lessons were; (i) Governance weaknesses at the LG level resulted in corruption and misuse of NAADS funds, therefore the project design tried to build governance and anti-corruption safeguards at all levels; however, this did not resolve procurement issues completely, or prevent ineligible expenditures that the project encountered during implementation. (ii) The need to enhance NAADS interface with the local governments to improve ownership, and the need to better coordinate NARO-NAADS linkages to expand outreach of extension services via use of research results. Joint missions conducted with other development partners enabled the preparation team to receive shared knowledge from ongoing projects supported by DANIDA, U.S. Agency for International Development, and IFAD. The project design was based on a clear, if convoluted logic connecting its components from technology generation to supporting agribusiness services and building market links. However, there were shortcomings. First, there was only a limited articulation of how the various parts of the design were intended to work together in and through various parties including research organizations, government departments and local authorities. There were also significant weaknesses in terms of financing allocations; i.e. nearly US$318 million (48 percent of the total project cost) was allocated to Component 3, and 75 percent of which was to be financed by the government to be allocated to NAADS. This design feature left the project highly exposed to a risk of policy and institutional shifts, especially in a political environment in which the role of NAADS came under high public scrutiny during preparation and approval. Implementation arrangements were also weak, i.e. the decision to have two parallel NARO and NAADS Secretariats implement the project without a single PCU for overall coordination and reporting roles significantly weakened the research & extension links that the project tried to build. M&E design was also weak and the M&E arrangements were incompletely defined. Diffused responsibility ultimately meant that accountability was compromised. In terms of risk mitigation measures, the low‐probability and high‐impact risk of policy change in the NAADS mandate was not identified. While it would not be possible to instill sufficient mitigation mechanisms to deal with such a significant policy change initiated by the government, there were lessons pointing towards variability of agricultural responsibilities that might have been referenced in the design. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The ICR noted that (p. 34), the World Bank supervision in general was sound in addressing major changes in institutional capacity and policy environment as well as addressing production shocks faced by beneficiaries, demonstrated by the timely responses to the extension policy shift and FAW‐related crises to conduct two Level 2 restructurings of the project and closing day extensions. The World Bank team carried out close supervision with all three supervision task team leaders being based in Kampala that helped particularly coordinate responses to shocks such as drought, new plant diseases, and pests, which threatened to derail the project. Communications and consultations with the client were reportedly regular, open, and transparent with reporting ratings and candid discussions on key issues. There were adequate number of supervision missions (10 implementation support missions in total) supported by a seasoned team of local and international experts who advised the client on M&E, procurement, and FM. Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) A key area that could have been addressed better by the task team is M&E. An M&E specialist as a full‐time member of the World Bank team could have provided timely support to NAARO, NAADS, and then the MAAIF and enhanced the quality of M&E utilization for measuring progress in the project. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design There were several weaknesses with the M&E design: (i) The intermediate indicator on adoption of technologies was not clearly defined, i.e. it was not clear which technologies were considered in the measurement and how they were measured; (ii) SLM activities was monitored only as number of ha, not in terms of outcomes and impacts, indeed outcome level indicators for the global environment objective were not included; (iii) PDO‐level indicators were incompletely defined (livestock productivity) and had baseline values that were erroneous (agricultural income) or outdated (crop yields), thus introducing ambiguity in benchmarking progress and project evaluation; (iv) institutional arrangements for M&E was delegated to four entities (NAADS, NARO, MAAIF, and Uganda Bureau of Statistics-UBoS) initially that made tasking and decision making challenging. b. M&E Implementation The M&E framework was inadequately implemented due to several reasons: (i) an M&E manual to outline the timing, responsibility, and methodological approach of surveys was not developed in time; (ii) a digital management information system (MIS) was not developed to maintain records during the lifetime of the project; (iii) ad hoc consulting firms were used instead of UBoS to conduct surveys and produce reports; (iv) the high turnover in M&E specialists and change in implementing agencies did not allow for consistency in approaches used for tracking, managing, and processing data. c. M&E Utilization At the Mid-Term Review stage, M&E data on project outputs were used to identify pre‐restructuring progress and delays in the implementation of planned activities. M&E data played a role in refining the activities of the project at review and planning meetings that helped to revise end targets for activities (eg. MARO’s activities on technology generation). Nevertheless, weaknesses in data transfer from NAADS to MAAIF prevented the project from utilizing real‐time feedback on progress toward PDO targets on indicators for number of beneficiaries, adoption, yields, and incomes. Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental Safeguards. The project was considered Category B at appraisal, triggering two safeguard policies: Environmental Assessment (OP/BP 4.01) and Pest Management (OP 4.09). The ICR (p. 31) reported that an Environmental Assessment was conducted. Until the Mid-Term Review, MAAIF faced challenges in implementation of the ESMF, due to the lack of safeguards capacity, a delay in training of project implementers, delay in implementation of the Strategic Environmental and Social Assessment, and absence of reporting on environmental issues in the quarterly and semiannual progress reports. Based on these shortcomings, environmental safeguards were rated as Moderately Unsatisfactory. This rating was upgraded to Moderately Satisfactory after NARO appointed a specialist for implementation of safeguards, screening of planned civil works under ESMPs and the successful demonstration of Integrated Pest Management practices. The MAAIF also developed a draft environment and social safeguards (ESS) management policy, guidelines, and plans for operationalizing the system. The guidelines include the ESS Manual; Environmental and Social Risk Management Procedure including the Environmental and Social Grant screening checklist and risk register template; and management plans for biodiversity, waste/hazardous waste, pests, natural resources, and stakeholders. Social Safeguards. The ICR reported that (p.32) NARO appointed a social safeguards coordinator and institutionalized Environmental and Social Safeguards in all its projects. Focal points at the district level were identified for all nine ZARDIs to address issues on HIV/AIDS, child labor, sexual harassment and gender, and grievance redress mechanisms being established in all NARO programs. In addition, NARO has also developed a Gender and Diversity strategy to guide the process of addressing project related gender concerns, which the World Bank has recommended to be expanded to the Agriculture Cluster development Project. Social safeguards were rated Moderately Satisfactory through the project period, but towards the end this rating increased to Satisfactory due to the realization of the above-mentioned progress. b. Fiduciary Compliance Financial Management (FM). The FM of the project had several shortcomings. The internal control system was weak due to insufficient supporting documentation for expenditures, low quality of the planning and budget follow‐up, and delays in implementing external and internal auditors’ recommendations. During the early implementation period, there were several instances of ineligible expenditures on input provision (technology uptake grants used for input provision). This issue was resolved with ineligible expenditures of US$1.36 million being identified and refunded to IDA. Due to increased activities and resources to NARO, coupled with its reporting weaknesses, staffing levels at NARO headquarters and ZARDIs were to be enhanced through recruitment of one accountant to be based at headquarters and three accountants to be assigned to all nine ZARDIs to support district‐level activities. In 2017, when the auditor general issued a qualified opinion on NARO’s financial statements, the Bank team made it mandatory for NARO to seek approval from the Ministry of Finance for budget reallocation and increases in expenditure ceilings for per diem allowances, seminars, and workshops to curtail irregular and unsupported payments. In general, the Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) MAAIF was receptive to the World Bank’s comments on its FM, but it was slow at implementing the recommended actions due to capacity constraints. The FM was rated as Moderately Satisfactory during project implementation due to these weaknesses. Procurement. The ICR noted that (p.32), initially there was misalignment of work plans and procurement and several cases of potential mis-procurement were noted. In particular, additional procurements outside the agreed Procurement Plan were undertaken, and the management overrode procurement and internal control procedures at the district and sub-county levels, with the absence of procurement records, prequalified suppliers, non‐involvement of contract committees, and single sourcing being common violations. This issue was one of the main reasons for the government decision to separate advisory/extension service provision from input delivery in NAADS. After the project restructuring, there were delayed procurement activities caused by insufficient capacity of MAAIF on procurement, bureaucracies in the procurement cycle players, management of contracts, lack of contract management information from users, and insufficient funding for some initiated procurements. These challenges were mostly addressed by preparation of a manual to elaborate procurement arrangements under the MAAIF, hiring of a dedicated procurement specialist and the monthly monitoring of progress reports by a procurement coordinator. Procurement ratings were in general moderately satisfactory. c. Unintended impacts (Positive or Negative) None were reported. d. Other The project was designed to increase women’s participation and voice in Farmer Groups. At project closing, 1.68 million project beneficiaries were women. The evaluation survey carried out in 2017 indicated that 94.3 percent of women reported that their voice had been considered for decision making in farmer groups, substantially exceeding the project target of 65 percent. More than half the households reported joint decision making on purchase, sale, and utilization of assets (land, livestock, and farm equipment). Also, the project supported the development of a gender strategy by NARO, which was integrated in all project activities and other programs implemented by the agency. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) 12. Lessons The ICR provided some important lessons; the most relevant ones follow with some modification of language. In dynamic, short‐term political environments, large‐scale, flagship projects can minimize their exposure to policy risk by reducing the concentration of key large size activities under one implementing agency. This was a large financial scale project with national coverage. However, nearly 50 percent of the total project cost was allocated to Component 3, 75 percent of which was to be financed by the government to be allocated to NAADS. This design feature left the project highly exposed to a risk of policy and institutional shifts, especially in a political environment in which the role of NAADS came under high public scrutiny during preparation and immediately after the project’s Board approval. In retrospect, the project could have reduced the concentration of big‐ticket and politically visible activities of technology promotion, beneficiary selection for uptake grants, and matching grants for commercialization under one implementing agency, to reduce the risk of disruptive institutional change. In countries where parliamentary approval is required, project effectiveness can be accelerated by bringing parliamentarians on board during the preparation stage. In Uganda, parliamentary approval is a condition for effectiveness. Under the project the effectiveness date was delayed by 12 months owing to a lengthy parliamentary ratification process. An early communications outreach to key policy makers like the Uganda Parliamentary Agriculture Committee could have accelerated the ratification process. The establishment of a single PCU can enhance coordination and administrative efficiency of complex projects with a multiplicity of implementation actors. At appraisal, NARO and NAADS were jointly responsible for Component 2 and R&E interfaces and partnership arrangements were planned at the national, zonal, and district levels. However, an explicit coordination unit was not created within the MAAIF, to which both Secretariats would have reported independently. This oversight resulted in the slow mobilization of Component 2 activities such as joint strategic planning, M&E, and SLM until restructuring, when corrective action was taken and a PCU was established within the MAAIF that considerably improved implementation of activities. It is good that the Bank seeks to improve government capacity, but effective PCUs can often still play this role in addition to more effective coordination and oversight. Careful assessment of the need for a PCU must happen early in the design process. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UG-ATAAS (P109224) The report was comprehensive and followed the guidelines and was focused on results. The project’s theory of change was adequately presented and clear in terms of how the ratings had been reached. The ICR’s lessons were useful. The weaknesses are as follows: (i) although they are not rated, the global environment objective outcomes were not assessed and presented in the efficacy section and SLM activities were measured and presented in terms of outputs only; (ii) it was not clear from the ICR why the big policy changes were initiated by the government undermining the decades of institutional support provided by the Bank and the donors on extension services. Explaining this aspect would better clarify the adequacy of the project design on extension. a. Quality of ICR Rating Substantial Page 18 of 18