Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Report Number: ICRR0021715 1. Project Data Project ID Project Name P101508 BR-RJ Sustainable Rural Development Country Practice Area(Lead) Brazil Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-77730,IBRD-82000 30-Nov-2015 89,253,818.25 Bank Approval Date Closing Date (Actual) 10-Sep-2009 30-Nov-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 39,500,000.00 0.00 Revised Commitment 98,735,596.11 0.00 Actual 89,253,818.25 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ebru Karamete J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The formulations of the project development objectives (PDO) for the Rio de Janeiro: Sustainable Rural Development Project were identical in the Project Appraisal Document (page 5) and in the Loan Agreement (page 5), which were “to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory, thus contributing to the higher-order objective of increasing small- scale farming productivity and competitiveness in those areas.” Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) This review will assess the project’s achievements by determining the extent to which it was able “to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory” (which will be referred to as Objective 1 in Section 4 of this review), and “to increase small-scale farming productivity and competitiveness in specific areas of the Borrower’s territory”(which will be referred to as Objective 2 in Section 4 of this review). The project’s PDO was amended during implementation by adding an objective. This will be discussed under the heading of “Restructuring” in Section 2e of this review. The additional objective will be referred to as Objective 3. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 27-Oct-2011 c. Will a split evaluation be undertaken? Yes d. Components The Project had three components: 1: Supporting Rural Production and Competitiveness (Appraisal Estimate: US$66.1 million, Additional Financing: US$108.1million, Actual Cost: Not Available*). This component supported rural beneficiaries via the following: (i) pre-investment activities to strengthen capacity of rural beneficiaries to prepare investment proposals; and (ii) grants to eligible beneficiaries and communities to implement approved investment proposals to improve productive farming systems; as well as compliance with environmental regulations and adoption of agro-ecological and environmentally sound practices; and erosion control, rural roads rehabilitation and maintenance activities. In October 2011, through a Level I restructuring, emergency rehabilitation activities were included to support the areas affected by the natural disaster in January 2011 and a reallocation of US$18.77 million under this component to emergency recovery activities in the Serrana Region was carried out. In October 2012 an Additional Financing of US100 million was approved to restore the portion of the original loan amount that had been used for the emergency rehabilitation activities, as well as to scale up the project scope into new municipalities (13 new) and micro-catchments (166 new). *The actual cost by component at project closing was not clear in the ICR but when asked by IEG, the Bank project team did not clarify this issue. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) 2: Strengthening Institutional Frameworks (Appraisal Estimate: US$5.2 million, Additional Financing: US$4.84 million, Actual Cost: Not Available). This component supported market-driven agricultural development by (i) strengthening rural institutions to provide better services and coordination with other public and private sector stakeholders through implementing specific activities (institutional sub-projects) identified in an institutional sustainability plan and contributing to the implementation of a national policy in support of territorial development; (ii) improving public and private financial support mechanisms through the enhancement of links between the supply and the demand of financial resources for sustainable rural development activities; and (iii) carrying out participatory research to establish a new and effective Sustainable Services Research Network System to conduct agriculture-related research and induce innovation. As mentioned above, additional financing in 2012 scaled up project activities into new municipalities and micro catchments 3: Project Coordination and Information Management (Appraisal Estimate: US$7.6 million, Additional Financing: US$10.5 million, Actual Cost: Not Available). This component supported the project management and coordination functions, including monitoring and evaluation (M&E), and dissemination of key sustainable rural development information before and after restructuring in October 2011. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Total project costs estimated at appraisal was US$79.0 million, and the actual cost at project closing was US$89.3 million. Financing: The project was financed by an IBRD loan (IBRD-77730) of US$39.5 million, and an additional financing loan IBRD-82000) of US$100.0 million. At project closing disbursement under the IBRD loan US$38.7 million and the additional financing loan was US$50.5 million. In September 2017, US$ 40 million from the additional financing was canceled as a result of the delays due to the acute financial crisis faced by the State of Rio de Janeiro, which blocked project implementation for almost one year. The remaining US$9.5 million was canceled at project closing, as some of the planned roads tenders could not proceed due to delays in procurement procedures. Beneficiaries would provide US$18.1 million at appraisal, but the ICR did not report on the final beneficiary contribution and the project team did not clarify this point to IEG. Borrower Contribution: The borrower planned to provide US$39.5 million but ended up not providing any funds. The reason for lack of borrower contribution was not reported by the ICR. The project team subsequently stated that Borrower contribution was zero due to the financial crisis. Dates: The project was approved on September 10, 2010; and it became effective in about six months (March 8, 2010). The project planned to close on November 30, 2015 but closed 3 years later, on November 30, 2018. The reason for the extension was to facilitate first the emergency recovery efforts, as Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) well as expansion of project scope to new areas via an Additional Financing. In addition, delays occurred after the financial crisis that led to judicial power seizure of the project’s designated account for about a year. Restructurings: The project went through one Level I restructuring on October 27, 2011, to amend the PDO, the allocation of US$18.77 million within Component 1 to a separate activity called “Emergency Subprojects”, and revision of the results framework to reflect the introduction of new activities to support emergency recovery efforts after the flooding disaster that affected the Serrana Region which was part of the “specific areas” covered by this project. This restructuring also authorized a one-year extension of the closing date. On October 1, 2012 an Additional Financing of US100 million was approved to: (i) enable the completion of original project activities (19 percent of the proposed additional loan amount) by restoring the portion of the original loan amount that had been used for the “Emergency Subprojects” namely rehabilitation activities; (ii) scale up the project scope into new municipalities (13 new) and micro-catchments (166 new); (iii) two-year extension of closing date. Three level II restructurings that didn’t require board approval were made under the project. The first two restructurings (on May 22, 2013 and June 29, 2015) both included revisions in components and costs, reallocation between disbursement categories and change in institutional arrangements. The last restructuring that was approved on September 1, 2017, was done to cancel US$40.0 million from the additional financing loan, reduce some indicator targets, change the implementing agency to the newly created State Secretariat of Agriculture, Livestock, Fisheries, and Supply, and reallocations between disbursement categories. This review is carrying out only one split rating based on the PDO revision on October 27,2011. While a second split could also be appropriate because of the reduction in most PDO indicator targets with the restructuring in 2017, the reduction in the ambition of already weak indicators for the PDO was not significant. 3. Relevance of Objectives Rationale The project’s original development objectives were substantially relevant to the World Bank and the Brazil country strategies in general at appraisal and closing. However, the original PDO was vague in terms of what an “increase in integrated and sustainable farming systems approaches” meant in terms of scale, the extent to which these farming systems were expected to be “approached” by small-scale farms which the PDO explicitly stated were the group that more efficient farming systems should benefit. At the same time the PAD and the original PDO indicators proposed in the results framework provided some, albeit different, clarity on the meaning of the original PDO. Specifically, the PAD (page 5) defined the farming system as: ”a population of individual farms systems that have broadly similar resource bases, enterprise patterns, household livelihoods, and constraints, and for which similar development strategies and interventions would be appropriate. Their analysis emphasizes horizontal and vertical integration, multiple sources of household livelihoods, and the role of the community, Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) the environment and support services.” The ICR (page 3, footnote 2) defined “improved production systems” as those that resulted in sustainably better agroforestry, crop, or livestock quality and yields.” These objectives were relevant because they addressed the challenges facing the agricultural sector the State of Rio de Janeiro. While agriculture is important in general in the state, agriculture is especially vital to economic and social well-being in the three administrative regions: the North and Northwest and the Serrana administrative regions. During the time of appraisal these three regions housed more than half of the state’s rural population and were responsible for 60% of agricultural employment, as well as the largest concentration of family owned small farms in the state (PAD page 2). Despite its importance, the agricultural sector faced the challenges of low productivity, poor linkages to markets with high demand for agricultural products, a weakened natural resources base, poverty, and the inability of public institutions to adapt to the evolving demands of the rural sector. The factors contributing to these issues included weak farmers’ organizations; the widespread use of inefficient and unsustainable agricultural practices; poor infrastructure, markets, and agro-industrialization processes; and the limited reach of public policy in rural areas (PAD page 2-3). The project's original development objectives were relevant to country strategies and priorities. During the time of appraisal Brazil’s state and federal governments had established a policy agenda that supported rural poverty reduction by integrating sustainable environmental and social practices and increasing the agricultural production and diversification of family farming. The government of Rio de Janeiro granted increasing supply of agricultural credit and agricultural TA to these regions. The state government was implementing a number of other programs in support of its rural development strategy, including: (i) the State Credit Program for Agricultural Production and Diversification (Moeda Verde), (ii) the State Microcatchment Program for Sustainable Rural Development (Rio Rural), and (iii) the National Smallholder Agriculture Program (PRONAF). The project complemented and built around the ongoing activities of the state. These included pilot activities supported under the Global Environment Facility (GEF) that promoted the long-term conservation and rehabilitation of agro-ecosystems, implementation of sustainable land management practices; and other donor-funded rural and environmental operations (i.e., KfW’s Pro-Atlantic Forest Program, SOS Mata Atlântica Foundation, Conservation International-Brasil, and the Critical Ecosystem Partnership Fund), and state-funded programs (i.e., Rio Rural and Frutificar e Cultivar Orgânico. Arguably this project’s PDO was therefore also relevant to other development partners. Overall the relevance of the original PDO was substantial but marginally so because it was vague, although clarified by the indicators proposed to measure whether or not the PDO was achieved. Addition to project objectives. As noted already in Section 2a of this review, in October 2011 the PDO for this project was amended “to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory and help re-establish an agricultural productive environment in areas of the Serrana Region affected by the January 2011 natural disaster, thus contributing to the higher-order objective of increasing small-scale farming productivity and competitiveness in those Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) areas.” Based on the urgency of providing support for the flooded areas in the Serrana Region, along with the temporary reduction in support for the achievement of the original objective, the revised objectives were substantially relevant to pursuing the strategic development objectives of the government and the Bank and safeguarding those objectives in the face of a natural disaster. At the project's close the project’s development objectives were and remained relevant to the Bank country partnership strategies, specifically the third pillar - a more competitive Brazilian economy - of the World Bank’s Country Partnership Strategy (FY 08–11); as well as third focus area of the Country Partnership Framework (FY18-23) on inclusive and sustainable development that plans to promote socio economic development of small rural producers and vulnerable groups. The relevance of this project’s objectives before and after restructuring in October 2011 are rated substantial by this review. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective “To increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory” Rationale Theory of Change: The project’s activities on productive planning, training and capacity building as well as investments in agro- ecological and natural resource management activities targeting farmers under component one as well as extension and adaptive research activities under components one and two would contribute to achieving this objective. Coordination and alignment of public policies in support of territorial development, rural sector institutional changes, the creation of a long-term financing mechanism for sustainable rural development Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) activities and a state-wide participatory research network under component two would improve the ability of public institutions to adapt to the evolving demands of the rural sector; which would also contribute to the achievement of the objective. To measure the achievement of this objective, the PAD had proposed three PDO indicators: (a) number of farmers transitioning to more productive farming systems; (b) land under improved production systems; and (c) improved product quality (measured in terms of number of beneficiary farmers adopting Good Agricultural Practices (GAP), number of small scale farmers or enterprises certified’ as agro-processing and artisanal enterprises adding value). In terms of defining and measuring the adoption of sustainable farming system, one would expect both environmental sustainability parameters such as recovering the productive capacity of soils, the protection of water resources, or the conservation of biodiversity. However, none of these aspects were monitored by the project. The indicator recording the number of farmers adopting good agricultural practices (GAPs) was monitored with data for a year only and then dropped as part of the October 2012 restructuring according to the ICR (page 73), although the Project Paper on this restructuring states that the indicator was moved to be an intermediate outcome indicator. Outputs:  7,127 km of rural roads were rehabilitated and/or maintained corresponding to 119 percent of the revised target of 6,000 km (original target was 1,300 km and additional financing target was 2,500 km).  75 participatory research projects supporting sustainable rural development were carried out, representing 150 percent of the target. State Agriculture Research Enterprise operated in 30 municipalities and 35 hydrographic micro-basins, involving 52 direct beneficiaries (experimental farmers), 3,939 indirect beneficiaries (farmers), and 28 researchers.  38,221 environmental and productive investment proposals (subprojects) were financed (6 percent above the revised target). The original target was 24,400 proposals.  59,651 beneficiaries were trained on key concepts of integrated and sustainable farming systems, exceeding the target of 50,000. The number of women beneficiaries trained was 13,670 compared with a target of 7,800.  3,870 stakeholders were participating in development committees across all levels equal compared to a target of 4,000  370 Micro Catchment Development Committees were established, accounting for 3,870 stakeholders.  370 micro-catchment development plans were prepared as against the 366 for original target.  A strategy and action plan (ISP) formulated to strengthen rural institutions in the State of Rio de Janeiro meeting the target. Outcomes  223,152 ha of agricultural land under improved production systems was below the original target of 266,000 ha but exceeded the final target of 160,000 ha.  37,172 small-scale farmers and family farms adopted more productive and sustainable systems exceeding the target of 35,000 farmers. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508)  5,280 small-scale family farms adopted more productive and sustainable systems were headed by women (less than the target of 6,000). This indicator was included during the 2017 restructuring.  One important caveat was that there was no breakdown of the above figures in terms of type of adopted farm or improved system. The ICR dropped this indicator and therefore there was no discussion of number of farmers adopting good agricultural practices.  In terms of client satisfaction, a Practices Evaluation Survey was carried out in 2018 with among 61 beneficiaries in the Northwest region and 42 in the Serrana. A 91.8 percent satisfaction rate was achieved in the first region and 100 percent in the second. These results exceeded the target of a 75 percent satisfaction rate. Due to inadequate progress in terms of achievement against the original PDO indicators the efficacy of this objective is rated modest. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective “To increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory” Revised Rationale Theory of Change: This did not change because neither the objective nor the activities designed to achieve the objective changed. While the original Objective 1 did not change, some of the outcome targets were modestly revised and new indicators were included in the results framework after restructuring "to monitor the implementation and results of the emergency activities" (Restructuring Paper, September 2. 2011, para 13) . The outputs and outcomes listed above indicate all the achievements towards Objective 1 before and after restructuring in October 2011. The efficacy of achievements against the revised indicators is rated by this review as substantial. Revised Rating Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Substantial OBJECTIVE 2 Objective “To increase small-scale farming productivity and competitiveness” Rationale Theory of Change Increased productivity and competitiveness (leading to improvements in the livelihoods of small-scale farming) would be achieved through investments in improved technology in small-scale farming, as well as improvements in rural roads to reduce transport costs for inputs and outputs, and linking small-scale farmers to at least one marketing chain based on a certification program that identified small-scale farmers producing high quality agricultural products. Outputs The same outputs discussed under Objective I also contributed to the achievement of this objective. Outcomes No PDO indicator was designed to measure increased productivity such as crop yields, thus no robust information on "increased small-scale farming productivity" was reported by the ICR. The economic analysis section of the ICR (page 56), however, reported yield results stemming from the participatory research activities but it was not clear if these were statistically significant or representative figures for small-scale farmers. In addition, no control group comparisons were included in the analysis. The Bank project team did not elaborate on this issue. The Impact Assessment mentioned statistically significant positive impacts on income but the ICR did not provide any figures. The economic analysis covering 155 sub-projects, found an annual incremental net income per hectare of US$275 on average (ICR page 55), without any information on the average incremental income to be able form any conclusions on the relative importance of this income increase. Also there was no control group comparisons mentioned in the ICR. The project initially included the PDO indicator "improved product quality in at least 50 percent beneficiaries receiving investment support, measured by farmers adopting good agricultural practices, farmers/enterprises certified, agro-processing and artisanal enterprises adding value”. The ICR did not report on these indicators. It was not clear at which point during implementation these were removed from the results framework. The project team did not elaborate on this issue. Other data reported by the ICR on the achievement of competitiveness part of the objective was that a total of 3,359 small farmers were included in (or with improved links to) at least one productive chain as a direct result of the project, corresponding to 129 percent achievement of the target (2,050). After the additional financing the target was 2,600. It was not clear what “improved links“ meant, for example whether these farmers established market access via new contracts with processing companies or larger buyers. Due to lack of sufficient evidence this objective is rated modest. Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective "To increase small-scale farming productivity and competitiveness” Revised Rationale The Theory of Change, outputs and outcomes for this objective were the same as before restructuring in 2011. Revised Rating Modest OBJECTIVE 3 Objective “Help re-establish an agricultural productive environment in areas of the Serrana Region affected by the January 2011 natural disaster”. This additional objective was introduced at the project’s restructuring in October 2011. Rationale Theory of Change: Heavy rains and floods in 2011 caused substantial destruction in rural areas of the Serrana Region. This additional objective was aimed at temporarily redirecting some of this project's resources to affected rural areas and providing an immediate response to restore physical access through rural road rehabilitation, restoration of damaged houses, and the restoration of productive assets. All those activities were within the scope of the original project. It was expected that including geographical targeting and the prioritization of productive activities in the project's PDO would allow the financing of emergency flood recovery activities managed by the local authorities in the Serrana region. No specific PDO indicators were included in the Results Framework; nevertheless, the corresponding intermediate outcomes were consistently monitored by the M&E system. Outputs:  2,277 emergency subprojects were financed (99 percent of the target) reaching a coverage of 1,908 farmers affected by the natural disaster (95 percent of the target).  890 km of roads (111 percent of the target) and 46 small bridges (115 percent of the target) were rebuilt Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508)  34 communities were served with soil conservation patrols assisting 4,858 beneficiaries.  Producer associations were supported with the acquisition of ‘mini-patrols’10 responsible for the management of small machines and equipment made available to small producers to rehabilitate roads and other public infrastructure. Technical assistance was reinforced, operators of such equipment trained, and associations qualified The Participatory Research Units in the Serrana region was strengthened where restoration of degraded areas was taking place, emphasizing agroecological transition, organic production, and the adoption of new technologies by experimental farmers. Outcomes: The ICR reported that (para. 30) 89 percent of the beneficiaries stated that as a direct results of project support, they were able to recover from the damage caused by the catastrophe within six months. The ICR also noted that most technicians and beneficiaries interviewed recognized positively the role of the emergency committees in local social organization. Rating Substantial OVERALL EFF TBL OLD Rationale OVERALLL EFFICACY The achievement of the first objective, to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory, is rated modest before the project's restructuring in 2011 and substantial after restructuring due to meeting or exceeding indicators and targets. The achievement of the second objective, to increase small-scale farming productivity and competitiveness, is rated modest before and after restructuring due to lack of robust evidence on either small-scale farmer productivity increases or evidence of increased competitiveness. The achievement of the third objective, to help re-establish an agricultural productive environment in areas of the Serrana Region affected by the January 2011 floods, is rated substantial after restructuring as a result of positive impacts reported by the ICR. No overall rating is given here as there is a split between the two periods. This is illustrated in Section 6 in reaching an overall rating for Outcome. Overall Efficacy Rating Not Rated/Not Applicable 5. Efficiency Economic Efficiency: The ex-ante economic and financial analysis covered all project activities and assumed three main benefit streams: increase in yields. Thirty-three farm models representative of the most prevalent Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) small-scale farming activities in the focal and replication areas of the Project were identified and used to estimate the impact of project-supported activities on farm profits. The per hectare farm profit increase ranged from 9% to 182% depending on the product. The economic Internal Rate of Return (EIRR) for the Project at appraisal was estimated at 33.6%. The ex post analysis randomly selected 155 sub-projects representing 1 percent of total sub-projects implemented. In terms of value chains, the sample included 54 sub-projects for horticulture; 51 PIDs for milk production; 31 PIDS for; 11 PIDs for coffee production; 6 PIDs for raising small animals; and 2 PIDs for forestry (palmito). The results generated by the various investments were increased production, lower production costs, and increased incomes. The average annual incremental net income per hectare was estimated as US$275, but its significance is unclear because it is not compared to any baseline for income per hectare. IRR of the improvement of two sample roads evaluated (around 7–9 km improved) ranged between 20 percent and 280 percent but the ICR provides no assessment of the reasons for the wide distribution of IRR results. The ICR reported, however, that based on the roads evaluated, there was; (a) a 92 percent reduction of days of non- trafficability; (b) in reduction of production losses (20 percent in the case of milk and 36 percent in the case of vegetables); (c) reduction of 50 percent of travel time due to increased traffic speed; (d) reduction of 40–60 percent in the average cost of vehicle maintenance; and (e) 20–50 percent reduction in average fuel consumption. Nevertheless, the basis for this information and its veracity was not clarified in the ICR. The ICR, based on these results, along with other project costs and carbon co-benefits (without evidence in the ICR) concluded that the economic net present value (NPV) and economic internal rate of return (IRR) were respectively US$301 million and 36 percent (ICR, para 38). The ICR noted in the same paragraph that "These results are similar to the indicators of the economic analysis carried out for the project at appraisal". Overall the economic analysis of the project in the ICR had a number of shortcomings and was hence not convincing. The shortcomings included (a) average annual incremental net income per hectare for sample beneficiary farms with individual development plans (PIDs) was not compared to a baseline nor to a counterfactual; (b) the economic analysis of the more than 7,000 km road rehabilitation program was based on a sample of two roads "(around 7-9 km improved)" raising questions about the veracity of the conclusions; (c) the efficiency of the disaster relief program was not assessed by the ICR. It was also unfortunate that the impact assessment did not provide any information on the efficiency of the project. Administrative and Operational Efficiency: The project closing date was extended for 3 years mainly due to external events including floods and the resultant emergency recovery efforts included in the project, a strike of employees of the implementing agency for about a year, an increase of project scope through an additional financing, and a financial crisis that led to judicial power seizure of the project's designated account for about a year. There were acute capacity constraints, governance issues and bureaucratic obstacles particularly due to the various crisis. In particular the PIU’s chronic inability to overcome procurement weaknesses made the procurement function especially challenging. PIU’s procurement capacity problems contributed to cancelation of some large size contracts. In addition, major delays in procurement review and approvals by the State Court of Accounts (Tribunal de Contas) negatively affected the efforts to maintain timely procurement processing and the scope and quantity of procurement packages; in particular planned road works and bridges were especially negatively impacted. Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Due to shortcomings in the economic analysis and administrative and operational inefficiencies, the project's efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome As mentioned in Section 3e this review is carrying out only one split rating because of the PDO revision in 2011. A second split on account of subsequent relatively small changes in indicators was not considered necessary. The following table provides the basis for the overall outcome for the project of moderately satisfactory based on the previous assessments of relevance of objectives, efficacy and efficiency. Derivation of Overall Outcome Rating. Rating Dimension Original Objective Revised Objective Relevance of PDO Substantial - PDO 1: To increase the adoption of integrated Modest Substantial and sustainable farming systems approaches - PDO 2: To increase small-scale farming Modest Modest productivity competitiveness - Additional PDO: help re-establish an agricultural productive environment in areas of the N/A Substantial Serrana Region affected by the January 2011 natural disaster Efficiency Modest Moderately Moderately Outcome Rating Unsatisfactory Satisfactory Outcome Rating Value 3 4 Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Disbursement (%) 10.7 89.3 Weight Value 0.3 3.6 Total weights 3.9 Overall Outcome Moderately Satisfactory a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Institutional Sustainability: The ICR argued (paras 96-97) that the decentralized approach adopted by the project, which is the transfer of powers from State Secretariat of Agriculture, Livestock, Fisheries and Supply’s (SEAPP) center to its regional and sub-regional offices, including to State Rural Extension Agency (EMATER-Rio) and State Agricultural Research Enterprise (PESAGRO) ensured increased local capacity building and outreach. The project also developed partnership models with NGOs, farmer/producer organizations and cooperatives, demonstrating how smallholder producers could develop and maintain market links. Thus the sustainability of the institutional approach was ensured despite the fiscal constraints. Financial Sustainability: The project financed sub-project proposals in a demand driven fashion; funds came from various resources including public funds, payment for environmental services (in carbon, water, and biodiversity), agricultural credits, and private partnerships in the areas of sustainable business and socioenvironmental responsibility. The project team did not clarify if there would be additional funds provided by the government or another program to continue the project’s approach to provide financing for investment proposals in the future. The ICR also reported the unfavorable fiscal situation of the State Government that may potentially affect the sustainability of the outcomes. In addition, the market risk (prices, quality of products, honoring contracts) is another risk that could impact profitability of beneficiaries particularly for exported products. 8. Assessment of Bank Performance a. Quality-at-Entry The project was the successor of the GEF funded Integrated Management of Agroecosystems Project and was designed to build on the existing productive and social base. The aim was quite ambitious and complex as it tried to incorporate three aspects: to execute a strongly market oriented agricultural operation to simultaneously support farmers’ competitiveness and protection of global biodiversity resources. According to the PAD (page 9) the design drew on the lessons from other Bank projects in the rural sector in other states in Brazil, as well as on the global lessons compiled in the Bank’s 2008 World Development Report focusing on agriculture. In addition, the project took certain features from the “productive alliance” model which was expanding under World Bank-supported rural operations in Latin America at the time. The main lessons from those projects were on decentralization (implementation and Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) supervision responsibilities at the municipal level supported by adequate institutional support); and one- time matching grants as incentives for technology adoption and innovation with a multi-sectoral focus. As mentioned earlier, the weaknesses of the results framework included a vague statement of PDO in terms of what an “increase in integrated and sustainable farming systems approaches” meant in terms of scale, the extent to which these farming systems were expected to be “approached” by small-scale farms. In terms of defining and measuring the adoption of sustainable farming system, one would expect both environmental sustainability parameters such as recovering the productive capacity of soils, the protection of water resources, or the conservation of biodiversity. However, none of these aspects were monitored by the project. No indicator was included to measure the quality and effectiveness of institutional capacity building, particularly at the local level. Project’s M&E framework included an Impact Evaluation (IE). The project mainly supported sub-project investments financed through a matching grant mechanism with beneficiary contribution, while retaining the micro-catchment as the organizing unit and geophysical location. However, as noted in the ICR (para 91), this complex structure was particularly challenging for a state with limited experience with such market-oriented methodologies and approaches, in addition to limited institutional capacity, which became evident during implementation. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision According to the ICR (para 93) the supervision team worked in a difficult operating environment trying to find quick solutions to natural disasters and financial crises, and in that respect the team was quite effective. In addition, fiduciary and safeguard issues were managed by experienced specialists, and particularly the procurement issues were ) addressed in general given the acute capacity constraints, governance issues and bureaucratic obstacles. One area that the supervision team could have addressed was the adequate revision of the results framework and outcome indicators particularly during the Additional Financing (see Quality at Entry for lacking PDO indicators). In addition, the Additional Financing amount proved to be extremely ambitious, much larger than the state could absorb. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) a. M&E Design Contrary to the broad statement of the PDO, the PDO indicators demanded that (a) “at least 50% of farmers in the project area had transitioned towards more productive farming systems” and (b) “improved product quality” (assessed by three measures namely “number of farmers adopting good agricultural practices”, “number of farmers or enterprises certified” for something undefined, and “number of agro- processing and artisanal enterprises adding value”), and (c) “number of farmers linked to at least one value chain”. None of these indicators made any reference to their relationship to integrated or sustained farming systems. To the extent that these achievements could be assessed the vague PDO was given a specific meaning but that meaning contrasted significantly with the intent of the original PDO. There was also lack of clarity on the design and methodologies to collect and process data. The Management Information System was conceptually ambitious and based on participatory principles that would build on existing information systems and databases from the Rio GEF project to monitor the project’s physical and financial progress. Baseline surveys and evaluation studies would be outsourced/shared, with some coordination activities and field surveys to be conducted by state and federal institutions (with recurrent costs covered). b. M&E Implementation During implementation, development of the originally planned Management Information System was abandoned due to restrictions established by the state government on hiring of consultants. The shortcomings with the M&E indicators were not resolved during the various restructurings. In the event an Impact Evaluation (IE) was the main M&E function carried out using treatment and control groups and randomized sampling, covering 120 micro-basins in 35 municipalities. Given that the project’s objectives were vague and not measurable, the IE used economic indicators for micro-basins based on information from agricultural producers. Based on information that economic indicators in micro-basins differed between the treatment and control groups due to the project’s investments, but that differences between the treatment and control groups may also be due to other factors, the contrasts due to the other factors were also taken into account in the impact evaluation (ICR, Annex 5, para 6). The results for indicators that were statistically different and for those that were not statistically different were presented in Annex 6 of the ICR. c. M&E Utilization Project monitoring data and periodic progress reports were used as inputs to management decision making as well as key research and reporting deliverables. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) Environmental Safeguards: The project was classified Category B, which required an Environmental Management Framework. The project triggered six safeguard policies: Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Pest Management OP/BP 4.09), Physical Cultural Resources (OP 4.11/BP), and Forests (OP/BP 4.36). Involuntary Resettlement (OP 4.12) was triggered on a precautionary basis. The project did not lead to any involuntary resettlement. The Environmental Assessment (EA) and Environmental Management Framework (EMF) had been prepared by the Borrower, and the final versions were approved before appraisal. The EA described the organizational framework for safeguards, which built on the capacity established within the implementation agency for the ongoing Rio GEF Project. It also listed all activities with potential negative impacts. The EMF proposed a plan for avoiding, minimizing, and mitigating negative impacts identified in the EA. The EMF also included a framework for screening project activities in relation to potential negative impacts on cultural property, as well as a Resettlement Policy Framework (RPF). The ICR reported (para. 80) that environmental impacts were in general positive, including enhanced soil nutrition and reduced erosion through improved cropland and grazing management; rehabilitation of degraded forest areas, sustained conservation of natural resources, leading to improved biodiversity; reduced use of pesticides in croplands; and water conservation benefits and climate co-benefits, although no evidence was provided supporting these claims. No measures were needed to preserve physical cultural resources because no project activity placed cultural assets at risk. The ICR reported that (para. 79), environmental safeguards compliance was rated satisfactory throughout the project’s life. b. Fiduciary Compliance Financial Management: Financial management performance was rated moderately unsatisfactory mid- 2018 (after that moderately satisfactory), due to late submission of interim financial reports (IFRs) and project financial data being registered in two separate systems that required constant reconciliations. Then the rating was upgraded to Moderately Satisfactory after mid-2018. Except for the 2014 audit report, all other audit reports were submitted late. Apart from the 2012 and 2014 audit reports which expressed qualified opinions, all other audit reports expressed unqualified opinions, but no ineligible expenditures were identified. The ICR did not describe why those reports had qualified opinions. The last audit report was not ready at the time the ICR was written. Procurement: The PIU’s continuing inability to overcome procurement weaknesses made the procurement function especially challenging, as did major delays in procurement review and approvals by the State Court of Accounts, which negatively affected the efforts to maintain timely procurement processing. Planned road works were particularly affected by that. The World Bank responded through repeated procurement training, frequent adjustments to procurement strategy and planning, more frequent procurement supervision, and contracting a consultant familiar with World Bank procurement rules to support the PIU. Thus, procurement ratings were mostly in the unsatisfactory range throughout until restored to Moderately Satisfactory in the final year. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) c. Unintended impacts (Positive or Negative) No unintended impacts were reported by the ICR. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Due to shortcomings with the Quality of M&E Substantial Modest M&E framework Due to weak TOC and results Quality of ICR --- Modest framework and limited evidence under efficacy 12. Lessons The ICR provided several lessons, many of which overlapped and mainly related to demand driven approaches for mobilizing investment support in rural areas. Such lessons have already been learnt under many projects in the past. However, one lesson that stands out relates to the importance of accurately assessing the impact of project benefits. Thus IEG concluded that the following was a key lesson: M&E design needs to be adequate to accurately capture project benefits. The project supported rehabilitation of more than 7,000 km of rural roads rehabilitation, which is known to be a crucial precondition for market access in rural areas. Based on two roads 7-9 km long the project estimated that the benefits of the road rehabilitation program were a 92 percent reduction of days of non-trafficability, a reduction of production losses (20 percent in the case of milk and 36 percent in the case of vegetables), reduction of 50 percent of travel time due to increase in traffic speed, reduction of 40–60 percent of the average cost of vehicle maintenance, and 20–50 percent reduction in average fuel consumption. This review questioned the veracity of these benefits determined on the basis of information from only two stretches of roads of 7-9 km underlines the lesson that a project's M&E program needs to be designed to accurately assess project outcomes. IEG also concluded the following related lesson: A complex project requires a clear Theory of Change and a robust results framework. This project included many aspects ranging from environmental sustainability to productivity as well as marketing linkages, road infrastructure and decentralization, and local capacity building. When such complexity is introduced this should be supported by a very clear objective and well-defined Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR-RJ Sustainable Rural Development (P101508) theory of change as well as a robust results framework as a basis for measuring outcomes. Unfortunately, the project failed to do that. Even though the project financed numerous investment proposals, sufficient outcome evidence on increased productivity and improved competitiveness was not collected. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain To retrospectively assess achievements of this project on the ground which were not measured for the ICR and were therefore not taken into account in assessing this project's overall outcome. 14. Comments on Quality of ICR The report followed the guidelines in general but had significant weaknesses; (a) the efficacy section was weak, some results were presented in an anecdotal manner with no or limited discussion of attribution issues; (b) the project’s theory of change was weak (c) the presentation of the impact evaluation in the ICR provided little quantitative data on the project's impact; (c) some of the ICR’s lessons were unclear and not based on project actions or results presented in other parts of the report. The project team provided incomplete responses to some of IEG’s queries related to project costs, indicators, efficacy and efficiency . a. Quality of ICR Rating Modest Page 19 of 19