Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Report Number: ICRR0021675 1. Project Data Project ID Project Name P108443 BR SP Sust Rural Dev & Access to Markets Country Practice Area(Lead) Brazil Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79080 30-Sep-2015 78,000,000.00 Bank Approval Date Closing Date (Actual) 25-May-2010 30-Sep-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 78,000,000.00 0.00 Revised Commitment 78,000,000.00 0.00 Actual 78,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Keith Robert A. Oblitas Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Project Objectives at Appraisal: To support the borrower to increase the competitiveness of family agriculture in its territory while improving its environmental sustainability. Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Source: Loan Agreement, September 27, 2010 Revised Project Objective: To support the borrower in improving access to markets for rural producers’ organizations in its territory and in improving its instruments and policies that contribute to environmental sustainability of family agriculture. Source: Amendment to the Loan Agreement, September 29, 2015 b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval c. Will a split evaluation be undertaken? Yes d. Components Component 1: Supporting Small Farmers’ Business Initiatives (Project costs estimated at Approval: US$ 58.80 million; Actual Costs at Project Closing: US$ 47.99 million) Implementing Small Business Investments through matching grants to eligible producer organizations to increase competitiveness (productivity, quality, value added and marketing) while improving agricultural practices; and (ii) information, training , and technical assistance to formalize and strengthen producer organizations to prepare viable business proposals for funding and for enhancing organizational, financial, marketing, implementation and monitoring capabilities. Component 2: Strengthening Institutional and Physical Infrastructure (Project costs at Approval: US$ 61.40 million; Costs at Project Closing: US$ 58.81 million) 2.1 Policy Frameworks, Markets and Rural Extension through an online agri-business information center and strengthening the Rural Extension Directorate’s capacity to advise producer organizations on marketing, management and production technologies. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) 2.2 Municipal Rural Infrastructure: grant financing for infrastructure sub-projects to rehabilitate/maintain priority sections of rural roads, and develop municipal roads management plans. 2.3 Improving Environmental Sustainability: through improved environmental practices in rural production systems to ensure sustainability of the productive base. Component 3: Project Management (Project costs at Approval: US$ 9.60 million; Costs at project closing: US$ 4.83 million) Improving effectiveness in administering, coordinating, supervising and monitoring of project activities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs. Total project costs estimated at appraisal were US$ 130.0 million. Actual costs at project closure were US$ 111.63 million, 86 percent of the appraisal estimate. The reduced overall project costs primarily reflect reduced costs in component 1.2 (strengthening producer organizations) – a reduction of US$ 9 million to 30 percent of appraisal costs; and rural infrastructure – a reduction of about US$ 6 million to 70 percent of appraisal estimates. Financing The proect received an IBRD Loan of US$ 78.00 million, all of which was disbursed. There were no co- financers. The borrower contribution was US$ 34.7 million as against the contribution estimated at appraisal of US$ 52.0 million. Beneficiaries are estimated to have contributed (very approximately) US$ 19 million in cash, kind or labor. Dates The project was approved on May 25, 2010, and made effective on December 22, 2010. The Mid Term Review was in December 2013. The original closing date was September 30, 2015, but was extended by 3 years to September 30, 2018. Restructurings and Project Reformulation There were two Level 2 restructurings (in 2016 and 2017) and one Level 1 restructuring in 2015. Concerning the Level 2 restructurings:  March 24, 2016. Level 2 Restructuring. For the project’s environmental activities, (a) added a disbursement category of US$ 1.0 million to cover payments for environmental services, reducing the allocation for environmental sub-projects by the same amount; and (b) reallocated US$ 1.35 million from that same category to the goods, works, consultants and training category. Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443)  August 21, 2017. Level 2 Restructuring: Extended the project closing date by another 12 months. Restructuring Level 1: A more comprehensive (Level 1) Restructuring was undertaken on September 29, 2015. The main changes were:  Revision of the project objectives (section 4).  Extension of the project period by 2 years.  Alignment of the project with Government’s environmental laws and policies.  Dropping the Development Indicator on the increase in the agricultural area where improved land and water management practices are implemented. 3. Relevance of Objectives Rationale When the Sao Paulo: Sustainable Rural Development and Access to Markets Project was approved in May 2010, the project’s objectives were “to increase the competitiveness of family agriculture in its territory while improving its environmental sustainability.” These objectives were revised in a project restructuring in September 2015 to become: “improving access to markets for rural producers’ organizations in its territory and in improving its instruments and policies that contribute to environmental sustainability of family agriculture. Both the original and revised objectives encompass two thrusts: the first related to agricultural productivity, and the second on environmental sustainability. However, the focus is different. Under the first thrust, the objective is to improve competitiveness, at face value a comprehensive goal which would include improved marketing but also any improvements in yields, production costs and other contributing factors. The revised objectives narrow down the goal to accessing markets. As concerns environmental sustainability; there are also significant differences between the original and revised objectives. The original objective is a broadly cast goal to improve environmental sustainability, while the revised objective narrows the focus to improving instruments and policies for environmental sustainability. A common denominator, however, is that both formulations of the project’s objectives are consistent with Sao Paulo’s needs and the policies of both Government and the Bank in addressing these needs, as reviewed below: Sao Paulo, Brazil’s richest state, has the second largest economy in Latin America and a booming GDP growth rate averaging 5 percent per year, broadly consistent with the national GDP growth rate. The rural Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) sector dominates both the state’s economy and the welfare of rural people. The state produces 25 percent of national agricultural output, and within the state itself agriculture is at the core of the economy and employment. At appraisal of the project - the Sao Paulo: Sustainable Rural Development and Access to Markets Project - agriculture represented over half of the state’s economic activity in 60 percent of the state’s municipalities, and 80 percent of the state’s employment. However, there were still large disparities in incomes within the agriculture sector, associated in particular with farm size and technologies used: 80 percent of agricultural units were small family-based operations occupying 20 percent of total farming area, while 80 percent of farm area was owned by large commercial farms. Family farming faced twin challenges – (i) improving productivity and (ii) reducing environmental degradation. Low productivity was due to a variety of constraints, amongst them, low market participation due to low market knowledge, poor infrastructure, especially roads and marketing infrastructure, limited access to credit and technical assistance, weak organizational capacity, and limited agricultural extension services. Environmental degradation, especially soil erosion resulting in reduced soil fertility and sedimentation of rivers and reservoirs, is the other key challenge. Addressing these challenges were priorities for both the Government and the Bank. The Bank’s overall strategy for Brazil at the time of appraisal was set out in the Country Partnership Strategy for 2008-2011 targeting three pillars: a more “Competitive Brazil;” a more “Equitable Brazil; and a more “Sustainable Brazil.” The PDO was also well aligned with the World Bank's priorities at closure.The Brazil Country Partnership Framework (FY18-23), the policy document at time of project closure, defines two ongoing challenges: (i) creation of productive and well-paid jobs, translating to increasing productivity; and (ii) better management of natural resources including land and water resources management. A core focus for both thrusts is to promote the socioeconomic development of small rural producers and vulnerable groups. Sao Paulo’s rural sector policy agenda, broadly consistent with national policies, shares these goals, which are primarily expressed in policy and technical documents on forest management, ecological and biodiversity preservation; and on productivity, value chains and marketing. Theory of change: The project’s thrust areas – productivity (marketing) and environmental sustainability – are aligned with Bank and Government policies, and the causative linkages between project activities, outputs and outcomes are expressed practically. Notwithstanding, there are several areas in the log-frame and in discussing efficacy where actions and outputs could have been better focused. First, actions to improve agricultural yields or reduce production costs are only sporadically mentioned and impacts are not elaborated, yet agricultural extension could be a significant complement to market improvements. Second, the degree to which smallholders benefited from the project (a specific target of both objectives) could have been analyzed further. Third, the possible impacts of the various “instruments and policies” changes (actions to outputs to intermediate outcomes) under the revised environmental objective are not assessed. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Nevertheless, the project’s Theory of Change was realistic. Intended project actions were carried out, mostly exceeding targets, and outputs were also measurable and achieved. For some first-round outcomes (such as increases in sales or the issuance of environmental policies) the impact, as to be expected, is less discernible, although the information available indicates that the enhanced actions and outputs have had concrete results. Thus, the project’s theory of change has been substantial and notwithstanding some environmental areas where impact is unclear, the overall Relevance of the project is assessed Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase the Competitiveness of Family Agriculture Rationale Background The project had two original objectives (objectives 1 and 2), which were replaced at project restructuring by revised objectives. The objectives were Objective 1: to Increase the Competitiveness of family agriculture; Objective 2: Improve Environmental Sustainability of family agriculture; Objective 1 Revised: Improve Access to Markets for Rural Producers’ Organizations; and Objective 2 Revised: Improve Instruments and Policies that Contribute to Environmental Sustainability of Family Agriculture. While there are significant differences between the original and revised objectives, the revised objectives have thematic commonalities to the original objectives. Thus, objective 1 Revised narrowed the focus from “competitiveness” under objective 1 to “market access,” both supporting increased productivity; and objectives 2 and 2 Revised are both environmental, objective 2 being in very broad terms, and objective 2 Revised narrowing the scope to environmental instruments and policies. Project beneficiaries are family farms and their rural organizations (ICR page 3) EFFICACY OF OBJECTIVE 1: Increasing the Competitiveness (of family agriculture) The main outputs (achievements) and comparison with targets for Objective 1 were: Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Outputs  Sales of participating producer organizations increased by 11 times the value of non-participating producer organizations. Average annual revenues of households in the participating farmer groups were US$ 1,800 higher than the revenues of non-participating households. (these achievements and targets also apply to objective 3.)  The project directly benefitted 16,600 families, compared with a target of 7,000; and 3,100 families were members of community sub-projects) (applies also to objective 3.)  Sub-projects of traditional communities identified and financed by the project: actual 87 sub-projects; target 49.  Number of producer organizations trained in management and accounting: actual 676 organizations, target: 831. (applies also to objective 3.)  Sub-projects of traditional communities identified and financed by the project: actual sub-projects 87, target 49.  Rural roads rehabilitated: actual 672 kilometers; target 500 kilometers. (applies also to objective 3). (Data on agriculture actions and yields are not provided in the ICR) Efficacy The efficacy of objective 1 was monitored by one indicator: increases in in total sales value for participating organizations. (The same indicator was used for objective 3.) Based on survey data from the World Bank’s Development Impact Evaluation Unit (DIME), the project substantially increased productivity - the value of total sales increased by 87 percent, and average net annual revenues of households which participated in a project marketing organization were US $ 1,800 higher than non-participants. Key factors attributed in the ICR to have increased productivity were that a larger share of participating households’ production was sold through marketing organizations at better prices. It is not possible to disaggregate impact from the project’s market improvements from impact due to possible agricultural improvements. Nevertheless, marketing improvements (through training, extension services, establishment of producer organizations, and assistance in establishing businesses and market chains) dominated the project activities, and can be expected to have had a stimulative impact on productivity. The Efficacy of Objective 1 in Increasing Competitiveness was Substantial. Agricultural Productivity While these results indicate that the project made a substantial impact on the competitiveness of family agriculture (higher productivity and incomes), it is likely that measures to increase agricultural yields, in parallel to the drive to improve marketing, would have provided a further boost to productivity and market performance. The increase in the value of sales of 87 percent – a near doubling of value – is a significant achievement but rural development projects that focus on increasing farm yields can also have good returns. This suggests that the project’s almost exclusive focus on marketing may have substantial room for further revenues if agricultural extension is added to the marketing initiatives. The Rural Extension Directorate, a competent organization, provides general agricultural extension in Sao Paulo, and tried to provide the more specialist extension needed by the project’s producer organizations. But greater impact on productivity is Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) likely if the agricultural extension services are more targeted and receive specialist training. Previous Bank- assisted projects in Sao Paulo have emphasized agricultural production, with marketing receiving much less attention. An appropriate blend of marketing and agricultural extension might be a possibility, but there is no indication that such options have been considered. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective Improving Access to Rural Markets Revised Rationale Objective 1 Revised: to “Improve Access to Markets for Rural Producers’ Organizations.” replaced the former objective 1 (on agricultural competitiveness) by specifically focusing on one aspect of agricultural productivity - market access. Key outputs and efficacy are reviewed below: Outputs (Some of the outputs listed under objective 1 also contribute to this objective.)  Number of business initiatives that are “market viable,” and financed under the project: 287 viable business initiatives established against an original target of 300 initiatives.  Beneficiaries of registered business initiatives: Initial target: 4,300 beneficiaries; actual: 13,500.  Staff trained in management and value chains: original target: 500 staff; actual 600.  Value chain studies carried out: initial target and actual: 4 studies.  Study of potential for traditional communities to access markets. Done as targeted.  Creation of system for developing and monitoring business proposals: Done as targeted.  Development of system for assessing quality of business proposals: System developed. Efficacy The monitorable indicator for the efficacy of objective 1 Revised - increases in total sales value for participating organizations – was the same as the indicator used for objective 1. As noted for objective 1, the value of sales increased considerably, growing by 87 percent. Incomes also improved - average net annual revenues of households which participated in a project marketing organization were US $ 1,800 higher than non-participants. Most of the intended outputs for objective 1 Revised were achieved and could be expected to have boosted the value of agricultural sales The Efficacy of Objective 1 Revised in Improving access to markets for rural producers’ organizations was Substantial. Revised Rating Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Substantial OBJECTIVE 2 Objective Improving Environmental Sustainability (of family agriculture) Rationale For the project’s second objective to “Improve the Environmental Sustainability of family agriculture,” the monitorable indicator for achievement was to increase the area of agricultural land under improved land and water management to 0.8 million hectares (PAD, page 34, results framework). A number of actions were undertaken, of which key outputs were as below: Outputs 1. Hectares planted under agro-forestry: 600 ha. No target 2. Number of beneficiaries of agro-forestry systems: 607 beneficiaries; no target. 3. 25 environmental sub-projects were implemented: target, 24 sub-projects. 4. Number of beneficiaries of environmental sub-projects: 821 (revised target 731). 5. Number of rural producers trained in environmental management:1078 producers; target 800. 6. Number of events to disseminate environmental sub-projects: 468 events; no target. Efficacy The 0.8 million hectare target was not achieved. Land and forest improvements and soil conservation together (primarily agro-forestry) achieved less than 10 percent of this. However, the outputs above were actions that would have helped the development of implementation experience for future land and water management. But these benefits are countered by the large shortfall in the total area developed. The Efficacy of Objective 2 was Modest. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective Improve Instruments and Policies that Contribute to Environmental Sustainability Revised Rationale Under the restructured project the large scale environmental improvement on 0.8 million hectares intended under Objective 2 was replaced by the narrower goal of Objective 2 Revised – to improve instruments and policies that contribute to environmental sustainability. Two monitorable indicators were chosen to assess achievement: Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443)  A Rural Environment Cadaster to be approved and made available for farmers; and  Three systems for Payments for Environmental Services to be piloted, approved and adopted by farmers. Outputs: 1. The Rural Environment Cadaster was approved and made available to 340,000 family farms, against an informal target of 330,000 farms. Six million hectares were registered. 2. Three Payments for Environmental Services sub-projects were piloted using different agro-ecologies, rural demographic situations, and approaches, and have provided a learning experience for further development. These outputs were supported by technical, training and monitoring actions including: 1. Technical environmental staff trained: 1135 staff; target 1100. (also relevant to objective 2) 2. Implementation of socio-environmental management plan. 3. (Some of the actions for Objective 3 would also have contributed.) Efficacy Both of the revised objective 2 monitorable indicator targets were achieved. The Cadaster provided a data base for planning and financing land use practices, and for monitoring compliance with agricultural and environmental management norms. The Payment for Environment Services pilots helped develop systems for environmental management through financial incentives, setting the stage for wider application. Although the revised Objective 2 has a narrower scope, the “policies and instruments” created improved the capacity for environmental planning and land and water use and will enable better resource management in the future. The Efficacy of Objective 2 Revised was Substantial. THE PROJECT's OVERALL EFFICACY Efficacy of the Original Project (objectives 1 and 2) Under the original project, the value of agricultural sales increased considerably demonstrating the Substantial Efficacy of Objective 1’s aim to increase agricultural competitiveness. Objective 2 was to improve environmental sustainability and the monitorable indicator was to increase the area of agricultural land under improved land and water management to 0.8 million hectares. Achievement (less than 100,000 hectares) was well below this. Land and water environmental management capability would have improved through implementation experience and training, but provide limited mitigation against the large shortfall in the area developed. The Efficacy of Objective 2 was Modest. Taking both Objectives 1 and 2 together, the Efficacy of the original project was Modest. Efficacy of the Restructured Project (revised objectives 1 and 2) Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) The Efficacy of Objective 1 Revised – to improve access to markets - was Substantial as manifested by the increase in value of agricultural sales noted above. The Efficacy of Objective 2 Revised – to improve instruments and policies that contribute to environmental sustainability of family agriculture was also Substantial, as the environmental cadaster established an implementation framework for environmental planning and management, and the environmental services projects piloted incentives-based environmental management. Taking the Revised Objectives 1 and 2 together, the Efficacy of the restructured project was Substantial. Revised Rating Substantial OVERALL EFF TBL OLD Rationale Overall Efficacy of the Project Efficacy was substantial for objective 1 but modest for objective 2, resulting in a modest efficacy for the original project. However, the substantial efficacy for the revised project (revised objectives 1 and 2, each objective with substantial efficacy) covers the larger part of project implementation (three quarters of the project's disbursements), and the increase in competitiveness under the original project and market access under the revised project represent a significant increase in productivity. The project's Overall Efficacy is assessed Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The project’s overall economic rate of return is estimated in the ICR as 28 percent, which is based on an extensive field survey by the Bank’s Development Impact Evaluation Unit (DIME). The ERR estimated in the ICR is slightly greater than the ERR estimated at appraisal of 26 percent, which used approximately the same methodology. There is a marked difference between the average annual net revenues from marketing per family for participants in the project (US$ 1,600/family) and the net income from sales of a non-participating family (US$ 440/family). Implementation Efficiency: Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) The project took three extra years to implement, an increase in the project’s duration of 60 percent. Ultimately, though, most implementation targets were met, and most achievements exceeded appraisal targets. Cost Effectiveness: The project’s Investment costs for sub-projects related to business investments or community market related investments were 22% lower than costs for approximately similar investments under Brazil’s Santa Catarina Rural Competitiveness Project. For the project’s environmental sub-projects, costs per family were 28 percent lower than those of the Rio de Janeiro Sustainable Rural Development Project, reflecting the project’s less costly investment in agricultural equipment and practices. Most project activities exceeded targets, while total actual project costs (IBRD, borrower and beneficiaries combined) at US$ 113 million were lower than the appraisal estimate of US$130 million. Overall Efficiency: The project was economically viable with a strong ERR and significant increases in incomes for project beneficiaries. With unit investment costs lower than costs of comparable projects, it was cost-effective. The project required three extra years to implement but most implementation targets were met or, more usually, exceeded. The project’s Efficiency was Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  26.00  Not Applicable 100.00 ICR Estimate  29.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome As a restructured project with revised objectives and a change in efficacy, a “split” evaluation is undertaken, taking account of both the project before restructuring and the project after restructuring. The outcome of the project based on the original objectives was moderately satisfactory, based on its substantial relevance and efficiency, and the substantial efficacy of objective 1 (increasing competitiveness), but the modest efficacy of objective 2, the environmental sustainability program. Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) The project after revision of the objectives had a satisfactory outcome as relevance, efficiency and efficacy were all rated substantial. Overall outcome under the split evaluation assigns a numerical score of 4 for the moderately satisfactory outcome for the original objectives, weighted by 0.26, the share of total project disbursement at restructuring, and the score of 5 for the predominantly satisfactory outcome of the revised project objectives weighted by 0.74 (disbursements to project closure after restructuring). The overall outcome of the project is, thus scored 4.7, rounded to 5 – the score for a Satisfactory Overall Outcome of the Project. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The Government provided strong supporting environment in policies, laws and financing; and the executing agencies have prioritized their support to the project. There is a need to provide continued involvement of high-level experts in marketing, agricultural production and other technical fields, including state-of-the art international expertise, in order to adapt to an increasingly demanding marketing environment. Such institutional support – from government, the implementing agencies and marketing/production specialists - seems likely given experience to date. But the most fundamental factor for sustainability is the quality and motivation of the community organizations. The business initiatives and environmental services subprojects require the continued interest of participants. For members of business initiatives, profitability is generally strong and incentives for participants and their organizations to continue their activities is likely. This is less sure for the environmental services subprojects. The ICR provides little data on the profitability of the environmental subprojects, although it comments that the sub-projects are mostly maintaining producer interest. Nevertheless, the sub-projects are only at pilot scale and environment sourced benefits may have lower profits than predominantly business entities, so incentives are not yet fully proven. Taken overall, however, government and agency commitment, and the interest to date of farmers provides a good base for going forwards and the project’s risk to development outcome is moderate. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank identified and prepared an innovative project pioneering development of marketing in a lead role in rural development projects and showing the considerable value-added from market improvements as the primary project focus. Institutional arrangements, extensive training, the means of reaching communities, and technical features were thought through in detail, resulting in a project that required little change in technical features during implementation. Marketing expertise was, however, limited, Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) primarily relying on the combined experience in marketing within the team. The task team commented to IEG that while the team was able to harness a workable capability in agricultural marketing, it would have been better to have had more involvement of market specialists. There were two principal shortfalls. First, while the focus on marketing proved to be an effective and viable approach to rural development, on-farm sources of agricultural growth received little attention, yet growth in crop yields, even if subsidiary to the marketing focus, would likely have given further impetus to productivity. Second, the time taken to develop arrangements between the project’s implementing institutions, to conduct training at all levels and to motivate and guide community organizations as they started their businesses, was significantly underestimated and the resultant delays were the main source of the project’s three year extension. Quality of Entry was Moderately Satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank mounted a concerted effort to revise and improve the project as experience was gained. Missions were regular, though could have benefitted from greater access to marketing specialists. The supervision team was proactive in adjusting implementation, including restructuring, to change modalities or address bottlenecks as they became evident. In the first several years, effort was, appropriately, placed on building the government and farmer organizations’ capacity for implementing the project. A Bank retiree living in Sao Paulo was hired to provide continuous support to the implementing agencies, which was helpful for the intensive guidance needed by a project covering new ground. There were two notable gaps: first, there is little indication in the ICR that agricultural extension needs were considered; and second, the broad intentions regarding government actions to carry the project forwards over the medium and longer term was not articulated. In general though, the Quality of Supervision was Satisfactory. Taking account of both quality at entry and supervision performance, the Overall Quality of the Bank’s Performance was Moderately Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design M&E was the responsibility of the two main implementing agencies – the Rural Extension Directorate and the Secretariat of Environment - each monitoring its respective implementation activities. M&E built on existing data bases and on field surveys. The Bank’s Development Impact Evaluation Unit (DIME) undertook a separate survey of the project’s marketing and income impacts. A key weakness was the formulation of the Monitorable Indicators for the project objectives. For the productivity/marketing objective, which was not revised, one indicator was chosen – increases in the total sales value for participating organizations. Sales value provides significant information, but it would have included agricultural productivity changes, which might have been substantial. The indicator could also have disaggregated marketing in categories such as processing and market chain values, and urban and international markets. On the environmental side, the initial objective to increase environmental sustainability was too broad and was measured by hectares planted under agroforestry with an overly ambitious target. b. M&E Implementation The two agencies collected, analyzed and disseminated data, prepared annual reports and coordinated studies. The DIME study was methodologically designed and based on extensive field survey work. The Payment for Environmental Services program was not monitored as the PES projects were started late in the post-restructuring period. In effect therefore, the environmental part of the project was unable to be monitored. c. M&E Utilization The ICR advises that project monitoring data and progress reports were used in management decision making, supervision, training, and seminars. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Environment The project was classified as Category B and triggered the following safeguards: Environmental Assessment (OP/BP 4.01); Natural Habitats (OP/BP 4.04); Pest Management (OP/BP 4.09); Physical Cultural Resources (OP/BP 4.11); Forests (OP/BP 4.36); Involuntary Resettlement (OP/BP 4.12); and Indigenous Peoples (OP 4.01). The project had positive environmental impacts due to the environment related actions taken under the project. The pilot projects are in early implementation and are intended to test an environmental management model for a variety of environmental management situations depending on local conditions. Actions are diverse, responsive to the local needs of each sub-project, and address several of the safeguards referred to above, including natural habitats, conservation or regeneration of forests, nature reserves, soil conservation, protecting downstream water supplies, and pest management. For the business-related sub-projects, environmental screening was undertaken. A likely more rigorous environmental review was at the time of clearance of a sub-project - environmental review by local governments was an integral part of the investment proposal. The ICR advises that project activities did not affect cultural resources. Social A land acquisition and relocation framework was prepared, and sub-projects were screened in accordance with the framework. An Indigenous People’s Participation Framework was also issued for similar guidance. The ICR advises that land acquisition was not required as land alongside rural roads was provided voluntarily. Gender Participation As noted in the ICR, there were no explicit features in project design to facilitate gender participation ("The project's omission of an overt gender focus is inexplicable." ICR para 48). Monitoring and data related to gender participation was also absent. This contrasts with the project's predecessor - Land Management III - which had a strong agenda for women's participation. The task team considers that a special focus on women may be less necessary in a marketing project as women are already fairly integrated, and even predominant in activities such as packaging produce, and some sub-projects are managed by women. However, as also commented on in the ICR (para 48) explicit focus on gender would have been a better safeguard, enabling systematic assessment of gender issues and opportunities and subsequent design and implementation, as needed, of a targeted women's participation program, or specific project features for women's inclusion. b. Fiduciary Compliance Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Financial Management A private firm was hired to design and implement a financial management system interlinked between agencies. project management staff in the rural extension directorate and state secretariat of environment had high quality staff and, in the case of the extension directorate, prior experience with the Bank. Audit reports were prepared in accordance with Bank requirements and were unqualified. Procurement Procurement was slow initially, due to delays in contracting, especially for consultants, and to slow start-up of project activities. The Bank’s procurement oversight, with the same staff throughout the project duration, provided quality training, mentoring and review of procurement processes, which significantly upgraded procurement capability. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons 1. In designing an agricultural marketing project, inclusion of measures to increase agricultural yields could substantially increase the project's impact. The project included some, but very limited, agricultural extension, relying on the general agricultural extension services of the state extension agency. Yet with more targeted agricultural support Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) services, crop yields would likely have increased further, substantially enhancing the overall impact of the project on productivity by increasing crop yields as well as market value added. 2. Specialist marketing and business experience are required to develop and implement an agricultural market development project. The preparation and supervision teams had a broad array of disciplines including in agribusiness, and staff had some general exposure to marketing in the course of their work, but this was mostly in a generalist capacity. More specialist inputs would have been desirable. (For instance, specialist skills in market chains, business development, commercial relations, assessing commercial opportunities and diversifying market options.) 3. Opportunities to boost participation of women should be sought for in a rural development/marketing project. The project, unlike its predecessor, the Land Management III Project, which had a proactive and well-regarded gender focus, had no specific program or features supporting participation of women, nor even any data collected regarding gender participation. A rural development and marketing project involving community organizations is well situated for actions and processes that can foster greater social inclusion, and a gender participation program to that effect would have been desirable. 4. If a project's main institutional features and main component(s) are not ready for implementation from Effectiveness or earlier, with core needs such as procurement already staffed and trained, there is a risk of implementation delay. The project, while ultimately successful, required an extension of the originally planned 5 ½ years by a further 3 years. The primary reason was the unreadiness of the implementing agencies, both in their institutional linkages and in staff skills and experience, and there was a complete lack of experience amongst the farmer organizations. Procurement procedures and other operational capabilities were also lacking, and the agencies and farmer organizations had little experience in marketing. An initial two years integrated in project design to build implementation capabilities would have been more responsive. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) The project could be reviewed, together with other agricultural marketing projects and with rural development projects that include marketing, to assess: the impact of agricultural marketing; marketing’s role as part of broader initiatives such as agricultural extension and rural infrastructure; and how market development can be inclusive, benefitting poorer families and women as well as higher income or more influential groups. 14. Comments on Quality of ICR The ICR provides a candid assessment of the project and its performance and provides thorough coverage of the various sections in the ICR template. Noteworthy features are: (i) the report provides an informative account of the sequence of revisions and restructurings of the project; (ii) efficacy is discussed by objective rather than by component or in general terms); (iii) the section on issues affecting the project (paras 49 to 60) is thoughtful and analytical; (iv) the results matrix at annex 1 generally provides informative background explanation on each achievement/target; (v) the ERR analysis at Annex 4 is clear; and (vi) the supporting information provided in Annex 7 enables reduced clutter in the main text. There are several areas where improvements could have been made. First, the actions under the project to increase agricultural productivity and the appropriate level of agricultural extension services could have been considered. Second, the project’s limited focus on gender is raised but has little discussion on what the issues were and what could have been done to better address gender. Third, environmental and social safeguards could have included more complete information on environmental clearance processes and compliance. And, fourth, in the efficacy and/or efficiency sections, presentation of supporting data, including any significant differences, between the project’s key impact results and the DIME survey results would have given greater confidence in the project’s impact. Overall, though, the report’s significant virtues outweigh the several gaps mentioned above, and the Quality of the ICR is Substantial. a. Quality of ICR Rating Substantial Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR SP Sust Rural Dev & Access to Markets (P108443) Page 20 of 20