Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Report Number: ICRR0021786 1. Project Data Project ID Project Name P115351 Samoa Agriculture Competitiveness Enhanc Country Practice Area(Lead) Samoa Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-50890,TF-12179 28-Mar-2017 12,097,450.70 Bank Approval Date Closing Date (Actual) 29-Mar-2012 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 8,000,000.00 5,000,000.00 Revised Commitment 12,889,316.35 4,999,584.70 Actual 12,097,450.70 4,999,584.70 Prepared by Reviewed by ICR Review Coordinator Group Paul Holden John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) The Project Development Objective (PDO) was to support fruit and vegetable growers and livestock producers to improve their productivity and take greater advantage of market opportunities. Financing Agreement (p. 5) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components Component 1: Improved Livestock Production and Marketing (estimated total cost at appraisal, US$4.73 million (PAD p. 7)), of which the project would finance US$3.6 million; actual financed cost was US$3.48 million (ICR p. 56). This component aimed to improve the quantity and quality of output through: • Increasing access to upgraded breeding cattle; • Improve locally grown feeding fodder to enhance livestock nutrition; • Provide technical advice on cattle raising; • Establish mobile and static slaughter facilities to improve meat quality and hygiene. It also aimed to finance investment on farms through matching grants, commercial loans from the Development Bank of Samoa and equity. If the Small Business Enterprise Center (SBEC) approved farmers’ business plans, they would be eligible for SBEC loan guarantees. Component 2: Improved Fruit and Vegetable Production and Marketing (estimated cost at appraisal, US$3.02 million, (PAD p. 7) of which the project would finance 1.8 million; actual financed cost was US$2.14 million (ICR p. 56) This component aimed to improve the quality and quantity of local produced fruit and vegetables through: • Providing farmers with improved planting materials; • Providing technical advice and promotional materials, including those for developing export markets; • Organic certification. Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Component 3: Institutional Strengthening (estimated cost at appraisal, US$6.09 million (PAD p. 7) of which the project would finance US$5.7 million. The actual financed cost was US$6.6 million (ICR p. 56) This component aimed to strengthen the technical capacity of government and non-government agricultural institutions providing extension services and undertaking agricultural research. It also included training Ministry of Agriculture and Fisheries (MAF) who monitored project implementation, as well as designing and implementing a monitoring and evaluation system (M&E), which would be compatible with the existing M&E systems of the MAF. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The total cost of the project at appraisal was estimated to be US$16.16 million. The project was financed through a grant of US$5 million from the Global Food Crisis Response Program and an IDA credit of SDR5.2 million (equivalent to US$8 million). An additional US$260,000 was provided by the recipient and US$2.9 million from the beneficiaries (ICR p. 13). At appraisal, the total cost of the components was US$11.1 million, with an additional US$900,000 being allocated for physical contingencies and US$1 million for price contingencies. The actual disbursed amount was: IDA credit US$7,097,866; Grant US$4,999,585 for a total of US$12,097,451. The differences in amounts disbursed compared with appraisal cost arose from changes in the exchange rate. Special Drawing Rights (SDRs) appreciated against the US dollar by more than 10 per cent over the life of the project. In addition, the Samoan Tala (SAT) depreciated against the US dollar over the same period. As a result, disbursements in SAT were 107 per cent of appraisal estimates. Other actual financing amounts disbursed were; borrower US$220,000; Local beneficiaries US$2,200,000. In SAT, the total beneficiary contributions were 90 per cent of appraisal estimates because fewer farmers participated in the Matching Grant Program. The project was approved on March 29, 2012 and became effective on May 4, 2012. The closing date for the trust fund grant amount of US$5 million was May 30, 2015, which closed on time and was fully disbursed. The original closing date for the IDA credit was March 28, 2017. The IDA credit component of the project was restructured (Level 2) on February 13, 2017 when the results framework was modified, there was a reallocation between disbursement categories and the closing date was Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) extended to March 31, 2018. The information on the restructuring and the revised targets is contained in the Implementation Status and Results Report (ISRR) dated December 22, 2016. There was a further Level 2 restructuring on December 21, 2017, which included a reallocation between disbursement categories and another extension of the closing date to December 31, 2018. There was a further Level 2 restructuring on December 21, 2017, which included a reallocation between disbursement categories and another extension of the closing date to December 31, 2018. 3. Relevance of Objectives Rationale The Samoa Country Partnership Strategy (CPS) 2012-2016 had, as part of its first priority (Rebuilding Economic Resilience and Encouraging Inclusive Growth) the goal to “improve agricultural productivity and strengthen opportunities for small and medium scale farmers.” It points out that “about 60% of the population is employed in agriculture, which provides a key safety net” (CPS p. 20), though it fails to mention that most of the employment is on a subsistence basis which is outside the market economy. The CPS points out that “the overarching theme running through the Government’s Samoa Development Strategy and the proposed Bank Group Program is the importance of building resilience against shocks” (p. i). It notes that the agricultural sector is dominated by small scale subsistence farming, which provides a safety net, and therefore resilience. The Regional Partnership Framework for FY 2017-2021 (RPF) (World Bank, 2016, Regional Partnership Framework FY17- FY21) (also ICR p. 15), which encompasses nine Pacific island countries, including Samoa, does not explicitly focus on agriculture beyond pointing out that it is important for livelihoods. Active projects in Samoa mentioned by the RPF include the Agriculture and Competitiveness Enhancement project. The RPF fails to mention that while the agricultural and fishing sector employs about a third of the workforce, much of the activity is subsistence and agriculture and fishing contributes only 14% to GDP. While 25 per cent of rural households are engaged in formal markets, where produce is sold for cash, (PAD p. 2) most of this is on a part-time basis. Merely 6 per cent of rural households receive their primary income from selling produce (Asian Development Bank, Reform Renewed: A Private Sector Assessment for Samoa, 2015, hereafter referred to as ADB 2015). The importance of the sector is clear, but the relevance of moving to a viable commercial model for the designated sectors is not clear. The reference in the regional strategy "to fully exploiting the available economic opportunities ... (including) increasing incomes from agriculture" infers that the priority is in sectors with the best opportunities for commercial returns. It is not clear, given the sectors chosen for this project, that this is apparent. The ICR (p. 24) points out that improved nutrition was an important secondary and complementary goal of the project through encouraging increased consumption of local fruit and vegetables and that the project is likely to have positive long- term impacts on diets, thereby reducing obesity, particularly among women. On the basis that an aim of the project was to Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) enhance productivity, with a secondary goal of improving health of the population, the relevance of objectives is rated substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improved productivity of fruit and vegetable growers Rationale The causal chain was based on the assumption that enhanced fruit and vegetable planting stock, better use of available technology, greater on-farm investment, training, and increased access to finance would lead to uptake by growers and thus higher productivity of fruit and vegetable growers. Investment was directed at field rock removal, improved drainage of fields during the rainy season, better irrigation during the dry season and the use of new banana stocks imported from South Africa. On the surface, this was a plausible causal chain. However, the underlying hypothesis on which the project was designed was short on important detail that the project does not address. The counterpart Ministry for the project was the Ministry of Agriculture and Fisheries, which had a history of poor advice and weak agricultural extension services, including the supply of unsuitable planting stock. (Samoa: Consolidating Reform for Faster Economic Growth, Asian Development Bank, 2008, hereafter referred to as ADB 2008). A further weak link in the causal chain was a failure to consider the negative incentives arising from communal ownership of customary land. Field rock removal and improved drainage and irrigation requires significant investment. Uncertainty of ownership implies that investors risk losing the fruits of their labor when they do not have long-term control over the land they are working, with a risk that the benefits would not accrue to the investor (see ADB 2008, for examples of this happening.) The inputs for this objective were nursery screening trials for new fruit and vegetable varieties, training provided to farmers and a matching grant program provided through the Development Bank of Samoa and the Small Business Enterprise Center. Outputs under this PDO were enhanced planting materials, new varieties of crops, “climate smart” farming practices and technologies and improved access to finance for farmers. The two original intermediate results indicators for this PDO were (i) the number of subsistence farmers using improved fruit and vegetable varieties (target 600) and (ii), the cumulative number of Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) business plans prepared by fruit and vegetable farmers and approved for grant financing (target 500). The original PDO indicators were (i) The percentage increase in average fruit and vegetable yields of growers participating in the Matching Grant Program (MGP), (target 100 per cent against a zero baseline); and (ii) The percentage increase in the average fruit and vegetable yields of targeted subsistence and farmers (target 50 per cent increase by the end of the project). Actual achievements for the intermediate indicators was 134 and 230 respectively. Well short of the original targets. The increase in average yields of fruit and vegetable growers in the MGP was 53 per cent significantly below the 100 per cent target. The result for subsistence farmers was a 36 per cent increase, well short of the original 50 per cent target. Rating Negligible OBJECTIVE 1 REVISION 1 Revised Objective Improved productivity of fruit and vegetable growers (unchanged) Revised Rationale Outputs and activities remain the same. Following the February 2017 restructuring, the intermediate indicators for this objective were revised substantially. The target for the cumulative number of business plans by fruit and vegetable growers was revised downwards from 500 to 150 and the number of farmers using improved fruit and vegetable varieties was revised down from 600 to 200. As outlined above, the actual achievements were 134 and 230 respectively. In addition, the outcome indicators were significantly revised down, with the percentage increase in average yields of fruit and vegetable growers in the MGP reduced to 50 per cent (from 100 per cent), with an actual achievement of 53 per cent, and that of subsistence farmers being revised down to 30 per cent (from 50 per cent) with an actual achievement of 36 per cent. It is difficult to ascertain the reason for the dramatic reduction in productivity targets given the adjustment took place close to the project's completion. By this time, a significant portion of the funds had been disbursed. As of March 31, 2017, the disbursement percentage was 73.1 (May 2017 ISR, p. 2). In addition, the considerable amount dedicated to this component is difficult to justify against the marginal progress on the adjusted targets. Only 150 business plans and 200 farmers using improved varietals is very low in the circumstances. Likewise, the results show that 68 per cent of the productivity improvement was achieved by farmers that received training alone as opposed to support through the MGP which suggests the matching grants were a premium priced support mechanism that did not provide as much benefit as was intended. Crudely applied, the Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) investment in fruit and vegetable growers was around $USD 14,000 as a proportion of the overall component spend. Given the small number of improved varietals introduced (7 as indicated on pg.17 ICR), uptake was small. Likewise, with a combination of mechanized services and fertilizer, the improvement in crop yields at these costs is also marginal. Given these shortcomings, even with the project targets being met it is difficult to see how progress against the objective is anything other than modest. Revised Rating Modest OBJECTIVE 2 Objective Improved productivity of livestock producers Rationale The causal chain for this objective was that higher productivity breeding stock would be imported and that improved investments, increased access to finance and training in better husbandry practices would lead to higher productivity of livestock producers and thus, there would be a larger proportion of locally produced beef and pork making its' way to market. Training needs would be identified by skill gap analysis. The training would be provided to the Ministry of Agriculture and Fisheries, which in turn would run training sessions for farmers. An anomaly, given the emphasis on private sector development in the government’s strategy, was that the Ministry of Agriculture and Fisheries ran the pig breeding unit and “sales of livestock from the MAF’s breeding farms would be made on a cost recovery basis.” (PAD p. 20). The construction of mobile and static slaughter facilities was also proposed to improve the quality and the hygiene of meat that in turn would increase the demand for locally produced beef and pork. There were no outcome indicators for this objective in the PAD. The ICR therefore used the intermediate results indicators as outcome indicators. For cattle, the indicator was the calving rate, with a target of a 60 per cent increase, with an actual rate of 68 per cent being achieved (results table ICR). However, the ICR points (p. 18) that MGP farmers "were able to increase their calving rate from 48 to 68 percent". Thus, it is unclear whether the average increase amongst all farmers was 68 per cent or if this was the maximum achieved percentage increase. For pigs, the average litter size increased, with the target being an average litter size of 7 compared with a baseline of 5 pigs per litter. The actual litter size achieved was 8. However, this figure only covers 90 farmers and there is no explanation of whether the increase was uniform or if some farmers performed better than others. This is particularly important when the benchmarks for each litter is small, well short of global benchmarks of 10 pigs per litter. Likewise, no indicators were developed that measured the number of animals produced annually, so that the per-litter number does not provide a full picture of actual improvements when the benchmark expectation for sows is two litters per year. This limitation in the data inhibits conclusions on the progress of the livestock sector improvements. There was no meaningful Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) additional material covering the other livestock farmers (ICR pg.18) given the small sample. The speculative reference to improvements in chicken producers does not sufficiently hold and the sheep sector is still in its infancy. An additional intermediate results indicator used in the ICR was the number of matching grant program and subsistence farmers owning improved livestock or adopting improved animal husbandry practices. The target was 400, though this is confusing in the ICR as the note explains that the indicator was added in the 2018 restructure and thus the notion of an original target seems superfluous (results table ICR pg.44) . The actual number at closing was 882 as a result of a surge in participants in the program, but it remains unclear if this surge was throughout the project or was towards the end. As a standard indicator, this result is of limited use given it does not sufficiently explain what the 'improved' practices were, nor which farmers were applying them and to what extent. There is some reference to improved feed, stockyards and new fencing (pg.19 ICR), but no breakdown of which approach had impact and what was expected of the participating farmers. While there are clear shortcomings, progress against this objective is rated Substantial. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective Improved productivity of livestock producers (unchanged) Revised Rationale Post-restructuring Outputs and outcomes remain unchanged and are provided in the overview above. As part of the February 2017 restructuring, the procurement of one of the two mobile slaughter units was dropped and a smaller scale static slaughter unit was proposed instead of the construction of the proposed abattoir. Most animal slaughter was previously undertaken under unhygienic conditions. This was a result of delays in the implementation due to the complexity of construction and limited MAF capacity. In spite of these shortcomings, progress was still made in enhancing the productivity of the sector and thus the rating is Substantial. Revised Rating Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Substantial OBJECTIVE 3 Objective Greater advantage of market opportunities taken by fruit and vegetable growers Rationale The causal chain for this PDO was based on certification of locally produced fruit and vegetables, which would lead to increased demand and a larger percentage of locally produced fruit and vegetables being sold domestically (PAD p. 5). While both the PAD (pp. 3-4) and the ICR (p. 13) mention exports, the focus of the project was on domestic markets. Furthermore, export prospects were not encouraging for a number of factors related to Samoa’s agricultural sector. The largest potential export markets were Australia and New Zealand, both of which have extremely strict phytosanitary standards. In order to export to these countries, fruit and vegetables must be heat treated, yet the government owned high temperature forced air fruit and vegetable facility was both too small and inefficient to process significant volumes of fruit and vegetables for export (ADB 2008). While the focus of the project was on domestic consumption (PAD p. 24), failing to realize the export potential of Samoa’s agricultural output limited the scope for sector expansion. Furthermore, one the largest exports of agricultural products was coconut oil. Yet the plantations of coconut trees date from the German colonial period and yields have been declining because of the age of the trees (ADB 2008). Investment in new coconut plantations is needed urgently, which the project failed to address. There were two PDO outcome indicators for this objective. The first was the percentage increase in the value of sales of fruit and vegetables by vegetable growers participating in the Matching Grant Program. The original target was 40 per cent while the actual achieved at completion was 68 per cent. This indicator was not revised in the restructuring. The second PDO indicator was the percentage share of locally produced fruit and vegetables sold by domestic retail and food service channels. The original target was 30 per cent, compared with a baseline of 5 per cent. By March 2017, the increase was 52 per cent (ISR p. 2). This compares with a 10 per cent increase that had been achieved by end October 2016. No explanation was provided for the very large change in the 5 months to end March 2017. While the PDO targets were met, progress against this objective is questionable. The support to fruit and vegetable farmers reached 134 beneficiaries (pg. 60 ICR). The project was aimed significantly at import substitution (ICR p. 17), but import statistics suggest that the impact of the project was minimal with respect to the impact on imports of fruit and vegetables. The baseline proportion of locally produced fruit and vegetables sold domestically was 5 per cent, while the increased output of producers participating in the program was 68 per cent. This would imply an increase of approximately 8.4 per cent in the proportion of local purchases of fruit and vegetables in total sales of domestically produced fruit and vegetables. Thus, the inference that there was a dramatic change in imports as a consequence of the project is not realistic. The changes identified in the 2018 Market Survey are reassuring, but it remains unclear what proportion of this change is likely to have come from the small number of project recipients (134 farmers) (pg. 20 ICR) This reasoning is supported by an examination of import data. According to World Integrated Trade Solution Statistics, imports of vegetables into Samoa in 2012 were US$19.3 million, Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) amounting to 5.6 per cent of total imports, while in 2017, vegetable imports were US$19.3 million, amounting to 5.4 per cent of total imports (see https://wits.worldbank.org/CountryProfile/en/Country/WSM/Year/2017/TradeFlow/Import/Partner/all/Product/06- 15_Vegetable). Therefore, the increase in PDO indicators is not consistent with import data results and the increase in the proportion of local purchases of domestically produced fruit and vegetables has been modest at best, even accounting for the Market Survey data. As a result, the efficacy rating for this objective is modest. Rating Modest OBJECTIVE 3 REVISION 1 Revised Objective Greater advantage of market opportunities taken by fruit and vegetable growers (unchanged) Revised Rationale Description of outputs and outcomes outlined above applies. In the February 2017 restructuring, the outcome indicator for the percentage of locally produced fruits and vegetables sold by domestic retail and food service channels was revised down to 20 per cent but at completion, the actual percentage was 51 per cent. These data were obtained from market surveys (ICR p. 41). However, as the overview above suggests, the accuracy of the market survey data and the very limited number of beneficiaries suggest there were a range of other factors at play in measuring any import substitution effects. As above, the rating for this objective remains Modest. Revised Rating Modest OBJECTIVE 4 Objective Greater advantage of market opportunities taken by livestock producers Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Rationale The causal chain underlying this objective was that the building of mobile and static slaughter units under a new regulatory framework governing meat hygiene, in addition to improved animal husbandry ,would lead to better quality meat, which in turn would increase the demand for locally produced meat and result in greater market opportunities for livestock producers. There were two outcome indicators for this PDO. The first was the percentage of locally produced beef sold by domestic retail and food service channels. The baseline was a 40 per cent share and the original target was 55 per cent. The indicator here has both technical and data issues. The indicator itself is confusing given that it does not explicitly articulate if we are interested in the proportion of locally 'consumed' beef produced by local producers or if we are interested ni the proportion of local beef that makes its' way into the supply chain. This is an important difference and the ICR uses these two different indicators interchangeably which makes assessment difficult. The challenge of the indicator is revealed in the ISR for May 2017 which reports that this proportion was 29 per cent in October 2016 and 95 per cent at end March 2017. This is not a credible increase over the 5 month period. Moving then to the ICR (p. 33), it reports that faulty methodology was used to obtain the baseline, which was inconsistent with the Market Survey data which suggests that 84 per cent of domestic consumption was coming from local producers, which casts further doubt on the achievement data. The second outcome indicator was the per cent increase in the value of sales of livestock producers participating in the matching grant program. The original target was 40 per cent, lowered to 30 per cent (but without explanation of why) and with an actual number of 46 per cent achievement at project completion. These indicators are difficult to interpret. Local producers in the matching grant program increased the value of sales by 46 per cent. At the same time, 94 per cent of locally produced beef was sold domestically, compared with a baseline of 40 per cent at the start of the project (acknowledged as problematic in the ICR) . The data are difficult to reconcile. In addition, there is no comparison on the increase in value of sales of livestock by MGP participants with any increase for non-MGP participants. This makes it impossible to know if a simple improvement in extension services and growing domestic demand encouraged farmers to prioritize livestock improvements or if the other aspects in the MGP approach were fundamental to the change. We know that access to hygienic slaughter facilities makes a difference to all farmers given the introduced legislation, so there is some evidence that the value of sales would equally apply to farmers not in the MGP program. Thus, the rating for this objective is Modest. Rating Modest OBJECTIVE 4 REVISION 1 Revised Objective Greater advantage of market opportunities taken by livestock producers (unchanged) Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Revised Rationale Outputs and outcomes remain as above. The second outcome indicator, the percentage increase in the value of sales of livestock by participants in the MGP program was revised downward to 30 per cent (no clear explanation as to why), with an actual achievement of 46 per cent at completion (ICR p. 42). Given the measurement issues outlined above and the limited increase in market share by local producers, this objective post-restructuring remains modest. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The causal chains associated with each of the objectives were plausible, although there were weaknesses in design of both the inputs and the outcome indicators. While productivity indicators showed improved outputs, several issues associated with their design calls into question their accuracy, and they do not hold when compared with other indicators, including import data. Furthermore, overall productivity data based on value added per worker in the sector (https://data.worldbank.org/indicator/NV.AGR.TOTL.CN?locations=WS) suggests that productivity improvements were higher in the 2006 – 2012 period than they were during the project period. The cancellation of one of the mobile slaughter units and the static slaughter unit compromised the objective of better hygiene. With three of the four objectives rated modest and one rated substantial, the overall rating is modest. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) While there were some minor differences in the level of achievement against the revised targets, the overarching issues regarding Efficacy remain the same across both periods and thus the overall rating remains unchanged as Modest. Overall Efficacy Revision 1 Rating Primary Reason Modest Low achievement 5. Efficiency Originally, the project aimed to assist 1,072 livestock farmers and 513 fruit and vegetable farmers through the matching grant program, as well as 583 subsistence fruit and vegetable farmers through improved technical advisory services. In addition, it had planned to construct two mobile slaughter units and one static slaughter unit. By closure, 740 farmers had entered the matching grant program, with 594 having completed their projects, and 882 farmers, some of whom were in the matching grant program, had undergone training in animal husbandry. At the same time, 200 Ministry of Agriculture and Fisheries staff had received training. However, only one mobile slaughter unit was constructed, and the static slaughter unit was not constructed at all. Significant delays were a feature of the project. This was in part due to delays at its commencement as a result of cyclone Evan, which impacted Samoa eight months after the commencement of the project. The ICR (p. 24.) states that this impacted implementation by at least six months. Overall, the closing date of the project was extended by a cumulative total of 21 months. The ICR (p. 24) notes that slow procurement resulted in some activities not being completed by project closure. At appraisal, the financial and economic internal rates of return were estimated to be 13.5 per cent and 15.3 per cent respectively, based on a discount rate of 12 per cent. The ICR (p. 59) states that it was not possible to use a standard approach to utilizing economic and financial analysis in order to calculate the ex post rate of return on the project because of the demand driven nature of the project that resulted in actual activities being substantially different from those envisaged at appraisal. The ICR (p. 59) states that the ex post ERR was calculated based on a tracking of ex ante business plans prepared by the beneficiaries of the matching grant program. Benefits to farmers were calculated from a survey that undertook to capture how the project positively impacted participants. These included: • Productivity gains through enhanced on-farm investment and access to upgraded livestock; • Productivity gains through investment and access to high yield crop varieties; • Increased access to extension services, although this is an input, not an output, and as noted above, extension services supplied by the Ministry of Agriculture and Fisheries had not been highly regarded by farmers (ADB 2008). However, as the M&E section points out, the data collected were unreliable, which casts doubt on any calculations of the ERR. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Based on the modest rates of return on the project, the project delays, the failure to build a second mobile slaughter unit and the failure to build a static slaughter unit, efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  15.30  Not Applicable 100.00 ICR Estimate  13.30  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The calculations of the final outcome are based on the split rating shown in Table 1 below. The project was rated Substantial for Project Relevance, Modest for the overall Efficacy rating and Modest for Efficiency.There were only minor differences in the ratings for specific objectives in the two periods under Efficacy and thus the outcome rating is unchanged for both periods as Moderately Unsatisfactory. Table 1: Overall Outcome Ratings Rating Dimension Objectives Relevance of Objectives Substantial Ratings Pre-restructuring Ratings Post-restructuring Efficacy Modest Modest Efficiency Modest Modest Outcome Rating Moderately Unsatisfactory Moderately Unsatisfactory Outcome Rating Value 3 3 Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Disbursement Percentage 62% 38% Weighted Value 1.86 1.14 Final Outcome Rating Moderately Unsatisfactory a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome There are a number of risks to development outcomes, the most significant being: • The ICR (p. 36) points out that by April 2019, none of the project staff remained in the Ministry of Agriculture and Fisheries, with a concomitant loss of institutional memory; • The lack of skills and capacity in the Ministry, which employed only one veterinarian for the entire country; • The lack of an adequate maintenance budget for equipment purchased under the project; • The continued threat of natural disasters, a feature of Pacific island countries, which have the potential to severely impact the agricultural sector; • Based on analytical work undertake outside the World Bank Group (ADB 2008, ADB 2015), IEG identifies the lack of commitment to private sector development by the Ministry of Agriculture and Fisheries and the continued substantial presence of the public sector in agriculture, including the failure to privatize the agricultural estates as adding to the risks. At closure, 20 per cent of the Matching Grants were incomplete and 21 per cent of farmers who had obtained loans from the Development Bank of Samoa were in arrears (ICR p. 28). This suggests that a significant number of farmers who took out loans were not generating sufficient income to service their debts, casting doubt on their sustainability. 8. Assessment of Bank Performance a.Quality-at-Entry At entry, the project did not sufficiently account for a number of critical factors in promoting agricultural production and output. The government of Samoa was the largest participant in the agricultural sector, owning two large estates totaling Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) over 12000 hectares, only some of which was leased out to private farmers. However, there was widespread acknowledgement (see ADB 2008 & ADB 2015) that the leasing process was non-transparent, and that the estates were inefficiently run. Furthermore, while the government sector plans and the PAD (p. 2) acknowledged the export potential for organic fruit and vegetables, the communal land tenure system as well as the reluctance of the land estates to seek organic certification limited the export potential of the sector, something that the project did not address in detail and was insufficiently incorporated into project activities. A parallel land reform project initiated by the ADB experienced substantial difficulties over the same period as the agricultural competitiveness project. See for example, ((https://www.rnz.co.nz/international/pacific-news/271408/calls-for-more-talks-on-samoan-custom-land-project), yet the competitiveness project appears to have ignored a highly complementary initiative The two largest agricultural export crops were coconut oil and nonu juice, yet the project did not explain how these products were to fit into the export market mix. It is evident that production for both of these products could not keep pace with demand (ADB 2008) yet there is limited explanation of why the designated sectoral focus was chosen over other more established industries. There were also issues with scale. The project design documents do not sufficiently articulate how the improvements in the given sectors would grow to scale in the longer term. This is inferred rather than explained and thus there is little evidence that foundation aspects of the sector were being established through the design and implementation. An additional factor was the inefficiency of the government owned Agricultural Stores Corporation, which was slated for privatization in 2012 and as of 2018 had still not been privatized. The project missed an opportunity to include this as part of conditions attached to its implementation. Quality at entry was therefore compromised by a too narrow focus in the produce sector on fruit and vegetables and a failure to take into account the longer-term strategic viability of sectors that were already successful as demonstrated by their export performance. In addition, there was insufficient detail and rigor in how to measure the different aspects of the project to determine which characteristics are likely to most improve the viability of the sector in the longer term. The considerable shortcomings mean the rating for Quality at Entry is Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The ICR (p. 35) reports that the “the Bank was well focused on achieving its PDO” (p. 35), undertaking 13 full implementation missions. However, the fact that the project was restructured only one month before its original closing date Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) belies the contention that supervision was proactive. The ICR (p. 29) points out that demand for matching grants under the fruit and vegetable component was significantly less than anticipated and that this should have been identified sooner. The ICR (p. 30) also notes that the Project Coordination Unit (PCU) was not integrated into the Ministry of Agriculture and Fisheries, which reduced the effectiveness of coordination. This could have been addressed during supervision and adds to the conclusion that the restructuring should have taken place earlier. A further issue was the failure to coordinate with other development partners, particularly with the ADB, which in 2014 provided a grant to Samoa to provide business support services to agricultural businesses focusing on agro-industrial production. There is no reference in the ICR, or in the restructuring documents to any attempt by the project to coordinate or synergize with this project, nor with an ADB land reform project. Given these shortcomings, the rating for Supervision is Moderately Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design There were a number of weaknesses in the design of the M&E framework. The measures of productivity of both the growers of fruit and vegetables and meat producers were deficient. For the former, aggregate measures did not adequately track specific varieties. As the ICR points out (p. 32) the measure “proved too general which… were hard to reconcile into one average”. Farmers used non-standard measures, such as “bunches of cabbages” or “bags of tomatoes” which resulted in data inconsistency. For livestock, indicators for the calving rate and the size of litters for cattle and pigs respectively did not adequately capture whether output had actually increased; they did not measure the total output of animals, nor their survival rates. A further problem with M&E design in the livestock sector was the inconsistent methodology used to obtain the baseline data, compared with that used to obtain the results of the project. In addition, terminology around the indicators is confusing and the use of percentage changes and percentage shares did not make sense for many of the indicators. An additional gap in M&E design was the failure to track the repayment rates for agricultural producers who had borrowed under the SBEC 100 Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) per cent guarantee program. The ICR (p. 39) points out that over 20 per cent of farmers who borrowed from the Development Bank of Samoa were in arrears on their repayments. b. M&E Implementation The Ministry of Agriculture and Fisheries were responsible for collecting M&E data with the help of the project staff. The design aspects of the M&E framework referred to above hampered M&E implementation, which was further compromised by lack of record keeping by farmers. The ICR reports (p. 32) that only 57 per cent of crop farmers and 34 per cent of livestock farmers kept records, which made data suspect and greatly added to the burden on project staff, who had to individually compile records on farms that they visited. As a result, M&E implementation was challenging c. M&E Utilization Market surveys were used to inform the evolution of the project. These resulted in the project targets being changed in the restructuring. However, there are two key points; first the restructured targets were in a number of cases set unrealistically low, and were exceeded at closure by a wide margin, which calls into question the accuracy of the data obtained from the surveys. Second, the delay in restructuring to only a month before the original closing date calls into question whether M&E data were used proactively. The design of the indicators was suspect. Furthermore, the failure, in spite of 13 supervision missions, to identify the poor record keeping of farmers, which compromised the indicator data and added to the burden of the PCU reduced the effectiveness of the M&E framework. Cross correlating M&E indicators with trade data would also have revealed issues with the framework. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Two environmental safeguards were triggered, namely the need for an environment assessment and pest management policies. The implementation of these was governed by an Environmental and Social Management Framework. The main risk identified was associated with the envisaged static slaughter unit, but since it did not progress, the risk oversight became redundant. (ICR p. 33). b. Fiduciary Compliance Fiduciary compliance was rated as moderately satisfactory based on reports from the World Bank financial management and procurement specialists, who also provided implementation support. A 2016 audit report found no fiduciary issues with the project. Nevertheless, the ICR (p. 30) reports that for substantial periods, there were no procurement staff on the project team, which resulted in significant delays. c. Unintended impacts (Positive or Negative) --- d. Other Staff costs for preparation and supervision amounted to US$1.42 million, more than 10 per cent of the value of the project. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment There were significant performance problems in the fruit and vegetable Moderately Outcome Moderately Unsatisfactory sector and substantial measurement Satisfactory issues in achievements against the livestock activities. Substantial shortcomings with design Moderately and inconsistent supervision in Bank Performance Moderately Unsatisfactory Satisfactory making the required changes to the project in a timely way. Quality of M&E Modest Modest Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) Quality of ICR --- Modest 12. Lessons The ICR offers the following lessons, with some amendment of language by IEG: It is important to ensure that the project team is closely integrated with the teams from the implementing Ministry and that it is adequately staffed. In the case of the Samoa project, the team was too small for the amount of supervision required (ICR p. 31) and the lack of integration resulted in issues around the ownership for the counterpart ministry; It is important to consider local capacity and counterpart capacity in project design and if necessary, budget for supplemental consultants to assist with implementation. In the case of the Samoa project, the difficulty of attracting and retaining project staff affected the efficiency of the project and its implementation (ICR p. 37) and there were capacity deficiencies in Ministry of Agriculture and Fisheries staff. It is important that the design of outcome indicators takes into account data availability and the consistency of data used for monitoring and evaluation. In the case of Samoa, inconsistencies and lack of availability of data hampered implementation, which was not anticipated in project design and necessitated project staff collecting data from farmers. IEG adds the following lesson: Failure to draw on analytical work by development partners and/or their projects in the same sector can compromise project design and the selection of local counterparts. In the case of Samoa, the design of the project needed to better employ existing analytical work from all development partners. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Samoa Agriculture Competitiveness Enhanc (P115351) On the positive side, the ICR is rich in detail regarding aspects of the project and the problems that it encountered. Nevertheless, there were substantial shortcomings. Much of the interesting narrative gives the project the benefit of the doubt and we are left wanting more detail on metrics and actual agricultural outcomes. Currently it is too inward looking and sees much of the success as delivering the committed activities rather than identifying developmental transformations. In addition, at 37 pages of main text, plus 28 pages of appendices, the ICR is too long. There is a lot of repetition and material that does not necessarily add to the performance story. The main text is over twice as long as the maximum recommended (15 pp.) by the Bank’s OPCS Guidelines. The ICR undertook a split rating on the basis of significantly downgraded indicator values, but it does not explain the drivers for the substantially lowered targets. Furthermore, while the ICR does capture some of the inconsistencies of the project data, which led to it being overly generous in its rating of various aspects of the project, it does not explore the issues candidly. Nor does it look much beyond the indicators as the basis for its reporting. There are major technical issues related to the agriculture sector in Samoa that deserved greater coverage, particularly with regards to how activities informed broader agricultural change and very little of the ICR looks at aspects of scope. This is a significant omission in a country like Samoa which is a small island state. The ICR did not address or acknowledge other analytical work out there, nor use this information to effectively make a case for its illustration of achievements. a. Quality of ICR Rating Modest Page 21 of 21