Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Report Number: ICRR0021848 1. Project Data Project ID Project Name P143921 Adriatic Sea Env. Pollution Control (I) Country Practice Area(Lead) Western Balkans Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-17706,TF-17727 15-Mar-2017 4,642,342.60 Bank Approval Date Closing Date (Actual) 11-Jun-2014 15-Feb-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 6,770,000.00 6,770,000.00 Revised Commitment 6,770,000.00 6,770,000.00 Actual 4,656,617.13 4,642,342.60 Prepared by Reviewed by ICR Review Coordinator Group Helen Santiago Fink J. W. van Holst Pellekaan Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) The project development objectives (PDO) of the GEF Adriatic Sea Environmental Pollution Control Project was to (a) reduce the discharge of pollutants with transboundary importance, particularly nitrogen, in selected Hot Spots of the eastern Adriatic Sea; and (b) to improve the capacity in Croatia and in Bosnia and Herzegovina (BiH) to prepare pollution control projects in selected localities of Dalmatia and Herzegovina and to strengthen the capacity to monitor sea water quality ("Global Environment Facility Grant Agreements" between Croatia and Bosnia and Herzegovina (BiH) dated respectively September 8, 2014 and September 10, 2014. As the ICR (para 3) notes, this project was designed as the first project of a programmatic operation. The design, approach, and objectives followed those agreed to for the Adriatic Sea Environment Program (ASEP) by the World Bank, GEF, and the countries in the region upon completion for the assessment of hotspots. The broader programmatic objective was to launch ASEP's implementation through technical assistance to support preparation of investment proposals and policy development based on the two demonstration investments in this project which were seen as "serving as pilot projects fostering innovation and subsequent replication " (The Scientific and Technical Advisory Panel, administered by UNEP, advises the Global Environment Facility, PAD, page 64). For the purpose of assessing this project's achievements this review has divided the PDO into two sub-objectives, namely Objective 1 - "Reduce the discharge of pollutants with transboundary importance, particularly nitrogen, in selected Hot Sorts of the eastern Adriatic Sea", Objective 2 - "Improve the capacity in Croatia and BiH to prepare pollution control projects in selected localities of Dalmatia and Herzegovina", and strengthen the capacity to monitor sea water quality." On September 8, 2016, targets for the project's objective indicators were substantially reduced, thereby reducing the level of ambition of the project. This triggered the need to undertake a split assessment of the project's outcome. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 08-Sep-2014 c. Will a split evaluation be undertaken? Yes d. Components Page 2 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Component 1. Demonstration investments to reduce nutrient discharges and improve water quality monitoring capacity (appraisal cost US$27.77 million, of which US$4.98 million is GEF financing; actual cost US$27.64 million, of which US$3.28 was GEF financing) - One demonstration investment project involving the installation and upgrading of leachate management systems was planned in each country of Bosnia and Herzegovina (BiH) and Croatia. In Croatia, the demonstration project for the leachate collection and treatment plant was originally set for the Diklo landfill in Zadar but after the project restructuring, it was moved to the Sitnica landfill on the island of Korcula. Whereas, in BiH, the demonstration project was planned at the Mostar’s landfill and was not affected by the restructuring, although it ultimately was not executed. Component 2. Technical Assistance (TA) (appraisal cost US$1.98 million, of which US$1.60 million was GEF financing; actual cost US$1.20 million of which US$1.20 million was GEF financing) - TA focused on the preparation of proposals for possible funding by the EU, including all related analysis required by the EU, such as a Strategic Environmental Assessment of the overall program for leachate treatment and the assessment of relative sources of nutrients to derive policy recommendations. The technical assistance in both countries was carried out with the assistance of consultants, who supplementing formal training programs, provided 'hands on' capacity building assistance for national staff. predominately by consultants. - In Croatia, consultant services were intended to assess sources of nutrients (pollutants), including point and nonpoint sources in two sensitive areas/hotspots (Northern Dalmatia’s Zadarska County and northern part of the Sibensko-Kninska County and Neretva Delta area), as well as to provide an analysis of the policy, legal and/or institutional reforms and address related water quality problems. Additionally, preliminary designs and project documentation, for investments in leachate treatment and management systems to comply with EU policies in selected localities in Dalmatia were prepared. - TA activities in BiH were targeted for the preparation of preliminary designs and project documentation, including preparation of tender documentation to access European Union (“EU”) funds, for investments in leachate and wastewater treatment and management systems to comply with EU requirements in selected locations in Herzegovina and Neum. Component 3: Project Management and Dissemination (appraisal cost US$0.22 million, of which US$0.19 million was GEF financing; actual cost US$0.167 million, all of which was GEF financing.) This component was intended to support the project management teams predominately in BiH, including office expenses related to project administration, translations, communications, local and abroad travels, banks fees, audit reports, office supplies and equipment as well as repair and maintenance of office equipment. The salaries of civil servants of the recipient countries were not to be financed. It also included about 1% of grant funds specifically reserved for evaluation and dissemination under IW-Learn related activities. In addition, it provided incremental operating costs for project management as well as monitoring and evaluation for the Project Management Team in BiH. Page 3 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs: The total costs at project approval were $29.97 million. Actual disbursements totaled $29.02 million at project closing (ICR, Annex 3, p. 43). The project costs were not revised as part of the restructuring. Financing: The Bank’s lending was through a Sector Investment Loan (SIL). This was supplemented by a GEF grant to Croatia and BiH totaling US$ 6.77 million, under respective grant agreements. The actual GEF grant disbursed was US$4.65 million. Of the original total GEF grant amount of US$6.77million, US$4.33 million was to be allocated to the Republic of Croatia, through a subsidiary agreement to the Environmental Protection and Energy Efficiency Fund (Fond za zaštitu okoliša i energetsku učinkovitost - EPEEF), and US$2.44 million to the Government of BiH, to be implemented by the State Ministry of Foreign Trade and Economic Relations (MoFTER). Borrower Contribution: Total government contribution was estimated at US$23.20 million, which did not change through the project, with US$22.78 million to be contributed by the Republic of Croatia and US$0.41 million from the Government of BiH. The total actual amount disbursed was US$24.36 million, fully contributed by the Republic of Croatia (ICR, Annex. 3). Dates and Restructuring: The project was approved on June 11, 2014. It underwent one restructuring in September 2016, that involved the following: 1) an extension of the project closing date by 23 months from its original date of March 15, 2017 to February 15, 2019 2) ‘Component 1: Demonstration Investments to Reduce Nutrient Discharges and Improve Sea Water Quality Monitoring Capacity' was revised and involved a change of location of the demonstration investment in Croatia from the Diklo landfill in Zadar to the Sitnica landfill on Korcula Island. 3) The indicator for the reduction of nutrient discharges was reduced from 130,000 tonnes to 70,000 tonnes per year as was the estimated number of direct beneficiaries from 235,000 to 140,500. The indicator for the latter part of Component #1, 'Improve Sea Water Quality Monitoring Capacity', did not change and remained at 3,000 per year; similarly the component cost was not modified. 4) Support/TA for carrying out an assessment of source of nutrients and TA for preparation of preliminary designs and project documentation to access EU funds for investment in leachate collection and management systems (letter to Croatia Ministry of Finance, Sept. 20, 2016) A mid-term review was conducted in December of 2017, "though conducted with some delay, confirmed and kept the revised Results Framework according to the 2016 restructuring and monitoring in relation to Croatia. For BiH, there was no discussion to restructure the project to address the challenges faced in BiH at the time" (ICR, p.25). Page 4 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Carryover work involving the construction of a final element of the leachate management system at the Sitnica Landfill continued beyond the Bank’s official closing date with expected completion by August or September of 2019. The costs of the work were fully covered by the Environmental Protection and Energy Efficiency Fund (EPEEF) of Croatia. 3. Relevance of Objectives Rationale The project’s objectives were to: (a) "to reduce the discharge of pollutants with transboundary importance, particularly Nitrogen, in selected hotspots of the eastern Adriatic Sea; and (b) to improve the capacity in the Republic of Croatia and Bosnia and Herzegovina to prepare pollution control projects in selected localities of Dalmatia and Herzegovina and to strengthen the capacity to monitor the sea water quality." - The GEF Adriatic Sea Environmental Pollution Control Project (I) was born out of the Adriatic Sea Environment Program (ASEP). Under ASEP, a regional Technical Assistance (TA) program funded by the Global Environmental Facility (GEF) and implemented by the World Bank in 2011, to support preparation of environmental policies and demonstration investments as well as a study on the ‘Rapid Assessment of Pollution Hotspots for the Adriatic Sea’. The Republic of Croatia and BiH were the two hotspot priority Adriatic Sea Riparian countries selected to be part of the first regional and programmatic pilot operational projects. - The project's objectives aligned with the EU water and wastewater policies and with conditions for the availability of EU grant funds, specifically the Instrument for Pre-Accession Assistance funds in candidate countries and cohesion funds for EU member countries. The recipient countries national policies, adopted as part of the Mediterranean Strategic Action Plan (SAP MED) of 2000–2015, are complementary to the project’s and Bank’s objectives. - The PDOs were also aligned with the Country Partnership Framework for Croatia FY19-24 (CPF). As per the CPF, all new IBRD financing is aimed at building essential institutional capacity (p.3) and prioritized strengthening Croatia’s abilities in addressing environmental and climate issues, specifically Objective Five of the CPF which focuses on improving water, wastewater, and solid waste delivery and management. - Finally, this project's objectives were aligned with the BiH CPF in that its long-term efforts to create stable and sustainable economic opportunities, in part through the prevention of the degradation of natural resources, includes reducing pollution and protecting the environment (page 43). Also, as an aspiring EU member, the project may contribute to BiH’s membership process and development trajectory. The CPF (page 34) focuses on strengthening the country systems for financial management and public procurement were aligned with the project's second objective, and remains, particularly relevant given the project's shortcomings. However, reducing water pollution at a country and regional scale goes beyond simply improving the leachate management systems at two local sites and enhancing the ability to secure funds. It is essential to address the root causes of the pollution at Page 5 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) point and non-point sources, which highlights the need to improve waste management operations of all sectors, including industry, business, farms, tourism as well as the practices of waste collection systems and behavior change on the part of households. While the ambition of the project was modest, it had the potential to establish a basic organizing framework for addressing important challenges associated with the risks of water quality deterioration in the Adriatic Sea and hence avoid the negative impacts on both local communities and tourism development in the region. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To reduce the discharge of pollutants with transboundary importance, particularly nitrogen, in selected Hot-Spots of the eastern Adriatic Sea. Rationale Theory of Change (TOC): The theory of change (TOC) is well portrayed in Figure 1 of the ICR which illustrates how the reduction of the levels of pollution of the Adriatic Sea on account of nutrient load mitigation at specific landfill sites in the countries of Croatia and BiH can be advanced through investments in the construction of leachate management systems in landfills in the respective recipient countries. The appropriate treatment of landfills will reduce the nitrogen load of water leached from them which could potentially reach the Adriatic Sea and cause major water pollution that will endanger the health of coastal communities and jeopardize the future of the tourism industry along the Adriatic coastline. The core objective of this project is to finance activities that will reduce the discharge of pollutants with transboundary importance, particularly nitrogen in selected hotspots adjacent to the Eastern Adriatic Sea Outputs: - In Croatia, the Diklo landfill in Zadar was selected for leachate management improvements but was not constructed prior to the 2016 project restructuring because a land rights dispute could not be solved. The site was changed after restructuring in 2016 to the Sitnica landfill on the island of Korcula where land title transfer issues could be resolved. Page 6 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) - In BiH, the landfill site of Mostar was identified for the leachate systems installation, however, it was never constructed due to legal and governance complications related to the refusal of the municipal authorities to comply with World Bank procurement guidelines before the project closed. - Preparation of various technical and TA documents, including contracts for the Zadar landfill, Strategic Environmental Assessment (ICR, p. 55) Outcomes: (ICR, Annex 1, p.39) - In Croatia, before restructuring, there were no achievements at the Diklo landfill. Following the 2016 restructuring and a delayed start on a smaller landfill at Sitnica to replace Diklo, the equivalent of 20 tons of nutrient load reduction per year was achieved at the project's close. It was expected that 15,500 persons would be direct beneficiaries of the improved landfill at Sitnica (ICR, Annex 4, page 44) - In BiH, the project did not achieve any nutrient load reduction either before or after restructuring in 2016 due to the inability of municipal; authorities to gain Bank approval to use the GEF funds to construct the leachate management system in Mostar. - For Croatia and BiH combined the outcome was one relatively small demonstration project where the reduction of 20 tons of nutrient load represented only 29% of the revised total target of 70 tons per year for the Sitnica and Mostar landfills which was estimated to directly benefit about 15,500 persons or 11% of the revised total target at both sites of 140,000 persons for the restructured project This review rates the achievement of Objective 1 before restructuring as negligible. Rating Negligible OBJECTIVE 1 REVISION 1 Revised Objective To reduce the discharge of pollutants with transboundary importance, particularly nitrogen, in selected hotspots of the eastern Adriatic Sea. There was no revision in the PDO, but the indicator for the number of beneficiaries in Croatia was amended due to the change in location for the landfill site from 110,000 persons at Diklo in Zadar to 15,500 persons at Sitnica in Korcula (ICR, page 44, footnote 38). Page 7 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Revised Rationale Theory of Change (TOC): The project aimed to reduce the level of pollution in the Adriatic Sea through nutrient load mitigation measures, specifically the construction of leachate management systems, at selected landfill sites in the countries of Croatia and BiH. Outputs: - In Croatia, the construction of a leachate management system was, as noted above in the context of Objective 1, eventually achieved at the Sitnica landfill after a change from the original location of Diklo after the 2016 restructuring. - In BiH, the construction of a leachate management system was not achieved due to governance and procurement issues. Outcomes: (ICR, Annex 1 p.39) - After the 2016 restructuring in Croatia, 20 tons/year of nutrient/nitrogen load reduction was achieved at Sitnica which met the target. Additional details are presented in the context of Objective 1 above. - BiH was not affected by the restructuring, nevertheless, Objective 1 was not achieved in BiH on account of governance and procurement challenges. The completion of improved leachate management at Sitnica with the reduction of nutrient/nitrogen load at a rate of 20 tons per year in association with the lack of any achievements at the Mostar landfill after the 2016 restructuring was assessed by this review as only a modest achievement of Objective 1 for this demonstration project. Revised Rating Modest OBJECTIVE 2 Objective To improve the capacity in Croatia and Bosnia and Herzegovina to prepare pollution control projects in selected localities of Dalmatia and Herzegovina and to strengthen the capacity to monitor sea water quality. Rationale Page 8 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Theory of Change (TOC): The project provided technical assistance aimed at informing and educating local authorities in both countries on the technical aspects of addressing water pollution through a series of training sessions and workshops, including participation at two IWLEARN GEF conferences to learn about best practices in water pollution controls and to better contribute to improved environmental conditions for the Adriatic Sea. Additionally, TA was provided to the riparian countries of BiH and Croatia in the procedures and processes of developing applications for EU investment funding aimed to enhance their institutional capacity to reduce the level of pollution of the Adriatic Sea. Strengthening the capacity to monitor seawater quality enhanced the ability of the local authorities in the region to gather evidence on the levels of nutrient discharge and to plan future pollution control actions for the Adriatic Sea. Outputs: ICR, Annex 1 p.39-40 - Six capacity-building workshops were organized in total; three on the preparation of documentation to apply for the EU funds and three (3) on sustainable planning and management of solid waste, landfill leachates, and wastewater. - Both Croatia and BiH government authorities participated in the GEF Biennial International Waters Conference (IWC) (IW Learn) in Sri Lanka in 2016 and Morocco in 2018 - Four technical assistance contracts were implemented, all related to the Croatian demonstration site, involving the analysis of the policy, legal and/or institutional reforms conducted, including the Croatian national waste and leachate management plan and Strategic Environmental Assessment for leachate management plan (for Croatia). - Much of the TA to prepare future project proposals for EU funding was delivered by consultants which begs the question: How much direct benefit was provided to the government authorities in the recipient countries and in the improvement of their respective institutional capacities to undertake that work in the future. This ICRR therefore questioned whether the TA outcome indicator (number of investment proposals prepared) was the best choice to measure the improvement of local institutional capacities. The Bank task team explained to IEG that the technical assistance was directed towards strengthening municipalities in both recipient countries. The TA consisted of training for building institutional capacity as well as workshops on the preparation of projects and the preparation of funding applications. - Provision and installation of equipment to strengthen regional capacity to monitor the seawater quality Outcomes: - A total of ten project investment proposals were prepared for EU funding. Seven were for Croatia and three for BiH (ICR, paras 46 and 50) exceeding the total target of six proposals. It was also learned from discussions between IEG and the Bank task team that three of the proposals prepared for the EU in Croatia were funded and are in the process of tendering. No information was available in the ICR on funding of proposals for BiH. - Valuable knowledge transfer from the project to local stakeholders resulting in 42 activities that included six waste management centers in Marišćina, Kaštijun, Biljane Donje, Bikarac, Lećevica, and Babina Gora and 36 landfill remediation Page 9 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) sites approved for EU co-financing and an additional 23 proposals under preparation for potential financing in the programming period 2014–2020 (ICR, para 47)). - According to the ICR, by 2019, 43,267 sea water quality measurements had been made annually in Croatia from 105 sampling stations compared with a target of 3,000 sea water measurements from 60 sampling stations. The stations are monitored every two weeks and data are reported on the Sea Bathing Water Quality website in Croatia (ICR, para 48). Based on the evidence obtained this review rates the achievements of this objective before restructuring as substantial but marginally so because of inadequate evidence about the capacity building results from the TA program.. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective To improve the capacity in Croatia and Bosnia and Herzegovina to prepare pollution control projects in selected localities of Dalmatia and Herzegovina and to strengthen the capacity to monitor sea water quality Revised Rationale There was no change in Objective 2 following the project's restructuring in 2016 Theory of Change (TOC). The theory of change was the same as stated for Objective 2 above. Outputs. These were the same as for Objective 2 above Outcomes. These were the same as for Objective 2 above Based on the evidence obtained this review rates the achievements of this objective after restructuring as substantial. Page 10 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The following is a summary of the efficacy of each objective assessed in this section pre-restructuring: Objective 1: Nutrient Load Reduction - Pre-restructuring for Croatia and BiH: Negligible. No nutrient load reduction was recorded due to neither of the leachate management systems having been constructed. Objective 2: Technical Assistance - Pre-restructuring: For Croatia and BiH - Substantial. Technical assistance provided valuable capacity building support to local municipalities in both BiH and Croatia. Sea Water Quality Monitoring- Pre- restructuring: For Croatia - Substantial. The sea water quality monitoring outcomes exceeded the target. Not applicable to BiH Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale The following is a summary of the efficacy of each objective assessed in this section post-restructuring: Objective 1 Revised: Nutrient Load Reduction - Post-restructuring: For Croatia - Substantial but for BiH - Negligible - hence overall Modest. The project completed of one of two leachate management systems in Croatia that underpinned the objective's nutrient load reduction and pollution reduction long-term outcomes of the theory of change (ICR p.7) Page 11 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) Objective 2 Revised: Post-restructuring: For Croatia and BiH - Substantial. Technical assistance provided valuable capacity building support to local municipalities in both BiH and Croatia. Sea Water Quality Monitoring- Post-restructuring: For Croatia - Substantial. The sea water quality monitoring outcomes exceeded the target. Not applicable to BiH Overall Efficacy Revision 1 Rating Primary Reason Modest Low achievement 5. Efficiency Economic analysis: At appraisal, a project-level ex-ante cost-benefit analysis of two demonstration leachate management investments in Croatia and BiH was undertaken. This resulted in a Net Present Value (NPV) of expected benefits estimated at US$ 6.21 million and a 24% Internal Rate of Return (IRR) for a total cost of US$ 4.58 million and an O&M equivalent to 12% of the investment cost per year. These demonstration investments were the most cost-effective for this type of waste, costing US$3.2 per kg of nutrient compared with an average of US$53 per kg of nutrient for nine other Baltic countries. In the ex-post economic analysis in the ICR the NPV was estimated at US$25.90 million (at 5 percent discount rate) and the IRR of the demonstration investment at 33.4 percent, for a total cost of US$4.75 million. The project implementation delay had an insignificant impact on the economic results of the restructured project, namely a 6.5 percent decrease in NPV and only 0.03 percent decrease in ERR. (ICR, para 59) A benefit transfer approach using economic values for ecosystem services by transferring available information from studies already completed in two other locations (Romania and Bulgaria based on household incomes were used but without any evidence that the circumstances in Romania and Bulgaria were the same as in Croatia) and calculating a unit value of ecosystem services per household per year , which was estimated at US$343.98 in 2010 prices (para 19 ICR). Using a similar approach, the benefit of improved quality of bathing water in support of recreational and tourist activities in Croatia, was estimated at US$47.64 in 2010 prices per person per year for the project area on Korcula. In comparison with the ex-ante cost-benefit analysis, the estimated ex-post cost-benefit analysis displays 40 percent increase in the IRR and over 310 percent increase in the NPV in Croatia despite the changes in the project demonstration locations, extended project implementation period, and reduced number of the population affected. The ex-post economic analysis demonstrates significantly better results than the ex-ante analysis because of the specifics of the new location for Croatia and its higher direct impact on human health through avoided pollution of the drinking water wells in the Project area. Financial analysis: Page 12 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) The project financed investment at one municipal site, namely Sitnica on Korcula Island in Croatia, located within two municipalities, Blato and Vela Luka, which received grant resources for physical investments. The Project only financed TA in BiH as the infrastructure work was cancelled; hence, its financial capacity is not assessed. The municipal budgets (2019–2021) of Blato and Vela Luka were analyzed to assess their capacity to sustain adequate funding for incremental O&M expenditures (investments costs are provided as a grant) of the monitoring of the closed Sitnica landfill. As these costs comprise about 0.2 percent of the total annual costs of the municipalities and the operators are public entities, the financial risk associated with the financing of the O&M costs is insignificant. Each of the municipalities is capable of covering the annual O&M costs on their own. The future costs associated with the closure of cell #4 of the Sitnica landfill after reaching its maximum capacity are expected to be grant funded, as is typical for such type of investments in Croatia. Even if the grant is not secured, the Blato Municipality would have the capacity to finance it. Inefficient Design and Implementation Despite acceptable traditional measures of efficiency and use of economic resources as noted above, the project suffered from inefficient design and implementation. For example (a) poor judgment at appraisal to design improvements to the landfill site in Mostar when it was well known that the municipal government was in disarray and that it was extremely unlikely that the procurement process for establishing the demonstration landfill site would be efficiently completed, and (b) inefficient project preparation of landfill plans in Croatia because the site chosen at appraisal needed to be changed because a land rights dispute could not be resolved from Zadar to Sitnica where an improved landfill management was eventually completed. These design and implementation weaknesses reduced the project’s overall efficiency. This was coupled with the fact that the actual project cost (US$29.02 million) was almost equivalent to the appraisal cost (US$29.97 million) for approximately only half the expected project results (i.e. one relatively small landfill with a leachate management system constructed instead of improved leachate management systems at two relatively large landfills). This review has rated the overall efficiency of this project as modest Efficiency Rating Modest Page 13 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  24.00  Not Applicable 0 ICR Estimate  33.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The overall outcome rating based on a split rating is moderately unsatisfactory derived from the following analysis. Derivation of Overall Outcome Rating Rating Dimension Original Objective Revised Objective Relevance of Objectives Substantial Efficacy - Objective 1: Nutrient Load Reduction Negligible Modest - Objective 2: Technical Assistance and Sea Substantial Substantial Water Quality Monitoring Overall Efficacy Modest Modest Efficiency Modest Modest Outcome Rating Moderately Unsatisfactory Moderately Unsatisfactory Outcome Rating Value 3 3 Amount Disbursed (US$m) 1.21 3.52 Disbursements (%) 26 74 Weighted Rating Value 0.78 2.22 Weighted Outcome Rating Value 3.00 Overall Outcome Rating Moderately Unsatisfactory (3.00) The rating of "modest" for Overall Efficacy in the above table is because the rating of "substantial" for Technical Assistance and Sea Water Monitoring is marginal on account of the limited evidence provided for the capacity building in Croatia and BiH in the Page 14 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) ICR. Information is provided on the completed training programs and number of prepared funding applications (ten in total), however, there is no evidence regarding enhanced capacity of institutions associated with the task of developing improved landfill management systems. Also, the TA evidence is not disaggregated by country and it was only learned in discussions with the Bank/TTLs that three of the Croatian prepared applications had been funded. No other information is available. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome Several of the development risks were anticipated in the PAD (page13), yet not acted upon adequately in the project design. Specifically, implementation entity risk was estimated to be moderate which was the average of the two implementing agencies despite the fact that both PMTs had previous experience, the complexity of the proposed program was foreseen and hedged by reinforcing the TA component, by involvement of key stakeholders working with experienced consultants to develop investment proposals, the project failed to fully mitigate this risk. The fact that TA was primarily delivered by consultants suggested that the direct benefit to government authorities could have been limited. The Bank task team explained to IEG, however, the broad technical assistance program focused on strengthening institutional capacity (see Section 4). The PAD (page14) stated "the procurement for several contracts has already been initiated during preparation to increase both PMT capacity and speed up implementation. Implementation capacity is also considered similar in both countries given the limited demands the project presents (3-4 contract for each country)." The latter statement was not supported by the evidence because Croatia was already an EU member as of 2013 and at higher level of institutional capacity than BiH, given its early stage in the membership process. More so, the PAD (page 17) outlined the procurement risk and rated it as Substantial for Bosnia-Herzegovina, which was a key debilitating factor in implementation. This reality was reflected in Objective 1 not being achieved in BiH and suggests shortcomings in the implementation and governance risk analysis in the PAD. The BiH CPF, (page zcviii), clearly highlights the risks associated with the country’s complex, decentralized governance structure that may thwart preparation and implementation of development programs. The CPF Objective (page v) of ‘Increasing Public Sector Efficiency and Effectiveness’ speaks to the need for strengthening governance, increasing efficiency and management of public finance and enhancing public service delivery. Addressing issues of pollution control and solid waste management warrants strong public leadership coupled with inter-government coordination and technical capacities. The project’s shortcomings in focusing on these foundational elements resulted in poor mitigation of development risk. Page 15 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) 8. Assessment of Bank Performance a.Quality-at-Entry The Bank made considerable efforts at entry to build upon the work of the initial ASEP program and address CPS issues in alignment with government objectives. However, despite the preparatory investments (PAD, page iii) and the 2016 restructuring, the project fell short of fulfilling the objectives on account of gaps in the design to address structural country risks and clearly assess the capacity of the implementing agencies. For example, with respect to the improvements proposed at the Mostar landfill it must have been known at appraisal that it was extremely unlikely that the procurement processes needed for establishing the demonstration landfill site would probably not be efficiently completed by the designated institutions responsible. It must also have been known at appraisal that landfill plans at Diklo in Croatia were a risky prospect because a potential land rights dispute which in the event was not resolved and the demonstration landfill site was consequently moved from Zadar to Sitnica and eventually completed there. If these risks were not known at appraisal it suggests inadequate project preparation. By design, the project selected implementing agencies with adequate skills and experience to mitigate capacity risk and indicated the provision of TA as deemed necessary (PAD, page 41) to address shortcomings. The TOC/Results Chain of the ICR (p.7) identified TA for BiH only for the designs and tender documents for the studies for the leachate systems but did not include capacity building for procurement, contract management and inter-governmental cooperation (ICR, page 29). There is no information in the ICR, if the PAD’s offer of TA - as needed was provided during the course of the project, despite recognizing the skill limitations of the PIT in Mostar. There is also no information in the ICR if and how the use of and reliance on 'existing structures' and 'donor experienced entities' , as foreshadowed in the PAD (page 44), affected project performance. Based on the evidence in the ICR of a lack of adequate preparation of the two core investment projects and unremarkable implementation support strategies and plans this review has rated the project's quality at entry as moderately unsatisfactory Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Page 16 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) According to the Croatian government’s response to the World Bank’s draft ICR (ICR, Annex 6), the World Bank’s intensive supervision was one of the major contributing factors to achieving the targets and was the result of established close collaboration among project stakeholders. This collaboration led to the resolution of the land rights problems at Zadar, the choice of Sitnica for the demonstration project in Croatia, and its successful completion. The BiH authorities did not provide feedback on the World Bank’s draft ICR. According to the ICR (para 89), the World Bank team did its best to resolve the complicated situation in BiH, but the relationship deteriorated further when complaints arose during the rebidding process in August 2017, related to the design, supply, and installation of the treatment of leachate at the sanitary landfill in Mostar. According to the ICR the World Bank team failed to identify a restructuring or simplifying implementation arrangements that could have addressed this challenge (ICR, para 89). It is not clear, however, from the available information in the ICR why constructive and timely action was not taken to address the various deficiencies/discrepancies of the PIT Mostar landfill during the procurement process and throughout the project. Based on IEG's discussion with the Bank task team on these issues, the Mostar city authorities had little leverage and found it almost impossible to collaborate with the local utility company that served as project Implementing Team (PIT) for the construction of the leachate management system in BiH. It is understood that the PIT was intent on favoring certain companies and to ensure they would win the tenders as well as have competitors disqualified. The PIT staff insisted on their own consultants and operational procedures rather than abide by Bank protocol for the procurement of consultant services. Despite attempts by the Bank to find a resolution to the procurement problems, the staff at the PIT proved to be uncooperative. The Bank investigated the procurement discrepancies but was unable to gather sufficient evidence to make a case for corrective action. According to the Bank task team, it became clear that PIT staff preferred to lose the Bank grant rather than work cooperatively and within the scope of the Bank Agreement. In discussions with the Bank task team it was also learned the Bank suggested to change the demonstration site but the Ministry insisted to keep it in Mostar, in part due to its potential impact to reduce pollution/nutrient loads because of its location and significant population size. At one point, a mediator was contracted to liaise among the Ministry/PMT, PiT, and City and was able to secure initial permits for the landfill improvements, however, the process did not advance further. The Ministry/PMT and other stakeholders maintained expectations that the impasse with the PIT would be resolved and based on these expectations the project continued. The ICR does make it clear that the supervision of the project in BiH was made extremely difficult by the intransigence of the BiH authorities in the face of strenuous efforts by the Bank to resolve Page 17 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) complex procurement problems on numerous occasions. Despite its efforts the World Bank team failed to identify a resolution to the problems at Mostar. On balance this review rates the Bank’ supervision performance as moderately satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The ICR provides limited information on the M&E design, yet makes clear it was the responsibility of each PMT in the respective country, premised on their experience and capacities implementing similar projects (PAD p. 12). The ICR indicates the M&E design incorporated indicators aligned with outputs and outcomes in the theory of change to measure the reduction of nutrient load and number of preparations of investment proposals, seawater quality measurements and direct project beneficiaries. The selected indicators were specific and measurable. Yet, there is no mention of engagement with stakeholders to confirm their relevance or if other/additional indicators may have been valuable. Also there was no information nor discussion in the ICR of the extent the M&E design and process was embedded institutionally between the Croatia and BiH to address the trans-boundary nature of the project, despite, the discussion on this matter in the PAD (para 42, 46). The PAD (para 42, 46) outlines existing capacities, based upon similar project experiences on behalf of both recipient countries, as the basis to “carry out the M&E function under the [current] Project.” The project Steering Committees (PSC) as described in the PAD (para 44) were established after some delay; they were responsible for coordinating and monitoring joint activities and sharing knowledge and lessons learnt. The ICR (para 68) confirmed this role. Regarding baselines, there is no information in the ICR (Annex 5, p. 46) as to why the pre-structuring target was set lower than the baseline (130 tons versus 135 tons respectively). There is also no discussion of the additional level of rigor which could have been given to M&E design because the two proposed major investments in landfills were demonstration/pilot projects, as is advised by the IEG Guidelines (page 34). Page 18 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) b. M&E Implementation The ICR provides few details of the M&E implementation. The M&E products consisted of biannual reports submitted to the World Bank by the respective PMTs, yet the ICR does not provide information as to the process carried out by Environmental Protection and Energy and Efficiency Fund (EPEEF) in Croatia and Ministry of Foreign and Trade Relations (MOFTER) in BiH to monitor project progress and identify and respond to problems that emerged. In 2016, a project restructuring modified the targets for Croatia, following a new location for the leachate management system and revised indicators for the nutrient load target for Croatia as well as BiH. However, there is no explanation how the M&E process supported the project's restructuring and the key M&E findings that triggered its necessity. Also, the ICR does not provide explanations for the less ambitious indicator changes summarized in Table 5.1, namely "Changes during Project Implementation" (ICR page 46). c. M&E Utilization The ICR (para 94 ) is unclear as to the actual utilization of the M&E and the bi-annual reports. No information is provided if stakeholders were engaged or informed of the reports' findings and if the M&E findings had any impact on project implementation. Instead the ICR text offers a suggestion, “the lack of mechanisms to enforce agreements on the ground, and procurement challenges suggested the need for strengthening staff to manage the procurement processes in BiH. To this point, no information is provided in the ICR as to why staff, administrative and/or TA needs and changes were not identified nor adopted during the M&E process, despite the PAD’s (p. 41) acknowledgment of the level of governance risk, which ultimately led to the inability to fully achieve Objective 1. Moreover, the demonstration aim of the PDO#1 through construction of leachate management systems was not realized nor documented in the M&E process (and ICR) and failed to offer the learning and information sharing it intended. Yet, it was learned in discussions with the TTLs that the Korcula demonstration/pilot has been valuable for other small island coastal communities in Croatia addressing water quality issues affecting tourism as well as contamination of portable water. A briefing note is under development to capture the knowledge gained from this demonstration project . M&E Quality Rating Modest 10. Other Issues Page 19 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) a. Safeguards The project was classified as a Category B—partial assessment project triggering OP/BP 4.01 on Environmental Assessment. An Environmental Management Framework (EMF) was prepared for each country with public consultations. An unforeseen fire risk emerged within the Sitnica landfill that was addressed through the mitigation measures included in the already (??) developed Environment Management Program (EMP), although the ICR does not provide information on exactly how it was resolved. The safeguard on Physical Cultural resources (OP/BP 4.11) was triggered due to the cultural richness of the region, and the Environmental Management and Social Frameworks included provisions for 'chance finds'. No chance finds occurred during construction works. The safeguard for Involuntary Resettlement (OP/BP 4.12) was not triggered. The International Waters safeguard (OP/BP 7.50) was also triggered and the Project complied with the notification requirements working through the United Nations Environment Programme’s Coordinating Unit for the Mediterranean Action Plan. b. Fiduciary Compliance The Project met its fiduciary obligations,but it did experience some delays with audit compliance. As is clear from the assessments above, procurement was problematic in the BiH part of the project and ultimately led to the inability to construct the leachate management system at the Mostar landfill and achieve Objective 1. There was no mention in the ICR of any procurement problems in Croatia. c. Unintended impacts (Positive or Negative) No issues mentioned in the ICR and no issues identified by this review d. Other n/a Page 20 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Unsatisfactory . Unsatisfactory Moderately Bank Performance Moderately Unsatisfactory Unsatisfactory The project contained two demonstration investment projects that did not contain the level of M&E rigor required by the IEG guidelines. Quality of M&E Substantial Modest Also, the M&E process was largely delegated to the country PMTs; few details are provided in the PAD and ICR of the process. Quality of ICR --- Substantial 12. Lessons Based on the project’s design shortcomings, the ICR (p.30-2) provides a number of lessons. One lesson below combines elements from a number of the lessons in the ICR stands out and is summarized below. IEG has added a suggested lesson. - Agreements among stakeholders on critical necessary conditions for project success are needed in advance of project implementation. Without advance agreements between client countries and the Bank on the availability of crucial resources such as institutional capacity, projects will fail. In this project, the lesson was that expectations of stakeholder capacities and future levels of engagement were based on agreed responsibilities identified at appraisal (see PAD, page 41) but agreements were not sustained during implementation leading to a lack of ownership and accountability on the part of the authorities (see ICR, page 59). As a result important objectives such as the contracting of construction work for addressing the leachate problems at the Mostar landfill were not achieved. In such situations, a strong M&E role by the Bank is all the more important. IEG considers that the following related lesson is relevant. - Constructive collaboration based on mutual benefits and shared costs can be a basis for achieving communal environmental goals. The improved landfill site at Sitnica was successful in reducing nutrient leaching levels and has the potential for enhancing the re-cycling of many waste materials on a financially sustainable basis. The lesson is that the choice of the site was strategically appropriate and had the constructive collaboration between two local Page 21 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) municipal governments willing to share resources at one landfill site. This lesson of optimizing economies of scale among small municipalities can be the basis for a broader roll out of landfill and recycling management systems and technologies elsewhere in the Adriatic Sea area and other regions. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The project’s lessons and achievements (and lack of) offer valuable and broad/regional learning and practices. As per Annex 5 of the ICR, 2, "the main identified pollution source at the Croatian and Bosnian coast is solid waste. There are almost no sanitary landfills at the coast and there are numerous dumping sites. Due to the karstic nature of the terrain, leachates from waste dumping sites are quickly released into the sea increasing health risks of local population and endangering tourism activities". The lack of project waste management infrastructure in the Adriatic region, and particularly among small coastal and island communities, jeopordizes the groundwater, potable/drinking water and exacerbates sea water contamination, negatively impacting local populations, millions of annual tourists and Adriatic tourism destinations as well as the countries' economic development potential. The Sitnica landfill on Korcula Island, despite its small size, is a priority landfill-related pollution hotspot site. It is also representative of the pollution challenges faced by many communities along the Croatian coast and within the Adriatic Sea region. The Bank's demonstration investment project (construction of a leachate management system) helps mitigate one source of pollution on land and sea and alleviates the related health and economic problems. It also provides the opportunity to capture the lessons and practices learned for replication and scale in similar Adriatic Sea communities. According to the Bank task team, the demonstration project is already being considered and/or replicated in other localities confronted with similar problems. Better understanding of the reasons for the success of the Croatia demonstration project, its potential for scaling and the collective impact of pollution control measures by small traditionally low-capacity communities could bring valuable knowledge to the region and institutional partners. Moreover, learning from the failure of the BiH project and what impact one country and its specific conditions has on transboundary pollution mitigation and how best to address such issues in the context of the Adriatic Sea Environment Program (ASEP) could bring important insights to the EU, the Bank and other development partners. The "Further Guidance from the Scientific and Technical Advisory Panel (STAP)" in UNEP recorded in the PAD (pages 64-65) align in part with this recommendation. Page 22 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Adriatic Sea Env. Pollution Control (I) (P143921) 14. Comments on Quality of ICR The ICR is candid, content-rich and provides useful tables and annexes to illustrate the project's achievements. The ICR also captures some important lessons from this project. However, it is often weak in the quality of the analysis, such as the use of the 'benefit-transfer' approach for the economic analysis without any evidence that the circumstances in Romania and Bulgaria were the same as in Croatia. This ICR was also insufficiently detailed for the reader to understand the context and rationale of the Banks's inability to gain agreement with the BiH authorities on procurement issues associated with the Mostar landfill. IEG's understanding of these issues was, however, enhanced by a discussion with the Bank task team. The ICR also provides little information for the reader to understand that the investment projects were to serve as pilot projects for learning and replication as noted in an annex to the PAD (pages 64-65). Gaps in the ICR also exist in the elaboration of content of bi-lateral cooperation between Croatia and BiH on landfill management issues in the context of the broader consequences for the environment in the Adriatic Sea on account of the project's shortcomings in nutrient load reductions. Nevertheless, again, IEG's understanding of these issues was enhanced by the discussion with the Bank task team. a. Quality of ICR Rating Substantial Page 23 of 23