IEG Report Number: ICRR14683 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 04/28/2015 Country: Andean Countries Project ID: P098248 Appraisal Actual Project Name: Adaptation To The Project Costs (US$M): 32.72 33.58 Impact Of Rapid Glacier Retreat In The Tropical Andes L/C Number: Loan/Credit (US$M): 7.49 7.89 Sector Board: Environment Cofinancing (US$M): 3.63 3.32 Cofinanciers: FAO, Japan Ministry of Board Approval Date : 05/27/2008 Finance (PHRD Closing Date: 09/30/2012 03/31/2014 Grant), Agence Francais Developpement Sector(s): General water sanitation and flood protection sector (36%); General agriculture fishing and forestry sector (27%); General public administration sector (21%); Water supply (12%); Forestry (4%) Theme(s): Climate change (67% - P); Biodiversity (33% - S) Prepared by: Reviewed by: ICR Review Group: Coordinator: J. W. Van Holst Ridley Nelson Christopher David IEGPS1 Pellekaan Nelson 2. Project Objectives and Components: a. Objectives: This project's objective stated in the GEF Grant (the Global Environment Objective) was "to contribute to strengthening the resilience of local ecosystems and economies to the impacts of glacier retreat in the Tropical Andes, through the implementation of specific pilot adaptation activities that illustrate the costs and benefits of adaptation" (GEF Grant Agreement, Schedule 1). This project's objective stated in the Project Appraisal Document (PAD) was exactly the same as in the GEF Grant Agreement (PAD, paragraph 34). This Review will use the objective as stated in the GEF Grant Agreement against to assess the achievements of this project. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The following is a summary of this project's components based on the PAD (para 36 to 60). The project covered three Andean countries, namely Ecuador, Bolivia and Peru. Component 1: Detailed design of key selected adaptation measures (appraisal estimate US$1.1 million; actual cost US$1.41 million): The objective of this component was to complete the design of at least six strategic adaptation measures to be implemented under Component 2. The objective would be achieved through the following activities:  Sub-component 1.1–Design of glacierized basin impacts map . This sub-component would apply global climate circulation models developed and run by the Earth Simulator in Japan and use the data generated by project preparation funds to quantify impacts of climate change on glacier retreat, runoff availability, and water regulation at basin levels. Participating countries were to develop impact maps for the selected glacierized basins.  Sub-component 1.2–Detailed design of specific adaptation measures . This sub-component would overlay the impacts map designed under Sub-component 1.1 on the existing and/or planned regional government programs and investments to adapt to glacier retreat impacts. This activity would support the detailed design (as well as M&E) of specific adaptation measures.  Sub-component 1.3–Public outreach and dissemination of information . This was aimed at (a) improving public knowledge of the actual and expected local impacts of climate change on tropical glaciers and how their recession will directly affect associated catchments’ ecosystems and socioeconomic activities in the Andean region; (b) to dissemination of existing information on climate change, high mountain ecosystems, and glacier retreat recession, and their impacts on: (i) water supply systems for human consumption and agricultural and livestock use, and (ii) the energy sector; and (c) raising international awareness on the economic and social costs of tropical glacier retreat. Component 2: Implementation of pilot adaptation measures (appraisal estimate US$25.55 million; actual cost US$27.8 million). This component's objective was to invest in pilot adaptation measures addressing the most pressing environmental priorities in each country. The component included the following activities:  Sub-component 2.1–Implementation of pilot adaptation measures in selected communities and sectors highly vulnerable to the effects of glacier retreat . The following pilot adaptation interventions for each country were selected on the basis of magnitude of the impacts and cost and were designed during the project's first year. Ecuador - Pilot 1: Increasing water supply resilience for Quito - compensation of regulation loss in the Antisana watershed. The objective of this pilot was to implement a climate change adaptation strategy for water supply to the city of Quito and surrounding parroquias. It included (a) development of a new potable water supply project for the Quito metropolitan district, (b) implementation of climate change adaptation measures such as the adoption of water efficient devices for domestic use; and (c) implementation of a monitoring system to assess water availability and the evolution of the impacts of global warming. - Pilot 2: Integrated Watershed Management Plan for the Antisana micro catchments to better cope with the impacts associated with glacier retreat . The objectives were to: (a) compensate for the decrease in the effectiveness of water storage capacity, due to increases in precipitation variability in selected catchments; (b) minimize the potential negative effects of climate change on highly vulnerable and typically poor local communities in the area, and (c) develop a participatory páramos (mountain wetlands) management plan to protect and maintain healthy associated ecosystems and hydrological balance. The main overall activity was the development and adoption of a community based Catchment Management Plan. Bolivia - Pilot 1: Integrated Watershed Management in the Tuni and Condoriri basins , incorporating the impact of rapid glacier retreat. This pilot was to provide compensation for the impacts of glacier retreat on water availability through: (a) development of a strategic water management plan for the Tuni and Condoriri basins; (b) water storage schemes and other high-priority interventions to compensate for the loss of water regulation capabilities in glaciers; and (c) a monitoring and evaluation to identify lessons learned and to guide policy. - Pilot 2: Integrated Pilot Catchment Management Plan for watersheds affected by rapid glacier retreat in the Bolivian plateau and high valleys This pilot was to support adaptation of agriculture and livestock activities to the loss of water regulation and supply caused by glacier runoff in the Bolivian plateau and high valleys. Activities included: (a) building and operating small civil structures where water scarcity induced by glacier retreat is projected to stress local economic activities; (b) implementing reforestation and re-vegetation to decrease erosion rates, and promote infiltration; (c) applying water conservation practices for agricultural and livestock activities; (d) implementing a Water Management Plan with the help of local communities to make efficient use of reduced water resources. - Pilot 3: Mainstreaming Adaptive River Defense for the Huayhuasi and El Palomar Settlements . The objective for this pilot was to develop a social model for disaster prevention and reduction of vulnerabilities to climate change impacts. It aimed to decrease risk and vulnerability to extreme events (floods) of the Huayhuasi and El Palomar rural communities on the La Paz River through improved adaptive construction with communal participation for the regulation and control of the La Paz River. Peru - Pilot 1: Implementation of a Water Management Plan . This pilot is aimed at: (a) improving water use practices in the agricultural and livestock sectors; (b) improving water storage infrastructure at selected basin headwaters to address negative effects caused by temporary increase in runoff; and (c) the development of a reforestation program at the basin headwaters in Shullcas (Mantaro Valley, Junín) and Santa Teresa (Vilcanota–Urubamba Valley, Cusco). This pilot will also facilitate the creation of a protected natural area to protect and conserve water resources, biodiversity, and its scenic beauty. - Pilot 2: Implementation of an Agricultural Production Plan . This pilot was aimed at providing compensation for the reduction of water availability to the agricultural sector as a result of rapid glacier retreat. It would implement a plan to diversify agricultural production aimed at improving competitiveness and food security, reducing agricultural production losses, and implementing agricultural good practices adapted to the consequences of glacier retreat in the area. - Pilot 3: Implementation of an Integrated Water Management Plan . The objective was to improve the availability of water for human consumption in the Huancayo basin by rationalizing water use and through research on alternative sources of water supply. The following activities were contemplated: (a) implementing improvements in drinking water supply infrastructure; (b) implementing a strategy for the use of drinking water and agricultural water; (c) developing a program with local communities on the efficient use of water for human consumption. Component 3: Monitoring of glacier retreat in the region (appraisal estimate US$2.30 million; actual cost US$2.03 million).  Sub-component 3.1: Design and establishment of field stations to monitor tropical glaciers of economic relevance. This sub-component was aimed at financing the design, installation, and operation of eight glacier monitoring stations, located at or near tropical glaciers of economic relevance.  Sub-component 3.2: Provision of glacier monitoring equipment . This sub-component was to finance high- precision remote sensing equipment to monitor tropical glaciers and associated ecosystems through the use of the Japanese Space Agency satellite (Advanced Land Observing Satellite - ALOS or DAICHI). This component would support (a) data access from ALOS; (b) data compilation and storage; and (c) data interpretation and use.  Sub-component 3.3: Analysis and monitoring of the behavior of tropical glaciers in Peru and their associated mountain wetlands in light of climate change . This sub-component was to be funded by the Global Environment Fund (GEF) and the Special Climate Change Fund (SCCF) which would finance: (a) compilation and analysis of baseline data and analysis of the expected behavior of tropical glaciers in light of climate change and the analysis of watershed response (hydrology) for selected basins; (b) compilation and analysis of baseline data and analysis of the role played by páramos (mountain wetlands) in fixing atmospheric carbon, and providing a basis for planning adaptation measures; (c) monitoring of water cycles in specific glaciated basins of major economic relevance; (d) design and establishment of field stations to monitor the hydrology and carbon cycle of mountain wetlands. Component 4: Project management (appraisal estimate US$3.77 million; actual cost US$1.60 million). This component was to support the overall technical coordination of project activities (including the implementation of a technical monitoring system) as well as the project’s administrative and financial management. It would include goods, consultant services, travel, and operating costs for project management. It would also finance the project coordinator, the procurement specialist, other required personnel for project management, and the project’s external audits. Restructuring of Project (August 17, 2011) Component 5: Regional Activities (appraisal estimate US$0; actual cost US$0.74 million). An additional component 5 was defined without additional funding as part of a project restructuring. The ICR stated that this component was added to focus on the development of regional knowledge and learning on adaptation to climate change and glacier retreat . The General Secretariat Andean Community of Nations (Secretaria General de la Comunidad Andina - SGCAN), which was the implementation agency, requested this component in order to: (a) provide a more robust regional depth to the project, with targeted regional activities; and (b) ensure that the scientific knowledge, methodological approaches and operational expertise generated in each participating country would be better systematized, utilized and disseminated. The component would also promote South-South cooperation, bring worldwide cutting edge research institutions to the region, and support SGCAN to implement its mandate to promote integration among member countries on topics of regional relevance (see ICR, Annex 11). Reorganization of pilots and components : Adjustments were made to some of the pilot projects such as strengthened M&E and strategic planning focus (Ecuador), and strategic analysis of unaccounted for water in the local utility (Bolivia). Some components were renamed and re-arranged as part of the restructuring in order to make them more consistent with the project's logic and activities - as had been suggested during the Mid Term Review. The changes in the components were: (a) Sub-component 1.2 defined in the PAD to finance "Detailed design of adaptation measures" was incorporated into Component 2 which was renamed to "Design and implementation of adaptation measures". (b) Sub-component 1.3 defined in the PAD to finance "Public outreach and dissemination of information" was incorporated into the new Component 5 and renamed "Development of Regional Activities". (c) Sub-component 3.3b and 3.3c were defined, respectively, in the PAD to finance "Development of scientific baseline for high mountain ecosystem" and Development of specific guidelines for adaptation measures in high mountain ecosystems" which were incorporated into the new Component 5. While the restructuring resulted in no changes in the project's objectives, there were some relatively small changes in the project's indicators but they did not warrant a "split" evaluation of the project's outcome. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs: The total cost estimated at appraisal was US$32.72 million, while the actual cost when the project closed was US$33.58 million. Financing: The project was financed by the following sources - in order of magnitude: (a) Global Environment Facility/Special Climate Change Fund - appraisal estimate US$7.49 million, actual US$7.94 million; (b) other donors - appraisal estimate US$3.63 million, actual US$3.32 million; and CARE (an NGO) provided parallel financing estimated to be US$1.20 million at appraisal, but its actual contribution was US$1.84 million. Contribution by Participating Countries : Appraisal estimate was US$14.14 million, but the actual contribution was US$6.57 million. Dates: The project's closing date was extended on two occasions from the original date of September 30, 2012 to the final closing date of March 31, 2014. Additional financing of US$0.45 million from the GEF/SCCF to meet additional costs of monitoring glacier retreat was approved on July 10, 2010. The project was restructured substantially (including the addition of a fifth component as described above) on October 14, 2011. The restructuring implemented recommendations of the MTR in October 2010. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High "Observations on the extent of glaciers in Venezuela, Colombia, Ecuador, Peru, and Bolivia provide a detailed and unequivocal account of rapid shrinkage of tropical Andean glaciers since the Little Ice Age" (PAD, page 127). According to the PAD if tropical glaciers continue to retreat and eventually disappear from certain catchments it will cause changes which will significantly affect the availability of water for human consumption, for hydropower production, mining, and irrigation (page 29). The PAD also noted that the governments of Ecuador, Bolivia and Peru had already embarked on activities to develop their capacity to adapt to the impact of climate change following the retreat of glaciers in the Andes. The ICR noted that at appraisal this project's objective was consistent with the Bank's strategies in the Ecuador Country Assistance Strategy (CAS) for 2002-2007, in the Bolivia Interim Strategy Note (ISN) which was not published (see PAD, para 16), and in the Peru Country Partnership Strategy (CPS) for FY07-11. At completion the project's objectives remained relevant to the Bank assistance strategies for the three participating countries. In the Ecuador ISN for FY14-15 the first pillar emphasizes "sustainable and inclusive growth" which would not be possible if Ecuador's water supply is not adapted to the effects of climate change. Also, in 2009 Ecuador created the Climate Change Secretariat and in 2010 the Inter-Institutional Committee on Climate Change "to coordinate and enhance the implementation of climate change policy" and in 2012 launched the National Climate Change Strategy which is the basis on which the national climate change plan will be developed (ICR, para 51). The Bank's CPS for Bolivia includes climate change and disaster risk management in its results matrix and mentions glacier retreat as an important development threat (ICR, para 50). The ICR also mentions that "the project objective of adapting to impacts of glacier retreat is more relevant than ever in Peru, a country where the most populated city (Lima, over 8 million people), important infrastructure and tourist attractions are located on an arid coast which receives most if its potable water from mountain runoff" (para 51). There could be no doubt about the high relevance a project whose objective is ""to contribute to strengthening the resilience of local ecosystems and economies to the impacts of glacier retreat in the Tropical Andes". b. Relevance of Design: Modest This was a multi-country project designed to be implemented by an independent agency in collaboration with three participating governments, the World Bank and a number of specialized international agencies. According to the ICR, this institutional complexity translated into delays and inefficiencies, especially during the first two years of the project (para 30). Such complexity, while typical of multi-country projects, arose in this case because the responsibilities and obligations of the General Secretariat Andean Community of Nations (SGCAN), which implemented the project, and the three national governments were difficult to coordinate. In addition, as reflected in Section 2c above, the project involved numerous activities in three countries with a number of pilots which, as the ICR acknowledged, were not well appraised (para 87). The ICR concluded that the "design failed to consider mechanisms allowing for direct use of funds by key partners, which would have expedited the use of funds, simplified the procurement plan, and reduced SGCAN’s exposure to activities for which it lacked direct expertise" (para 33) . It was therefore not surprising that the project was restructured in October, 2011 after the Mid Term Review (MTR) to reorganize the components and give the project a regional (in contrast to a multi-country) perspective. The PAD's results matrix also had design shortcomings because it did not clarify the roles of the SGCAN and the participating countries in achieving the project's objectives and therefore failed to provide a results chain from project inputs through intermediate outcomes to final outcomes. Rather, the results matrix was a listing of the project objectives and components along with their designated indicators to measure the extent to which the different levels of objectives and components as well as indicators were achieved. The results matrix was, however, improved after the MTR by establishing a more logical narrative on the chain of activities leading to the project's intermediate and final outcomes (Annex 3 in the ICR). The project's improved design after restructuring became more relevant to its objectives. The revised results matrix improved the clarity of the project's results chain. The improvements reinforced a sound feature of the project, namely that activities flowed from "scientific research to monitoring, to implementation, to policy and provided incentives at national levels for (relevant) actors to get involved. The design managed to bring different stakeholders to the same table, placing glacier retreat and adaptation to climate change in mountain areas at the center of the discussion" (ICR, para 31). 4. Achievement of Objectives (Efficacy): The overall project objectives were "to contribute to strengthening the resilience of local ecosystems and economies to the impacts of glacier retreat in the Tropical Andes". For the purpose of assessing the extent to which the overall objectives were achieved this Review partitioned the overall objective into two sub-objectives, namely (a) contribute to strengthening the resilience of local ecosystems to the impacts of glacier retreat; and (b) contribute to strengthening the resilience of local economies to the impacts of glacier retreat. Outputs and outcomes will be identified for each of these sub-objectives . (a) Contribute to Strengthening the Resilience of Local Ecosystems to the Impacts of Glacier Retreat - Substantial Outputs  The project strengthened national meteorological services capacity to monitor glacier dynamics (PDO Indicator #3 and Intermediate Outcome Indicator #8) - 8 high-mountain meteorological stations were installed and are operational producing useful information. Moreover, in Ecuador, 15 additional hydro meteorological stations were also acquired, and in Colombia, an additional high mountain monitoring network was set up but not financed by this project. - With this equipment and other support the project contributed to strengthening the capacity of four national meteorological services to monitor glacier dynamics, namely INAMHI (National Meteorological and Hydrological Institute) in Ecuador, SENAHMI (National Meteorological and Hydrological Service in Peru), IHH and SENAMHI (Hydraulics and Hydrological Institute and National Meteorological and Hydrological Service) in Bolivia, as well as IDEAM (Institute of Hydrology, Meteorology and Environmental Studies) in Colombia (ICR, page 31).  According to the ICR (para 59) the following outputs were also attributable to the project: - Increased regional integration at the scientific, decision makers', and beneficiaries’ levels created opportunities for the exchange of knowledge and experience through 6 regional workshops. - The publication of two important Andean regional documents, namely the Andean Strategy for Integrated Water Resources Management, and the Andean Environmental Agenda, were both prepared under SGCAN leadership. - Support for AndesPlus, which promotes intensive cooperation among Andean countries to prepare a regional database and regional guidelines for regional adaptation projects, together with highly regarded European institutions working on glacier retreat.  The three participating countries (a) generated climate change scenarios using the Earth Simulator from the Meteorological Research Institute in Japan; (b) obtained and used images from the Advanced Land Observing Satellite (ALOS) to better describe glacier retreat; (c) prepared integrated watershed management plans, vulnerability analyses of critical sectors; and (d) developed information and monitoring systems (ICR, para 55 and Intermediate Outcome Indicator Nos. 1, 2, 4, 5, 6 and 7). Outcomes  The scientific knowledge generated by the project on glacier dynamics contributed to strengthening the resilience of local ecosystems to the impacts of glacier retreat. For example: Ecuador - The revised results matrix stated that "Five adaptation plans/strategies were prepared with information of the project setting the base for the implementation of adaptation activities. Knowledge generated by the project also informed the National Strategy on Climate Change and the National Plan on Climate Change among others (ICR, page 29). The plans/strategies included adaptation plans for the Antisana glacier in the Antisana Ecological Reserve. Bolivia - The glacier inventory for the Coordillera Real and development of a strategic water management plan for the Tuni and Condoriri basins informed the design and development of the Master Plan for increased resilience of water and sewerage for La Paz and El Alto against climate change and glacier retreat (ICR, page 31 and Restructuring Paper - Report No. 63456-LAC, page 11). Peru - The project contributed to the development/update of management tools in both selected basins. The climate change scenarios for the years 2030 and 2100 for the Mantaro and Urubamba basins, informed the regional climate change strategies for Junin and Cusco respectively, as well as the integrated water management plan for irrigation from the Shullcas river (ICR, page 31).  Strengthening the capacity of national meteorological institutions to identify protocols for information sharing, contributed to knowledge transfer among scientists on glacier dynamics within the Andes region (ICR, page 31).  Overall, the project supported and contributed to the formulation and/or provided detailed information on ecosystem (e.g. paramos, High Andean wetlands and native grasslands) management tools for different levels of climate change adaptation. These outputs were translated into strategies and planning instruments attributable to the project that were achievements beyond initial expectations, and will have long term significance (ICR, para 55). (b) Contribute to Strengthening the Resilience of Local Economies to the Impacts of Glacier Retreat - Substantial Outputs  Pilots implemented by this project generated relevant information for investments aimed at adaptation to the impacts of climate change  Information from these pilots was collected, elaborated and used as inputs in the planning and implementation of public/private investment programs and projects to improve their adaptation to climate change. Outcomes  The results of the project's pilots provided information for plans and policies of different levels of government and for private investors. Although this response to the project's pilots was beyond original expectations, the impact of the resulting plans, policies and investments cannot be measured yet. However, examples of the anticipated achievements attributable to pilot investments in each participating country are provided below. Ecuador - Development plans for the local governments of Papallacta, Quijos, and Napo captured lessons from pilot activities, leading to adaptation and hence resilience of these plans to climate change. The ICR stated that "the perception of the benefits of some adaptation activities implemented in Ecuador has been so positive and immediate (e.g. adaptive cattle management) that replication is already taking place, and it is expected that it will soon influence local and regional policy and investment decision and tools". It was also noted that the activities in Papallacta served as a "basis upon which the Environment Ministry is building a larger adaptation program in the area' (ICR, page 30). - The project also contributed to 'the formulation of strategies and investment activities of the water utility company through the development of the Adaptive Management Plan for the Pita-Puengasi water supply in Quito (ICR, page 30). Bolivia - Based on the positive results and lessons of the pilot project to identify unaccounted for water in the pilot district in El Alto, similar activities are being replicated by the La Paz and El Alto Water Supply and Sanitation Utility (EPSAS) and in other districts (ICR, page 31). - Also, final designs for two investments (identified as immediate actions to ensure water supply in the medium term for La Paz and El Alto) were developed by the project. These designs addressed the need for EPSAS to balance supply and demand of water (ICR, page 31) which was important for economic development. Peru - Adaptation activities and scientific modeling of glacier retreat stimulated informed investment decisions in the municipality of Santa Teresa which developed an investment on food safety based on the results of the studies of agro biodiversity and agro climatology under this project (ICR, page 31) - Based on the experience in Santa Teresa pilot, the district governments of Echarate and Maranura in the province of La Convencion in the Cusco region are developing adaptation projects, and hence their resilience to glacier retreat, with their own resources (ICR, page 31). 5. Efficiency: Neither the PAD nor the ICR provided an estimate of the economic rate of return on all investments in this project. The PAD provided an estimate of benefits based on the Ríos Orientales project as an example of investments aimed at supplementing the water supply for the capital of Ecuador (Quito). It showed that an accelerated demand for water could force the city of Quito to increase its investment in water supplies at the expense of other societal needs. Net benefits were not estimated but the analysis concluded that the net present value of future investments (with climate change) exceeded the net present value of current investments (without climate) change by 31 percent (PAD, pages 91 and 92). The ICR argued that the Andean environment is highly susceptible to climate change and glacier retreat and that the value of adaptation to the impact of climate change is considerable because of the measurable negative impact of climate change to Andean ecosystems, long run water supplies, sustainable economic development and human well being. These impacts were described as "tremendous losses that climate change could bring to the region" (para 61) but without any quantification. The ICR stated that it assessed the project's efficiency by assessing the cost effectiveness of investments in water supplies for human consumption and irrigation as adaptations to climate change in Ecuador, Bolivia and Peru. Ecuador - In Ecuador the Quito Water Supply and Sanitation Utility (EPMAPS) and the Fondo par la Proteccion del Agua (FONAG) implemented pilot activities to reduce degradation and promote recovery of water from the existing paramo ecosystems to safeguard Quito's water supply. An alternative would have been to look for untapped water catchments in more remote paramo ecosystems involving costly infrastructure to extract additional water for Quito which would need to be compared with a relatively small investment focused on conservation of existing water paramo resources. The ICR noted that such paramo restoration would also generate other benefits including carbon sequestration and enhanced biodiversity protection. The ICR provided no quantitative analytical comparison of the two alternatives (page 43). Bolivia - One of this project's core pilot investments was to improve water distribution efficiency, and reduce unaccounted water use for the La Paz and El Alto water utility (EPSAS). The ICR reported that for an investment of US$369,000 large reductions in water losses were achieved for District 4 in El Alto (from a 39.6% loss down to 26.5%), with a total reduction in loss of approximately 619 million cubic meters per year (equivalent to the water required annually for 18,697 new connections). According to the ICR the EPSAS has replicated this pilot in three other districts, yielding 1.3 million cubic meters per year water savings, and plans to expand it to all districts (para 63). However, the ICR provided no estimate of the increased value of water nor any improvement in the water fees collected as a result of the investments. Peru - A pilot project in Shullcas included the modernization of three irrigation schemes with a sprinkler system and small water reservoirs. For an investment of about US$500,000 about 190 ha of additional cultivation area became available, benefiting 347 families. The project also invested in the lining of an open canal costing about US$235,000 which (by reducing water losses) allowed an additional 144 ha to be used for intensive agriculture. This benefitted 179 families allowing two harvests per year. These investments were complemented by community development and capacity building on climate change for farmers, financed by CARE (ICR, para 64). The ICR stated that an analysis of the Shullcas irrigation rehabilitation based on actual costs and expected values of production resulted in internal rates of return of about 46 percent, but without showing any of the data on which the estimate was based (page 44). The ICR concluded that these examples indicated the project’s efficiency, because "significant outcomes at local (activities by municipalities), national (activities led by the Ministries of Environment), regional (Andean community strengthening and sharing of experiences) and even international (through a South-South exchanges) levels were completed with relatively low funding" (para 67). Although these examples undoubtedly represented "contributions to strengthening the resilience of local ecosystems and economies to the impacts of glacier retreat in the Tropical Andes" - namely the project's objective - they provided inadequate evidence of the project's efficiency in achieving these contributions. Improved data on glacier dynamics and the protocols for adaptation strategies to climate change derived from the analysis of the data were also expected to provide substantial value for money in the Andes Region assuming that there would be a sharing of information and protocols. However benefits to countries in the Andes Region were not as strong as had been anticipated because, despite SGCAN's request for a more regional approach which resulted in the project's restructuring, its performance on technical leadership and dissemination of information and adaptation protocols was not effective (see Section 9b in this Review). In summary, there is inadequate evidence in the ICR to assign more than a modest rating to the project's efficiency. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of this project's objective was rated high because of the need to adapt to the anticipated serious impacts of almost certain climate change. However the relevance of design was rated modest because of the complexity of the project, the failure of the project to achieve close coordination among partner countries, and the poor design of some of the pilots. Efficacy of the project's contribution in individual countries to strengthening the resilience of local ecosystems and economies to the impacts of glacier retreat was substantial. The efficiency with which the project's objective was achieved was modest because of inadequate hard evidence on benefits. Overall the project had moderate shortcomings and its outcome is therefore rated as moderately satisfactory . a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project's sustainability will depend on the capacity of the national institutions in the three participating countries to continue investment in the measurement of glacier dynamics and in the design of adaptation measures to reduce the impact of glacier retreat. . At appraisal the risk that future national administrations may not continue to support the activities under the project was assessed as "high" but by the time the project closed the Ministries of Environment in the three participating countries had provided strong support for the project and they are now endorsing a follow up project with the expectation of GEF financing.. The ICR asserted that strengthening of knowledge, generation of institutional capacity, policy influence, actor involvement, and deep social engagement will contribute to sustaining the project's development outcome (para 80). It also states that sustainability and scaling up of activities is guaranteed through the leverage of additional funds and efforts to strengthen policies and foster resilient investments. For example in Bolivia, scientific activities will continue through initiatives such as JICA’s “Proyecto Grande“, while EPSAS will receive support from the World Bank financed Pilot Program for Climate Resilience. In Peru, CARE together with the Productive Rural Agrarian Development Program (AGRORURAL), and with the support of the World Bank financed Sierra Irrigation Project, will continue activities with local communities in Shullcas to further strengthen irrigation project investments. The implementation of two new projects with the support of the Swiss Government and the leadership of the Ministry of Environment will continue to strengthen the capacity for measuring glacier retreat in Peru. Finally, monitoring and scientific research will be supported through the World Bank-financed Program for the Modernization of SENAMHI in the Climate Change Adaptation Project (ICR, para 81). As noted already a new GEF-financed Andean regional adaptation project (which will include Colombia) is under preparation (paras 49 and 84). Given the evidence in the ICR about ongoing and future investments stimulated by this project together with the evidence about the follow up project, the risk to sustained development outcomes for this project is rated as moderate a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Lessons from other Bank projects were identified during preparation (ICR, para 31), and reflected to some extent in the project's design (e.g. adaptation measures generating benefits for all stakeholders, comprehensive participatory approach, and governance structure) . At appraisal adaptation to glacier retreat was an important issue for the Andean environment. The Bank was able to mobilize GEF’s Special Climate Change Fund to respond to a priority for the three participating countries. The ICR states that this project "was arguably one of the first projects worldwide to pursue real investments and activities related to glacier retreat, and to do so with a regional approach" (para 86). However, the ICR also mentioned that "At the time of the project's design, plenty of the information needed to undertake these investments was still unknown, and the final achievement of objectives indicates that project design contained the minimum basis upon which to build a successful intervention. A considerable level of risk was thus taken, acknowledging that potential rewards were worth the effort" (para 86). Project design was therefore challenging and this led to a very complex set of activities. At the same time the ICR acknowledged that "the initial project design did not specifically include regional activities" (para 87). In addition appraisals of the six proposed pilot interventions were not prepared. The need for a substantial restructuring of the project in 2011 underlined some of the weaknesses of the project's design at entry. The risks of choosing SGCAN as the implementing agency were assessed in the PAD as "high". In the event SGCAN was constrained by its own charter and did not evolve as a dynamic leader on regional adaptation to climate change. Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: The ICR stated that there were distinct differences in the Bank's performance before and after the MTR. According to the ICR, before the MTR "supervision was carried out in a general, detached way. Coordination and procedural difficulties were jeopardizing the project, and hard decisions to better define or discard and identify alternatives for some of the anticipated pilot activities were delayed. Little progress was achieved, and this was also partly justified by certain lack of ownership and guidance from the participating countries" (ICR, para 95). A significant change occurred following the appointment of a new Bank team to conduct the MTR. The MTR resulted in the formal restructuring of the project. According to the ICR (para 90) after the project's restructuring the Bank's task team tightened its supervision efforts, organized many detailed field visits and frequent video and audio conferences, placing the participating countries at the center of the discussions. In addition, country based Bank staff was added to the supervision team to strengthen oversight and provide even further guidance to the SGCAN and its project Implementation Unit (PIU), and to the Ministries of Environment in the three participating countries. The procurement, financial management, disbursement, social and environmental safeguards, and legal Bank team members all engaged directly with the client. This high level of attention was maintained by the task team throughout the remainder of the project's implementation period and ultimately was one of the factors contributing to project improved implementation Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Three governments participated in this project. There were periods, particularly during the first two years, when progress at the participating government level was slower than expected. At the start of the project delays were due to turnover of government staff, lack of ownership and inadequate leadership. Nevertheless after the restructuring the performance of all governments in terms of their ownership of the project, their leadership and response time on implementation issues improved substantially. According to the ICR the result was a remarkable performance by all governments (ICR, para 95. As a result the last three years of the project's implementation witnessed strong engagement and leadership from the Ministries of Environment which devoted specific staff to coordinate and manage activities, and significantly contributed to achieving project objectives. The ICR stated that participating countries ultimately took ownership of the project’s outputs and results, and are eager to continue learning from them and from each other (para 95). However, their financial contribution declined from an estimated US$14.4 million to US$6.57 million. Despite the weak start the overall performance of Governments is rated as moderately satisfactory Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The project was implemented by the General Secretariat Andean Community of Nations (SGCAN) which is an entity within the Andean Community of Nations. SGCAN, with the support of its PIU for this project, was in charge of the overall inter-institutional coordination of project activities (ICR, para 66). From a fiduciary perspective SGCAN, with PIU support and its own staff and resources, managed to carry out its core functions. The ICR noted that the first years of project implementation saw a somewhat cumbersome and slow administrative process, which resulted in significant delays. This situation was reviewed during the MTR, and SGCAN agreed to change some of its procedures, which allowed it to speed up operations and prevent delays (ICR, para 97). However, despite its improved efficiency the SGCAN fell short of providing leadership on technical issues. Its performance in country dialogue on the environment and climate change issues was also fairly limited. In addition, according to the ICR, SGCAN did not participate in supervision missions, did not provide guidance on the definition of activities, preparation of terms of reference for studies, or engage in problem solving and dissemination of lessons - such as adaptation protocols (para 98 and Annex 6). This was because SGCAN was not created to work on regional commercial and trading matters and, although it had experience implementing environmental projects, it had never implemented investment programs for its member countries (ICR, para 98). The Bank's project appraisal team was apparently too quick to assume that SCAN, as a respected regional political organization, could handle the broad responsibilities of managing the implementation of a technically challenging multi-country project. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD stated that the SGCAN would be responsible for M&E. National Technical Specialists (responsible for the overall monitoring and evaluation at country level) were enlisted by SGCAN to support the M&E activities but they were based in the three national Ministries of Environment (MOEs). A monitoring and evaluation system and guidelines were developed for the project’s operational manual. The NTSs were managed and coordinated by the project's regional coordinator located in the SGCAN office in Lima. Moreover, the NTSs were individuals who had earned the trust of their governments, through the Ministries of Environment, and often traveled to the pilot locations with MOE staff. The NTS reported officially to SGCAN but also reported more regularly to the MOEs, and received full support from MOEs (most frequently they were actually hosted inside MoEs buildings, using their facilities and interacting with their civil servants and decision makers). The NTSs also submitted biannual project progress reports to the Bank demonstrating project development and physical performance indicators (para 70). b. M&E Implementation: According to the ICR implementation progress was recorded in semi-annual reports. Most of the information was gathered by NTSs who also made preparations for Bank supervision missions. Detailed supervision Aide Memoires reflected this information and described the implementation of M&E methodology. The MTR, which used M&E reports, proved to be pivotal for the project because it identified the restructuring needed to improve project performance and scope. The M&E system also produced a number of documents such as the end-of-project report, systematic country based reports, and detailed reports on the results of all the project's components (ICR, pp 41-42) c. M&E Utilization: The project's complexity meant that many steps needed to be taken before activities were implemented and outputs achieved. This was not reflected in the M&E system and results framework before the MTR because it focused mainly on outputs. After restructuring, the adjusted results framework improved the description and follow-up of the sequential steps to achieve outcomes, and was frequently used by the different project stakeholders to report on progress and identify weaknesses, and informed decision making, as discussed in Section 3 of this Review. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: This Project was classified as category B in terms of its environmental vulnerability. Safeguard policies triggered by the project were an Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Pest Management (OP 4.09), Forests (OP/BP 4.36), and Indigenous Peoples (OP/BP 4.10). The project complied with all safeguard policies, but no major impacts or issues of material compliance were identified According to the ICR an environmental and social management framework was developed at appraisal. This framework was generic and focused on the positive impacts promoted by the project, but it provided only limited guidance on actual implementation. During the MTR some shortcomings related mainly to proper documentation of processes and plans implemented by CARE were noted. These shortcomings were rectified by SGCAN. The framework was updated and turned into an easy tool, dedicated training was offered to the NTSs, and closer Bank guidance and supervision was also provided. The updated environmental framework provided clarity for the preparation of simplified environmental impact assessments, mitigation activities and monitoring for small water and sanitation works. It included a sample pest management plan, sample terms of reference for the preparation of a forestry plan, and procedures to adequately address and document the requirements of the Indigenous Peoples safeguard among others. From a social point of view, Project implementation was highly participatory and supportive of rural farming communities. Pilot activities had no social adverse impacts, and positively helped enhance the resilience of the involved communities to changes in water runoff and glacier retreat. b. Fiduciary Compliance: Procurement: According to the ICR "close supervision and extensive Bank support were needed to improve the overall procurement capacity of the PIU within SGCAN and of the participating entities in each country - especially in Bolivia and Ecuador (para 47). Initial procurement shortcomings such as weak contract management and low participation in bidding processes were apparently largely due to lack of experience with Bank procedures and were identified and addressed in each country. By the closing date the PIU had demonstrated adequate capacity to ensure compliance with Bank procurement standards and procedures (ICR, para 47). This Review concluded that procurement performance at the project's close was Satisfactory. Financial Management (FM): The ICR concluded that the project provided reasonable assurance that Grant proceeds were used by the SGCAN for the intended purposes, by ensuring adequate FM arrangements on matters such as budgeting, accounting, internal controls, funds flow, financial reporting and auditing (para 46). In addition the last FM supervision mission concluded that SGCAN had retained qualified staff to manage the project's finances and there were adequate internal controls during implementation. In addition, SGCAN was able to provide timely and reliable information required to manage and monitor implementation. The SGCAN also submitted the final audit reports and auditors issued unqualified opinions on the Project’s financial statements (ICR, para 46). This Review concluded that FM performance was Satisfactory. c. Unintended Impacts (positive or negative): The Government of Colombia was impressed by this project's results and information generated by the project was shared with the Colombian authorities. In addition Colombia has offered to provide leadership for the follow-up project. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Moderately This Review rates the project's Satisfactory efficiency as modest due to inadequate evidence on the project's benefits. The rating of efficiency was lower than the unstated, but implicit, rating of efficiency in the ICR. Risk to Development Moderate Moderate Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR listed a number of lessons, although some of them were conclusions rather than generally applicable lessons. The following is a summary of the most significant lessons in the ICR.  The ability of countries to increase their long term resilience to climate change is directly linked to their capacity to generate and analyze data to assess vulnerability to future events . Therefore robust, well managed and well funded institutions with appropriate analytical capacity are crucial for relevant adaptation to climate change.  Legal agreements for projects should allow for flexibility in implementation arrangements so as to avoid the need for subsidiary agreements if modest changes are required for those arrangements during implementation. Project legal agreements should therefore provide for operational flexibility. This Review suggests another lesson, namely:  A technically qualified and experienced project implementing agency is critical for a project 's success. The choice of implementing agency for this project was not appropriate because SGCAN (even with the support of a PIU) and despite its sound credentials as a regional organization, did not have the institutional capacity to provide the vision and leadership on technically complex environmental issues. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provided a candid and comprehensive coverage of the project's evolution during its implementation and restructuring. It also included a clear description of the project's achievements and weaknesses as well as the performance of the Bank and the three governments. However, the analysis of efficiency could have been much stronger and in particular the ICR provided no rating for efficiency. The project's heavy emphasis on the generation of knowledge about the impact of climate change on glacier retreat, the design of protocols, and the inclusion of pilots to test a number of protocols to adapt to the effects of glacier retreat placed an onus on the project to analyze the net benefits of pilots carefully. A comprehensive analysis of the effectiveness of pilots in addressing climate change could therefore have been an important part of the ICR's assessment of the efficiency of this project in reaching its objective. For example the benefits and costs of the options for improving Quito's water supply could have been estimated. For the pilot in El Alto the value of water savings through improved distribution systems could have been estimated and compared with the investment costs required to improve the distribution of water. In addition, the basis for the estimated internal return for the Shullcas irrigation infrastructure rehabilitation in Peru (the only quantitative measure of efficiency in the ICR) was missing from Annex 4. Finally, the benefits and costs of "adaptive cattle management" in Ecuador (an outcome of the project hailed as important) could probably have been estimated with a minimum of survey work. a.Quality of ICR Rating : Satisfactory