ICRR 14026 Report Number : ICRR14026 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/27/2013 Country : Armenia Project ID : P063398 Appraisal Actual Project Name : Municipal Water And US$M ): Project Costs (US$M): 25.56 51.95 Wastewater Project L/C Number : C3893; C4514 Loan/ US$M ): Loan /Credit (US$M): 23.00 44.29 Sector Board : Water US$M): Cofinancing (US$M ): 0 0 Cofinanciers : Board Approval Date : 05/04/2004 Closing Date : 02/28/2009 02/28/2012 Sector (s): Water supply (71%); General water sanitation and flood protection sector (20%); Sewerage (5%); Central government administration (4%) Theme (s): Other urban development (40% - P); Urban services and housing for the poor (40% - P); Pollution management and environmental health (20% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Toneema M. Haq Roy Gilbert Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The Development Credit Agreement (DCA, Schedule 2) stated: "The objectives of the Project are to improve the quality of water and wastewater services in the Armenia Water and Sanitation Company (AWSC) Service Area by providing efficient and sustainable water and wastewater services and strengthening the capacity and sustainability of AWSC" (Schedule 2). The Project Appraisal Document (PAD, p. 7) stated: "The project development objective is to improve (i) the quality of services provided to customers in AWSC's service area; and (ii) the sustainability of AWSC." This review will use the objectives as stated in the Development Credit Agreement . The Additional Financing, approved in 2008, did not change project objectives . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: There were three main components in the Project : Component A : AWSC Management Improvement (called Management Strengthening in PAD ) (Estimated Cost : US$ 9.74 million, Additional Financing Cost : US$8 US$9 US$ 20. 20 .29 million ) US$ 8.23 million, Actual Cost : US$20 This included two subcomponents : management services and strengthening the Contract Monitoring Unit (CMU). About 84% (US$17 million) of this component financed the management contract . The rest financed the second subcomponent, operating costs of the CMU . The Additional Financing extended the management contract for two years. A third subcomponent, the preparation of the lease contract for the future after completion of the management contract, was eventually financed by a European Bank for Reconstruction and Development grant . AWSC received US$19 million in parallel financing from the EBRD. The lease contract transfers the operational and commercial risks from the government to the private partner, AWSC . US$ 3.33 million, Additional Financing Cost : US$6 Component B : AWSC Investments (Estimated Cost : US$3 US$ 6.37 million, US$ 7.22 million ) Actual Cost : US$7 This included five subcomponents : general assistance, redundancy and training, laboratories, meter revolving fund, and assistance to condominiums . General assistance financed equipment, machinery, vehicles and chlorine for AWSC operations. Over 75% of the Additional Financing for this component financed the general assistance subcomponent, including carrying out AWSC fixed asset inventories and reviewing its operations and maintenance costs. The redundancy program financed the severance packages of 610 employees terminated during the first two years of the project. Some of this subcomponent financed salaries of remaining staff to encourage performance . Since AWSC organized technical and administrative training programs at its own cost, no funds from the Additional Financing were used for this, though a small amount (US$0.57 million) had been originally allocated in the Additional Financing. The laboratories subcomponent equipped chemical, bacteriological and meter laboratories . The Additional Financing added funds to this subcomponent . The meter revolving fund financed the supply and installation of about 2,000 meters for low income households . Finally, the last subcomponent, assistance to condominiums, replaced leaking pipes and valves in the basements of apartment building, even though this was not legally under AWSC responsibility. The Additional Financing provided additional financing for this fifth subcomponent. US$ 12. 12 .49 million, Additional Financing Cost : Component C : Investments in Branch Systems (Estimated Cost : US$12 US$ 12. 12 .21 million, Actual Cost : US$24 US$12 24 .45 million ) US$ 24. This included two subcomponents : the immediate program and investments . The Additional Financing only financed the investments subcomponent . The immediate investment program financed the rehabilitation of two water treatment plants in Dilijan, the construction of two water reservoirs in Sevan and the replacement of leaky pipes in these two cities. The main investment program financed the rehabilitation of pumping stations, storage reservoirs, and installation of water distribution pipes in 14 urban areas. It also financed energy efficient pumps and the replacement of distribution pipes in other urban and rural areas . Finally, it financed the supply and installation of chambers for residential water meters transferred from private to public land . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost . The project cost at closing of US$ 51.95 million is equal to the actual cost of the original credit (US$26.05 million) plus the actual cost of the Additional Financing (US$25.9 million) The rationale for the Additional Financing is explained in the Project Paper (p. 3): "The need for additional funding from the Bank to scale up the on-going project was anticipated at appraisal of the project and the management contract was prepared on the basis of a six year program. However at the time, because of limitations in IDA availability, funding was provided for the first four years with the option of an extension . The additional funding was therefore intended to fund an extension of the management contract and the remaining investments identified earlier ." Though the total project cost by component was US$51.95 million, it was US$52.61 million by disbursement category, a difference of US$ 0.66 million. The Project Team clarified this difference as being due to changes in SDR -US Dollar exchange rates. Financing . There was no official cofinancing in this project, but the EBRD provided parallel financing to the AWSC, including that for the lease contract which was grant financed by the EBRD . Borrower Contribution . At approval the Bank planned to finance 90% of the total costs and the Government of Armenia (GOA) 10%. This changed slightly at closing with the Bank financing 85% of total project costs and the government 15%. Dates . In 2008, the Bank provided Additional Financing of US$ 20 million with a closing date of 12/31/2011, two years and 10 months after the original closing date of 2/28/2009. The restructuring of 12/28/2011 extended the original closing date by two more months from 12/31/2011 to 2/28/2012 in order to use cost savings accumulated under the project due to SDR-US Dollar currency exchange fluctuations . 3. Relevance of Objectives & Design: a. Relevance of Objectives: The Progress Report on the Country Partnership Strategy for Armenia for FY 2009-FY2013 extended the Country Partnership Strategy (CPS) for FY2009-2012 by one year. According to the CPS, sustainable access to improved water is part of Goal 7: Ensure Environmental Sustainability . By recruiting an international water operator to manage Armenia Water Supply Company (AWSC) under a management contract, the sustainability of the overall water supply system was improved. Relevance of Objectives is High . b. Relevance of Design: The management contract and the technical assistance to Armenia's water and sanitation company were designed to improve the the utility's efficiency and sustainability . The rehabilitation of water treatment plants, replacement of leaky pipes, and other investments are linked to the objective of improving the quality of the services provided to the customers in the utility's service area . However, wastewater is not mentioned in the Results Framework of the project even though it is in the title of the project and is also referred to repeatedly in the PAD, the DCA. and the ICR too. The PAD, for instance, highlights the underfinancing of wastewater as a key sector issue (PAD p. 4), foresaw project investment in wastewater through component C (PAD p. 9), and reports the benefits of improved wastewater collection in its economic analysis (PAD pp. 17-18). In comments on this ICR Review, the Region nevertheless regarded this project exclusively as a water supply operation . Even though non-revenue water losses were at a very high level in 2004 at project appraisal, at 74%, this was not explicitly addressed in the PAD or the management contract and no reduction targets were set . The Project Team maintains that this was not addressed since dealing with water losses would require massive investments and highlights that Armenia is a water abundant country . The Project Team further explained that the benefits of carrying out the investments in reducing technical water losses would be lower than the costs associated with such investments, and therefore such non -revenue water reduction programs would not have been justifiable economically or financially. Instead, the project aimed to address this issue by focusing on reducing the commercial losses and by replacing pumped water systems /sources with gravity systems /sources where feasible, and thereby reducing the energy and operating costs associated with pumping in the system . Relevance of Design is Modest . 4. Achievement of Objectives (Efficacy): Objective 1: Improve the quality of water and wastewater services in the Armenia Water and Sanitation AWSC ) Service Area by providing efficient and sustainable water and wastewater services is rated Company (AWSC) Substantial . Outputs 1. Percentage of cities with minimal (unspecified) daily hours of water supply service increased from 68.1% (in 2004) to 94% (in 2012), exceeding the original target set in 2011 of 90%. According to the ICR, the original targets for this and all other indicators were set in 2011, one year before project closing . This was later clarified by the Project Team who confirmed that the original targets were set in 2004 and baseline values were established by the management contractor (and verified by an auditor) in 2005. 2. Percentage of block apartment building with meters increased from 38.6% (2005) to 82.1% (2012), exceeding the original target set in 2004 of 75%. 3. Ratio of water volume billed on the basis of metering to the total metered and normative billed volume increased from 25% (2005) to 85.8% (2012), exceeding the original target set in 2004 of 65%. 4. Average domestic metered consumption increased from 81 liters per capita per day (2005) to 100 (2012), and further to 108 by 2013 exceeding the original target set in 2004 of 94. According to the project team, this increase was due to decrease in illegal water connections and more efficient billing practices . The ICR reports the target as 70, but this was explained by the project team as a typographical error . 5. Average daily production increased from 668 liters per capita per day (2005) to 670 (2012), meeting the original target set in 2004 of 670. 6. Ratio of metered final consumption (m3) to water production increased from 6.8% (2005) to 13.7% (2012), exceeding the original target set in 2004 of 7.8%. 7. Electricity consumption decreased from 0.43 KWh (2005) to 0.26 KWh (2012), exceeding the original target set in 2004 of 0.30 KWh. Electricity consumption was reduced by 40% overall (ICR, p. 21), mainly due to conversion of pumping systems to gravity, installation of more efficient pumps and implementation of rehabilitation activities distribution networks (ICR, p. 62). 8. Percentage of water disinfected went from 60.9% (2005) to 97.3% (2012), exceeding the target set in 2005 of 95%. The project financed chlorination equipment installed at production plants so this indicator referred to disinfection of water produced . 9. The ICR reports that 259,505 new piped household water connection resulted from the project . The Project Team explained that the 259,505 new connections included not only households in new communities that joined the service area, but also the households in communities where very old and rusted connections and deteriorated pipes were rehabilitated and replaced. 10. 99,938 piped household water connections benefitted from rehabilitation works 11. 34,748 people in rural areas and 285,055 people in urban areas were provided with access to improved water sources 12. The Beneficiary Survey (2011, based on a representative sample of 400 households distributed proportionally among the regional cities covered by the project ) reported that 42% of customers reported water supply service of 21 to 24 hours per day and 46% reported service of less than 12 hours per day. For the lower income quartile the average was 13.4 hours and for the higher income quartile the average was 15.9 hours (ICR, p. 23). Total water production was 159.9 million cubic meters and total consumption was 26.5 million cubic meters. Also, non-revenue water is not monitored though it was highlighted in the PAD as being abnormally high . In fact, during the implementation of the project, non -revenue water increased from 74% in 2004 to 83% in 2011, manly due to physical leaks and commercial losses (ICR, p. 19). Outcomes 1. The weighted average of daily hours of drinking water service went from 6.04 hours (2005) to 15.02 hours (2012), just meeting the target set in 2004 of 15 hours. This can be attributed to investments in Component C (investments in branch systems) and output #5 above. 2. The weighted average water bacteriological safety compliance increased from 93.8% (2005) to 98.2% (2012), exceeding the target set in 2004 of 98.1%. This can be attributed to the laboratories subcomponent under Component B (AWSC investments) and output #8 above. 3. The percent of individual subscribers billed on the basis of metered consumption increased from 40.2% (2005) to 82.2% (2012), exceeding the target of 77.8%. This outcome is related to the sustainability part of the objective, since improved billing should lead to sustainability in the sector . It is attributed to the meter revolving fund subcomponent in Component B (AWSC investments) and output #2 above. 4. The Beneficiary Survey (2011) reported that 76% of households reported an improvement in the constancy of service since 2004, while 18% said it remained the same. 5. The Beneficiary Survey found that 43% of customers were satisfied with the response time to problems reported to AWSC. This is called a "noticeable improvement" in the ICR (p. 23) but the baseline is not specified . The efficiency and sustainability of water services have been achieved . While the ICR did not provide clear outcome evidence of the intended improvements in wastewater services, the information provided by the Project Team indicates that the sewerage systems in AMSC's service area had not deteriorated because; (i) the management contract mandated the private operator to improve water supply services while maintaining the sewerage system intact; and (ii) project financed interventions in highly deteriorated sewerage networks to prevent potential contamination of drinking water supply . Objective 2: Improve the quality of water and wastewater services in the Armenia Water and Sanitation Company AWSC ) Service Area by strengthening the capacity and sustainability of AWSC is rated Modest . (AWSC) Outputs 1. Increased revenue collected from domestic subscribers rose from 166 Armenian dram (equal to US$0.29) per month 2004 to 502 dram (equal to US$1.28) per month in 2012, exceeding the target set in 2004 of 260 dram per month. 2. Subscribers with debt outstanding for more than 4 months decreased from 79.5% of subscribers (2004) to 26% (2012), exceeding the target set in 2004 of 78%. It is unclear from the ICR why the target was nearly the same as the baseline. 3. The average price of m3 metered and billed to domestic subscribers increased from 100.41 Armenian dram (equal to US$0.18) in 2004 to 179.8 dram (equal to US$0.46) in 2012, meeting the target set in 2004 of 180. 4. Total cost of staff as % of revenues decreased from 70.2% (2004) to 59.3% (2012), exceeding the target of 60%. 5. The redundancy program reduced staff by 610 employees and increased salaries of remaining staff . Outcomes 1. The AWSC working ratio decreased from 194.9% (2004) to 117.8% (2012), slightly exceeding the target set in 2004 of 118%. The working ratio is defined as operating expenses divided by operating revenue, excluding subsidies and grants. 2. The Beneficiary Survey reported that 74% of the respondents believed that the water system management improved between 2004 to 2011 while 21% said it remained the same and 5% believed it worsened (ICR, p. 52). 3. The Beneficiary Survey reported that 43% of households considered the current water bill adequate, while 40% considered it "rather high" and 17% "very high." (ICR, p. 23). 4. While the management contract introduced some built in financial incentives and technical targets for the private operator to strive for gains in achieving operational efficiency and financial sustainability, the ICR reported that AWSC is "at risk of bankruptcy." (ICR, p. 18) and this was a major issue for the failure to achieve this objective . The Project Team provided further explained that a 2012 audit report (not available at project completion ) noted that: Revenue increased by 6.36%, while Cost of Sales increased by 5.72% only, which is a positive sign for the entity ’s performance. Administrative Expenses and Other Expenses decreased by 9.9% and 35%, respectively, indicating that there are some improvements in overall cost control of the entity . 5. Efficiency: The PAD calculated both an Economic Rate of Return (ERR) and a Financial Internal Rate of Return (FIRR). Since most specific investments in the project were to be determined after the management contract with the private utility operator began, the economic and financial analyses were based on "engineering estimates guided by experience " (PAD, p. 69) for such similar projects as the Yerevan Municipal Development Project (Credit 4038). The economic benefits that could be measured were resource savings, mainly energy by decreased electricity consumption and bulk water use, from improved operational efficiency . The ERR assumed an average life time of 30 years on investments and was calculated at 11% based on these resource savings . The PAD mentioned that important benefits such as coping expenditures, reduction in water -borne diseases, and reduction in sewerage-related impacts, were difficult to quantify due to limited data availability .The PAD also discussed the cost-benefit analysis of the redundancy component (US$0.46 million, 1.8% of total project costs) for the project due to expected reduction in staff during the second year of the management contract . The ERR of this component was estimated at 12.2%. The percentage of total project costs covered by the ERR is not available in the PAD . The economic analysis in the ICR mentions that Component C (investments in branch systems ) led to energy savings. That component was US$12.49 million, or 49% of total project costs at appraisal . The ICR calculates the rate for energy savings from this component (US$24.45 million or 47% of total project costs at completion ) as 14%. The ICR estimated the ERR of the management contract which was US$ 16.85 million or 32% of total costs at closing. The ERR for this is estimated at 12%. The ICR states that details on calculation of Net Present Value (NPV) and ERR "will be submitted later after making respective calculations " (p. 76). The Project Team computed additional economic benefits over a year after project closing . The ICR analysis was updated to include all costs and more comprehensive set of measurable benefits including energy savings, welfare gains to new customers and reduced coping costs for existing customers . This resulted in an ERR of 16% with an NPV slightly above US$42 million. The financial benefits were based on increased revenue collection resulting from increases in tariffs and collection rates, offset by declines in sales volumes . The FIRR was estimated at 14% based on efficiency gains expected from the management contract which was US$ 7.68 million or 30% of total project costs. Despite this, ICR estimates of the NPV and FIRR of future AWSC cash flows (using total project cost as outflow and total of collections and operating subsidies as inflows ) are minus US$3.6 million for a discount rate of 12% and minus 6% respectively (ICR, p. 36). Appraisal estimates were US$12.5 million and 14% respectively (PAD, p. 18). The ICR candidly admits that "As the NPV and FIRR of the project are good indicators of the sustainability of AWSC operations, the achievement of this PDO is rated unsatisfactory ." (P. 20) and furthermore, "Obviously AWSC is at risk of defaulting on its loan payments in the near future " (p. 21). Efficiency is rated Substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 11% 49% ICR estimate Yes 14% 47% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on High Relevance of Objectives, Modest Relevance of Design, Substantial Achievement of the first objective and Modest Achievement of the second objective, and Substantial Efficiency, the final Outcome rating is Moderately Satisfactory. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The greatest risk to development outcome is the nonpayment of and lack of adjustment to tariffs, making it difficult for AWSC to become independent of government subsidization . Another substantial risk is the future of the management contract. Currently, the management contract, which began in 2004, is valid until the end of 2013. Due to AWSC's poor financial situation, it is unlikely that the management contract could be turned into a lease contract . In a lease, the operator's costs must be covered by collections and the shareholders would have to finance the working capital. The management contract is heavily dependent on subsidies : "Throughout the project period AWSC has received subsidies that represented about 30 percent of its total revenues " (ICR, p. 21). AWSC is on the verge of bankruptcy and highly dependent on the Government to finance its operating subsidies and dependent on external funds to finance its manager. Raising tariffs to cover operating costs would be difficult since over half (57%) of the beneficiary survey respondents already said the tariff was "rather high" or "very high" (ICR, p. 24). a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The Bank supported a "simple and well-focused project" (ICR, p. 25) that used the public private participation model that was well adapted to the water sector . The signing of the management contract was a condition of credit effectiveness and the appointment of the manager of AWSC foresaw a fair and transparent selection . The Bank did not prepare financial forecasts of the operations of AWSC and selected "inadequate financial covenants." (ICR, p. 25). This lack of preparation has led to "an AWSC that is highly dependent on operating subsidies and has no clear action plan for moving towards financial sustainability " (ICR, p. 25). As of the latest audit report (May 2011), AWSC is on the verge of bankruptcy, as mentioned in Section 4, above. As noted in Section 3b, above, better preparation was needed to ensure the success of the second objective of sustainability of AWSC. Some data important to the Project was not collected . Information on AWSC technical operations such as water production, water sales, and non -revenue water (NRW) were not collected at the appraisal stage . The issue of water loss should have been add ressed since the PAD mentioned abnormally high levels of unaccounted for water, know as non-revenue water. As mentioned in Section 3b, above, no NRW reduction targets were set either in the PAD of management contract . Internationally accepted levels of NRW are 20% and below (ICR, p. 19). The NRW for AWSC was 74% in 2004 and increased to 83% at the end of 2011. The Project Team later explained that the cost-benefit analysis of addressing the technical losses showed that such interventions would have not have been justifiable both economically or financially, but this should have been clarified at entry . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: As noted in Section 11a below, the Bank supervision team flagged that the Environmental Management Plan (EMP) was not being followed and this was corrected during project implementation . The project was implemented as designed, delivered on time, within budget and in compliance with relevant guidelines and safeguards . The Bank team contributed to the implementation of an institutional reform in the water sector through a well functioning public private partnership arrangement /management contract between the government and a private operator . Supervision missions paid insufficient attention to the financial weaknesses of AWSC and this failure led to its bankruptcy by project completion . Better supervision and use of financial indicators to raise "red flags" (ICR, p. 20) was needed to monitor progress towards the objective of AWSC sustainability and mitigate the high risks to the development outcome of the project that these failings entailed . On balance, quality of supervision is rated moderately satisfactory . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: In terms of safeguards, a national requirement was introduced to carry out supervision of civil works to track environmental compliance (see Section 11a, below). The Government embraced the Public -Private Partnership model by transferring the responsibility for setting water and wastewater tariffs and monitoring performance of operators to the Public Services Regulatory Committee, a semi autonomous body . It implemented the management contract successfully and intended to apply the Public Private Partnership model to the AWSC . It did, however, create a dependency situation with AWSC by extending subsidies to cover its operating expenses, and thus, "is partially responsible for not taking actions for putting AWSC on the path of financial sustainability " (ICR, p. 26). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The implementing agency is the Armenia Water and Sanitation Company (AWSC). It delivered quality construction in a timely fashion and within budget, while meeting fiduciary and safeguard requirements . AWSC organized staff training to improve management operation and procured equipment for operations and maintenance of the investments program . It was, however, very dependent on the Government for operating subsidies. There is not much evidence in the ICR of AWSC's implementation performance . The ICR has a brief section on Borrower Performance mainly commenting on government performance, and not a separate section on implementing agency performance . AWSC still relies heavily on Government subsidies to cover its operating and management costs . It has been in compliance with its financial covenants only because its operating subsidies have been extended by the Government. Due to AWSC's poor financial situation and limited future cash generation capacity, it is not possible to evolve the management contract into a lease contract, similar to what has been successfully implemented in the above-mentioned Yerevan Water and Wastewater Project . The latest audit report on AWSC (May 2011) expressed great concern that it has yet to recover its operating costs from collected user charges, leaving it open to "risk of bankruptcy" (ICR, p. 18). A more recent audit report from 2012 (see Section 4, above) had a more optimistic outlook. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Three categories of outcome indicators were used to monitor achievement of the objectives : technical, commercial and sector investments . Technical indicators measured service reliability, improved water quality and greater operating efficiency . This was measured by increased weighted average of the daily hours of drinking water service and weighted average of water bacteriological safety compliance . Surprisingly for a problem identified at appraisal, there was not an indicator for unaccounted -for water. The commercial indicators measured increased water company cash generation through reform of billing and collection and closer alignment of service tariffs with costs as well as better cash flow . This was measured by the percent of individual subscribers billed on the basis of metered consumption and the company working ratio (operating expenses divided by operating revenues ). However, actual payments made by customers were not monitored . The sector investments were to be decided upon by the manager and all investments had to be economically justified and meet all applicable safeguard policies . The technical and sector investments indicators were well designed but the commercial indicators on AWSC finances were missing. There was too much focus on technical indicators and none on financial indicators necessary to monitor progress achieved on AWSC sustainability, and indeed its survival . Other indicators that would have been useful to monitor progress in the Project were non -revenue water indicators (discussed in Section 8a, above) and indicators measuring "time savings, reduction of coping costs, health improvements and environmental improvements" (ICR, p. 29), as well as indicators for wastewater . b. M&E Implementation: The targets were revised in the 2008 Project Paper to monitor progress under the Additional Financing and set clearer targets to be reached by the management contract . Non-revenue water, admittedly "an indicator commonly used for measuring the efficiency of a WW utility company " (ICR, p. 13) was not added. Neither were any indicators for wastewater. Financial indicators were not reconciled with the AWSC audited financial statements, which turned out to be a major shortcoming. c. M&E Utilization: There is not much discussion in the ICR on how well the data from the indicators was utilized . The 2008 Project Paper "set clearer targets" (p. 13), but the ICR does not report revised targets . M&E was of little or no use for the financial monitoring of AWSC. M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: OP/BP 4.01 Environmental Assessment was triggered by this project . The project was classified as environmental category FI (financial intermediary) but the 2008 Project Paper revised this classification to Category B (potentially adverse environmental impacts ). The ICR notes (page 14) that the project closed with satisfactory performance on the Bank's safeguard policies ." Only investments classified as Category B or C (minimal or no impacts) were supported under the project . An Environmental Management Plan (EMP) framework was developed during project preparation and provided guidance on screening of individual activities . The EMP framework, however, was not followed during the earlier stages of project implementation and individual investments were not screened and classified. This was noted by Bank supervision teams and the quality of environmental monitoring of works improved. A new national requirement introduced an entity to carry out supervision of civil works to track environmental compliance. No major environmental issues were encountered due to the small scale of works and good construction practices . Such issues as maintenance of sanitary zones and handling of asbestos were addressed satisfactorily. There was no discussion in the ICR of safeguards applying to wastewater . b. Fiduciary Compliance: Financial Management . The ICR notes (page 26) “Indeed, the Bank: (i) ensured that Armenia Water and Sanitation Company (AWSC) delivered quality construction on time and within budget and complied with fiduciary and safeguards requirements…� The Bank closely supervised the financial management of both the Contract Monitoring Unit of the project as well as AWSC, the implementing agency . Annual financial management supervision missions were carried out. Implementation Supervision Reports consistently reported financial management ratings of satisfactory (with the exception of the report of August 2005 which was rated marginally satisfactory ). Audits were timely and unqualified for the Contract Monitoring Unit . However, audits for AWSC had qualified disclaimers for 2005 and 2006 and qualified exceptions for 2007 and 2008. Audits for 2009 and 2010 were unqualified. However, the latest available report (May 2011) expressed "doubt about the ability of AWSC that had accumulated large financial losses in recent years, to continue as a going concern " (ICR, p. 16). In response to this, the Bank requested a time bound action plan for implementing the recommendations of the auditor's letter of May 2011. This action plan was not available for review at the time of project closing . AWSC was in compliance with the financial covenant of not incurring any debt unless its net revenues are at least 1.5 times its estimated maximum debt service requirement for any succeeding fiscal year (ICR, p. 16). This was only possible due to the operating subsidies extended by the government . Procurement . A condition of credit effectiveness was the award of the management contract . After a fair and transparent bidding process among four pre -qualified operating companies, an experienced manager (SAUR of France) was selected. Out of 132 contracts, 77 were subject to ex-post review and 27 were reviewed ex-ante. Supervision reports were regularly carried out and never identified governance issues that could adversely affect the procurement process. Disbursement . No issues were reported by the ICR . Most disbursement categories (works, goods, consultants services, refunding of PPF) spent amounts that were close to what was anticipated at appraisal . Operating costs were higher than expected (129% of appraisal) and redundancy payments, as mentioned in the section below, were lower than expected (63%). c. Unintended Impacts (positive or negative): A potentially controversial program aimed at reducing the AWSC work force by 25%, the redundancy program, was financed by the Project. The identification of staff to be terminated was left to the manager . A total of US$0.25 million was spent on this subcomponent and 610 employees were terminated in the first two years of the project . Since statutory severance packages were lower than originally budgeted (US$0.46 million originally budgeted), salaries of the remaining staff were increased . are There were no specific activities in the Project aimed at gender issues . However, the ICR notes that women are the main users of water for domestic tasks are thus likely to be the main beneficiaries of improved water reliability . d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant High Taking into account the risk of AWSC's Outcome : bankruptcy reported by the project audit. Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lesson from the ICR Sound baseline information is needed for project evaluation, including its economic justification . Time savings (time spent fetching water), reduction of coping costs (related to booster pumps, individual storage tanks, household disinfection equipment, water boiling or purchase of bottled water), health improvements, and environmental improvements were difficult to measure in this case, due to uncertainty about baselines . Lesson from IEG In order to evaluate results more fully, a water supply and wastewater sector project needs data on total water production and consumption as well as total wastewater collection, treatment and disposal . It should also include data on unaccounted for water . 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned . 15. Comments on Quality of ICR: Though portions of the ICR are candid, there is not much outcome evidence . There are several shortcomings: (1) The ICR includes a thorough Borrower ICR that provides a different context of the evaluation. The Borrower ICR reports (ICR p. 57) that wastewater disposal systems were constructed and rehabilitated (later clarified by the Project Team that this was financed by EBRD ), but that existing treatment plants do not operate. They are deteriorated and cannot be rehabilitated anymore " ICR (p. 65) but the ICR itself provides no information about this, or assessment of the performance of this wastewater component /subsector. (2) The ICR does not correctly reproduce the original project objective by omitting the reference to 'wastewater or to AWSC's services in general (that include wastewater) (ICR pp. 2 and 19); (3) the ICR does not rate the government performance and implementing agency performance separately (p. 26) as required by the guidelines and which would have provided more insights into the performance weaknesses on the borrower side; (4) other shortcomings in the text of the ICR include: (i) inaccurate objective re-statement that deleted reference to wastewater and ignoring wastewater completely (even though this was in the title of the project ); (ii) cost tables improperly labelled; (iii) data inconsistencies between results framework that shows 259,505 new household connections achieved (also given as figure for the target) and the Annex 2 Outputs per Components that reports that 259.505 "community water points rehabilitated" (ICR p. 32); (iv) reporting target values that are identical to actual results for nearly half the project's outputs (ICR p. 32) with no explanation in Annex 2 Outputs per Components ; (v) typographical errors throughout, even in the outcome indicators listed in the front (indicator 6 reported a target of 70 liters per capita per day domestic metered consumption as a target when it was actually 100, as clarified by the project team; (vi) no discussion of safeguards as applying to wastewater . a.Quality of ICR Rating : Unsatisfactory