Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) Report Number : ICRR0020956 1. Project Data Project ID Project Name P096586 UA - ENERGY EFFICIENCY Country Practice Area(Lead) Ukraine Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80640 31-Mar-2016 201,000,000.00 Bank Approval Date Closing Date (Actual) 17-May-2011 31-Mar-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 200,000,000.00 0.00 Actual 200,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Richard L. Berney Fernando Manibog Christopher David Nelson IEGSD (Unit 4) PHPROJECTDATATBL Project ID Project Name P129704 CTF Ukraine: Energy Efficiency Project ( P129704 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 1,000,000.00 Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) Bank Approval Date Closing Date (Actual) 08-Nov-2012 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) in the Legal agreement states that the project objective is “to contribute to improved energy efficiency by industrial and commercial companies, municipalities, municipal sector enterprises and energy service companies.” The Legal agreement goes on to state that the project will support this outcome by “facilitating sustainable financial intermediation for the financing of energy efficiency investments” (Loan Agreement Schedule 1, page 5 ). The PAD states the identical Objectives (on page 7). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project consisted of one component, a loan to the financial intermediary, the State Export-Import Bank of Ukraine (UEB). (At Appraisal US$200 million; Actual US$200 million): UEB was to on-lend the funds to support to energy efficiency subprojects at interest rates and repayment periods that were more attractive than those being provided by Ukraine’s commercial banking sector, because the identified savings would be more than sufficient to repay the loans. The UEB would also on-lend funds to qualified commercial banks which would, in turn, use these funds to lend for smaller energy efficiency projects. Since the subloans would be driven by demand from individual enterprises, there were no specific subproject targets for individual energy intensive industries. While the Bank’s loan agreement didn’t establish onlending rates and repayment durations, it did establish a maximum subloan size of US$30 million. The ICR lists the type Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) of energy efficiency investments that the project would support, including: • modernization of inefficient and obsolete equipment and facilities; • installation of highly energy-efficient industrial equipment and processes for new production capacities, in industries where current energy use considerably exceeded current best practices; • utilization of waste gas and heat and excess pressure; • improvement of systems that involves a suite of measures to increase energy efficiency; • reduction of energy loss losses in municipal sector enterprises; • preparation of energy efficiency-related studies and technical assistance; • reduction of energy losses in buildings; and • implementation of other energy efficiency subprojects satisfactory to the World Bank. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total investment supported by the Bank’s loan was US$539 million. Thus the Bank’s US$200 million loan had a leverage ratio of 2.7. (ICR Table 3, page 18) Financing: The project provided US$200 million to the State’s Export Import Bank (UEB) for onlending to individual energy-saving subprojects. EUB also on-lent to a commercial bank (Megabank) for it to onlend to smaller EE projects. Borrower Contribution: The borrower, UEB, made no direct contribution to the onlending financing, but made a significant (but unqualified) contribution through its financing of all the onlending implementation costs. Dates: The project was approved on 12/07/2011, and was expected to close five years later, on 03/31/2016. The project was restructured at the end of 2015, about a year after the mid-term review. A new PDO target indicator was added and an existing one was modified. The closing date was extended by one year to 03/31/2017 to compensate for the slowdown in lending during the country’s 2014-2015 financial crisis. 3. Relevance of Objectives & Design a. Relevance of Objectives Ukraine’s heavy industry had been established during the Soviet Union period, when energy was inexpensive and energy costs were not considered an important element in the Government’s investment decision making. Since low-cost energy was readily available, technology needed to reduce energy demand was not considered to be important. Although Ukraine now faces world market energy pricing, its manufacturers have not had the resources to modernize their production facilities and have continued to use the existing energy-inefficient technologies. As a result, Ukraine is among the most energy-intensive economies in the world, with an intensity that is more than double that of the EU-12 countries. Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) The project’s single Development Objective, to contribute to improved energy efficiency by industrial companies and commercial companies, municipalities, municipal sector enterprises, and energy service companies, was highly appropriate for the country conditions at the time of appraisal, and continue to be highly appropriate today. The FY08-FY11 CPS assigned a high priority to supporting energy efficiency projects in the Core Program, as a major pillar for sustaining growth and improving competitiveness. The FY12-16 CPS and the FY17-FY21 CPS both supported the continuation of activities to improve energy efficiency. The FY17-FY21 CPS confirms that the WBG will continue to focus its support in the energy sector on energy efficiency interventions, including district heating, and on the establishment of energy efficiency financing mechanisms for these goals. The project covered all energy-using entities, except for state and local Government buildings that were not supplied by municipal authorities. There were no restrictions for lending by sector, region, or the technology used to increase energy efficiency. This broad scope covered all possible energy users, thereby enabling the borrower to apply a completely flexible approach to its lending targets when circumstances changed. The financial conditions established for these sub-loans (particularly the debt service ratio of 1.3) were appropriate for the country conditions at the time of appraisal, although when the country’s economy collapsed in 2014-1015, they were seen as too restrictive, and were modified. The change in target Outcome PDO from “Projected lifetime fuel savings”, measured in Mega Joules (MJ) to the “Extent of Energy Savings” in the Results Framework, measured in Tonnes Oil Equivalent (MTOE), had no effect on the evaluation of the project’s outcome, since both measurements are interchangeable. Rating Revised Rating Substantial Not Rated/Not Applicable b. Relevance of Design Project design was similar to those used in the two previous Ukrainian Export Development Projects, in that it used a strong national bank to channel funds to local participating Banks for onlending to industry. The Results Framework (RF) was deficient in that it failed to make the distinction between Outputs and Outcomes. All of its result indicators were considered (erroneously) to be Outcomes, None were identified as Outputs. But only one of the indicators in the Results Framework is actually connected to the Project Development Objective of improving energy efficiency (Outcome: the Extent of Savings in Energy Consumption). As mentioned earlier, the added Outcome Indicator, Project Lifetime Energy Savings is an alternative way of saying “the Extent of Savings in Energy Consumption”, it adds nothing to the reader’s knowledge about the achievement of the PDO. The other two RF Outcome indicators, the volume of energy efficiency sub-loans, and the number of sub-loans should have been identified as Intermediate Outcome indicators, useful to judge the project’s progress, as are the volume of loan commitments and loan disbursements, and the number of participating banks. Indeed, it is not clear why the second participating bank was necessary for project success, especially when it was scheduled to begin participating in the project in year 5, which was the original target year for project completion. Institutional issues had been addressed in two concurrent energy sector projects (the Hydropower Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) Rehabilitation Project and the Power Transmission Project), They were not addressed in this project, and were not included in the PDO or the Results Framework. Rating Revised Rating Modest Not Rated/Not Applicable 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To contribute to improving Ukraine's energy efficiency by industrial companies, commercial companies, municipalities and municipal sector enterprises, and energy service companies. Rationale Output: The project effectively facilitated the financial intermediation for the financing of energy efficiency investments. • US$200 million in subloans were provided for energy saving investments via credit lines through participating banks. • There were121 subprojects, in seven industrial sectors. • The largest number of subprojects were in agriculture and food/beverages. According to UEB, around 45 percent of credit line funds were disbursed to small and medium enterprises (SMEs), with large enterprises receiving the rest. • The Borrower (UEB) signed subsidiary loan agreements with two commercial banks to support their onlending for energy efficiency projects. It implemented lending to one of the banks. The other bank had to drop out of this program as a result of Ukraine’s 2014-2015 financial crisis. • Four loans were provided to Energy Service Companies, in support of improved efficiency in municipal heating operations, twice the project’s original target of two. A fifth loan was made directly to the operator of a municipal system, thereby meeting the appraisal target of five loans to municipal enterprises. The total amount of these five loans was US$0.36 million. Outcome: The project investments achieved an energy saving of 664,000 metric tons oil equivalent (MTOE), equal to 24.8 billion Megajoules of energy, over the lifetime of the energy saving investments, exceeding by about 10% its appraisal target of saving of 600,000 MTOE. The five loans that went to Municipal enterprises save 15,429 MTOE, (and 15,429metric tons CO2), a little over 2% of the total savings. All the other loans were to industrial enterprises. It should also be noted that, although the Results Framework makes no reference to greenhouse gas (GHG) Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) emissions, the ICR quantified both the GHG emissions saving per project over ten years and the dollar cost per unit of GHG reduction in the tables in Table 3 (Output by Sector), on the basis of which it estimates the total CO2 savings to be over 680,000 metric tons CO2, which is also an important project outcome. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency The project was slow to disburse and the closing date had to be extended by one year. All the funds were disbursed by the extended closing date.A cost-benefit analysis was carried out for a representative sample of 20 subprojects, which represent 63 percent of total investment cost in the project. All of the projects have a ERR above 10%, without including CO2 reduction benefits. The weighted average ERRs is 15.9% without carbon benefits and 21.7 percent with carbon benefits (with CO2 valued at the Bank guidelines' level of US$30 per metric ton CO2). All sampled subprojects also have FRRs higher than the 10% threshold. The weighted average for the sample as a whole is 19%. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 20.70 Appraisal  12.60 Not Applicable 100.00 ICR Estimate  21.70 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's development objectives were, and still are, substantially relevant to Ukraine's long-term development prospects The project design was modest: although it reflected the project's objectives, much of the target criteria was poorly focused. The project substantially achieved its investments and outcome targets. Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) Efficiency is also substantial. a. Outcome Rating Satisfactory 7. Rationale for Risk to Development Outcome Rating The risk of failure of the subprojects financed by the project is negligible. Enterprises that made these investments have every incentive to maintain them, and could not, in any event, return to the use of their previous obsolete equipment. Given the substantial energy savings that the subprojects have generated, the borrowers are highly likely to maintain their new facilities. Ukraine is undergoing major energy reforms, which have increased energy prices and made businesses and municipalities more cost conscious. Currently, several additional energy-related regulations are being considered in the Ukrainian Parliament, which, If approved, would encourage further energy efficiency activities. Both UEB (the borrower) and Megabank (the project’s on-lending bank) have continued to provide energy efficiency financing to their clients, showing commitment to the project’s goals. UEB has also expressed readiness to finance further energy efficiency investments from its own funds, when there is a good subproject, a reliable client, and relatively low risks. a. Risk to Development Outcome Rating Negligible 8. Assessment of Bank Performance a. Quality-at-Entry The project was built on the successful results and lessons of the previous financial intermediation projects in other sectors in Ukraine. By the time of the project’s effectiveness, several subprojects were already in the pipeline. Nevertheless, the first two years saw substantially fewer sub-loans completed than initially projected, due, primarily to the change in Ukraine’s economic situation. The design of the RF was poorly conceived, but did not impact the project activities or outcome. Quality-at-Entry Rating Satisfactory b. Quality of supervision The twice-a-year supervision missions provided timely responses to project issues. Management played a key role in resolving complex project issues, particularly with regard to changing the financial requirement for sub- Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) loan borrowers. Implementation support missions often included field visits to beneficiaries in different regions, which enabled discussions with loan recipients on both the loan process and the effects of the energy efficiency financing on their business. Project performance ratings were consistently realistic. During the political and banking crisis of 2014–2016, the Bank continued close monitoring of the situation and, together with UEB, found ways to strengthen the disbursement process, which had stalled in 2015. The team was able to recognize the temporary nature of the negative factors affecting project implementation, and was able to agree with the borrower on the adoption of a set of amendments to the conditions of the credit line that broadened the range of eligible enterprises. This enabled a resurgence in subloans as the economic crisis eased. Project performance in the last two years demonstrated that both institutional arrangements with a strong local bank partner and the terms of the credit line were effective in attracting a good response by the local enterprises. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. Assessment of Borrower Performance a. Government Performance During the banking crisis of 2014–2016, the Ministry of Finance (MoF) and the National Bank took appropriate measures to keep the banking sector afloat and demonstrated high quality of supervision over the sector. Both paid close attention to the financial condition of UEB and, in 2017, recapitalized UEB so that it could stay within NBU’s revised capital coverage limits Government Performance Rating Satisfactory b. Implementing Agency Performance Overall, the performance of UEB was highly satisfactory. There were no major changes in the staffing of UEB’s PIU throughout the implementation period, which created conditions for maintaining the institutional memory and avoiding undue procedural delays, sometimes observed in other Bank projects. The PIU maintained continuous dialogue with the PBs to support preparation of sub-loan requests in accordance with the project requirements, and was sensitive to the concerns of the enterprises involved in the project. It also maintained a constant dialogue with the World Bank on the best ways of adjusting the project to the changing needs of the clients without risking its sustainability. Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) Implementing Agency Performance Rating Highly Satisfactory Overall Borrower Performance Rating Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E framework was generally aligned with the PDO and consisted of indicators commonly used in energy efficiency projects. It included a combination of indicators, measuring both the financial progress of the project and the extent of energy savings achieved as a result of the project’s implementation. However, the inclusion of specific targets for the individual gas, electricity and fuel oil project saving were of little value in evaluating the project progress, since there was no interest in trying to dictate the ratio of investment in each type of savings. Eventually the number had to be revised at the time of project restructuring when the proportion of each target did not conform to the preconceived ideas of the ratios of the various project types. b. M&E Implementation The data was effectively assembled and transmitted to the Bank by UEB on a timely basis. Overall, the data requested did not create any additional burden for UEB, since it was essentially the same data that UEB required of all its borrowers in the course of its own portfolio reviews. It was regularly submitted in the project’s progress reports. Its quality and timeliness was satisfactory. c. M&E Utilization The intermediate indicators were used in the Implementation Status and Results Reports, where they were instrumental in keeping track of, and assessing the project’s progress. They were critically important in the identification of issues related to implementation progress, where they established the basis for the adjustments in lending conditions needed to accommodate the requests from borrowing enterprises. Some of the changes in the development indicators made to the Results Framework during the restructuring in 2015--including the intermediate results indicators of the number of municipalities covered by the Project, the number of energy service companies participating in the project, and the number of municipal sub-projects-- were also useful for focusing borrower’s attention on subsidiary project objectives during project implementation. M&E Quality Rating Substantial Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) 11. Other Issues a. Safeguards The project was classified as OP/BP/GP 4.01 Category FI, because it provides funds through a national financial intermediary (FI) for subprojects that may result in adverse environmental impacts. For these projects, the Bank requires that each FI screen its proposed subprojects and ensure that sub-borrowers carry out appropriate Environmental Assessment (EA) for each subproject and ensure that the project meets the environmental requirements of appropriate national and local authorities and is consistent with this OP and other applicable environmental policies of the Bank. However, PCB control was not initially included in the project’s safeguard documentation, but was added to the project’s updated Operations Manual to ensure the proper PCB management in all subprojects that included reconstruction of old utilities and buildings. UEB undertook environmental screening of the sub-loan applications to determine the appropriate environmental risk category, and was responsible for ensuring that subprojects financed under the project underwent environmental screening to ensure their conformity with both the Ukrainian environmental regulations and the World Bank’s safeguard policies and procedures. The sub-borrowers were then responsible for (i) carrying out environmental analysis and confirming that the proposed subprojects complied with national environmental guidelines and (ii) obtaining the necessary clearance from the appropriate licensing authorities. The Bank also conducted regular environmental and social safeguard compliance monitoring and supervision through regular reviews of sub-borrowers’ activities. No environmental or social risks/issues related to safeguards were observed during the project’s implementation. b. Fiduciary Compliance The procurement reviews were made on a regular basis and found that all procurement procedures were conducted in accordance with commercial practices and followed a competitive approach. UEB monitored the compliance of sub-borrowers to ensure that all contracted firms met the World Bank’s eligibility criteria and ensured the check of all firms against the World Bank’s list of debarred firms before awarding a contract. The ICR (page 29) reports that throughout the implementation period, auditing and reporting were carried out on time and in a manner satisfactory to the World Bank. c. Unintended impacts (Positive or Negative) The level of expertise gained by UEB during the implementation of the project (as well as during the implementation of other three lines of credit to UEB, totaling US$385 million) million has made it a much more efficient and effective institution for providing onlending to both private and public sector institutions. As a result of the successful implementation of the loans in its energy portfolio, and its enhanced understanding of how to select good projects for financing, UEB has developed a solid pipeline of projects Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) amounting to US$80 million, ready for further energy efficiency financing, including using new municipal financing mechanisms, introduced in 2016 through budget decentralization measures. d. Other None 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Negligible Negligible --- Outcome Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The following lessons were abstracted from the ICR: • It would be useful to pre-appraise several financial intermediary institutions for participation in the project, so as to avoid the problem encountered in this project where one of the two these financial institutions originally chosen had to drop out. • The project could have benefitted from early-stage assistance in promoting the project among potential clients, including assistance in developing a countrywide communication campaign. UEB lack of specific experience in communicating the benefits of the EE financing was one of the elements that led to lower-than- expected interest in EE investment during the first few years of the implementation. • The project relied exclusively on a direct onlending financing model for extending financing to companies interested in pursuing EE investments. Future projects should explore additional financing models, including partial risk guarantees, leasing, and possible types of leveraged financing. Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UA - ENERGY EFFICIENCY (P096586) • In some circumstances, such as in Ukraine, which at the time of project appraisal had an effective financial system and a strong public sector bank, a project can succeed without the need for additional technical assistance or policy reforms. 14. Assessment Recommended? Yes Please explain It would be useful to compare this project’s results to similar projects that also included TA and an explicit policy reform agenda. 15. Comments on Quality of ICR The ICR covers all the important project aspects, including rationale, external events, problems, outcomes, and safeguards, although at 26 pages, it is unnecessarily long. Non-project related matters, such as paragraph 70 on activities of other IFI, and potential future Bank projects were distractions from the primary purpose of the ICR, which is to inform about the project outcome, and the factors (both positive and negative) that contributed to this outcome. There are also quite a few repetitions of relevant information, and many details, such as explanations of the ERR/IRR calculations, the borrower’s comments, and deconstruction of the subproject sector categories, that could have been relegated to annexes without any loss in understanding of the project outcome a. Quality of ICR Rating Substantial Page 12 of 12