Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) Report Number: ICRR0021616 1. Project Data Project ID Project Name P131659 ESRP Country Practice Area(Lead) Comoros Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H8660 30-Apr-2017 4,495,613.08 Bank Approval Date Closing Date (Actual) 06-Sep-2013 30-Apr-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 5,000,000.00 0.00 Revised Commitment 5,000,000.00 0.00 Actual 4,495,613.08 0.00 Prepared by Reviewed by ICR Review Coordinator Group Fernando Manibog Peter Nigel Freeman Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) is to contribute to improvement in the electricity sector's commercial and financial performance. (Financing Agreement dated September 24, 2013, Schedule 1, page 5) Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project had three main components: Component 1: Assistance for the commercial and financial performance recovery of the state- owned electricity enterprises (appraisal estimate, US$3.5 million; revised cost, US$4.15 million; actual cost, US$3.91 million) This component covered mainly covered MA-MWE (Water and Electricity Utility in Comoros) and EDA (Anjouan Power Utility). The main activities included the hiring of external experts as senior managers to lead the commercial and financial restructuring of the utilities. The component also sought to enhance commercial and financial performance through the purchase and installation of management and information systems (MIS) in these utilities, 80 units of network metering equipment, 4,000 boxes to protect customer meters, and fuel metering equipment for fuel tanks. The component also financed: (a) A Poverty and Social Impact Analysis (PSIA) to assess the potential impacts of electricity payment enforcement; and (b) communication and awareness campaign to address negative practices within MA-MWE and among its customers. Component 2: Electricity sector governance (appraisal estimate, US$0.90 million; revised cost, US$0.10 million; actual cost, US$0.04 million) This component was intended to provide technical assistance, capacity-building, and studies related to energy policy and planning, legal and regulatory framework, development of a new tariff and subsidy structure, as well as the framework for adjustments. Component 3: Project management (appraisal estimate, US$0.6 million; revised cost, US$0.75 million; actual cost, US$0.57 million) This component supported the overall management, coordination and monitoring of the project implementation activities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) Project Cost. The original amount approved was US$5.00 million, of which US$4.50 was actually disbursed. Financing. An IDA credit was provided for this project through an Investment Project Financing (IPF) instrument. Borrower Contribution. The Borrower did not provide any contribution. Dates. The project was approved on September 6, 2013 and declared effective seven months later on March 13, 2014. A mid-term review was conducted on October 27, 2015. The project was restructured on March 2, 2017, i.e., about two months before the original closing date of April 30, 2017, when only about half of the approved IDA financing had been disbursed. The Level 2 restructuring involved changes in components and costs as well as the implementation schedule, adjustments in the results framework, reallocation among disbursement categories, and extension of the closing date. The original closing date of April 30, 2017 was extended by one year and the project closed on April 30, 2018. 3. Relevance of Objectives Rationale Country Context At project appraisal in 2013, Comoros was a fragile country with huge development challenges. This condition remains today, although its poverty rate is low compared to its regional peers. Its gross domestic product (GDP) per capita was US$645 in 2015, and the country ranked 159 out of 188 countries on the 2015 Human Development Index. The country’s economy is small, undiversified, and constrained by poor public service quality, including electricity. Since its independence in 1975, Comoros has experienced 20 coups or attempted coups, frequent changes in government, and a strong separatist movement in Anjouan, all of which have contributed to political fragility. The country’s weak institutional capacity persists as a major bottleneck in the ability of the Government of Comoros (GOC) to implement development projects and deliver basic services. When the project was appraised, the country had just emerged from a program of international debt relief, hence the Government and international financial institutions were putting a high priority on the commercial and financial performance of state-owned enterprises (SOEs). At appraisal, the country’s moderate growth path was being constrained by a severe crisis of high costs and extreme unreliability in the electricity sector. Sector Context The two state-owned electricity providers are: (1) MA-MWE - the Water and Electricity Utility in Comoros (Gestion de l’Eau et de l’Electricité aux Comores), which provides electricity to the islands of Grande Comore and Mohéli); and Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) (2) EDA - the Power Utility in Anjouan (Electricité d’Anjouan), which serves only the island of Anjouan. At appraisal, these utilities were able to provide electricity services only a few hours per day. Most households depended on kerosene and candles for lighting and spent a large part of their income on batteries to power radios and small appliances. Health centers were unable to provide adequate medical care. Power cuts made water supply unreliable even in the capital. Businesses remained uncompetitive because of unreliable supply despite electricity tariffs of over US¢20 per kWh that were significantly above the regional average of US¢15 per kWh. The country’s decades of fragility, very limited institutional capacity, and lack of investments have resulted in high electricity costs and dismal service delivery. High costs are the result of high system losses, poor collection rates, costly generation, and poor governance. The electricity SOEs are in severe financial distress: two thirds of electricity generated remain unpaid, leading to a steady decline in cash flow, lower fuel purchases than needed, expensive diesel-based power generation, deteriorating operations & maintenance, saturated and poorly maintained transmission and distribution infrastructure, and a vicious circle of consumers unwilling to pay due to poor service. Government Strategy The project’s objective is highly relevant to the GOC’s strategy. Based on a World Bank-supported energy diagnostic, GOC developed and adopted its energy sector strategy in the Sector Policy Note entitled Document de politique de l’énergie électrique et des produits pétroliers de l’Union des Comores. The Note highlighted the need for (a) comprehensive corporate governance reforms at MA-MWE; (b) investments in renewable generation including micro-hydro, wind power, and geothermal energy to reduce the reliance on thermal power generation; (c) reforms of the sector legal and regulatory framework including on tariffs and purchasing power parity; (d) upgrading of petroleum storage facilities to improve safety and security of supply. The PDOs remain relevant to GoC’s Strategy for Accelerated Growth and Sustainable Development 2015–2019 (Stratégie de Croissance Accélérée et de Développement Durable), which asserts that the country’s difficulty in attracting investors is partly due to the recurrent shortages of water and electricity. Bank Strategy The project is the World Bank’s first financial engagement in the energy sector of Comoros. The PDO is highly relevant to the latest Country Partnership Strategy (CPS) for FY2014–2017, extended to FY2019, which indicates that the acute power supply situation is due to several factors “all linked to the operational and technical performance of MA-MWE” (page 23). The CPS aimed to tackle the commercial and financial performance of MA-MWE under Pillar 2 (‘Shared growth and increased employment’), CPS Objective 9 (‘Improvement in the commercial and financial performance of the electricity sector’). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) EFFICACY_TBL OBJECTIVE 1 Objective To contribute to the improvement in the electricity sector's commercial and financial performance. Rationale Theory of Change In assessing the project’s theory of change, it is important to take into account the respective roles of the main international donors to Comoros. The African Development Bank has taken the lead role in financing physical infrastructure, mostly through new generators and grid rehabilitation. The World Bank was to lead the support for improvements in utility performance. Finally, the European Union was to help in developing small-scale solar capacity. Within the World Bank’s specifically defined role, the project’s technical assistance activities (external experts, MIS, metering equipment, protection of meters, awareness campaigns) were expected to help in specifically addressing electricity theft and unpaid bills. In turn, positive results in these areas would be expected to lead to helping mitigate massive revenue losses at the utility level and relieve the fiscal strain at the sector level. Figure 1 of the ICR articulates clearly, logically and credibly the causal chain between the project’s inputs and its intermediate and final outcomes, although it would have been more complete if the constraining factors had been also indicated. For example, more effective metering and an operational MIS could be directly linked to improved collection rates and reduced system losses. Attribution is also strong, given the aforementioned roles assigned to the various aid agencies, with the World Bank specifically designated to help address commercial and financial performance issues. Outputs The following outputs were fully achieved as targeted:  A commercial and financial recovery plan was prepared. Correspondingly, commercial and financial software were procured and installed, and made operational. 4,000 prepaid meters were installed. This target was added when the project was restructured.  The Poverty and Social Impact Analysis (PSIA) was completed. The study resulted in a better understanding of the social impacts of electricity outages. It also expanded public knowledge on issues related to social imbalances that are associated with tariff increases, and the undifferentiated enforcement of arrears payment. The PSIA was instrumental in enabling senior managers to advocate successfully against unpopular measures, e.g., those that would result in small revenue impacts but have significant negative effects on the poorest and most vulnerable households. The following outputs were partially achieved:  Only the prepaid meters were installed, out of the original target of 80 network meters and 4,000 protection boxes, and 1 fuel metering equipment. The rest of the equipment was dropped during the project’s restructuring due to the reduction in project funding that resulted from the depreciation of the SDR against the Euro (ICR, page 16). Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) Only a subsidy study was completed. A tariff study, the drafting of an energy sector policy, and new planning processes were dropped during restructuring, in order to enable more funds to be reallocated to priority activities linked to improving commercial and financial performance. Several activities that had not yet started were also dropped, including a study on the institutional and regulatory setting, a study to set up a methodology for the collection of energy statistics, a study to review the fuel supply chain and diversification operations, as well as a communication campaign for the Ministry of Energy, These activities originally comprised about 20 percent (US$0.9 million) of the original total cost, which at restructuring was adjusted to US$0.10 million. Outcomes The following outcomes exceeded their targets:  The combined cash recovery index of MA-MWE and EDA increased by more than 50 percent. Total cash recovery increased from 30.25 percent to 48.90 percent (49.40 percent for Grand Comore and 43.00 percent for Mohéli), which is 5.2 percentage points higher than the expected level of 43.70 percent based on the project’s combined targets on collection and electricity losses. (ICR, paragraph23, page 14) The collection rate of MA-MWE (based on a sliding 12-month average) improved significantly. From a baseline of 55.0 percent, the cash collected as a percentage of billed electricity achieved 77.4 percent at project closing, thus exceeding the target of 70.0 percent. For MA-MWE, the commercial results for the year 2017 show a collection rate at 78.3 percent on Grand Comore and 67.6 percent on Mohéli, hence the average of 77.4 percent for the company overall. The overall result is 7.4 percentage points higher than the target. (ICR, paragraph 23, page 14) These financial performance improvements resulted from: (i) changes in commercial procedures; (ii) the installation of 4,000 mobile payment- compatible, tamper-proof prepaid meters; (iii) institutionalization of GOC’s direct payment to MA-MWE for electricity used by the administration and public services; (iv) debt settlement and settlement of cross-arrears between the GOC-owned fuel supply company, MA-MWE, and the State; (v) MIS implementation; and (vi) a load management system, which helped to improve client relations and bill payment discipline.  The electricity losses per year in the project area was also reduced above target. Compared to a baseline of 45.0 percent and a target of 37.5 percent, electricity losses were reduced by almost a quarter to 36.8 percent. For MA-MWE, the commercial results for the year 2017 show a system loss rate at 36.8 percent on Grand Comore and 36.4 percent in Mohéli, averaging 36.8 for the company overall. This result is 0.7 percentage points lower than the target. (ICR, paragraph 23, page 15) These improvements resulted from changes in commercial procedures—including improved monitoring of large medium-voltage customers—and MIS deployment at MA-MWE. The number of direct beneficiaries—defined as the number of MA-MWE and EDA household electricity consumers—was exceeded significantly. Compared to the target of 200,000 consumers (of which 50 percent female), 354,613 households benefited (while meeting the 50 percent female target). The following outcome was dropped at restructuring: Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659)  Although the subsidy study was completed, the adoption of new utility governance arrangements and tariff and subsidy framework was not implemented under the project. GOC decided to drop this indicator from the project since associated measures would be pursued under another project financed by the African Development Bank. The funds for these activities were reallocated to the project’s component to strengthen commercial and financial performance, thus enhancing the results (both outputs and outcomes) that were fully achieved or exceeded, as indicated above. The ICR (paragraph 25, page 16) added two outcomes that were not part of the original Results Framework:  Government subsidies required to maintain adequate supply were reduced as a result of the improvement of MA-MWE’s commercial performance. The State’s combined explicit and implicit subsidies to the utility—that is, the sum of direct subsidies and residual losses absorbed by the utility was equivalent to 0.4 percent of GDP on average during 2015–17—were much lower than the level of 1.7 percent of GDP on average during 2011–13 observed at appraisal. The additional cash flow available for fuel purchases allowed MA-MWE to almost double power supply over the period of implementation. Power supply increased from 39 GWh to 69 GWh, bringing power supply much closer to the economy’s needs and reducing the share of unserved demand. Rating Substantial OVERALL EFF TBL OLD Rationale Overall Efficacy Rating Substantial 5. Efficiency Economic Analysis At the project’s completion, the economic internal rate of return (EIRR) was calculated at 18 percent and the net present value (NPV) at US$1.2 million using a 12 percent discount rate (the NPV is US$3.8 million at 6 percent discount rate). This exceeds the appraisal estimate of 16 percent EIRR and NPV of US$0.6 million NPV using a 12 percent discount rate. The project’s main economic benefit is greater energy efficiency, with the improved revenues collection raising the actual price paid by users per kWh, which in turn provides incentives for more efficient use of electricity. The economic analysis at both the appraisal and completion stages assumed that the change in the collection rate comprised (a) two-thirds resulting from a purely financial transfer from existing consumers to MA-MWE and (b) one-third resulting from more efficient energy use, thus making incremental Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) supply available for users who would not have been served otherwise. The ICR (paragraph 32, page 18) recognizes that institutional and operational factors may have contributed into the improvements in bill collection and points out that it is not technically possible to assess the exact contribution of efficiency gains to the increased collection rate. The main financial benefit of the project is to restore payment discipline, resulting in an incremental revenue transfer from electricity users to MA-MWE, which has made MA-MWE less dependent on Government subsidies. The project’s contribution to a significantly higher power supply has enabled MA-MWE to acquire larger volumes of fuel for production. Consistent with gains in end-use efficiency, the country’s overall growth in electricity consumption has slowed down to 4 percent per year during the 2013–2017 period, compared to 9 percent per year during 2009–2013. In summary, with a relatively small amount of financing, the project has achieved significant commercial and financial improvements. Administrative and Operational Efficiency Project implementation was significantly delayed. Procurement was slowed down by the new government’s frequent and disruptive changes of the senior management in MA-MWE. The submission of critical financial reports and commercial and performance data were delayed due to the absence of an administrative and financial senior manager at MA-MWE. The disruption from the frequent changes in MA-MWE’s senior management also resulted in major delays in implementing the Recovery and Development Plan (RDP), which remained incomplete. These delays led to the necessity of extending the project's closing date. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 16.00 Appraisal  75.00  Not Applicable 18.00 ICR Estimate  80.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the project’s objective is high. The efficacy of achieving the PDO is substantial. Efficiency is substantial. The overall project outcome is rated satisfactory. This ICR Review did not apply a split rating, in line with the IEG Guidelines (page 47) indicating that “If activities were re-aligned or resources were re-allocated to change the target or beneficiary coverage, and the ICR made a convincing case that the level of ambition or difficulty was similar, then this can support a decision not to apply a split rating and instead assess the project based on objectives under the revised coverage or the revised definition of beneficiaries.” In this project’s case, the PDO was not changed, the level of ambition was the same Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) (with governance-related goals pursued under an African Development Bank-financed project instead), and resource realignment was the main effect of the project’s restructuring. Moreover, the main thrust of this relatively small project’s PDO was to contribute to improvements in commercial and financial performance, not to achieve major sector reforms per se, as was reflected in the respective shares of the various components. That said, the ICR’s split rating is valid in its assumptions and methodology, which led to the same result that the overall project outcome rating is satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Technical Risks. The project’s main achievement—the new MIS and commercial management system for MA-MWE—is at significant risk because the one-year duration of the IT provider’s maintenance warranty is insufficient. Even a small incident could force the utility to roll back the newly installed IT system and infrastructure. Given the limited local capacity, the MIS maintenance warranty clearly needed to continue beyond the project’s April 2018 closing date. However, the Bank’s operational policy does not allow for an advance payment of the warranty extension against the provision of a bank guarantee by the software provider to secure satisfactory performance. There was no clear resolution to this issue by the time the project closed. The sustainability of project results is also threatened by the departure of international experts who played critical roles in applying best practices to operational and commercial procedures in the utilities. The experts departed and were not replaced due to the lack of Government funds. Commercial Risk. The utilities continue to rely on costly fossil fuels for power generation, which makes their commercial and financial performance vulnerable to cuts in Government fuel subsidies (as in 2015). Institutional Risk. The utilities continue to lack adequately trained staff. There is also a persistent lack of transparency and competition in the recruitment of senior managers, which could hinder commercial and financial performance improvements. Governance Risk. There is slow progress in institutionalizing good sector governance, which is a major risk since some of the project’s achievements could be reversed by incoherent investment planning and implementation. These risks may be partially mitigated through a planned follow-up project with an IDA allocation of US$20– 36 million. The project would finance some transition from a costly fossil fuel-based generation infrastructure toward the greater use of renewable energy sources as well as network upgrades to further improve MA- MWE’s financial and operational performance. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) 8. Assessment of Bank Performance a. Quality-at-Entry The World Bank assisted in preparing an energy sector diagnostic in 2011-12. The project’s design was focused and straightforward, focusing on key sector bottlenecks, and tailoring the activities and targeted results accordingly. It adequately took into account the fragile country environment, the respective areas of focus of the various development partners, and the client’s limited implementation capacity. The design was also adaptive and highly flexible, reflecting the best practices for operations in FCV contexts. The specific features included: (a) a realistic selection of objectives to facilitate strong and focused client ownership; (b) a deliberately lean program focused on billing and bill collections; and (c) close attention to implementation arrangements. However, there were two moderate shortcomings. First, governance risks were underestimated at the design stage. There were inadequate mitigating measures to address the failure in the sector’s governance framework when the Government did not meet the commitments it made during project preparation. As an example, the ICR (paragraph 51, pages 22-23) indicates that "after a first meeting of the Steering Committee in June 2015 and the signature of the amendment of the performance contract between MA-MWE and the GoC in March 2016, the Government did not monitor the execution of the contract or called for new meetings of the SC" ,which resulted in delays to several key activities under Component 2. Second, the project was also overambitious in its timelines for procurement and implementation. A less ambitious approach would have been more appropriate, given the limited project funding, MA-MWE’s inadequate implementation capacity, DGEME’s weaknesses in sector governance. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR (paragraph 69, page 26): “The World Bank responded to emerging issues through just-in-time advice, technical assistance…” For example, the PSIA resulted in several benefits, as described in Section 4 above on Efficacy. The project’s level 2 project restructuring and one-year extension was timely. The World Bank alerted the client and proposed constructive solutions on the following issues: (a) the disruption and delays in procurement and management decisions due to the frequent changes in the head of the PIU and MA-MWE’s senior management team; (b) the negative impact of the cutbacks of state fuel subsidies on electricity supply in 2015 and 2016; (c) MA-MWE’s financial vulnerability due to the lack of incentives for the independent contractor of the new diesel generators to minimize fuel consumption; and (d) high electricity cost of production and possible additional World Bank’s engagement in system rehabilitation and solar photovoltaic installations. Supervision missions were held twice a year and included the right skills mix. The Bank’s field presence increased significantly during project implementation, which facilitated closer implementation support. Starting initially with only a local operations specialist and a consultant, the country office expanded to include four staff, which improved the Bank’s policy dialogue and project Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) implementation follow-up. Issues were identified in a timely manner and time-bound action plans were recorded in the Implementation Status and Results Reports (ISRs) and aide-memoires to facilitate follow- up. The two TTL transitions did not negatively affect the project. Global Practice collaboration within the Bank was also useful. Cross-sectoral expertise was brought in to assist in SOE debt management and public expenditure management, which helped improve financial management at MA-MWE. The ICR (paragraph 72, page 27) indicates that “the project was adequately supervised despite exceptionally difficult circumstances of project implementation in a volatile and disruptive governance context.” This included four different project coordinators, four Managing Directors (MD), four commercial directors, three technical directors, and three Chief Financial Officers of the utility. The MD and project coordinator changes were particularly disruptive because they slowed down ongoing utility reforms and led to month-long interruptions of procurement activities as the Government insisted that only MA-MWE’s MD could be accredited to sign contracts and procurement documents. In addition, donor coordination was also weak. Some activities originally under the project were shifted to other donors, but according to the ICR (paragraph 53, page 23): "The anticipated mutual reinforcement of the project’s commercial and financial results and AfDB’s technical improvement (rehabilitation of generation, fuel storage, and distribution network) never materialized." Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s indicators were adequate and selected appropriately. The M&E framework captured relevant results. However, its design would have benefited from the inclusion of intermediate indicators related to MA-MWE’s financials. (ICR, paragraph 58, page 24) Although commercial performance and governance were adequately captured in the results indicators, financial performance was only indirectly covered. Moreover, the PAD erroneously defined the results indicator for project beneficiaries as household connections; instead, the target figure should be individual beneficiaries, which is a much larger number than household connections. b. M&E Implementation Despite limited data availability, the measurement of project implementation progress based on commercial performance indicators was timely and adequate. All other indicators were qualitative and Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) binary (mostly ‘Yes/No’), which the PIU measured continuously and adequately. (ICR, paragraph 59, page 25) c. M&E Utilization The M&E framework regularly collected data which was not previously available, thus significantly improving data availability for decision making, both for GOC and the Bank. This is especially true for the utility’s commercial performance indicators. (ICR, paragraph 60, page 25) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project did not trigger any Bank safeguard policies. It was assigned a Category C for environmental assessment purposes. b. Fiduciary Compliance At the project’s closing date, financial management arrangements were compliant, all audit reports have been received, and the latest independent audit of the project’s financial statements, expenses and balance sheet received and unqualified opinion. Earlier, the project had faced delays in the submission of project audits. The last ISR, for example, rated financial management risk as Unsatisfactory because of delays in the submission of project audits and concerns regarding the risk of failing to implement the project’s budget plan in the absence of back-up project management and financial management persons with the authority to sign off and provide continuity in PIU’s daily operations. According to the ICR (paragraph 63, page 25), these issues were resolved by the time the project closed. Procurement. Project procurement complied with the Bank’s guidelines. However, procurement was slow and caused significant delays in implementation. Although the PIU’s procurement staff were training in the Bank’s guidelines for contract preparation and bidding, delays nonetheless resulted from the frequent Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) changes in the head of the PIU, the lack of staff, and GOC’s multiple levels of approvals required for most contracts. According to the ICR (paragraph 64, page 25), despite the delays, there were no issues of procurement non-compliance during project implementation. c. Unintended impacts (Positive or Negative) A positive, unintended impact was that--aided by lower global fuel prices--the project helped to create a significant fiscal space by reducing the need for budgetary transfers to state-owned enterprises while maintaining the ability of the utilities to operate. According to the ICR (paragraph 42, pages 20-21): "The sum of the GoC’s subsidies to the utility and MA-MWE’s losses equivalent to 0.4 percent during 2015–17 were much lower than the level of 1.7 percent during 2011–13. This freed up fiscal resources to spend on human development and other GoC priorities." d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons were derived from the project’s implementation experience (with some paraphrasing by IEG): The World Bank’s analytical engagement and strong client buy-in are particularly important in fragile and low-capacity countries. MA-MWE’s Recovery and Development Plan (RDP), prepared during appraisal and adopted by the GOC soon after the beginning of the project, ensured that both scope and design of the project were targeting the most critical levers for project success. Moreover, the key parameters of the RDP subsequently served as the basis for the performance Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) contract between MA-MWE and the GOC adopted a year later. Without the analytical foundation and official adoption of the RDP, the project would neither have the needed relevance nor the client buy- in to successfully pursue the most critical utility reforms. Simple, well-targeted interventions based on proven solutions to critical operational challenges can have significant positive impacts within a short time frame, even in the most difficult country context. The simple project design and avoidance of activities that are not critical to achieving the PDO were instrumental in achieving results within a difficult governance context. An even more focused approach exclusively aimed at utility governance may have allowed a faster and more effective achievement of the PDO. If adequately communicated and targeted, measures to reduce electricity theft and nonpayment can yield large financial benefits without triggering significant opposition from vested interests. The project’s support of targeted communication campaigns was key in MA- MWE’s successful reduction of nontechnical losses. Women’s groups can be effective in fostering positive change by appealing to social norms and individual responsibilities, especially in an FCV context with weak enforcement of property rights and chronically poor payment behavior for public services. The engagement and empowerment of women’s groups to lead campaigns against electricity theft and meter fraud was particularly powerful within the context of the matrilineal Comorian society. If it is not explicitly addressed, the lack of provisions for IDA-funded warrantee contracts and maintenance services that extend adequately beyond a project’s closing date can become a major threat to project sustainability. The World Bank’s inability to finance long-term warrantee contracts or maintenance services beyond a project’s closing date is a major limitation that needs to be addressed appropriately during project preparation. Warrantees of MIS and related hardware typically last 5–10 years, hence projects must either be extended accordingly or governments have to credibly ringfence the necessary funds in their budgets. However, for typical FCV environments such as the Comoros, the latter is not an option due to the perceived risk of Government default on contract payments. In an FCV context, the intervention of resident experts can make the difference between project success and failure. The intervention of experts must be well prepared and carefully managed to ensure: (a) the definition of a robust executive mandate including decision-making authority and detailed key performance indicators; (b) the recruitment and embedding of the right experts within the client organization; and (c) robust provisions for retaining experts and transferring knowledge. Teams tend to underestimate the complexity of these challenges while overestimating the capacity of PIUs to meet these requirements. In this project, the experts’ value added was Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) reduced by GOC’s categorical refusal to grant experts more than just an advisory mandate. The lack of a managerial role made the experts vulnerable to the frequent change of MA-MWE’s senior management. Given the PIU’s lengthy recruitment and unsuccessful attempts to hire financial and technical resident experts, the services of a specialized recruitment agency were utilized. Although costly, suitable experts were identified sooner, contracts were negotiated faster, and the experts were retained or quickly exchanged as needed. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well prepared and thorough. The country, sector and FCV context was well articulated and provided solid justification for the project. The ICR was evaluative and candid in its analytical approach. A large emphasis was placed on providing concrete evidence behind the intermediate and final outcomes; in this regard, it identified two additional outcomes that went beyond those in the original results framework. The figure and tables provided were useful in helping understand this relatively small but complex institutional strengthening and capacity-building project. Albeit long, the ICR put a lot of effort in delineating the project’s implementation record and how specific issues evolved and were resolved. The ICR’s format and internal substance within each section complied generally with the Bank’s guidelines on ICR preparation. The lessons, particularly those related to the Bank’s convening role and the importance of cross-sectoral collaboration (i.e., between the Bank’s energy and social protection sectoral staff) were valuable and had broad replicability in FCV setting. Annex 3, page 38 of the ICR did not provide the original amounts per component at the time of approval, although the figures were on Table 1, page 9 of the main text. a. Quality of ICR Rating Substantial Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ESRP (P131659) Page 16 of 16