Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) Report Number: ICRR0021743 1. Project Data Project ID Project Name P150113 Tonga Cyclone Reconstruction Country Practice Area(Lead) Tonga Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-54770,IDA-H9620,TF-17580 30-Jun-2018 12,740,914.88 Bank Approval Date Closing Date (Actual) 28-May-2014 30-Nov-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 13,800,000.00 1,800,000.00 Revised Commitment 13,759,590.39 1,762,032.72 Actual 12,743,134.13 1,762,032.72 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Stephen Hutton Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objectives (PDOs) as stated in the Financing Agreement (Schedule 1, page 5) and the Project Appraisal Document (PAD, page 4) were; Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) (i) Restore housing, community facilities and basic services to the affected population of Ha'apai: (ii) Strengthen the country's resilience to natural disasters. The revised PDOs as stated in the restructuring paper of June 28, 2017 were: (i) Strengthen the climate resilience of housing and selected community facilities for the affected population of Ha'apai; (ii) Strengthen the government's preparedness to recover from natural disasters. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 28-Jun-2017 c. Will a split evaluation be undertaken? Yes d. Components There were three components (PAD, pages 6-9). 1. Housing and Key Community Facilities Repair, Reconstruction and Climate Resilience. Appraisal estimate US$11.20 million. Actual cost US$9.29 million. This component aimed at reconstructing the damaged houses and community facilities. There were four sub-components.  providing contractor-built two-room climate-resistant dwelling for 200 vulnerable households (vulnerable households defined as households who were unlikely to recover from the effects of the disaster event on their housing without special assistance),  providing subsidies for activities using the Self-Supported Recovery (SSR) approach (Under this approach, eligible households were responsible for reconstructing the damaged houses). Activities in this component: (i) rehabilitating damaged houses of 350 households; (ii) repairs and retrofitting houses with reparable damages for 400 households; (iii) small retrofitting for 150 houses with undamaged buildings. The SSR approach was substantially scaled back during implementation, and instead the design supported contractor-build housing.  providing logistical support through purchase of a barge for the Ministry of Infrastructure for transporting reconstruction materials to project areas. 2. Technical Assistance and Training for Enhanced Disaster and Climate Resilience. Appraisal estimate US$1.50 million. Actual cost US$1.74 million (including a contingency subcomponent of US$0.30 million). This component provided technical assistance for implementing component one activities and for Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) improving the government's long-term recovery program for disaster management. There were five sub- components.  Technical assistance for: (i) establishing an on-site one-stop recovery center; (ii) preparing pre- approved building plans; (iii) implementing a communications strategy: and, (iv) installing mobile advisory units.  Training tradespeople, building inspectors and other stakeholders on climate-resilient building techniques.  Updating the National Building Code of Tonga and implementing a public awareness campaign.  Mapping of coastal hazards for reconstruction planning and resilience building within the Ian- affected areas. This activity was dropped as there were no alternate sites on the island for moving houses, that conformed to the existing zonal rule (building within 20 meters of the coastline).  Technical assistance for establishing the disaster recovery framework through: (i) documenting lessons from past reconstruction projects: (ii) updating legislation and institutional arrangements: (iii) developing an operational manual for recovery implementation arrangements; (iv) consultation with stakeholders on the draft Housing Recovery and Reconstruction Policy prior to Cabinet adoption: and (v) capacity building for improving post-disasters and damage assessments. 3. Project Support. Appraisal estimate US$1.10 million. Actual cost US$1.74 million. This component provided technical support for construction and support for fiduciary management, social and environmental oversight and monitoring and evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate US$13.80 million. Actual cost US$12.76 million. The actual expenditure on component one activities was 17% lower than estimated at appraisal due to a combination of factors, including scaling back of activities associated with rehabilitation/reconstruction of houses (due to the shift from self-construction to owner-construction) and exchange rate changes during implementation. The costs of component two and three activities were 16% and 58% higher than appraisal estimates, respectively. The increase in cost of component two and three activities was met through reallocation of spending between components. Project financing. The project was financed by an IDA credit of US$6.00 million and IDA Grant of US$6.00 million. There was co-financing of US$1.80 million from the Global Facility for Disaster Reduction and Recovery (GFDRR) Trust Fund. With this, the total financing for the project was US$13.80 million. Amount disbursed US$12.74 million. The amount disbursed in US$ terms was lower than at appraisal, due to the exchange rate changes between the US$ and SDR during implementation. Borrower contribution. Appraisal estimate US$2.09 million. Actual contribution US$0.78 million. Dates. The project approved on May 28,2014, became effective on July 1, 2014 and was scheduled to close on June 30, 2018. Other changes. There were three project restructurings (one Level 1 and two Level 2). Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113)  The first Level 2 restructuring on December 14, 2016, extended the closing date of the Trust Fund (TF) from December 31, 2016 to June 30, 2018, to align the closing date for the TF with that of the IDA Credit/Grant.  These changes were made with the Level 1 restructuring on June 28, 2017: (i) The PDO was modified (discussed in section 3); (ii) The number of beneficiaries was reduced following the beneficiary selection process: (iii) Activities pertaining to basic services were dropped and activities pertaining to mapping of coastal hazards were dropped for reasons described above: (iv) Activities associated with supported self-recovery funding were scaled back; and (v) The revised PDO was made more specific regarding building disaster recovery capacity.  The Level 2 restructuring on June 17, 2018, extended the closing date by five months for completing activities associated with disaster recovery preparedness, that had been delayed due to the disruption caused by Cyclone Gita in February 2018. The project closed on November 30, 2018. Split rating. Since the PDOs were changed through a Level 1 project restructuring, this review is based on a split rating of objectives, when 80% (US$11.01 million) was disbursed before restructuring and after restructuring when 20% (US$2.74 million) was disbursed. 3. Relevance of Objectives Rationale Original Objectives. The cyclone Ian (TCI), that hit the South Pacific region in January 2014, had a devastating effect on Tonga. The cyclone passed directly through the Ha'apai Island Group (62 dispersed islands north-east of Tongatapu) in Tonga. Of the 6,616 people in Ha'apai, about 5,500 people were seriously affected. The combined physical loss and economic losses were estimated to be around US$49.50 million (about 11% of Tonga's GDP). Housing, community facilities, and basic services were heavily damaged by the cyclone, and so restoring these facilities was an important part of disaster reconstruction. Strengthening Tonga’s resilience to natural disasters was important, given high disaster exposure and vulnerability and weaknesses in existing disaster risk management capacity and systems. Government strategy. The objectives were well-aligned with the government strategies. The National TCI Response Plan, endorsed by the cabinet on January 30, 2014, outlined the immediate relief and recovery priorities in the housing sector (the sector that bore the brunt of damages and losses from the cyclone). The plan highlighted the need for reconstructing houses and assisting affected families move out of tents and evacuation centers into safe housing with adequate water and sanitation facilities. Tonga's Strategic Development Framework for 2011-2014, called for integrating disaster risk management and climate change into program planning. The National Infrastructure Investment Plan for 2013-2023 underscored the need for integrating Climate Change Adaptation and Disaster Risk Management in infrastructure planning. Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) Bank strategy. The PDOs were well-aligned with the Bank's Country Assistance Strategy (CAS) for 2011- 2014. The second pillar of the CAS highlighted the need for strengthening environmental and disaster risk management and climate change adaptation through rehabilitating damaged houses and improving the capacity for disaster management. The PDOs were aligned with the Bank's "Regional Partnership Framework" of 2016, covering nine Pacific Island countries, including Tonga. This framework emphasized the strategic importance of addressing climate risks under " Focus Area Three. Protecting incomes and livelihoods”. The original PDOs - restoring houses and community facilities that were damaged - was framed in terms of outputs, rather than outcomes (such as restoring services provided by these assets). There was a disconnect between project activities and intended outcomes of improving resilience to natural disasters at the national level in Tonga, given the relatively limited project size. There were issues with the original PDO, when the local population interpreted the PDO to mean that housing would be restored directly by the project, while the design envisioned providing housing through a mix of directly and indirectly supported self-reconstruction efforts. Revised objectives. The Bank’s original intention was that households living in houses of absentee house- owners, when the cyclone struck (including households living under informal arrangements or unregistered houses) would be eligible for support, through subsides for self-supported reconstruction efforts. Some households interpreted the PDO as meaning that the project was to provide them with occupancy rights in houses of absentee house-owners for 30 years. This interpretation was neither acceptable to the government nor to the absentee house-owners. The project design was revised during implementation to clarify that households were ineligible for occupancy rights they did not have under the original project, but that they would have secure occupancy rights for four years. The revised PDO was reframed to cement this change and avoid an interpretation that the objectives guaranteed housing to all affected parties. However, the revised PDO also introduced some vagueness, as it avoided any mention of restoration or reconstruction even when the project activities were clearly designed as reconstruction. The problem was not that houses in Tonga needed to have strengthened climate resilience (as if this were retrofits to existing facilities) – it was that houses had been destroyed and needed to be rebuilt. The revised PDO also narrowed the scope of the second part of the PDO, by moving away from the general national goal of building resilience to natural disasters, to a more localized goal of strengthening Ha’apai’ s preparedness to recover from future natural disasters. This better reflected the remaining activities that could feasibly be completed at project closure, but was less ambitious, and did not reflect the need for resilience building outside of preparedness for disaster recovery. The relevance of objectives is rated Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) EFFICACY_TBL OBJECTIVE 1 Objective Restore housing, community facilities and basic services to the affected population of Ha'apai Rationale Theory of Change. Original design. Self-supported reconstruction of houses and contractor-executed works for vulnerable groups, reconstruction of community facilities, and providing basic services were expected to contribute to the restoration of houses and community facilities to the affected population of Ha’apai. Supported self- construction was adopted as an approach shown in other countries to accelerate reconstruction, lower costs, and reduce risks. Technical assistance activities, such as preparing building plans at the national level, updating the National Building Code, mapping of coastal hazards for reconstruction planning and training building inspectors on climate-resilient techniques, were likely to contribute to the intended outcome of strengthening Tonga’s resilience to natural disasters. Government’s lack of commitment to self-supported reconstruction efforts led to largely dropping this approach and a shift towards contractor construction of houses during implementation. This change consumed project resources and contributed to: (i) dropping activity associated with repair of the Lifuka market and delaying water and sanitation investments: (ii) reducing the number of houses that could be reconstructed, and (iii) The government’s decision to concentrate on reconstruction activities and hold off on technical assistance activities (because of the problems in reconstruction following the scaling down of the supported self-recovery activities), contributed to slow progress on the policy front. Revised design. Project activities aimed at reconstruction of housing using contractors were relevant for the objective strengthening the climate resilience of housing and community facilities for the affected population of Ha’apai. Likewise, the technical assistance activities aimed at capacity building were likely to contribute to the narrower objective of building Ha’apai’ capacity to recover from natural disasters. Outputs (pages 13-19). 1,031 houses were built, repaired or retrofitted to climate resistant standards at closure, less than the original target of 1,100. (The original target was revised following a detailed damage assessment of the number of houses eligible for assistance) and actual delivery was larger for minor repairs on houses and smaller for reconstructed houses. This included the following.  Reconstruction of 196 damaged houses by contractors of vulnerable households (target 200 houses). No houses were reconstructed for households not classified as vulnerable using the Self-Supported Recovery (SSR) approach (target 350). Rather, the project delivered 204 one-room houses contactor build houses. The combined 400 contractor build houses were built to more resilient standards, enforced by providing pre-approved contractors with housing plans prepared by the project using approved designs. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113)  Repair and rehabilitation of 314 houses with moderate damages (target 400 houses).  Repair and rehabilitation of 275 houses with minor damages (target 150).  42 households were reimbursed for repair work undertaken prior to the availability of project funds.  Six building inspectors were trained and deployed to Ha'apai to develop repair plans, supervise repairs and inspect the work of inspectors.  The ICR (paragraph 37) notes that as envisaged at design, eligible beneficiaries were to have access to a Recovery Center and mobile advisory units, through which they would receive technical assistance from an SSR firm for building to resilient standards. Following the government's decision to hire contractors for building houses for beneficiaries, the SSR firm procurement was cancelled and support to the remaining SSR households was provided in a fragmented fashion by Building Inspectors and by a Community Liaison officer in Ha'apai.  Restoration work on the one large community facility assisted by the project - the Lifuka market - was not completed due to budget overruns on housing construction, that left a shortfall for this activity. The ICR notes that although the damage assessment and rehabilitation plan were prepared under the project, construction of the facility that was completed in July 2017, was funded by the Asian Development Bank.  18 small community facilities were rehabilitated as targeted and asbestos-containing materials were removed from the community facilities, rehabilitated under the project. Supervision of these facilities by building inspectors helped to ensure that the works would meet building code standards and that they were more climate resilient.  Water supply services was to be restored to 1,100 housing units. The ICR (paragraph 46) notes that “these services were omitted from the original housing contracts’ without a clear explanation, but were added for some houses through change orders. Water systems and guttering were installed in 125 units, significantly short of the target.  No sanitation facilities were restored for the housing units under the project, as compared to the target of 1,100.  Considering electricity as a basic service, connecting houses to electricity in 178 contractor-build houses was added at government request, during implementation . This activity was dropped during implentation for budgetary reasons. Outcomes.  The project supported improved climate resilience in housing for 5,400 beneficiaries. This was short of the target of 5, 800 beneficiaries. 27% of the beneficiaries were females, short of the target of 40%.  Restored community facilities are able to be used for religious and cultural functions.  No evidence was provided that water supply services were restored, given that only that only water tanks and guttering were installed in the selected housing units. While the project exceeded its target for minor repair work, it fell short of the target for number of reconstructed houses, due in part to the move away from supported self-reconstruction to contractor- delivered reconstruction. The one-room houses built under the project faced problems of social unacceptability. Rating Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) Modest OBJECTIVE 1 REVISION 1 Revised Objective Strengthen the climate resilience of housing and selected community facilities for the affected population of Ha'apai; Revised Rationale The outputs discussed above were also relevant to the revised PDO. The National Building Code activity was updated in a consultative manner and may lead to more resilient construction in the future, but this had not been approved by Cabinet as of project closure. Revised Rating Modest OBJECTIVE 2 Objective Strengthen the country's resilience to natural disasters. Rationale Outputs.  The Tonga Housing Recovery Policy was developed as targeted, and this has the potential to improve future post-disaster housing reconstruction, but the policy was yet to be approved by Cabinet when the project closed. According to the clarifications provided by the team, the policy identifies a range of approaches and criteria for deciding among them and promoted self- supported reconstruction, as the default approach for reasons of lower cost, speed and resilience building. Government is demonstrating a willingness to adopt this approach: after cyclone Gita in 2018, the government provided cash transfers for temporary shelter and provided the option for self-supported reconstruction. A key decision for the government is to appoint a lead agency for coordinating housing recovery, given that there is no housing ministry. Once this decision is made, the policy will need to be approved by the cabinet.  A House Identification Guide (a compilation of designs and other technical information from past Tonga housing reconstruction projects) was prepared for the Ministry of Infrastructure, to serve as a reference for future resilient housing designs. This guide was made available to the Ministry of Infrastructure when the project closed.  The activity associated with coastal hazard and risk maps for the affected islands was dropped, for lack of alternate sites on the island for moving houses, that confirmed to the existing zonal rule. This made the hazard mapping an unnecessary cost. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113)  Equipment and software for facilitating future post-disaster building assessments were acquired and equipment and protocols for building damage assessments were updated, as targeted.  The building code activities discussed above were also relevant to this objective. Outcomes. The indicators were output-oriented. The ICR (paragraph 51) states that the technical, policy and regulatory tools have increased resilience. However, there is no clear evidence that the project activities reduced Tonga’s vulnerability to disaster risks. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective Strengthen the government's preparedness to recover from natural disasters. Revised Rationale The outputs and outcomes described above were also relevant to the revised PDO. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale 1,031 houses in total were delivered under the project, as compared to the revised target of 1,100 houses (The total included 400 reconstruction of destroyed or heavily damaged houses and repairs on 631 houses with moderate or minor damages). Given project design changes and cost overruns, many activities were dropped, and the revised design ended up financing a narrow set of activities and others were delayed and so did not reach the point of cabinet approval. Overall Efficacy Rating Primary Reason Modest Low achievement Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) 5. Efficiency Economic analysis. There was no economic analysis at appraisal, given the emergency nature of the project. Lack of data precluded an ex-post economic analysis of investments in housing at project closure (ICR, page 20). However, a benefit-cost analysis for the privately-owned assets created by the project, would have been difficult, given that there was no liquid market for these assets. The ICR (paragraph 65) notes that housing reconstruction investments from previous cyclones provided some evidence of a high rate of return from reconstruction interventions. However lack of data precluded quantitative estimation of the avoided cost of disaster damages, due to more climate-resilient structures. Cost effectiveness considerations. High costs are to be expected when operating in remote parts of small states. However, even given this, the project was not cost effective, with the actual cost of two-room and one room contractor-build housing units 29% and 51% higher than the original estimates – which in turn were roughly double the costs in nominal $US terms of a similar World Bank housing reconstruction project in Tonga a decade earlier. The overruns were due to a combination of factors, including design changes and government decision to engage contractors as opposed to self-supported reconstruction, which had been expected to be more cost-effective. The cost overruns contributed to dropping activities relating to rehabilitating community facilities and providing basic services in housing units. As discussed in section four, the number of beneficiaries that received assistance for reconstruction work was lower than targeted at appraisal. Administrative issues. There were issues associated with overpayment to contractors and ineligible expenses (discussed in section 10b). Issues relating to occupancy rights and high turnover rate of the project management unit contributed to implementation delays. There was not enough consultation with the local population on the decision pertaining to the government decision to build one-room houses, that were considered to be culturally inappropriate. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original objectives. The original PDOs are rated as Substantial, in view of the disconnect between project activities and intended outcomes. Efficacy of the two original sub-objectives – to restore housing, community facilities and basic services and to strengthen the country’s resilience to natural disasters is rated as modest, given the limited achievements. Efficiency is rated as negligible, given the significant cost overruns and reduced scope of project activities. Revised objectives. The revised objectives were narrower in scope and represented activities which could realistically be completed within the remaining time frame. The revised objectives are rated as Substantial. The two revised PDOs - strengthening the climate resilience of housing and selected community facilities and strengthening the climate resilience of selected community facilities - were still rated as modest. Efficiency is negligible. Considering the ratings discussed above and weighing them by the shares of Bank disbursement before and after restructuring (0.80*2 + 0.20*2 = 2), outcome is rated as Unsatisfactory, reflecting major shortcomings in efficacy and efficiency. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome Institutional risk. There is a risk to the ongoing benefits, given that some people living in the rehabilitated houses only have occupancy rights for four years, after which the houses would revert to the control of the original owner. Although agreements from nonresident house-owners had been obtained for four years, given that construction of houses was mostly completed by January 2016, there is risk of uncertainty going forward for the housing beneficiaries targeted under this project. Government risk. There is also a risk to the ongoing benefits given that the updated Tonga Building Code and the National Housing Reconstruction Policy were yet to be approved by the relevant authorities when the project closed. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) 8. Assessment of Bank Performance a. Quality-at-Entry Given the emergency nature of the project, preparation of this project was appropriately expedited and the project was prepared in four months. This project was prepared based on the experience of prior Bank-financed reconstruction projects in Tonga (Post Tsunami Project in 2010 and the Cyclone Emergency Recovery Project in 2002) and drew upon the lessons from other large-scale Bank-financed reconstruction projects in Aceh and Haiti. Lessons incorporated at design included; (i) Unlike prior Bank- financed reconstruction projects in Tonga, this project envisioned supported self-recovery reconstruction activities for cost-effectiveness considerations, based on international practice. However, although the approach was technically sound, the approach did not have sustained government buy-in due to political factors and was discarded during implementation; and (ii) incorporating capacity-building activities of government agencies at the national and district levels of practices that could be used for future disaster recovery efforts in partnership with private sector and civil society. Several risks were identified at appraisal. These included High risks associated with the emergency nature of the project and the speed of preparation under compressed procedures, risks associated with logistics of reconstruction efforts, given the relative isolation of the Ha'apai Island Group. Overall project risk was rated as High at appraisal. The implementation arrangements were appropriate, with the Project Management Unit (PMU) based in a recovery center in Ha'apai and the main PMU based in Tongatapu (the capital and main island). Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 10a). There were significant shortcomings at design. The project design stipulated long-term occupancy rights to households living in houses of absentee house owners. This generated significant resistance from the government and from the absentee house owners. There was not sufficient recognition of the difficulty of getting land consents from absentee house owners and resolving this issue contributed to the implementation delays. Although the financing agreement fixed August 31,2014 – three months after Bank approval – as the date by which the beneficiary list should have been completed, the list was not officially finalized until August 2017, due to a combination of factors, including uncertainty resulting from the land consent process, inconsistent household and housing damage assessment data despite multiple surveys, failure to authorize an objective third part to prepare the list of beneficiaries and irregularities in the qualifications process. The lack of a final list delayed the delivery of housing solutions and created confusion.for contractors and beneficiaries. The promised timeline of eight months for completing housing reconstruction was unrealistic. The short preparation time precluded sufficient consultation with stakeholders. There were shortcomings in M&E shortcomings (discussed in section 9a). Quality-at-Entry Rating Moderately Unsatisfactory Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) b. Quality of supervision Nine Implementation Status Results Reports were filed over a five-year project implementation period (implying twice a year supervision missions). There were three Task Team Leaders (TTLs) over the implementation period. The continuity of leadership was maintained, as the TTLs in all cases had been involved as team members, prior to assuming the TTL role. The support provided by the supervision aided in addressing delays associated with activities such as identifying project beneficiaries and obtaining land consents from non-resident owners and addressing issues of political uncertainties, following the snap election in November 2017. The support provided by the supervision team helped in safeguards and fiduciary compliance (discussed in section 10). There were significant shortcomings during implementation. The Bank was unable to generate sustained political commitment to the supported self-recovery approach. The ICR (paragraph 91) notes that it should have better anticipated the political risks ex ante and managed them when they arose. During implementation, there was inadequate consultation with the local population on the decision to build one- room houses, which were considered as culturally inappropriate. As the ICR (paragraph 89) notes, “a social assessment would have helped in identifying and ensuring sensitivity to cultural factors”. The shortcomings in M&E design were not rectified during implementation (discussed in section 9a). Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The key indicators for monitoring project performance – the number of houses that were reconstructed to climate resilient standards – were output-oriented. As there was no gender results chain in the Project, the PDO level indicator “direct beneficiary of which females” was not a meaningful measurement. There were no key outcome indicators pertaining to measuring outcomes associated with restoring community facilities and strengthening the country's resilience to natural disasters. The M&E design did not consider any beneficiaries other than households, so owners and users of community facilities were not assessed for their level of use and satisfaction. There were no indicators associated with the number of tradespeople, community members and government staff who were trained under the project. Given that the baseline data for monitoring project performance was not available, this data was to be collected as part of the impact assessment during implementation. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) b. M&E Implementation Following the restructuring of the project which revised the PDOs, indicators were modified and scaled back in view of the reduction in project scope. The rapid damage assessment prepared by the government, with estimates of houses damaged or destroyed, were refined over the next six months based on additional surveys (including a household survey). The shortcomings pertaining to lack of outcome indicators at design were not rectified during implementation. Staffing transitions in both the Project Management Unit and the Bank’s task team led to gaps in monitoring and reporting on the status of beneficiary households. Although the design envisioned a management information system as the key tool to be used by the Project Management Unit for monitoring, excel spreadsheets were used as the principal monitoring tools. This approach did not allow for easy tracking, cross-referencing or validation of project data. The ICR (paragraph 99) notes that after the first 18 months of implementation, the Ministry of Infrastructure’s detailed physical progress reports to the Bank became less regular. c. M&E Utilization The ICR provides no details on whether the data was used for purposes other than monitoring project performance. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a Category B project. Two safeguard policies were triggered at appraisal (PAD, page vii); (1) Environmental Assessment (OP/PB 4.01); and (2) Involuntary Resettlement (OP/BP 4.12). Environmental Assessment. The PAD (page 14) noted that adverse environmental or social impacts were expected to be minor, as the project activities did not involve medium or heavy civil works for reconstruction activities. An Environmental Management Plan was prepared and publicly-disclosed to address Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) environmental issues during implementation. The ICR (page 28) reports that there were no major environmental issues during implementation and there was compliance with environmental safeguards. Involuntary Resettlement. The PAD (page 14) noted that there was the possibility of relocation of beneficiaries and involuntary resettlement, due to tenure security issues and establishment of a landfill site. A Resettlement Action Plan (RAP) was prepared and publicly-disclosed at appraisal to address involuntary resettlement issues. The ICR (page 28) reports that there was no land acquisition or involuntary resettlement during implementation. b. Fiduciary Compliance Financial management. An assessment of the financial management arrangements of the implementing agency conducted at appraisal, concluded that the arrangements were satisfactory. Financial management risk was rated as substantial. Mitigation measures incorporated at design included, hiring an additional accountant dedicated to the project disbursement of supported self-recovery subsidies through vouchers in tranches and direct payments to approved suppliers for providing building materials and services to the eligible households (PAD, page 34). The ICR (page 29) notes that there were some irregularities in construction contracts during implementation (over payments made to four contractors based on fraudulent documents). The ICR notes that three of these firms made reimbursements and the government reimbursed for the fourth firm. This contractor was subsequently barred by the Bank following an investigation. Other than this, financial covenants were met. Unqualified audits were submitted in a timely fashion (ICR, page 29). Procurement. An assessment of the implementing agency's ability to address procurement issues was conducted at appraisal (PAD, page . The assessment concluded that the procurement risk was Substantial, due to the limited in-house capacity to address procurement issues. A procurement plan was prepared at appraisal and this plan was to be updated to reflect project requirements during implementation. The ICR (page 29) notes that there were no major procurement issues during implementation and procurement was carried in accordance with the Bank's procurement guidelines. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) Moderately Significant drawbacks in efficacy Outcome Unsatisfactory Unsatisfactory and efficiency. There were shortcomings both Moderately Moderately Bank Performance at Quality-at-entry and at Satisfactory Unsatisfactory supervision. Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR draws the following four lessons from the experience of implementing this project, with some adaptation of language. (1) Reconstruction of housing infrastructure is necessary but not sufficient for building resilience to natural disasters. This project contributed to constructing houses that are more resistant to cyclones and hence less likely to be destroyed or damaged. The project however affected only a small portion of at-risk assets and hence had limited impact on the overall vulnerability of Tonga to natural disasters. (2) Government provision of direct reconstruction of houses can establish expectations that have long term consequences. The project illustrated the political and fiscal implications of the government having previously assumed too much responsibility for risk management in the housing sector, with public expectations that the government would directly reconstruct houses destroyed by a natural disaster. This make it difficult to get government commitment for housing supply models that differed than this historic expectation. This implies the need for the Bank to be prepared to explain and persuade government of the long-term consequences of various housing recovery project designs and to seek to rebalance risks between households and the government. It also highlights the need for developing consensus around a housing strategy prior to an emergency, as it is difficult to do so after a disaster. (3) A careful consideration of country-specific conditions (including cognizance of cultural norms and sensitivities) is required for social acceptance. This project contributed to construction of two room houses for 200 vulnerable households and given the scarcity of resources, one room houses for 350 households whose needs were deemed to be less vulnerable. This decision created social and political challenges, as the one room houses were perceived as too small and "undignified". (4) The risks/challenges associated with reconstruction in remote locations can be anticipated and proper mitigation measures can be applied. Project efforts to facilitate access to trained labor and materials would be an important precondition for self-supported or owner-driven reconstruction efforts. 13. Assessment Recommended? Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Tonga Cyclone Reconstruction (P150113) No 14. Comments on Quality of ICR The ICR is well-written and candid. It clearly discusses the move away from the self-supported recovery approach and the time delays associated with getting consent from absentee house owners. The lessons drawn by the ICR from the experience of implementing this project were thoughtful, candid and constructive. The ICR is consistent with the guidelines and appropriately conducts a split rating of objectives. The ICR also did a commendable job of addressing and explaining the reasons for project implementation challenges and results but the layout of the report and the ICR structure made it difficult to describe the story in an accessible manner. The team could consider publishing a separate lesson note in a more accessible manner, given the depth of analysis presented in the analysis. The ICR largely framed the project’s results in terms of outputs rather than outcomes, more could potentially have been done to go beyond the evidence from the project indicators. a. Quality of ICR Rating Substantial Page 17 of 17