Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) Report Number : ICRR0021103 1. Project Data Project ID Project Name P149606 Road Safety Activity under on RSDP roads Country Practice Area(Lead) Nepal Transport & Digital Development L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-18701 15-Jul-2016 7,470,000.00 Bank Approval Date Closing Date (Actual) 08-Apr-2015 15-Jul-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 7,470,000.00 7,470,000.00 Revised Commitment 7,470,000.00 7,470,000.00 Actual 6,774,924.96 6,774,924.96 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Peter Nigel Freeman Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Outcome (PDO) as stated in the Grant Agreement (Schedule 1, page 5) and in the Project Appraisal document (PAD, page 11) was: "To reduce the risk of roadway departure crashes on selected sections of the roads targeted under the Road Sector Development Project (RSDP) and to strengthen the Government of Nepal's capacity for improving road safety." Note. The RSDP was a Bank-financed project aimed at providing connectivity to the poorest and least connected districts in the country. The project started in 2008 and was completed in 2017. Page 1 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) This review is based on the two sub-objectives of the project. (i) To reduce the risk of roadway departure crashes on selected sections of the roads targeted under the RSDP: (ii) To strengthen the Government of Nepal's capacity for improving road safety. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components There were two components. One. Capacity strengthening for the Implementation of Road Safety Action Plan. (Appraisal estimate US$0.80 million. Actual cost at closure US$1.10 million). This component aimed at strengthening the capacity for improving road safety with a focus on Nepal's Road Safety Action Plan: Activities included technical assistance for: (i) operationalizing the Nepal Road Safety Council and Secretariat: (ii) legislation, regulation and skills development: and, (iii) small scale road safety pilots to identify areas for scaling up. Two. Improved physical safety of RSDP roads. (Appraisal estimate 6.67 million. Actual cost at closure US$5.67 million). This component aimed at financing installation of about 70,000 meters of crash barriers in high risk sections of RSDP roads where steep drop-offs or other hazards made road departure crashes particularly dangerous. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. (Appraisal estimate US$7.47 million. Actual cost at closure US$6.77 million). Cost of component one activities were about 38% higher than estimated and cost of component two activities were 15% lower than estimated. There was savings in cost due to the US$ exchange rate appreciation against the Nepalese rupee during project implementation. The increase in cost of component one activities was met through reallocation of funding between project components. Project Financing. The project was financed by the Global Road Safety Facility Phase 2 Multi-donor Trust Fund administered by the Bank. Amount of the Grant US$7.47 million. Amount disbursed at closure US$6.77 million. Amount disbursed was lower than the appraisal estimate due to the cost savings on account of US$ exchange rate appreciation against the Nepalese Rupee during project implementation. Borrower Contribution. None was planned. Dates. The project closing date was extended by 12 months for completing the ongoing project activities through a Level 2 restructuring on June 6, 2016. The project closed a year behind schedule on July 15, 2017. Page 2 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) 3. Relevance of Objectives Rationale Road safety was a serious concern in Nepal. In years before appraisal, statistics from the Nepal Police Traffic Directorate indicated that there were approximately 13,582 road traffic crashes between 2012 and 2013, resulting in 1,516 fatalities, 3,986 major injuries and about 8,000 minor injuries. An estimate in 2007 indicated that these crashes cost Nepal between 0.4% to 0.8% of Gross Domestic Product (GDP) annually. The incidence of road accidents varied within the country. While roughly one half of all accidents occurred in the Kathmandu valley, accidents in the valley were less severe than accidents in the regions outside Kathmandu. For instance, the rate of fatalities per 10,000 registered vehicles outside Kathmandu ranged from seven to 33 times higher than the rate of fatalities observed in the valley. One reason for this disparity within the country was the mass casualty events that resulted when a single multi-purpose passenger vehicles (such as buses),loses control and plummeted from a steep hill or mountain. The PDO was consistent with the government strategy articulated in the National Road Safety Plan (NRSP) adopted in 2013. This strategy aimed at improving the road safety environment for road users and the pillars of the action plan were fully aligned with the United Nations' Global Action Plan for the Decade of Action for Road Safety for the 2011-2020 period. The PDO was well-aligned with the Bank Strategy for Nepal. The pillar one of the Country Partnership Strategy (CPS) approved in 2014 for the 2014-2018 period underscored the need for "Increasing economic growth and competitiveness" through among other things "improved transportation connectivity, internally and with India". The CPS also highlighted the importance of improving safe road connectivity for inclusive growth throughout Nepal (especially in the most remote regions of the country) (PAD, page 10). Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To reduce the risk of roadway departure crashes on selected sections of the roads targeted under the RSDP. Rationale Outputs (ICR, pages 8-12 and pages 22-27). • 73,000 meters of crash barriers were installed on selected sections of the roads targeted under the RSDP (based on the recommendations of a Road Safety Audit of the selected road sections) at project closure. This exceeded the target of 70,000 meters. The crash barriers were installed after two layers of quality checks: one at the factory by the Department of Roads engineers and the second at the site during Page 3 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) installation by supervision consultants to ensure that the barriers met the technical specifications. • 76% of the high risk locations along the selected RSDP were protected at project closure. This was slightly short of the target of 78%. Outcomes. • Indicators were output-oriented. A simplified impact evaluation experiment was conducted at project closure to develop a rough measure of vehicle accidents after the installation of crash barriers in five vulnerable locations and to visually market and record vehicle impacts on the newly installed crash barriers. Engineers were present to record the details including information on vehicle types, occupancy, photographs and other relevant information. The results of the evaluation indicated that at least seven hits were recorded and vehicles carrying a total of 270 passengers were protected from departing from the roadway and potentially falling into the valley. In the absence of official actual crash data before and after the project, it was difficult to ascertain the extent to which the project activity contributed to the PDO. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To strengthen the Government of Nepal's capacity for improving road safety. Rationale Outputs. • Technical assistance was provided and all nine activities of the National Road Safety Plan of 2013 were completed at project closure as compared to the target of six activities. The activities included: (i) Formulation of the new Road Safety Act: (ii) developing the amendment to the Vehicle and Transport Management Act (VTMA) and Transport Management Regulation (VTMR): (iii) preparing the amendment to Public Road Act and Regulations: (iv) developing the amendment to Transport Policy: (v) Training for conducting the road safety audit: (vi) Training for the staff of the Department of Transport (DOR) engineers, police and other stakeholders: (vii) distribution of directives for professional licensing: (viii) development of a bachelors' and masters' course curriculum on road safety: and, (ix) development of the Road Safety Database policy, software and database management. • 297,000 days of employment was created during the project implementation period. This included 38,260 days of employment for women. • The Nepal Road Safety Council Secretariat (NRSCS) was officially established by a Cabinet decision. Page 4 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) The secretariat was however not operationalized and staffed by project closure. • Of the two pilots that were to be undertaken, one pilot on the rural road network was completed. The second pilot on an urban area was not undertaken in view of the complexities in the urban environment that had not been considered at design. Outcomes. • Indicators were output-oriented. Given that there is credible evidence that objective was realized, this Review rates efficacy as Substantial. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale On both objectives, the rating is considered Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic Analysis. An economic analysis (cost-benefit analysis) was conducted for the project activity (associated with the installation of crash barriers). This component accounted for 89% of the project cost at appraisal and 83% of the actual project cost at closure. The analysis was conducted using national- level data as specific-crash related information was not available from the Road Sector Development Project. Economic benefits of the project were assumed to come from savings from accident reductions after the installation of crash barriers. Three casualty severities were considered: fatalities, serious injuries and minor injuries. The main assumption was that the installation of crash barriers would reduce the incidence of fatalities by 30% and that of serious and minor injuries by 25% and 20% respectively. Given that no follow up survey was conducted during implementation to record the changes in crash rates for the project roads, the ex post analysis was conducted using the data in the impact evaluation assessment. The average ex post Economic Internal Rate of Return (EIRR) was 36.5% as compared to the ex ante EIRR of 32%. Administrative and Operational Issues. The project experienced significant delays during implementation. The delays were on account of internal factors such as lack of capacity of local contractors (which necessitated a longer period than expected for installing the crash barriers), insufficient M&E capacity of the implementing agency, delays associated with testing of crash barriers and delays associated with securing the Cabinet's approval for the establishment of the Road Safety Council. These internal factors were exacerbated by exogenous factors such as an informal embargo by India which blocked the imports of crash Page 5 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) barriers and an earthquake in April 2015, which consumed most of the management time within the Government to respond to the emergency. The project activities for the most part were completed within the extended project closing time. There were no major cost overruns. The project was able to realize cost savings, mainly on account of the US$ appreciation relative to the Nepalese Rupee during project implementation. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 89.00 Appraisal  32.00 Not Applicable 83.00 ICR Estimate  36.50 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objective for the government strategy and the Bank strategy for Nepal is rated as Substantial. Efficacy of the two sub-objectives - to reduce the risk of roadway departure crashes on selected sections of the roads targeted under the RSDP and to strengthen the government's capacity for improving road safety - is rated as Substantial. Efficiency is rated as Substantial. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Technical risk. Given that the crash barriers were installed after two layers of quality check - in a factory by the Department of Roads engineers and the second at the site during installation to ensure that it met all the technical specifications, the technical risk associated with the quality of crash barriers is rated as Modest. Institutional Risk. The institutional risk is rated as Modest, given that the government has demonstrated its commitment to road safety, by allocating 10% of road maintenance funds to support road safety features of the road network. Page 6 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) 8. Assessment of Bank Performance a. Quality-at-Entry This project was prepared based on lessons from the prior Bank financed (Road Sector Development Project (RSDP). Lessons incorporated at design included, instituting joint monthly field supervision visits by the Bank team and the implementing agency - the Foreign Cooperation Branch in the Department of Roads (DOR) (PAD, page 13). The design for installing crash barriers on selected road sections was appropriately based on the recommendations of a comprehensive Road Safety Audit carried out before project preparation. The preparation of this project was overseen by the same Task Team Leader who had worked on the Road Sector Development Project and other Bank-financed projects in Nepal and this aided in collaboration with the counterpart agencies. Several risks were identified at appraisal including High risks associated with the weak management capacity of the Department of Roads and Substantial risks associated with political instability that could affect governance issues. Mitigation measures were incorporated at design (such as strengthening the implementation capacity and joint supervision missions to the road sites by the Bank and the implementing agency team and using the DOR's e-bidding procedures to reduce bidder collusion) and the overall risk was rated as Moderate at appraisal. Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in Section 10). The project time line for implementation (initially 15 months and later extended by a year) was unrealistic, in the context of the low implementation capacity in the country context. A more realistic timeframe would have supported arrangements for better monitoring of the intervention outcomes. There were also shortcomings in M&E (discussed in section 9). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Given that this was a relatively small project, the planned implementation period was short. Four Implementation Status Reports (ISRs) were completed over a three year project implementation period. The support provided by the supervision team aided in safeguards and fiduciary compliance and the support provided by the team aided in expediting project works and resolving the many issues and problems that arose during the implementation phase (Borrower's ICR, page 48, para 49). Given the shortcomings in M&E design, it is not clear why the supervision team did not put more effort into revising the framework for monitoring outcomes that could be specifically attributed to the project, during implementation. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 7 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E indicators for both the PDOs were output oriented, due to the lack of reliable crash data. It is not clear why the team did not work with the Department of Roads to arrange for a qualified consultant to carry out ex ante and ex post crash surveys on the road segments where the barriers were installed. It is not clear why there was no attempt at monitoring the institutional strengthening component of the project for the skills development supported by the project. There were no indicators aimed at the theory of change regarding outcomes associated with road safety. The project preparation team envisaged a simplified impact evaluation experiment for evaluating the effectiveness of the crash barriers (discussed in section 4). This experiment was not an effective substitute for actual measurement of the crash data before and after the project intervention. b. M&E Implementation It is not clear why the supervision team during project implementation did not attempt either to collect outcome-based data or establish either a more robust M&E system and methodology. The ICR (page 17) notes that a new road safety database was developed to digitize road crash statistics, but those data were not used for monitoring the performance of this project. c. M&E Utilization The M&E system was used for reporting the output-level indicators and informing the management on key decisions to be made for resolving implementation issues. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a "Category B" project. Other than environmental safeguards (OP/BP 4.01), one social safeguard policy was triggered: Indigenous Peoples (OP/BP 4.01). Environmental Assessment and Indigenous Peoples. The PAD (page 29) notes environmental issues that could arise under this project included, dust pollution, construction noise, minor disruptions to traffic along work routes and health and safety of workers. The Department of Roads (DOR) was expected to prepare Initial Environmental Examinations and Site-Specific Environmental Management Plans (EMPs) Page 8 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) where required (PAD, page 29). Given that the project activities were implemented in Road Sector Development Project road corridors, there were the possibility of presence of indigenous communities in the road corridors. The Environmental and Social Management Framework (ESMF) was to include framework for Vulnerable Community Development Plans (PAD, page 28). The ICR (page 17) notes that an Environmental and Social Management Framework (ESMF) that was developed by the DOR was updated for this project and site-specific Environment Management Plans were prepared (EMPs). As the project activities were confined to right-of-way along selected road sections, no land acquisition was required. The ICR (page 17 and 18) notes that compliance with safeguards was deemed to be satisfactory during project implementation. b. Fiduciary Compliance Financial Management. The financial management arrangements were similar to that of the Road Sector Development Project (RSDP) and the Project Coordination Team at DOR had extensive experience in working with prior Bank-financed projects (PAD, page 26). The ICR (page 17) notes that the financial managements were appropriate and there was compliance with the requirements as stated in the Grant Agreement during project implementation. Procurement. The Project Coordination Team in the Department of Roads (DOR) had implemented the RSDP and was familiar with Bank's procurement guidelines and procedures (PAD, page 27). The ICR (page 18) reports that despite some delays associated with procurement of crash barriers, there was compliance with procurement issues and there were no instances of mis procurement. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- Page 9 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) 12. Lessons The ICR (pages 19-21) draws the following main lessons from the experience of implementing this project, with some modification of language. (1) A simple design can be useful for executing projects in countries with relatively weak capacity. In this project, the design was simple and based on a proper assessment of the capacity of the Project's stakeholders. (2) The implementation agency's capacity and ownership can be key to successful operations. In the case of this project, the Department of Roads (DOR) and the Ministry of Physical Infrastructure and Transport (MOPIT) had prior experience in executing Bank-financed projects and were familiar with the Bank's requirements. This in conjunction with strong ownership of the implementing agency aided in completing the project and in completing the civil works on site. (3) For a road safety operation, soft components may be as important as the hard components and both components may be required for sustaining the overall objective of improving road safety. The experience with this road safety project showed that investments in road safety operation can provide leverage to influence and drive policy and regulatory reforms later. (4) Stand-alone road safety operations can have a chance of success in improving road safety compared with having a component or subcomponent focused on road safety under a road project. Despite a relatively small investment, this project attracted the attention of the authorities in the implementing agency and this was demonstrated by the commitment to project ownership within the government bodies. (5) A Road Safety Audit before project preparation can help in achieving positive results. The recommendations of the audit in this project before project preparation helped the team to better align the project activities and have a focused mitigation plan. However, at preparation, arrangements could have been made for a better M&E framework. (6) The project timelines need to be realistic and allow time not only for completing project activities but also allow time for evaluating the results of intervention. The limited time frame in this project left little time for a proper evaluation of results. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is concise and well-written. It candidly discusses the issues relating to the lack of baseline data, inadequacy of the M&E framework and of the delays during implementation which were well within the control of the project. The ICR draws reasonably good lessons from the experience of implementing this project. The ICR's description of the bank performance at supervision is rather brief and the ICR would have benefitted from a better description of the how the issues that arose during implementation were resolved. Page 10 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Road Safety Activity under on RSDP roads (P149606) a. Quality of ICR Rating Substantial Page 11 of 11