Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) Report Number : ICRR0021527 1. Project Data Project ID Project Name P121515 IN: NHAI Technical Assistance Project Country Practice Area(Lead) India Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79800 31-Jul-2016 24,014,004.51 Bank Approval Date Closing Date (Actual) 30-Nov-2010 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 45,000,000.00 0.00 Revised Commitment 24,014,004.51 0.00 Actual 24,014,004.51 0.00 Prepared by Reviewed by ICR Review Coordinator Group Peter Nigel Freeman Kavita Mathur Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives Original Objectives The original project development objective (PDO) was "to assist the National Highways Authority of India (NHAI) to adopt appropriate practices that would enhance its program management and operational efficiency" (Project Appraisal Document [PAD], Page 9; and Loan Agreement, Page 4). Revised Objectives Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) In the restructuring approved on December 10, 2015, the PDO was amended to read "to assist the Ministry of Road Transport and Highways (MoRTH) and the NHAI to adopt better program management and operational efficiency" (Restructuring Paper Page 6, and Letter of Loan Amendment Page 2). MoRTH replaced NHAI as the implementing agency following the poor performance of NHAI in the preceding five years before the December 2015 restructuring. The intention of the revised PDO was similar but since there was insufficient time remaining to measure "enhancement" this word was dropped. The new PDO also reflected the change of implementing agency. These revisions necessitated a split evaluation. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-Dec-2015 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components Original Components: The project had only one component, “institutional strengthening and capacity building” covering seven “areas of concern”: The seven areas of concern were (i) project planning, preparation, and implementation; (ii) documentation control, timely availability and accessibility of information, decision making, internal controls, and contract administration; (iii) skills gaps in the areas of research and development and contract administration; (iv) corporate governance; (v) asset management; (vi) safety awareness; and (vii) sustainability. These areas were further divided into numerous technical assistance (TA) activities. Revised Components: At the restructuring on December 10, 2015, the single component "institutional strengthening and capacity building" was retained but some of the original TA activities were dropped and a further 16 TA activities were added. In total 23 TA activities were identified by NHAI and 24 for MoRTH (47 in total). In particular, the Enterprise Resource Planning (ERP) was dropped. Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The original project cost was estimated to be US$ 55 million. The actual final project cost was US$ 29.99 million because the project was scaled back when the large ERP activity was dropped. Financing: The original IBRD loan amount was US$ 45 million. This was reduced by US$ 15 million to US$ 30 million in the restructuring of December 10, 2015. The final amount disbursed from the loan was US$ 24.01 million. Borrower Contribution: The original Government of India (GOI) counterpart contribution was US$10 million. This was reduced to US$7.62 million in a level 1 restructuring of December 10, 2015, when inter alia the proposed enterprise resource planning system (ERP) was dropped. The actual Borrower contribution was US$ 5.98 million. Dates: The closing date at appraisal was July 31, 2016. This was extended by almost two years in two increments to June 30, 2018. This was to complete existing work and to cover additional technical assistance activities. Restructurings: The project was restructured three times: • A level one restructuring on December 10, 2015. This comprised changes in the PDO, a change in the responsible implementing agency, changes in the results framework, components and costs so that the project was scaled back from US$ 55 million to US$ 37.62 million. In addition, there were changes in the financing plan, the institutional arrangements, legal covenants, financial management and procurement procedures, the safeguard policies triggered and the project environmental category, which was changed from C (not required) to B (partial assessment). • A level two restructuring on April 13, 2016 extended the loan closing date by one year. • A level two restructuring on June 30, 2017 further extended the loan closing date by 11 months and changed the implementation schedule, adding new technical assistance activities. 3. Relevance of Objectives Rationale Original Objectives: Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) The World Bank had been supporting national highway construction and rehabilitation in India for 25 years when the project was conceived, but despite project components to strengthen NHAI, the authority's performance in general and especially in the previous four projects financed by the Bank, showed evidence of serious problems including significant delays in preconstruction activities, poor quality planning and design, ineffective contract administration, a lack of safety awareness and large cost and time overruns. This was brought into focus by the Lucknow- Muzaffarpur National Highway Project, which was found to have shortcomings in its financial accounting and reporting systems and also experienced two fatal worksite accidents. The subsequent investigation by the World Bank's Integrity Vice Presidency identified significant evidence of practices by some contractors and consultants that could be a reason for sanctions. Against this background, the NHAI Technical Assistance Project was formulated to help address the systemic institutional challenges faced by the Authority. The physical expansion and improvement of national highways was a priority for the Government of India (GOI) supported by the Bank as laid out in the Country Assistance Strategy of FY09-12 and continued in subsequent strategy documents until the current Country Partnership Framework for FY18-22. It was also a significant feature of the Bharatmala Pariyojana, a 2015 initiative to improve the efficiency of freight and passenger movements across the country. However, the urgency to improve the efficiency of the NHAI was not shared by the agency itself nor perceived at the time by the GOI, and so the timing of the initiative was likely premature. Discussions with MoRTH were at best superficial at the appraisal stage. Revised Objectives: During the first four years of implementation, disbursements were minimal as the project was clearly not seen as relevant by NHAI, and World Bank management became concerned about the project's poor performance with over three consecutive years of "Unsatisfactory" ratings for progress towards development objective and for implementation performance. This led to a restructuring of the project when MoRTH replaced NHAI as the implementing agency. With the newly elected Central Government becoming actively concerned with improving NHAI’s performance, especially in resolving cost overruns and delays, MoRTH became the World Bank’s main interlocutor and helped to restructure the project to reflect the Government’s reform and development agenda for the national highways. The PDO was changed to adopt better program management and operational practices instead of enhance program management and operational practices since time was now limited to measure such enhancement, As the ICR notes, however, (page 11, footnote 13) the word "adopt" is subject to slightly different interpretations depending on the dictionary used, so the PDO even as revised was vague and consequently difficult to monitor. The project's activities were expanded to cover new categories relevant to the new Government's reform agenda such as greenhouse gas emissions and logistics. By project closure the PDO was substantially more relevant to the Government than it had been at appraisal. Rating Substantial Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To assist the NHAI to adopt appropriate practices that would enhance its program management and operational efficiency Rationale The Theory of Change for the original project design was that the proposed technical assistance consultancy activities (project preparation and management, corporate governance, resource planning, asset management, road safety management, training, and capacity building) would lead to improved systems, practices and programs that in turn would result in better project outcomes, such as better design review processes and the introduction of safety information and management systems. These outcomes were expected to lead to higher level outcomes such as increased mobilization of resources from the private sector, and a higher implementation rate for national highways with better quality. The NHAI would shift from a role as a builder to that of an asset manager. Outputs: The project had a single component covering the seven specified areas of concern described in 2(d) above. These areas were further divided into numerous TA activities. Eleven TA activities were awarded but only two were completed before the 2015 restructuring. The first was a guideline for Strategic Environment and Social Assessment and the second was a national road transportation strategy. An advance of US$ 4.5 million was released to the Designated Account shortly after the project’s effectiveness, but this was not a true disbursement as before the restructuring in December 2015 only US$11,000 was actually spent. The consultants typically started work without receiving advance payments, and their invoices were substantially delayed while their interim or final drafts were being reviewed. The implementation of the Enterprise Resource Planning (ERP) was dropped from the project at the 2015 restructuring following serious procurement delays after it was judged that this system could no longer be implemented during the life of the project. Delays had been due to disagreements on the procurement process to be followed, and two unsuccessful rounds of bidding. Eventually, the ERP was included in the ongoing National Highways Interconnectivity Improvement Project and is expected to be completed at the end of 2019. The design review and GAAP processes commenced but were completed after the restructuring. The implementation of Accident Information and Management System was completed. Outcomes: Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) The PDO indicators were inadequate to measure the achievement of the objective. Since no activities were completed, no outcome was achieved. Rating Negligible PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective To assist MoRTH and NHAI to adopt better program management and operational practices. Revised Rationale In the 2015 and 2017 restructurings, the Theory of Change was revised. The PDO was changed to "adopting better program management and operational practices". The project’s activities were expanded to cover three new categories: environmental protection and greenhouse gas emissions; logistics; and sector strategies. The higher-level outcomes remained the same. For NHAI, new financing options and modalities, the development of a road asset management system, and an improved preparation and design review process were included. For MoRTH the additional activities included a logistics efficiency and enhancement program, a model for analyzing the carbon emissions of highways, and a national program for intrastate bus transport reform were added according to the revised Theory of Change model. All these measures would help to improve the operational practices of both organizations. Outputs Organization and process transformation study was completed as targeted. Model concession agreements for tunneling and expressway projects were developed. Independent quality audit system was adopted by NHAI on June 30, 2018. Performance evaluation system for consultants and contractors was established but was not operational at project closing. The Technical Assistance on "Passenger Mobility Enhancement helped MoRTH in developing a national program for intrastate bus transport reform through its recommendations on adoption of a regulated competition regime, enhanced competitiveness of state road transport undertakings, including modernization of bus fleets and public transport technology enhancement. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) The Technical Assistance on "Strategy for hill transportation and infrastructure utilization/creation" was used to developed a master list of potential ropeway sites in two hill states and a three-year prioritized implementation strategy and road map for two projects. Outcomes Three of the four original PDO indicators were dropped, but the GAAP indicator was retained. The new and revised outcomes were: New financing options and contract modalities were developed and adopted. Two new financing options and contract modalities were developed and adopted by NHAI to assist with public private partnership arrangements. These were the Hybrid Annuity Model and the Toll-Operate-Transfer Model. In addition, Model Concession Agreements for expressway and tunneling projects were drawn up for use under the Bharatmala program and for other access-controlled highways. An Infrastructure Investment Trust for road projects was also introduced and listed on the stock exchange. The "Financing Options Study for National Highways Development," contributed directly to the issuance of rupee-denominated Masala bonds listed on the Singapore and London stock exchanges, where some US$ 460 million was raised. Further improvements were also made to prevailing contract agreements. Several bundles of projects covering nine stretches of road were awarded in 2018 with more to follow. Asset Management System developed for national highways, it was not fully operationalized. The idea was to upgrade the Road Information System to a Road Asset Management System. Three years of pilot data for 3,100 km of highways was entered into the system but it could not transition to an institution- wide planning and budgeting tool as it was only set up with pilot data. NHAI invited two bids, one for expanding the data collection system by an additional 27,000 km, and the other for establishing a unit to manage the system. However, both bids were cancelled due to limited market response. NHAI is planning to re-bid the proposals after re-packaging them to ensure wider participation. Improved Project Preparation and Design Review Process adopted. A manual for project preparation was prepared and approved by NHAI, while recommendations were made for improving design review and monitoring systems. This was followed by a further phase that resulted in business process changes that included the upgrading of the Project Management Information System to create dashboards and reports for project monitoring. Officials and staff were trained to use the system. There were measurable benefits from this process that included a 150 per cent increase in the award rate, and a 95 per cent increase in the road construction rate over five years. Eighty per cent of identified "problem" projects were put back on track and there was a 33 per cent improvement in the time to award smaller projects through delegation. Implementation of GAAP actions and adoption of mitigation measures within the project period was partially achieved. A final report on governance and accountability was prepared. However, it did not cover sanction regimes against vendors found to have engaged in misconduct. A subsequent pilot TA was launched to develop a performance evaluation system for contractors and concessionaires but at closure it had not been completed. Recommendations covering system transparency, user engagement and safety- related social accountability were implemented. The NHAI website was revamped to improve the interface Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) with the public and a toll-free corruption hotline was established. The ICR (Page 14) notes that in terms of actual adoption and implementation these results represented only a small fraction of the recommendations of the Corporate Governance Study. However, the ICR does not elaborate which of these recommendations were not pursued. A "Logistics efficiency and enhancement program" technical assistance made recommendations to MoRTH on integrating logistics with infrastructure development, including the development of green and brownfield freight corridors, the development of multi-modal nodes and logistics parks, as well as seamless interstate movements. The study also developed recommendations for modernizing the trucking industry. Interventions to reduce carbon footprint in developing highways: a model for analyzing the carbon emissions of highways. According to the ICR (Page 14) the study for MoRTH has contributed to (a) the identification of nine new highway corridors that have the potential to save travel time and GHG emissions; (b) savings through acquiring land only on one side of the right-of-way for expansion; and (c) the possibility of resurfacing roads with recycled materials. A noticeable factor was that activities related to institutional governance and internal management systems took longer for adoption and would have benefitted from a sub-component aimed at improving understanding through better dissemination. Revised Rating Substantial PHOVRLEFFRATTBL Rationale A large number of technical activities were completed to assist MoRTH and NHAI to adopt better program management and operational practices. These include inter alia improved project preparation and design review process, new financing options and contract modalities. The asset management system was developed for national highways, but it was not fully operationalized by project closure. However, there is some concern whether the recommnedatons of some of the studies will be implemented by NHAI or MoRTH (see section 7). Overall Efficacy Rating Substantial 5. Efficiency As this was a technical assistance project conventional cost benefit or cost effectiveness analysis was not applied. Nevertheless, efficiency could be assessed based on the time taken to produce the outputs and Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) outcomes. During the first five years of implementation the only work carried out was procurement-related, including some contract awards. Almost all the deliverables were achieved in the last 30 months of the project, in which the closing date had to be extended by nearly two years. The risk assessment was inadequate and mitigation measures were not effective. Stakeholder support risks were not even considered even though the NHAI was not enthusiastic and MoRTH scarcely consulted. The expectation that procurement could be carried out through NHAI's existing staff and procedures was also naïve, especially as NHAI was unfamiliar with the World Bank's rules for technical assistance projects. After the restructuring in 2015 and MoRTH became the implementing agency the efficiency improved so that many of the technical assistance sub-components were completed. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The methodology for split evaluations was applied: • For the original PDO and targets: Substantial relevance, Negligible efficacy, and Modest efficiency, result in an overall outcome rating of Unsatisfactory when assessed across the entire duration of the project. • For the revised PDO and targets: High relevance, Substantial efficacy, and Modest efficiency, result in an overall outcome rating of Moderately Satisfactory across the entire duration of the project. The overall weighted outcome rating was Moderately Satisfactory, based on the following: Before Restructuring After Restructuring Outcome ratings U MS Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) Numerical value of the outcome ratings 2 4 Actual disbursements US$ 4.56m* US$ 19.340 Shares of actual disbursements 19.464% 80.536% Weighted value of outcome ratings (row 2 X row 4) 0.389 3.221 Final Outcome Rating 0.389 + 3.221 = 3.610 rounded to 4.0 = MS Note: The ICR argues that although US$ 4.56 million was disbursed to the Borrower's designated account only US$ 0.11 million was actually spent during the five year period up to the restructuring. This is moot, however, as it makes no difference to the overall outcome rating of MS. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Government commitment and ownership. The expansion, delivery and improvement of national road projects has a high priority with the GOI. Accordingly, technical assistance sub-components that expedite this process have a strong chance of adoption and sustainability. These include new contract modalities, road finance including toll roads, better design reviews and project preparation procedures. However, the track record for technical assistance addressing challenging institutional reforms and difficult operational initiatives is not very positive and will likely not be pursued without strong World Bank follow-up and intensive engagement. Examples are IT infrastructure security, strategic environmental and social assessments, value engineering, road safety, performance rating of contractors and consultants, land acquisition and dispute resolution. The risk to the development outcome is compounded by frequent changes of senior officials and key staff. If the proposals are not implemented within a window of two to three years after being put forward, they are less likely to be adopted as ownership and commitment diminishes over time. A further issue is the lack of an adequate system to capture the new knowledge and ensure its dissemination to all relevant stakeholders. 8. Assessment of Bank Performance a. Quality-at-Entry The project was not as highly relevant as motivated in the PAD because it was formulated based on the World Bank’s perception of urgent need and this sense of urgency was not shared by NHAI. NHAI’s management and staff were preoccupied with meeting targets for delivery of the GOI National Highway Development Program, and the project received a low priority. MoRTH was also relatively uninterested Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) in the project at that time. During the project's preparation, MoRTH was not engaged in the dialogue about the problems (discussed in section 3) that led up to the formulation of the project (ICR Para 47). NHAI may have appreciated the potential usefulness of some of the TA activities, but for the most part, it perceived the project as an imposition. This was the fundamental reason for the project’s nonperformance during its first five years. None of the original PDO indicators were realistic. The PDO indicators were not formulated to measure enhanced program management and operational efficiency. Instead they were framed either as project outputs or the adoption of new practices (see section 9). Positive factors were the breadth of the activities conceived, which covered the full spectrum of issues that had become apparent in previous projects involving NHAI. There was also flexibility built into the design to accommodate changing needs as the project progressed. However, the project’s implementation arrangements had deficiencies. Implementation agency risks, especially procurement capacity, were underestimated, and the related mitigation measures were inadequate. Although NHAI had long experience with procurement of construction projects, its experience with technical assistance procurement was limited. The World Bank failed to appreciate the importance of strong engagement with MoRTH and giving it a role in project implementation. In addition, the project’s M&E design was poor, with an unrealistic PDO and key indicators that did not align with it. The ICR indicates (Page 24) that some of these deficiencies might have been avoided or diminished had the project had not been rushed from the concept stage to Board approval in only seven months, apparently without the usual formal quality or decision reviews. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The World Bank implementation support initially focused on expediting procurement and administrative issues. NHAI experienced high turnover in its senior management during 2010–14. These factors caused procurement processes to be very slow and key decisions to be delayed. From May 2013 to December 2015 the achievement towards the development objective was rated unsatisfactory and the December 2015 restructuring was drastic. While it could be argued that the Bank took too long to act, it had considered both the suspension and cancellation options, but the fact that a newly elected Central Government was in place made the likelihood of turning the project around much greater especially as the new Government had become much more involved in the project and was actively concerned with improving NHAI’s performance, especially in resolving cost overruns and time delays. US$ 15 million was cancelled inter alia for the ERP system, which could no longer be finished within the remaining time left for implementation. The PDO was amended to make it more realistic in the circumstances and MoRTH was brought in as the implementing agency demonstrating new energy for the project and restructuring it to reflect the Government’s reform and development agenda for national highways. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) After 2015, the World Bank team endeavored to provide timely and appropriate advice to counterparts to address issues as they arose. Once MoRTH was brought into the project, its content became more relevant to the World Bank’s current approach in the sector (ICR Page 11). The restructuring of 2015 added several activities under MoRTH related to mobilizing private sector finance for highways, reducing the carbon footprint of road projects, and enhancing the logistics efficiency of transport. The restructuring of 2017 added further activities related to private sector finance and climate change. The arrangements put in place in 2015 contributed to a major acceleration of the project's implementation especially for MoRTH initiated technical assistance. NHAI was less focused because of capacity constraints. Payment delays were reduced but the slow disbursement problem was never completely resolved. However, the procurement process was eventually speeded up. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Given the lack of commitment from NHAI's management the original PDO as framed was overly ambitious. In addition the original key indicators were inappropriate since they did not purport to measure either enhanced program management or operational efficiency. Rather, two of them were outputs and two were indicators of the adoption of new practices. The road safety indicator did have an outcome-oriented target but no quantitative measurements were proposed. In any case, since so many factors affect road safety there would have been a real problem in attributing any changes to the actions in the project. The institutional governance aspect (which was a major motivator for the project proposal) lacked both specific indicators and targets. The 2015 restructuring established a more realistic results framework. However, the framework suffered because of the lack of a clear definition of what was meant by "adoption" in relation to new practices and procedures, while there was also over-simplification because three new indicators were based on "yes/no" answers. The PDO indicators were not formulated to measure enhanced program management and operational efficiency. Instead they were framed either as project outputs (implementation of Enterprise Resource Planning was successfully completed; and implementation of accident information and management system and network safety management were successfully completed) or the adoption of new practices (design review process established and adopted at NHAI; and implementation of GAAP actions and adoption of mitigation measures were successfully completed within the project period). The target for the indicator "implementation of accident information and management system and network safety management" was a reduction in fatal road accidents Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) and construction site accidents on the national network. This indicator was inadequate as since any change in accident data could not be attributed to the project. b. M&E Implementation Prior to the 2015 restructuring the reporting of progress in the results framework was confined to comments about the status of procurement and the extent of implementation of the related activities. Following the restructuring the revised indicators were tracked but information provided was "yes/no" until the Country Management Unit requested more substantive information. The reporting on the indicators in the results sections of the implementation supervision reports (ISRs) did not add much value until the last two ISRs, which presented key issues and actions for management's attention. c. M&E Utilization No evidence could be found that the indicators were used to inform project management or decision making. M&E Quality Rating Modest 10. Other Issues a. Safeguards The original project did not trigger any environmental or social safeguards policies. During 2015 loan restructuring World Bank policies OP 4.01 (Environmental Assessment) and OP 4.12 (Involuntary Resettlement) were triggered and the environment assessment category was changed from C (not required) to B (Partial Assessment). The rationale was that although activities under the restructured project did not have direct adverse environmental or social impacts, the outcomes of the implementation of the recommendations of the various studies could potentially lead to significant environmental and social implications, entailing risks and potentially inducing adverse impacts. These changes led to the addition of one new activity that was not part of the original project, a Strategic Environment and Social Assessment (SESA), initiated in July 2016. The SESA met the objectives of recommending measures to facilitate the screening of the project’s activities It also sensitized NHAI, MoRTH, and their consultants concerning the need for upfront consideration of environmental and social issues at the design stage. Both the environmental and social safeguard performance were rated as satisfactory in the ICR. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) b. Fiduciary Compliance Financial management was rated as moderately satisfactory throughout the project due to the chronic slow disbursements prior to the 2015 restructuring and even afterwards there were delays due to the time it took both NHAI and MoRTH to review inputs and deliverables before payment could be effected. Most disbursements took place during the final two years of the project, after it had been extended by nearly two years. There were many complaints from consultants about the payment delays and the issues causing them were never fully resolved. The ICR (Page 20) cites four reasons for the delays: i) NHAI's lack of clarity about the purpose of some of the studies; ii) frequent changes of responsible officers; iii) NHAI's need to seek multiple levels of approval before releasing payments, and iv) NHAI's resistance to partial payments for approved drafts. Minor lags in producing financial reports also occurred. Procurement delays were highlighted in most Aide Memoires and Implementation Status and Results Reports. The delays were caused by inter alia inadequate competition for some contracts, NHAI wanting to follow its own procurement policies instead of the Bank's as stipulated in the loan agreement; setting qualification requirements that were too rigorous compared with international practice, requesting excessive supporting documentation, delays in decisions, and inflexible interpretation of rules during the evaluation process. NHAI managers and staff lacked experience with consultancy procurement other than for works supervision. This, together with NHAI’s weak ownership of the project, meant that according to the ICR (Page 22) its officials often turned to the World Bank for validation and facilitation of procurement processes, which contributed to delays and sometimes placed the World Bank in a position of going beyond its formal role. c. Unintended impacts (Positive or Negative) The World Bank’s ongoing engagement with GOI through the project helped to enable the launch of two new initiatives. One was the Green National Highways Corridor Project (P167350) to enhance the institutional capacity of MoRTH to adopt and implement green and climate resilient construction methods. The second was a Reimbursable Advisory Study to assist MoRTH in helping states to improve their public transport policies, and support state road transport undertakings to improve their service definition and planning, and to help increase the capacity of state public transport systems through innovative contracting methods. d. Other --- 11. Ratings Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The following useful lessons adapted from the ICR are pertinent: It is crucial for the World Bank to candidly and carefully assess the client’s commitment to a technical assistance project and its objectives. While the adoption of recommendations was significant in the formulation of operational and strategic technical assistance, there was little take-up of initiatives in the areas of core institutional reform and corporate governance. Stand-alone technical assistance loans, especially large ones spanning over five or more years, may have some limitations in achieving significant improvements in institutional performance. Nevertheless, several transformational logistics and connectivity initiatives in this project had their genesis in the various TA studies undertaken. The World Bank should consider a range of options when selecting an appropriate technical assistance strategy—such as a series of TA loans, smaller, focused non lending TA, components of TA embedded in larger lending operations, reimbursable advisory services, and development policy lending— and carefully tailor the choice to the specific circumstances. Moreover, frequent changes of senior officials and key staff may mean that windows of opportunity for institutional reform are limited. Projects with such objectives should be designed to move fast, within two to three years at most, while ownership and commitment last. For institutions that have deep-rooted cultures and ways of operating, an incremental approach to change, which builds upon past gains and mixes “low-hanging fruit” together with bolder aims, may be more effective. The concept of introducing e-governance in NHAI was a bold and ambitious move. However, its repeated failure and eventual exclusion from the project was necessitated due to a combination of lapses including: failure to build up the internal information technology (IT) capacity of NHAI, inadequate market assessment, and focusing exclusively on NHAI rather than adopting a sector wide approach. It is a mistake to assume that TA projects are easy to implement because they only involve consultant contracts. Consultancy contracts, especially ones related to institutional change, can be as challenging to procure and manage, if not more so, than works or goods contracts. Implementation arrangements must be carefully analyzed and appraised, with special attention to ensuring that the client agency has the requisite technical skills, absorptive capacity, and clear lines of accountability and decision making. In these aspects, it is also extremely important to consider the role of other relevant institutions besides the primary client. According to the World Bank staff involved, the project would have benefited more from enhanced Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) mobilization of the World Bank’s own subject matter experts in areas such as asset management, IT, performance monitoring, road safety, infrastructure finance, and logistics, among others. Such inputs potentially would have helped to provide quick, ‘just-in- time’ inputs at crucial stages to facilitate the formulation, implementation, and absorption of the TA activities. In some specialized areas, such as specific IT systems, the World Bank should consider mobilizing third-party expertise. Dissemination of TA findings and their links to practice should not be an afterthought. Too often, findings are not communicated to relevant line ministries, states, and other stakeholders in forms that are practically useful. With the rapid turnover of key personnel in government agencies, the knowledge generated, if not captured in a repository, is likely to be lost. Dissemination of information should be built into the project design and not be a one-time, end-of-project activity. Several such dissemination activities were planned under the NHAI Technical Assistance Project, but ultimately the pressure of completion pushed many of these to the back burner. 13. Assessment Recommended? No 14. Comments on Quality of ICR In general the ICR was well-prepared given that the there was such a wide spectrum of activities in the technical assistance project. The amount of detail provided on the technical assistance was, however, uneven, with some aspects covered in great detail while others (such as the corporate governance assessment of NHAI) had too little information. The ICR states (Page 14) "that in terms of actual adoption and implementation these governance actions represented only a small part of those recommended. However, the ICR does not clarify which recommendations were not adopted. The split evaluation for the outcome was appropriate although it is arguable whether it was justified to use actual disbursements to consultants instead of the transfer to the designated account as the amount in the calculation. In the end, however, this made no difference to the actual outcome rating. The rating for relevance at entry as high was questionable given the client's lack of commitment at that time, while the rating on Bank performance taken as a whole could have put more emphasis on the five years of negligible progress prior to the 2015 restructuring. On the other hand, the lessons on client ownership and alignment, the choice of instrument for the project, implementation arrangements and dissemination were insightful and showed that considerable thought had gone into their formulation. Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: NHAI Technical Assistance Project (P121515) a. Quality of ICR Rating Substantial Page 17 of 17