Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) Report Number: ICRR0021741 1. Project Data Project ID Project Name P083108 RAIL TRADE & TRNSPT FACIL Country Practice Area(Lead) Azerbaijan Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-75090,IBRD-82820 30-Jun-2012 421,893,773.59 Bank Approval Date Closing Date (Actual) 27-Mar-2008 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 450,000,000.00 0.00 Revised Commitment 421,893,773.59 0.00 Actual 421,893,773.59 0.00 Prepared by Reviewed by ICR Review Coordinator Group Elisabeth Goller Kavita Mathur Victoria Alexeeva IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (LA) (Schedule 1, page 7), the objective of the project was to improve railway services in Azerbaijan, as well as the competitiveness, financial sustainability, operating and cost efficiency, and capacity of the Azerbaijan Railways, in particular along the transport east-west corridor. This is practically identical to the project objective in the PAD (para 18), which reads as follows: to improve railway services in Azerbaijan, as well as the competitiveness, financial sustainability, operating and cost efficiency, Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) and capacity of the Azerbaijan State Railways (ADDY), in particular along the transport corridor toward Georgia (east-west corridor). This assessment will divide the project development objective (PDO) into the following two parts: (i) to improve railway services in Azerbaijan, in particular along the transport east-west corridor and (ii) to improve the competitiveness, financial sustainability, operating and cost efficiency, and capacity of the Azerbaijan Railways, in particular along the transport east-west corridor. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 27-Jun-2013 c. Will a split evaluation be undertaken? No d. Components The project was structured around the following four components (PAD, paras 25 to 31): Component 1: Rehabilitation of East-West Mainline (cost with contingencies at appraisal US$440.0 million, cost at appraisal of 2013 AF US$927.7 million; estimated actual cost US$952.0 million, which corresponded to the contract price of the activities under this component at project close, when the signaling and power supply contracts were still ongoing (ICR, para 14)). The original activities under this component were to rehabilitate 240 km of the East-West mainline in poor condition. It included track rehabilitation (cost at appraisal US$117.0 million, cost at appraisal of 2013 AF US$190.7 million, estimated actual cost US$215.0 million), an upgrade of the signaling system (cost at appraisal US$17.0 million, cost at appraisal of 2013 AF US$340.0 million, estimated actual cost US$339.2 million), and power supply rehabilitation and conversion from 3 kilovolt (kV) to 25 kV alternate current (AC), including new power stations and catenary (cost at appraisal US$307.0 million, cost at appraisal of 2013 AF US$397.0 million, estimated actual cost US$397.8 million). Changes under this component involved financing a new and state of the art signaling system instead of upgrading the existing one. The price of the new system was 20 times higher than the budgeted upgrading. At appraisal, it was assumed that the existing system could be upgraded to work with the new power supply system. This assumption was to be confirmed through a study carried out at the beginning of project implementation. The study concluded that the existing system was obsolete and that the new system was safer and more economic in the long term (2013 AF Project Paper, para 11.c). According to the task team, the amount of track rehabilitation works also increased. Component 2: New Mainline Locomotives (cost with contingencies at appraisal US$334.0 million, cost at appraisal of 2013 AF US$334.0 million; estimated actual cost: US$334.0 million, which according to the task team corresponds to the appraisal estimate because the Borrower fully financed this component and the Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) actual cost was not available). The component was intended to finance the purchase of about 50 new AC mainline electric locomotives to operate on the east-west corridor. The activity under this component did not change. Component 3: Modernization of Azerbaijan Railways Systems (cost with contingencies at appraisal US$17.0 million, cost at appraisal of 2013 AF US$27.5 million; estimated actual cost US$27.0 million). This component included (i) support for the full implementation of the International Financial Reporting Standards (IFRS) accounting system, (ii) provision of advisory services to the Azerbaijan Railways to implement its modernization program, and (iii) equipment, procedures, and training to prevent oil spills that may occur as part of railway transportation (e.g. in case of derailment) and create the response capacity of Azerbaijan Railways to address such occurrences. This component was scaled up by adding the rollout of the IFRS system and the purchase of IT hardware and accounting software. Component 4: Project Implementation (cost with contingencies at appraisal US$3.0 million, cost at appraisal of 2013 AF US$3.0 million, estimated actual cost US$3.0 million). The component included consultants services, training to the Project Implementation Unit (PIU) for procurement, financial management, financial audits, and monitoring, and equipment for project implementation. The activities under this component did not change. Note: ADDY, the Azerbaijan State Railways and project implementation agency, was corporatized in July 2009 to become the Azerbaijan Railways (ADY). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost According to the ICR (page 43), the estimated project end cost was US$1,291.7 million, which does not include the front end fee. With the estimated front end fee of US$1.5 million, the estimated project cost is basically in line with the AF appraisal estimate of 1,293.4 million (2013 AF PP, page 8). However, in paragraph 14, the ICR notes that the estimated cost of component 1 was US$952.0 million, corresponding to the contract price because some activities were still ongoing. This is US$24.3 million more than the cost indicated on page 43, and hence would bring the overall project cost to US$1,316 million or 102 percent of the AF appraisal estimate. The task team explained that both figures were estimates, but that US$952.0 million might be closer to the actual figure. Financing The Bank financed a total of US$421.89 million through an IBRD loan, which is 94 percent of the original loan and 63 percent of the loan with the 2013 AF. Not all loan funds were disbursed because the Bank decided not to extend the loan closing date further. This was due to the fact that the contractor of the Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) signaling system performed poorly and there was no clarity on the way forward for this contract (ICR, para 79). Co-financing The Czech Export Bank was expected to finance US$117 million (2009 PP, annex 2). The task team noted that the actual financing of the Czech Export Bank for the activities related to the project was US$215 million. Borrower Contribution At appraisal, the Borrower was expected to contribute US$345.0 million. With the 2013 AF, the expected Borrower contribution increased to US$623.4 million (2013 PP, para 23). According to the 2009 PP (annex 1), US$1 million of this amount was to come from the Azerbaijan Railways and US$ 117 million from the Czech Export Bank. The task team pointed out that the Borrower financed the balances of the signaling and power supply contracts. The estimate of the Borrower contribution at project close was US$431 million. Dates, Project Restructurings, and AF The project was approved on March 27, 2008, became effective on March 15, 2010, and was expected to close on June 30, 2012. The closing date was extended twice for a total of five years and six months. The project closed on December 31, 2017. The project underwent a restructuring and an AF with a restructuring. The 2009 restructuring, which took place before the project became effective, was necessary because the global financial crisis and the subsequent drop in oil prices worsened the economic and fiscal situation of the Borrower. This led to a revision of the overall scope of the railways reform program and delayed the conversion of the ADDY into a joint stock company, which was a loan conditions (ICR, para 77), and the signing of the original LA. The restructuring changed (i) the loan terms, (ii) the project financing plan to include the Czech Export Bank as co-financer of the counterpart funds under component 1, (iii) the disbursement percentage for the acquisition of locomotives under component 2, (iv) the key project implementation dates to reflect the delay in project signing, (v) the cutoff date the for retroactive financing, (vi) the baselines and project end targets of the two financial outcome indicators because the 2007 restated financial accounts became available after Board approval and showed figures that were significantly different from the baselines estimated at appraisal (according to the 2009 PP, para 19, the level of ambition of these indicators stayed the same), (vii) one intermediate indicator end target to adapt it to the changed reality, (viii) the name of the implementation agency, which had been transformed into a joint stock company, and (ix) minor other aspects in the legal agreement. The restructuring extended the closing date by 15 months, from June 30, 2012 to September 30, 2013, because of the delay in loan signing. The 2013 AF and restructuring were necessary because, as mentioned in the Components section above, the signaling system needed to be replaced instead of upgraded, which exponentially increased its cost. In Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) addition, the cost of the power supply system rehabilitation and conversion was 30 percent higher than originally budgeted. The AF provided a total of US$220 million to cover the additional cost of these systems. The AF and restructuring (i) changed the cost of some components, (ii) reallocated loan proceeds between disbursement categories, (ii) revised the financing plan by increasing the counterpart funds, and (iv) changed the results framework. The changes to the results framework reflected the project extension in the end targets, making several of them slightly more ambitious. The changes also included a revision of (i) the end targets of the two financial indicators, (ii) the baselines and/or project end targets of the two indicator targets capturing the railway traffic, and (iii) the baselines and/or project end targets of the two intermediate outcome indicators capturing the percentage of tracks in need for repair and the locomotive reliability. The task team explained that these changes were necessary because the original baselines were estimates and did not correspond to the actual trends. To implement the new signaling system, the AF and restructuring extended the closing date by four years and three month, from September 30, 2013 to December 31, 2017. Due to the fact that (i) the AF adjusted the project activities and costs to what was necessary under the project, (ii) reflected the project changes in the indicator targets, which increased in ambition, except for the targets of the indicators, for which the original baselines were not correctly reflected, and (iii) the AF took place early during project implementation when only 15.78 percent of the loan funds were disbursed, the application of a split rating is not warranted. 3. Relevance of Objectives Rationale The relevance of objectives is rated substantial. Country context. Azerbaijan's investments in the oil and gas sectors did not achieve the desired impact of job creation and employment, and the government decided to utilize oil and gas revenues to support growth and job creation in non-oil sectors, to ultimately reduce poverty in a sustainable manner. The strategic geographic location of the country and its existing transport infrastructure positions Azerbaijan as a regional transport hub. However, at appraisal, the railway sector faced major challenges, including deteriorating assets, reduced maintenance and operational capabilities, and inefficient and old-fashioned management practices. This reduced the railway system's capacity to meet the demand on a timely and cost-efficient basis, threatening the country’s transit revenues and increasing the financial burden of rehabilitating and maintaining the system. Therefore, to improve railway services and the competitiveness, financial sustainability, operating and cost efficiency, and the capacity of the ADY was highly relevant. Alignment with strategy. The high relevance of the PDO was reflected in the government strategy "Azerbaijan 2020: Vision for the Future", which identified investments in physical infrastructure as one of three measures to strengthen the country’s resilience to external shocks. It was also in line with the Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) government’s 2008–2011 Sectoral Program for the Development and Modernization of Azerbaijan Railways, which laid out the reform and investment agenda in the sector. The project contributed to the 2007–2010 Country Partnership Strategy (CPS) for Azerbaijan, which among others aimed at supporting sustainable development through (i) increasing the quality and transparency in public sector governance and (ii) supporting sustainable and balanced growth of the non-oil economy. The CPS noted that “the long term vision was for Azerbaijan to become a prosperous transit center for energy and goods flowing between Europe and Central Asia.” The project stayed relevant in the light of the 2016–2020 Country Partnership Framework, which aims at “contributing to the development of the main transport network for enhanced international and domestic connectivity.” However, considering the drastic decrease in the demand mainly for oil transport services by rail because of oil reduced production and the competing oil pipeline, the focus on the railway sector might have lost some of its relevance. Previous sector experience. This project was the first cooperation between the ADY and the Bank, but the Bank had been actively involved in other transport projects in Azerbaijan and had a close partnership with the Ministry of Transport (PAD, para 32). Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve railway services in Azerbaijan, in particular along the transport east-west corridor. Rationale The theory of change for the first objective was that (i) physical improvements in rail infrastructure along the east-west corridor, consisting of track rehabilitation, the conversion to an enhanced power supply system, and a new signalizing system, (ii) new locomotives, and (iii) enhanced operational, financial, and institutional capacity in the ADY, were expected to result in less tracks with a need for rehabilitation and speed restrictions, increased locomotive reliability and availability, and hence reduced train transit times. These service improvements along the east-west corridor were expected to increase customer satisfaction. They were also expected to increase the transported freight volumes under the assumption that freight demand on the corridor would continue to grow. Note: The indicator target end values are different on page 18 and in annex 1 of the ICR. The task team clarified that the outcome indicator achievements in annex 1 were automatically picked up by the system from Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) the last ISR, and that the correct and final values are on page 18 of the ICR. The correct intermediate outcome indicator achievements are in annex 1. Outputs and Intermediate Outcomes The following outputs and intermediate outcomes contributed to the achievement of this objective (ICR, pages 36 to 39 and paras 40 to 50): Physical rail infrastructure, systems and rolling stock  317 km rail tracks rehabilitated on the east-west corridor, which is more than originally envisaged (the task team pointed out that the rehabilitation increased and regularized train speeds and reduced energy consumption);  New signaling system partially installed but not yet operational, except for a test site (ICR, annex 5) (the task team noted that despite the plan to terminate the contract for the signaling system pointed out in the ICR (para 79), the ADY has reduced its scope and reached an agreement with the original contractor to complete two of the four sections. It is expected that these two sections will be completed by the end of 2019 or early 2020. For the other two sections, the new bidding is expected to start in late 2019 and the works to be completed by 2022. The task team also explained that for each section completed, the new signaling system can be used, leading to increased safety and speeds over that section);  New power supply system 90 percent completed on two sections and 70 percent completed on the other two sections of the east-west corridor (the task team pointed out that without the completion of the power system conversion, the system is not usable);  50 new AC electric freight locomotives purchased, of which eight have been delivered, 28 to be delivered in 2019, 22 in 2020, and two in 2021 (the task team pointed out that these locomotives have not yet been used);  Two locomotive depots for maintenance of the new locomotives under construction;  24 percent of tracks with speed restrictions on the east west corridor, not achieving the revised target of 10 percent, but coming close to the original target of 20 percent;  15 percent of tracks in need of renewal on east west corridor, which exceeds the revised target of 20 percent, but is significantly lower than the original target of 5 percent;  One locomotive able to cross Azerbaijan non-stop, compared to an original target of 50 and a revised target of 30 (the task team clarified that this is the locomotive used for testing and commissioning);  16,209 km of mean distance between operational failures of locomotives, which is significantly lower than the target of 100,000 km (the task team explained that this indicator applies to ADY as a whole Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) and that it depends on maintenance of existing locomotives, replacement of aging locomotives, and the purchase of new ones); Modernization, institutional and capacity  Corporatization of the ADY, transforming it from a government department to a "self-accounting" joint stock company, which was a condition of effectiveness for the loan;  IFSR accounting system operational, including hardware, software and training, and in use, which is in line with the intermediate outcome target;  Change management in the context of IT software implementation carried out (ICR, annex 5);  Five-year business plan for the ADY developed (ICR, annex 5);  Asset management system for the ADY developed (ICR, annex 5);  Revaluation of the ADY's assets completed (the task team explained that at appraisal the ADY's assets were not correctly valued);  Financial audit of the ADY through an international audit firm carried out;  ADY created the following four lines of business: freight, infrastructure, passenger services, and rail track and equipment construction and maintenance, in line with the intermediate outcome target;  Oil spill management response contingency plan developed and rolled out in line with the intermediate outcome target;  26 pieces of equipment for contingency response in case of oil spills purchased;  54 staff trained in contingency response; and  Followed up on annual audit qualifications during project implementation and the follow-up was ongoing by project closure, which is in line with the revised target. Outcomes This objective was only partially achieved, and is rated modest. By project end, only the track rehabilitation and the institutional and reform activities were fully completed and had the potential to positively impact the service quality. The performance in terms of achievement of the intermediate outcomes was mixed. In terms of outcome indicators, with 77 percent of locomotive availability for service, the revised target of 80 percent was nearly reached, and the original target of 75 was slightly exceeded. The task team clarified that this indicator captures the progress in the ADY's as a whole and not only for the east-west corridor. The Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) achievement can be partially attributed to some of the institutional and reform activities under the project, such as the creation of a line of business for rail track and equipment construction and maintenance and enhanced focus on maintenance. Its full achievement depends on the use of the new locomotives acquired under the project. The train transit time on the east-west corridor went down from 22 hours at appraisal to 18 hours by project end, which is significantly less than the target of 12 hours. This reduction can be reasonably attributed to the track rehabilitation. The ICR (para 56) also points out that there was a marginal benefit increase in passenger services because of the design changes made during project implementation, which increased the target speed for passenger trains from 100 km/h to 140 km/h. As seen under objective 2, the institutional capacity of the ADY improved. Even if the respective outcome indicator targets were reached, its financial sustainability was not enhanced. The operational and cost efficiency was also not enhanced. The ADY's improved institutional capacity and the partial service improvements in terms of train speed and locomotive availability did not lead to the expected increase in transport volumes. On the contrary, the overall transport volumes on the east-west corridor went down from the revised and original baselines of 28.2 and 20 million tons, respectively, to 10.5 million tons. This is much lower than the revised target of 25 million tons and the original target 23 million tons. This was due to the reduction in the demand for rail freight services, mainly oil transport services, because of reduced production rates and a competing oil pipeline. Finally, client satisfaction for freight services increased from an index of 100 in 2009, to 125 by project end, exactly reaching the revised target of 125 and exceeding the original target of 120. This indicator, again, refers to the ADY as a whole and not only to the east-west corridor. In addition, considering that the physical investments were only partially completed and most outcome targets were only partially achieved by project end, it is questionable how much of this improvement can be attributed to the project. Rating Modest OBJECTIVE 2 Objective To improve the competitiveness, financial sustainability, operating and cost efficiency, and capacity of the Azerbaijan Railways, in particular along the transport east-west corridor. Rationale The theory of change for the second objective was that improving the rail infrastructure, signaling and power supply systems, and the rolling stock would increase the ADY's handling capacity to meet future demand and increase transport speeds and reliability. The institutional reforms, especially the ADY's corporatization that was expected to increase its commercial drive, the financial management system, audits, training, and other institutional activities were expected to enhance the ADY's institutional and operational capacity. The ADY's increased handling, institutional, and operational capacity, in turn, were expected to lead to greater operating and cost efficiencies. It was also expected to increase the rail freight traffic volumes under the assumption that freight transport demand would go up. The increased traffic Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) volumes, together with greater operating and cost effectiveness and the ADY's capacity improvements, were expected to enhance its financial sustainability. Finally, the improved handling, operational, and institutional capacity, the greater operating and cost efficiency, and the enhanced financial sustainability were expected to enable the ADY to meet the future freight demand in a timely and reliable manner and at competitive tariffs and speeds, hence improve its competitiveness. Outputs and Intermediate Outcomes The outputs and intermediate outcomes that contributed to the achievement of this objective are the same as for objective 1. Outcomes This objective was only partially achieved, and is rated modest. The project partially strengthened the ADY's capacity. The ADY's operating and cost efficiency improvements took place, but the extent of their attribution to the project is questionable. The project's contribution to the ADY's financial sustainability was minor. The ADY's competitiveness modestly improved. Capacity of the ADY Through the reforms, institutional activities, systems, and training, the ADY strengthened its institutional and operational capacity, especially in terms of financial management and planning. The ADY's also created the capacity to respond to oil spills, which is likely to enhance its reputation with oil companies. The intermediate outcome indicators related to the reform and institutional aspects were achieved. The track rehabilitation allowed for higher and more steady traffic speeds, hence partially increased the ADY's handling capacity. The other project activities, such as the new signaling system, the conversion of the power supply system, and the new locomotives, were not completed nor in use by project end. These activities were expected to further increase the ADY's handling capacity. Operating and cost efficiency The ADY's operating and cost efficiency improved. The operating cost/net ton-km ratio was reduced by 12 percent and exceeded its target of 2.5 percent. The operating revenues/net ton-km increased by 10 percent and exceeded its target of 5 percent. The electric consumption per transport unit (kWH per 10,000 ton or passenger-km) decreased to 137 and exceeded both the revised target of 170 and the original target of 174. The net ton-km per new locomotive used on the east-west decreased to 101 and exceeded the revised target of 150 and the original target of 260. Although the ADY's increased institutional capacity and the track rehabilitation, which increased travel speeds and reduced the energy consumption, likely contributed to these achievements, it is not clear how much can be attributed to the project. This is because most of the physical investments on the east-west corridor, which were to play a key role in enhancing operational and cost efficiency, were not completed by project end and Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) most of the outcome indicators capture the progress of the ADY as a whole and are not specifically related to the east west corridor. Financial sustainability The ADY achieved the financial ratio targets under the project. However, they provide insufficient evidence that the ADY's financial sustainability increased because the ratios only give an insight on the ADY's financial health on two dimensions. The trends did also not move in the desired direction. An increasing operational cash flow to revenues ratio is desirable because it means that the ADY has greater amounts of operating cash available. With 10 percent, ADY's operational cash flow to revenues ratio exactly reached its revised project end target. However, 10 percent appears low compared to the original target of 34 percent. In addition, the ratio ranged from 13 percent in 2007 to 10 percent in 2012, with a peak of 21 percent in 2009 (2013 AF PP, page 19 - the data for 2013 and 2016 is not included in the ICR). This shows a declining trend after 2009 whereas a consistent and/or increasing trend in this ratio is a positive indication of good debtor’s management (https://www.readyratios.com/reference/cashflow/operating_cash_flow_sales_ratio.html). For the working ratio, meaning the operating expenses, excluding depreciation, divided by the total revenues, the most important consideration is again the direction is takes over time. An operating expenses to revenues ratio that is decreasing over time means that the company is operating more efficiently from period to period (https://yourbusiness.azcentral.com/operating-expenses-sales-ratio-indicate-10946.html). With 87 percent, the working ratio of the ADY also exactly reached its target, but it showed an increasing trend between 2007 and 2010, a drop between 2011 and 2012 (2013 AF PP, page 19) and an increase to 87 percent in 2017 (the data for 2013 and 2016 is not included in the ICR). The ratio is also far above its original target of 56 percent. The project, however, might have had some positive impacts on the ADY's financial health. The task team noted that the ADY's financial sustainability depends on several aspects, which include (i) good practices and an enabling environment for better fiscal control, (ii) careful debt management, monitoring, and adherence to debt ceilings, and (iii) maximizing revenues and optimizing expenditure. The project helped build good practices and an enabling environment for better fiscal control, and the respective outputs and intermediate outcome indicators were achieved. According to the task team, the ADY also carried out careful debt management and monitoring, but this is not attributable to the project activities. With respect to expenditure optimization, the track rehabilitation on the east-west corridor increased and regularized traffic speeds. This likely contributed to the reduction in electric consumption. The rehabilitated tracks also reduced the maintenance needs. However, the real operating cost savings were to come from the new signaling and power supply systems and the locomotives, which were not in use by project end. Operating revenues did not increase as a consequence of the project because the physical handling capacity was not increased as expected and the demand for rail freight services on the east-west corridor went down. Competitiveness The ADY's competitiveness modestly improved as a consequence of the project. The ADY strengthened its institutional, operational, and oil spill response capacity, but its physical handling capacity to meet the future freight demand in a timely and reliable manner was only partially improved. Similarly, by project end, rail Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) speed only moderately increased. The operating and cost efficiency improved, but there are questions about its attribution to the project. The project's contribution to improve the ADY's financial sustainability was limited. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Objective 1 and 2 were only modestly achieved, and the overall efficacy is considered modest. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency Economic Efficiency According to the PAD (pages 65 to 70), the cost-benefit analysis (CBA) at appraisal compared the cost of the investments against the economic benefits, using "do-nothing" and "do-something" scenarios. The analysis looked separately at (i) track rehabilitation, and (ii) the locomotives, conversion of the power supply system, and oil spill response capability. It used a discount rate of 12 percent. The analysis period is not clear in the PAD. For track rehabilitation investments, the benefits only considered track maintenance and energy savings. The analysis showed a net present value (NPV) of US$64 million and an economic internal rate of return (EIRR) of 27 percent. The sensitivity analysis still revealed robust returns. For the locomotive and power conversion component, the analysis focused on the traffic that would be lost without investments. It showed a NPV of US$186 million and an EIRR of 18 percent. The analysis indicated that this component was quite sensitive to the level of traffic, but the EIRR remained above the 12 percent discount rate in all cases. For the project overall, the analysis showed a NPV of US$250 million and an EIRR of 19 percent. At the time of the AF, the economic analysis was repeated for the electric power supply system, the cost of which had increased since the original evaluation, and the new signaling system. The analysis used a discount rate of 12 percent and an evaluation period of 25 years. The analysis compared the "do-nothing" and "do- something" scenarios. The benefits included (i) reduced accidents, (ii) reduced maintenance costs of the signaling system, (iii) reduced maintenance, renewal and energy costs of the electric power supply, and (iv) increased rail traffic capacity. The results showed an EIRR of 14.4 percent and a NPV of US$85.9 million. The sensitivity analysis with a 25 percent traffic growth still showed and EIRR of 13.1 percent (2013 AF PP, pages 14 to 18). Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) The ex post CBA used the same methodology of comparing investments against the economic benefits, using "do-nothing" and "do-something" scenarios. The analysis applied a discount rate of 8 percent, which is lower than the discount rate of 12 percent used for the ex-ante analysis, and an evaluation period of 25 years. It also used actual data as of December 30, 2017, forecasted the remaining targets until the completion of the project, and used the same approach as for the AF (ICR, annex 4). The analysis was carried out for a project cost of US$952, which included the track rehabilitation and signaling and power supply systems under component 1. The ICR (para 60) mentions that the analysis yielded a NPV of US$10.0 million and an EIRR of just 3 percent. The task team confirmed that the analysis yielded a negative NPV of US$110 million. With the discount rate of 12 percent used at appraisal, the negative NPV would be significantly higher. Administrative Efficiency The project had significant administrative inefficiencies. The project cost increased by 66 percent compared to the appraisal estimate of the original project. The closing date was extended by five years and six months. When the project closed, a significant part was not completed. Based on the significant shortcomings in the economic and administrative efficiencies, the overall efficiency is rated negligible. Note: The appraisal rate in the table below is the one of the appraisal of the AF. This is when the real needs had become apparent and the project was adjusted accordingly. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 57.00 Appraisal  14.40  Not Applicable 72.00 ICR Estimate  3.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) 6. Outcome The relevance of the objective is rated substantial in the light of the drastic decrease in the demand mainly for rail oil transport services. The project outcomes were only partially achieved and the efficacy is rated modest. The efficiency of the project was negligible. Hence, the project outcome is rated unsatisfactory. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome There is a high risk that the development outcomes will not be achieved, as follow:  Freight transport demand. The ICR (para 114) points out that the Government of Georgia is developing the Batumi Port, which is likely to have a positive impact on the demand of the east- west corridor in Azerbaijan. Other factors that might increase the demand for rail freight services are increased global and regional oil demands. However, the demand has plummeted over the last 10 years and the recovery might not be as quick and as complete as expected.  Financial sustainability. Given the limited demand in rail freight services on the east-west corridor, there is a strong likelihood that the ADY's financial sustainability might suffer.  Operational and cost efficiency. Without the completion of the signaling and power conversion contracts, the expected operational and cost efficiency improvements cannot be obtained. The ICR (para 79) mentions that the project was not extended because of lack of clarity on the future of the signaling system. However, the task team noted that the Borrower now has a concrete plan to complete this system and the power system conversion contract. The task team also pointed out that the railway sector constitutes a priority for the Government of Azerbaijan, hence there is a strong likelihood that the systems will be completed.  Capacity and institutional reforms. The ICR (para 113) points out that the institutional reforms were being continued. They were supported by a policy-based operation financed by the ADB, which focused on efficiency and financial sustainability. Although these reforms and improvements seem well grounded, a risk remains that they might not be continued or that certain aspects might be reversed.  Cost to deliver the remaining activities under the project. There is a significant risk that the completion of the signaling system and the power system conversion contracts will be more costly than anticipated, especially because some parts of the signaling systems might become obsolete, for example, computer software and hardware, which were customized by the contractor. Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) 8. Assessment of Bank Performance a. Quality-at-Entry The Bank performance at entry is rated moderately unsatisfactory. The Bank team soundly grounded the project design in the Borrower's Program for the Modernization of the Railway Sector. The preparation of this Program was financed through a trust fund managed by the Bank, which comprehensively reviewed the sector, provided recommendations for the modernization of the ADY, and identified priority investments to rehabilitate and expand the system. The project design also benefited from an adequate stakeholder engagement. The Bank team set realistic objectives and designed a relatively good results framework. The Bank team adequately handled the environmental, social, and fiduciary aspects at appraisal. The PAD is well-written. However, project preparation had serious shortcomings with respect to the technical assessments. The ICR (para 69) points out that the Bank team had limited time to thoroughly assess the technical implications of the Borrowers decision to convert the existing power supply system from 3 Kv DC to 25 Kv AC in enough detail. The task team clarified that the project initially was to finance track and signaling system improvements, with the locomotives expected to run on diesel. Late during preparation, the Borrower decided to convert the existing power supply system from 3 Kv DC to 25 Kv AC. This decision also impacted the requirements in terms of the signaling system. Instead of delaying the appraisal to carry out the necessary detailed technical studies, the Bank team deferred them to the project implementation phase. The cost figures for the power supply system conversion and the signaling systems were rough estimations. The PAD is silent on these details. The ICR (para 101) points out that due to the lack of an early technical assessment of the signaling system, the project costs were majorly underestimated, which impacted the project duration and its economic viability. It also recognizes that although the Bank team was led by experienced railways specialists, the project would have benefited from more focused technical expertise during preparation, especially for the signaling system. Finally, it points out the technical review of works phasing and interface requirements as another area that showed deficiencies, leading ultimately to project modifications and impacting the implementation performance. A further shortcoming relates to the demand forecast. The ICR (para 58) mentions that the Bank team misjudged the impact of the oil pipeline on the demand for rail freight demand. Because demand forecasting is difficult, it would have warranted special attention, but nothing is documented in the PAD. The Bank team adequately identified several critical project risks. In hindsight, however, the Bank team underestimated the overall complexity of the project and missed risks, such as (i) the inadequacy of the originally proposed upgrading solution for the signaling system, (ii) the non-materialization of the rail traffic demand, (iii) significant cost increases, and (iv) the compatibility of the proposed systems with neighboring countries. Some of the risk mitigation measures were also weak. For instance, providing technical assistance and the support of the reform program by various ministers is not enough to mitigate institutional resistance to change in the ADY and a change management activity could have been included under the project. Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank performance during supervision is rated moderately satisfactory. The Bank's task team leader changed four times, but according to the ICR (para 82), this did not negatively affect the support to the Borrower because the teams consistently engaged in high quality technical support. The ICR lists the following examples of Bank support to the Borrower: (i) methodologies, benchmarks, and technical recommendations, backed by case studies, to manage the power conversion contract, (ii) safety assessment and recommendations for improvements at level crossings, and (iii) a detailed road map for project completion when the contractor abandoned the contract for the signaling system and the Bank decided not to extend the closing date. The ICR (para 82) points out that the Bank's advice varied in terms of timeliness, effectiveness, and consistency. The task team clarified that consistency of advice referred to the fact that for certain issues, such as the full or partial replacement of electricity poles, the Bank engaged in lengthy discussions whereas for other technical issues, little is recorded in the project documents. In terms of effectiveness, the Bank team pointed out that certain advice, such as the creation of a steering of committee led by the Minister to monitor project implementation and resolve bottlenecks, resulted effective whereas the advice to solve the impasse related to the signaling system contract was not taken on board by the Borrower. Regarding the timeliness, some advice, such as the suggestion to create the steering committee, could have been provided earlier. In institutional terms, the Bank team supported the Borrower through measures, such as twinning efforts with the Lithuanian Railway Administration and the hiring of consultants to provide support on specific technical areas. The ICR (para 105) points out that the reform implementation process, however, was in many aspects an exercise of learning through doing. The ICR (para 82) notes that the support through the Bank's country office was highly satisfactory because of the continuous and timely engagement of the staff with the Borrower. The task team noted that there were two team members in the country office, which provided day-to-day support to the Borrower, clarified Bank rules and procedures, and solved operational issues. These team members had the trust of the Borrower and were able to engage in a frank dialogue. The Bank team undertook regular implementation support missions, which increased in frequency during key implementation milestones. The Bank team offered clear guidance to the ADY through the missions’ Aide Memoires (ICR, para 83). Although the physical project results were adequately monitored, the ICR (para 107) reports shortcomings in the monitoring of the achievement of the financial sustainability targets of the project, especially in the earlier years of project implementation. With respect to procurement, the Bank team did not recognize facts, such as large contracts with only one responsive bidder and the weak contract performance, as red flags, and hence did not exercise additional scrutiny. Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) The Bank team responded to the changed needs of the Borrower, especially after the 2008 economic and financial crisis, by approving the AF and restructuring. According to the ICR (para 105), the preparation and delivery of the AF were quick, and this did not jeopardize its design and appraisal quality. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design included 10 outcome indicators and six intermediate results indicators. These indicators were largely adequate to measure the achievements of the five dimensions of the PDO. Only the competitiveness dimension lacked a more direct measure. However, its achievement could be reasonably deduced from the other indicators. For the institutional dimension, only intermediate results indicators were available. These indicators were adequate due to the nature of the institutional interventions. Although the outcome indicators were of high quality because they measured real outcomes, they had some limitations in terms of attributability of the results to the project. This is because most of them measured results for the ADY as a whole and were not limited to the east-west corridor, where most project interventions were focused on. In addition, the indicator measuring the increase in freight transport volumes was based on the assumption that freight traffic demand would grow, which is outside the control of the project. All indicators had clear baselines and end targets, except for one, which looked like an oversight. However, the baseline values were only estimates. Several of them revealed incorrect during project implementation and required changes. This also affected project end target values. The M&E arrangements were clearly defined. The ADY was responsible for preparing progress reports with the project results. A monitoring consultant assisted the ADY in this task. b. M&E Implementation The ADY measured the project indicators regularly and reported them during most of Bank implementation support missions, including the first and last ones. The ADY experienced delays in submitting progress reports, and there were inconsistencies in the reporting. The indicators were not changed during project implementation, but, as previously mentioned, baselines and targets were modified. Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) c. M&E Utilization The ADY and the Bank team used the M&E data to agree on certain changes to the monitoring arrangements. These changes included reports showing changes in the number of kilometers under speed restrictions and the speeds on these sections. The ADY and the Bank team also used the M&E data to design the AF and restructurings and improve the indicator targets. The ICR (para 89) notes that the discussions between the ADY and the Bank team focused majorly on delays in the implementation of the different project components and that there was little evidence that the M&E data was used to initiate or influence the dialogue toward rectifying project progress and attaining outcomes and objectives. The ICR cites the lack of progress on the debt service coverage ratio as an example. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The original project was classified as category B for environmental assessment purposes because the anticipated adverse impacts were not found significant or irreversible and were expected to be successfully addressed through preventive actions or mitigation measures. The following safeguards policy was triggered: environmental assessment OP/BP4.01 (PAD, para 41). An Environmental Assessment and Environmental Management and Monitoring Plan were prepared. These documents identified the generic impacts of the track rehabilitation works and the purchase of diesel and DC locomotives and specified generic prevention and mitigation measures. With the 2013 AF, the project's classification remained the same. However, the Borrower prepared an Environmental Assessments for the power conversion contract and an Environmental Management Plan to replace the entire signaling system as part of the design supply and installation contract. During implementation, the Environmental Management Plan was amended to account for the acquisition of more environment-friendly AC electrical locomotives instead of diesel DC locomotives (ICR paras 92 and 93). According to the ICR (para 94), the environmental safeguards performance under the project was satisfactory. The operation of the three construction camps was satisfactory, with health, safety, and environment measures duly established. There were some shortcomings, such as inadequate waste burning practices, lack of proper transportation and disposal of waste, and safety for workers on-site. These issues were promptly rectified. b. Fiduciary Compliance Financial management. The project's financial management performance was largely satisfactory. The project had an adequate financial management system, which was well maintained and able to generate Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) all necessary accounting data. The ICR (para 96) notes that the PIU was in compliance with the financial management legal covenants. Acceptable financial management and disbursement arrangements were in place, required financial reports were provided on time, and the financial audit reports had unqualified opinions. However, there were delays in the submission of the financial audit reports. The project helped the ADY establish a culture of auditing in line with international practices. Procurement. The procurement performance of the project had serious shortcomings. The signaling system and the power system conversion contracts, two complex and high-value contracts, both only had one responsive bid. The ICR (para 98) points out that the fact that such high-value contracts resulted in only one bid being considered responsive suggests several possible weaknesses. These included (i) the technical requirements not being set properly, (ii) restrictive qualification requirements, (iii) an incorrect procurement strategy, or (iv) inadequate consultation with the prospective bidders with lack of bidders' awareness. The ICR (para 98) notes that there was a significant number of red flags throughout the procurement process from planning to contract implementation. The Borrower and the Bank team failed to recognize and hence mitigate them, especially the low participation of responsive bidders and the weak contractor performance. The contracts for the signaling system and the supervision consultants performed poorly, with the need to partially resolve the former. The ADY also revolved the contract for the conversion of the power supply system after the preparation of the ICR. The task team noted that complaints of fiduciary irregularities related to the contract for the signaling system prompted careful reviews by the Bank's Integrity Vice Presidency and fiduciary and technical staff to fully assess and minimize potential risk. Finally, the ICR (para 99) points out that the procurement processes incurred serious delays, and that overall, the procurement processes did not fully comply with Bank procurement procedures, and contract management and monitoring were unsatisfactory. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory Moderately Moderately Because of the shortcomings in Bank Performance Satisfactory Unsatisfactory the quality at entry Quality of M&E Substantial Substantial Quality of ICR --- Substantial Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) 12. Lessons The following are lessons from the ICR with some modification in language introduced by IEG:  Capable supervision consultants are vital for technically complex projects. The poor performance of the original consortium supervising the implementation of the railway systems negatively impacted the implementation progress, particularly on issues such as the interface between the system's subcomponents. Although the ADY did not renew the contract of the consortium, the ADY should have taken earlier and stronger steps to ensure adequate performance, including the possible early termination of the contract.  The success of institutional reform activities requires as a prerequisite, strong ownership and commitment at different government levels through the entire life of the project. Such ownership and commitment are required not only at the strategic ministerial levels, but also at the level of the directly concerned implementing bodies within institutions. Under this project, the ownership and commitment of the concerned implementation bodies was lacking during earlier stages of project implementation. This negatively influenced the performance of the project. The identification of champions to lead the institutional reforms can also contribute positively to the success of the reforms.  Capacity building and technical support planned in advance yield better results. Various capacity-building and technical support activities were introduced during project implementation. This approach offered a good level of flexibility when such activities were done on time. A pre-planned and well-structured capacity development program, with precise responsibilities, timelines, and a vision, designed at the time of project preparation, however, would have yielded better results.  Careful and diligent planning in the early stages of project implementation is important for the smooth implementation of technically complex project. Several activities and subcomponents under the project were either introduced, modified, or cancelled during implementation. Although a number of the changes were needed because of factors beyond the Borrower's control, some changes were required because of inadequate early planning. This negatively affected project implementation. The design of the activities in such a complex project and their implementation should be very carefully and diligently undertaken during the early stages of project preparation. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Two large project activities had still to be completed when the loan closed. This led to a high risk that the development outcomes would not be achieved. Therefore, it is recommended to carry out an assessment to Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RAIL TRADE & TRNSPT FACIL (P083108) verify if the project outcomes were eventually achieved. The ICR also includes a number of mistakes, mainly in reporting results, which should be confirmed in the field. 14. Comments on Quality of ICR The ICR is comprehensive and well-written. It presents well the different project changes and provides a relatively clear picture of the project implementation experience. Problems and shortcomings are discussed in a candid way. This is especially true for the sections on key factors during preparation and implementation and the procurement section. Environmental and fiduciary reporting is comprehensive and well done. The ICR sets out the theory of change in a graphical way, but does not follow it when presenting the outcomes and results. Nevertheless, the narrative that presents the project achievements is relatively clear. The project end values for several indicators are different in this narrative and in annex 1 of the ICR. Although, according to the task team, this is because the data in annex 1 is directly picked up by the system, the Bank team should at least have highlighted this difference in the ICR. Given the nature of some of the indicators, such as the financial ratios, the quality of reporting could have been enhanced by including comments with details on how the results needed to be interpreted and to which extent they are attributable to the project. The ICR also shows slightly different project component costs in different section. Considering that the project activities were not completed at ICR preparation, it is understandable that the project cost figures were estimated. However, again, the Bank team should have included explanation in the ICR and ensured consistency. Finally, the ICR also reports a small positive NPV whereas the actual NPV was negative. This did no impact the overall efficiency narrative and rating. a. Quality of ICR Rating Substantial Page 21 of 21