Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004975 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-50780; IBRD-81430 ON A CREDIT IN THE AMOUNT OF SDR 25.8 MILLION (US$ 40 MILLION EQUIVALENT) AND A LOAN IN THE AMOUNT OF US$ 30 MILLION TO GEORGIA FOR THE Second Secondary and Local Roads Project (SLRP II) December 23, 2019 Transport Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective Nov 08, 2019) Currency Unit = Georgian Lari (GEL) GEL 2.84 = US$1 US$ 1.38 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Cyril E Muller Country Director: Sebastian Molineus Regional Director: Lucio Monari Practice Manager: Karla Gonzalez Carvajal Task Team Leader(s): Nijat Valiyev, Aymen Ahmed Osman Ali ICR Main Contributor: Nijat Valiyev ABBREVIATIONS AND ACRONYMS AADT Average Annual Daily Traffic MESD Ministry of Economy and Sustainable Development ADB Asian Development Bank MRDI Ministry of Regional Development and Infrastructure CPS Country Partnership Strategy NBG National Bank of Georgia EA Environmental Assessment NCB National Competitive Bidding EIRR Economic Internal Rate of Return NPV Net Present Value EMP Environmental Management Plan OPRC Output and Performance-Based Road Contracting ESMF Environmental and Social ORAF Operational Risk Assessment Management Framework Framework ETCIC / Eurasian Transport Corridor PAD Project Appraisal Document TRRC Investment Centre, previously the Transport Reform and Rehabilitation PBC Performance Based Contract Center FA Financing Agreement PDO Project Development Objective FEWHIP First East West Highway PIP Project Implementation Plan Improvement Project FM Financial Management PIU Project Implementation Unit FMAR Financial Management Assessment POM Project Operational Manual Report FMM Financial Management Manual RAMS Road Asset Management System RAP Resettlement Action Plan FPU Foreign Project Unit RD Roads Department GEL Georgian Lari (currency unit) RPF Resettlement Policy Framework HDM 4 Highway Design and Maintenance SDR Special Drawing Rights model IBRD International Bank for SEWHIP Second East West Highway Reconstruction and Development Improvement Project ICB International Competitive Bidding SIL Specific Investment Loan IDA International Development SLRP Secondary and Local Road Project Association IFI International Financing Institution SLRP-II Second Secondary and Local Road Project ISP Implementation Support Plan SLRP-III Third Secondary and Local Road Project KRRIP Kakheti Regional Roads SRAMP Secondary Road Asset Management Improvement Project Project MENR Ministry of Environment and Natural TOR Terms of Reference Resources TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs ............................................................................................................ 11 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 12 C. EFFICIENCY ........................................................................................................................... 17 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 20 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ................................................................................... 22 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 23 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 29 V. LESSONS AND RECOMMENDATIONS ............................................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 32 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 37 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 39 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 40 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 44 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 45 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P122204 Second Secondary and Local Roads Project (SLRP II) Country Financing Instrument Georgia Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Roads Department, Ministry of Regional Development Georgia and Infrastructure (MRDI) Project Development Objective (PDO) Original PDO To improve local connectivity and travel time for selected secondary and local roads, and to strengthen the capacity of the Roads Department to manage the road network. Page 1 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 40,000,000 38,983,295 38,384,947 IDA-50780 30,000,000 27,486,253 24,117,036 IBRD-81430 Total 70,000,000 66,469,548 62,501,983 Non-World Bank Financing 0 0 0 Borrower/Recipient 17,500,000 0 0 Total 17,500,000 0 0 Total Project Cost 87,500,000 66,469,548 62,501,983 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 15-Mar-2012 24-May-2012 02-Feb-2015 30-Jun-2017 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 09-Jul-2015 51.00 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 26-Dec-2012 Satisfactory Satisfactory 13.55 02 26-Jun-2013 Satisfactory Moderately Satisfactory 22.48 Page 2 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 03 04-Jan-2014 Satisfactory Moderately Satisfactory 34.87 04 30-Jun-2014 Satisfactory Satisfactory 42.40 05 30-Jan-2015 Moderately Satisfactory Moderately Satisfactory 47.44 06 18-Aug-2015 Moderately Satisfactory Moderately Satisfactory 51.00 07 27-Feb-2016 Satisfactory Satisfactory 52.73 08 07-Sep-2016 Satisfactory Moderately Satisfactory 56.16 09 18-Apr-2017 Satisfactory Moderately Satisfactory 56.93 10 15-Dec-2017 Satisfactory Moderately Satisfactory 60.51 11 29-Jun-2018 Satisfactory Moderately Satisfactory 63.21 12 10-Jan-2019 Satisfactory Moderately Satisfactory 64.61 13 28-Jun-2019 Satisfactory Satisfactory 64.61 SECTORS AND THEMES Sectors Major Sector/Sector (%) Transportation 100 Public Administration - Transportation 2 Rural and Inter-Urban Roads 98 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Economic Policy 25 Trade 25 Trade Facilitation 25 Private Sector Development 33 Jobs 8 Job Creation 8 Regional Integration 25 Page 3 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Urban and Rural Development 41 Urban Development 8 Urban Infrastructure and Service Delivery 8 Rural Development 33 Rural Infrastructure and service delivery 33 ADM STAFF Role At Approval At ICR Regional Vice President: Philippe H. Le Houerou Cyril E Muller Country Director: Asad Alam Mercy Miyang Tembon Director: Laszlo Lovei Lucio Monari Practice Manager: Henry G. R. Kerali Karla Gonzalez Carvajal Task Team Leader(s): Joseph Melitauri Nijat Valiyev ICR Contributing Author: Nijat Valiyev Page 4 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Georgia lies on the southern flank of the Greater Caucasus mountain range, with Russia to its north, Armenia and Turkey to its south, and Azerbaijan to its east. The country is mountainous with elevations rising from sea level to 5,000 meters. At the time of the 2012 appraisal of the Second Secondary and Local Roads Project (SLRP-II), the national population1 was 3.7 million, of which 43.7 percent lived in rural areas. Poverty rates were falling but remained significant with 30.0 percent of the population living below the national poverty line. 2. The country had a well developed road network with a total of 20,930 km of roads. The network comprised 1,564 km of international roads, 5,466 km of secondary roads, and 3,750 km of core local roads. The international and secondary roads are managed by the Roads Department (RD) of the Ministry of Regional Development and Infrastructure (MRDI), while local roads are the responsibility of local municipalities. 3. In 2012, Georgia had mostly recovered from the August 2008 conflict and the 2008-2009 global economic crisis. The economy grew by 7.2 percent in 2011 due to an increase in exports, tourism, and continued high levels of public investment. This investment was mainly focused on the road network and aimed at boosting economic recovery by improving main road corridors and local connections, and by creating temporary employment. 4. National strategies for post crisis sustainable medium term growth were evolving during project preparation with the Government launching new initiatives to attract private investors in selected regions. In June 2010, the Government approved the State Strategy on Regional Development of Georgia for 2010-2017, followed it in 2011 with a ten-point Strategic Plan of Development2. The plan had goals of facilitating job creation and improving the welfare of the population. Among the priorities was the improvement of the country’s infrastructure, especially roads with a particular emphasis on the rehabilitation of secondary and local roads. This investment was expected to help maintain economic growth by promoting the transportation of goods within the country, increasing tourism, and revitalizing agriculture. Improved local connections would also provide the rural population with easier access to markets, with the capital investments generating employment. 5. Seventy percent of secondary and core local roads required significant improvement in 20123 despite progress having been made. Many years of underinvestment in rehabilitation and maintenance left a large rehabilitation backlog while the road administrators lacked experience in managing contracts and efficiently using limited resources. To address these issues, the Government substantially increased funding for road programs and directed significant external donor assistance into the sector. The largest allocations were made to improve the international road links, such as the East- West Highway (E60), but a significant part was invested in rehabilitating secondary and local roads. 6. The Second Secondary and Local Roads Project was designed to align with the investment priorities in the Country Partnership Strategy (CPS) for Georgia for FY10-13. The CPS identified two strategic pillars, (a) meeting post- conflict and vulnerability needs, and (b) strengthening competitiveness for post-crisis growth. The expected creation of 1 World Bank data https://data.worldbank.org/country/georgia 2 Strategic Plan of Development. Ten Point Plan of Modernization and Employment, 2011-2015. Government of Georgia, October 2011 3 World Bank PAD for SLRP-II, February 2012 Page 5 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 12,000 person-months of direct employment over the duration of the project would contribute to the first pillar, while improving road connections and reducing transport costs would directly contribute to the second pillar. The local construction industry could also expect growth opportunities through works contracts in nine regions. Figure 1 Roads Before Rehabilitation Chiatura-Gezruli Road Igoeto-Lamiskana Road Mtskheta-Shio Mgvime Monastery Road 7. The SLRP-II was part of the Bank-supported comprehensive program to improve rural connectivity in Georgia (see chart below). The program aims at rehabilitation of secondary and local roads and strengthening the institutional capacity to manage the road network for achieving the higher level objectives of the enhanced growth of regional YEAR 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 Secondary and Local Roads Project AF Kakheti Regional Roads Improvement Project Second Secondary and Local Roads Project Third Secondary and Local Roads Project Secondary Road Asset Management Project - Secondary and local - Secondary road - Secondary and local - Secondary and local - Secondary road road rehabilitation rehabilitation road rehabilitation road rehabilitation rehabilitation - Strengthened capacity - Improved road safety - Strengthened capacity - Strengthened capacity - Further strengthened in regions and at local and accident data of RD with introduction of MRDI with local capacity of RD with government level to collection of OPRC and design- roads asset OPRC and design-build manage the road - Strengthened regional build contracts management improvement and network capacity to manage the - Strengthened RAMS - Further strengthened maintenance of road network for RD RAMS for RD secondary roads - Improved road safety -Strengthened road through OPRC - Strengthened safety and introduction - Enhancement of procurement of a safe systems RAMS - Introduction of socio- approach. - Climate resilience economic impact support evaluation economies through improved transport links and ensuring sustainable management of the road sector. The program has Page 6 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) been systematically expanding to improved road conditions in more regions, as well as the scope of institutional capacity building activities. While the earlier projects helped to build capacity at regional and local levels, the Second Secondary and Local Road Project (SLRP-II) mostly concentrated on strengthening the RD’s capacity by introducing alternative contracting methods. These changes are now being carried forward in later Bank financed projects along with activities supporting road safety improvements and the development of a road asset management system (RAMS). In parallel to supporting secondary and local roads, the Bank continues to be a significant partner in the upgrading of the East West Highway, Georgia’s main transport corridor. Theory of Change (Results Chain) 8. SLRP-II aimed at improving the condition and safety of secondary and local roads and strengthening the capacity of the RD to manage the road network. The results chain, derived from the Project Appraisal Document (PAD), links the actions of the project to the desired outcomes. Table 1 Theory of change Activity Output Outcome Long Term Outcomes Improved local - Rehabilitation of - 225 km of roads in connectivity and travel secondary and local roads good condition times - Enhanced growth - Roads rehabilitated of regional using design-build - Use of two alternative Strengthened capacity of economies contracts contracting methods for the RD to manage the - Roads rehabilitated road works contracting road network using: - More efficient road and maintained using - Improved access to rehabilitation and OPRC services for the rural maintenance population - Localized road safety - design-build and OPRC - Social and - Road safety improvements made in contracting economic benefits improvement schemes five regions - impact assessments to arising from fewer and the use of road safety - Safer road designs understand outcomes and road accidents. audits using auditing efficiency - data driven road asset - Social and - Impact evaluation management economic benefits for - Two impact studies on rehabilitated - design stage safety women that are evaluations undertaken roads and Kakheti OPRC4 audits engaged in OPRC - identification and - RAMS capacity for implementation of small- - Equipment and RAMs improved with scale road safety schemes. consultancy for the road consultancy and new asset management system equipment 4Output and Performance-based Contracting for Roads (OPRC). Under the Output-and Performance-based Road Contract, most of the payments to be made to the contractor are based on measured "outputs" reflecting the target conditions of the road, expressed thro ugh "Service Levels” defined in the contract. The Contractor is also responsible for the design of the works which are necessary to reach the specified Service Levels. Page 7 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 9. Critical assumptions relevant to the results chain were: - The Government maintains its ability to co-finance the project. The return of conflict or economic instability are risks that may damage this assumption; - The domestic road construction industry has the capacity to absorb the many civil works contracts expected to be launched in 2012. The scale and geographical distribution of the work make it unattractive to international civil contractors, so success depends on smaller local companies; - The concept of OPRC, which required roads to be maintained over an extended period, will be sufficiently attractive to contractors to ensure competitive bids. - Only minor land acquisition will be necessary alongside existing road corridors. Some land acquisition for footpaths and drainage may be required and is planned for, but more extensive acquisition would trigger additional safeguards and require time. Project Development Objectives (PDOs) 10. The PDO, as stated in both the loan and the financing agreement5, was: “to improve local connectivity and travel time for the selected secondary and local roads and strengthen the capacity of the RD to manage the road network.” There were no changes to the PDO during implementation. Key Expected Outcomes and Outcome Indicators 11. For the purposes of assessment, the PDO statement was unpacked into two sub-objectives: (i) improved local connectivity and travel time for selected secondary and local roads; and (ii) strengthened capacity of the RD to manage the road network. 12. Outcome (Results) indicators for each of the outcomes were described in the PAD together with methodologies for measurement. “Outcome 1: Improved conditions on selected secondary and local roads, to be measured through two indicators, i) an increase in the percentage of Secondary and Local Roads in good or fair condition from 30 percent at present to 34 percent at the end of the Project, and ii) a decrease in the current average travel time by about 20 percent over the life of the Project. Outcome 2: Increased capacity of the RD to manage the road network, to be measured through one indicator: kilometers of roads managed under alternate contract methods (as a measure of RD’s capacity to manage road network through PBC or Design / Build / Operate).” 13. PDO indicators “Roads in good and fair condition as a share of total classified roads” and “Roads in good and fair condition as a share of total classified roads” were assigned to measure the first expected outcome: Unit of Indicator Name Core Baseline End Target Measure Roads in good and fair condition as a Value 30.00 34.00 X Percentage share of total classified roads Date 31-Mar-2012 30-Jun-2017 Value 0.00 20.00 Average decrease in travel time Percentage Date 31-Mar-2012 30-Jun-2017 5 Both agreements dated March 22, 2012 Page 8 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 14. The second expected outcome used the single PDO indicator “Roads managed under alternative contracting methodology” as a measure of the RD’s capacity to manage the road network: Unit of Indicator Name Core Baseline End Target Measure Roads managed under alternative Value 0.00 200.00 Kilometers contracting methodology Date 31-Mar-2012 30-Jun-2017 The full Results Framework is included in Annex 1. Components 15. SLRP-II had two components; Component 1 focused on rehabilitating and improving selected secondary and local roads while Component 2 focused on institutional strengthening. The two components broadly aligned with the two expected PDO outcomes, although the costs of the OPRC pilot and the design-build contracts (alternative contracting methods) were included in Component 1 while its results contributed to the second PDO outcome. Table 2 Components Component Planned Activities at Entry Estimated and Actual Resource Allocation At Entry Component 1: - Rehabilitate and improve 19 road sections with a total length of US$86.0 million total, Rehabilitation and 225 km, including piloting two design-build contracts. US$69.0 million Bank financing Improvement of Selected - Rehabilitate, repair, and maintain 200 km of roads in the Actual Secondary and Local Kakheti Region using the Output and Performance-based Road US$81.3 million total Roads Contract (OPRC) model. US$61.1 million Bank financing - Strengthen the capacity of the RD to manage and implement alternative contracting methodologies. - Support the development of a road management system. At Entry - Improve road safety engineering at the design stage, including US$1.5 million total, Component 2: road safety audits. US$1.2 million Bank financing Institutional Strengthening - Strengthen the capacity of the RD to identify and implement Actual and Project Management rural road safety improvements. US$1.8 million total - Strengthen the capacity of the RD to undertake and benefit US$1.4 million Bank financing from impact evaluations. - Support the RD and the TRRC with project implementation, including financial management. At Entry Actual Project Total US$87.5 million total, US$83.1 million total US$70.0 million Bank financing US$62.5 million Bank financing B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs 16. The Project Development Objective and the Key Expected Outcomes remained unchanged during implementation. Page 9 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Revised PDO Indicators 17. The project underwent one significant change during implementation. The mid-term review undertaken in February 2015 confirmed that the project was successfully delivering outcomes related to PDO Outcome 1, improved local connectivity and travel time, but faced a challenge in fully delivering the second outcome - strengthening the capacity of the RD. At the time of the review, only 20 km out of the target 200 km were managed under the alternative contracting methodologies (two design-build contracts). The more substantial and innovative OPRC pilot was delayed due to the cancellation of bidding when the lowest bid came in three times higher than the cost estimate. The main recommendation of the mid-term review was the extension of the project closing date by at least 24 months to support the RD during the implementation of the revised OPRC. 18. Following a request from the Government of Georgia, a level-II restructuring was undertaken, based on the mid-term review recommendation, and completed in June 2015. The restructuring extended the project by 24 months until June 30, 2019, which became the end target date for all indicators. Two project PDO Indicators and two Intermediate Result Indicators were revised to reflect the changed scope of the OPRC pilot and provide more realistic targets. At the time of restructuring, US$51 million (73 percent) of the funds had been disbursed and PDO 1 Outcome almost achieved. Table 3 Project Development Objective Indicators Substantially Revised at Restructuring Unit of Original End Actual (at Revised End PDO Indicators Core Baseline Measure Target restructuring) Target Value 0.00 20.00 40.00 44.00 Average decrease in travel time Percentage The target was increased to provide an improved but achievable Reason outcome. Value 0.00 200.00 20.00 137.00 Roads managed under alternative Kilometers The target was decreased to reflect the reduced scope of the contracting methodology Reason OPRC pilot. Intermediate Results Indicators Value 0.00 225.00 186.14 200.50 Roads rehabilitated, Rural X Kilometers The original end target included a “Physical Contingency” of 24 Reason km of undefined roads. All 19 selected roads were rehabilitated as planned so the contingency was not needed. Value 0 5 19 20 Number of safety audits Number The target was increased to reflect the increased scope of this conducted Reason activity. The End Target Date for all indicators was revised to June 30, 2019. Revised Components 19. The Components remained unchanged during implementation. Other Changes 20. The restructuring extended the closing date by 24 months until June 30, 2019 to allow the project to support the implementation of the OPRC. Rationale for Changes and Their Implication on the Original Theory of Change 21. The only driver behind the changes was to provide time for the OPRC to be successfully piloted. The additional two years allowed both PDO Outcomes to be achieved but did not have any impact on the Theory of Change. Page 10 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) II. OUTCOME Split Rating 22. The change to the outcome target of the PDO indicator “Roads managed under alternative contracting methodology” was significant, reducing the length of alternatively managed road from 200 km to 137 km. Changes were also made to another PDO Indicator and two Intermediate Results Indicators. To account for the changed outcome targets, the “split rating” method has been utilized to take into consideration both the original and revised targets. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 23. The relevance of objectives is the extent to which a project’s development objectives are consistent with the Banks CPF/CPS that are current at the time the operation is closed. SLRP-II closed in June 2019 when the Country Partnership Framework for Georgia, FY19 - FY22 was current. 24. The objectives of SLRP-II align with the Country Partnership Framework for Georgia, FY19 - FY22, which became effective towards the end of the project after almost all activities had been procured and most completed. Although the new CPF was looking well beyond the project, the objectives of SLRP-II were mostly aligned with this framework. In broad terms, the current CPF envisages support to Georgia across three areas: (i) enhancing inclusive growth and competitiveness, (ii) investing in human capital, and (iii) building resilience. It is with the first area that SLRP-II best aligns as access to markets and economic participation in the regions is improved through better connectivity and reduced travel times. The project also engaged private sector participation with advanced contracting methods and strengthened the RD’s institutional capacity to better plan and manage infrastructure initiatives. Through improving landslide-affected road sections, the project contributed to building resilience of the project road sections as well as increasing the resilience of communities living along and accessing those road sections for social and economic needs and services. The extent of this alignment is examined in more depth in Annex 6.2. 25. The objectives of SLRP-II aligned very well with the Country Partnership Strategy6 (CPS) that preceded the current Framework and was current at the time of the mid-term review and restructuring. This strategy involved, in part, “Enabling private sector job creation through improved competitiveness” and “The development of secondary and local roads targets the poverty disparities among regions and provides a platform for the integration of the rural economy.” The project’s PDO to improve local connectivity and travel time for the selected secondary and local roads and strengthen the capacity of the RD to manage the road network aligned well with these CPS goals. 26. During project preparation, SLRP-II was designed to support the Country Partnership Strategy for Georgia for FY10-13. Therefore at entry, the expected 12,000 person-months of employment directly addressed the CPS strategic goal of meeting post-conflict and vulnerability needs. The project also aligned well with the other strategic goal, to strengthening competitiveness for post-crisis growth, by facilitating improved road connections and reducing transport costs in the regions. 6 Country Partnership Strategy for Georgia for the Period FY2014‐FY2017, April 9, 2014 Page 11 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 27. The overall relevance of the Project Development Objectives is Substantial. This assessment takes into consideration the alignment of the PDO to the current CPF objectives. The very good alignment to the objectives of the CPS in place during most of the Project’s operations is noted. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 28. The project’s efficacy examines the evidence for the achievement of each outcome expected from SLRP -II, which are described above. A primary measurement tool to assess the project’s outcomes is the Results Framework, which compares expected outcomes (targets) with actual results. As some of the targets were substantially changed during the project, the split rating method is used to provide separate efficacy ratings by considering targets before and after the restructuring. In addition to the results framework, the outcomes are also evidenced by other quantitative and qualitative assessments. Of particular relevance are the findings of the two socio-economic impact evaluations7 covering the rehabilitated roads and the OPRC area, prepared with the support of SLRP-II. A separate discussion of the quality of the Results Framework and indicators is included in Section IV, A. Quality of monitoring and evaluation (M&E). Outcome 1: Improved connectivity and travel time for the selected secondary and local roads 29. 200.5 km of rural roads were successfully rehabilitated by the project using traditional and design-build contracts with a further 37.5 km of road sections rehabilitated as part of the output and performance-based road contract. Spread over nine regions throughout Georgia, the project roads directly serve about 122,000 residents and connect this rural population to markets, services, and the rest of the Georgian road network. A map is included in Annex 6.6 showing the location of each road. 30. Before intervention, the roads were in poor condition making it difficult for residents and businesses to access services and markets. The rehabilitation of the selected roads reduced or removed these difficulties by decreasing journey times and increasing the frequency of public transport. The rehabilitated roads are safer and sustainable as they are now in a maintainable condition. Each of these roads are under three-year district maintenance contracts. Figure 2 Roads After Rehabilitation Kutaisi-Tsageri Road Chalaubani-Signagi-Anaga Road Nokalakevi-Ledzadzame-Didi Chkoni Road 7 - Roads Department, Socio-economic Impact Evaluation of the Second Secondary and Local Roads Project (SLRP-II), December 2015. This evaluation studied 9 of the projects 19 non-OPRC roads. - Roads Department, Road Rehabilitation Impact Assessment, July 2019. This assessment studied the OPRC area. Page 12 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 31. Reductions in travel time are captured by PDO Indicator 3: Average decrease in travel time, and indirectly by PDO Indicator 1: Roads in good and fair condition. These indicators are discussed below. Travel time was also evaluated by the socio-economic impact evaluation which found the average time decreased significantly by 0.94 minutes per kilometer. 32. Progress towards better connectivity, the other part of this outcome, was measured using PDO Indicator 1: Roads in good and fair condition and is also demonstrated well by the findings of the Socio-economic Impact Evaluation of SLRP- II. The frequency of bus usage increased for most destinations (including schools, municipal centers, hospitals, and pharmacies) by about 4 percent. A larger increase (10 percent) is seen in the frequency of minibus journeys to schools. Correspondingly, walking decreased as a mode of travel for visiting markets, municipal centers, pharmacies, and schools. Local businesses reported having increased the distance driven by about 12 percent, and over 90 percent of both residents and businesses stated their access to various services has “considerably improved” or “improved.” A more detailed summary of impacts, taken from the Socio-economic Impact Evaluation report, is included in Annex 6.5. 33. PDO Indicator 1: Roads in good and fair condition. Unit of Original End Revised End Reported Indicator Name Baseline Measure Target Target Result Roads in good and fair condition Value 30.00 34.00 Not revised 34.00 Percentage as a share of total classified roads Date 31-Mar-2012 30-Jun-2017 30-Jun-2019 30-Jun-2019 34. The End Target for PDO Indicator 1 was achieved. The project rehabilitated 238 km of roads including those rehabilitated under OPRC. Of these 45.8 km were classified as local roads and 192.3 km as secondary roads. The secondary roads in good and fair condition increased from a baseline of 30 percent to 34 percent, when considering only the impact directly attributable to the project on the secondary road network. If improvements made through all IFI and state funded projects are considered, the RD reports 66 percent of all secondary roads are now in good or fair condition. This indicator is discussed further in Section IV. A. Quality of Monitoring and Evaluation (M&E) as the indicator name/description refers to the “total classified network” rather than the “secondary road network” which was measured under the project. 35. PDO Indicator 3: Average decrease in travel time. Unit of Original End Revised End Reported Indicator Name Baseline Measure Target Target Result Value 0.00 20.00 44.00 44.00 Average decrease in travel time Percentage Date 31-Mar-2012 30-Jun-2017 30-Jun-2019 30-Jun-2019 36. The methodology for Indicator 3 compared each non-OPRC project road with an average speed before and after rehabilitation. These speeds were weighted by road lengths to give a total travel time for all rehabilitated 19 roads at the start and end of the project. The baseline travel time of 5.97 hours was reduced by the end of the project to 3.34 hours, an average decrease in travel time of 44.1 percent (5.97- 3.34)/5.97 = 44.1%). Therefore, the result achieved the revised target for Indicator 3, exceeding the original target. 37. SLRP-II achieved intended Outcome 1, Improved local connectivity and travel time for the selected secondary and local roads, as demonstrated by the PDO Indicators 1 and 3. Outcome 2. Strengthened capacity of the Roads Department (RD) to manage the road network. 38. The project strengthened the capacity of the RD to manage Georgia’s road network in several diverse ways. The ability of the RD to efficiently manage and implement network improvements was improved by introducing OPRC Page 13 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) and design-build as alternative contracting methods to reduce costs and transfer risk to the private sector. Monitoring and planning capacity increased with the support of consultancy services and equipment for road asset management. Impact evaluations strengthened the RD’s capacity to assess how its investments affected the conditions of the rural population and to better demonstrate such outcomes. The project also supported small-scale road safety schemes and the use of road safety audits, thereby supporting and strengthening RD’s capacity for accident reduction. 39. One of the two alternative contracting methods introduced under SLRP-II was design-build for secondary roads. The project supported the RD in the preparation of feasibility studies, preliminary designs and bidding documents for the two roads, Mtskheta-Shio Mgvime Monastery road (8.24 km) and Chiatura-Perevisa-Sveri-Tvalueti-Gezruli road (10.03 km), both in Mtskheta Mtianeti Region. The contracts were bid, and each awarded to the same joint-venture contractor in 2014. Despite one of the partners having financial difficulties during implementation (unrelated to the use of a design- build contract) both roads were successfully completed. The RD has now moved forward with design-build contracts and included this contracting method in the Bank-financed Third Secondary and Local Roads (SLRP-III) and Secondary Road Asset Management Projects (SRAMP). 40. The other alternative contracting method introduced by SLRP-II was Output and Performance-based Road Contracting covering part of the rural road network in the Kakheti Region. Bank supported activities working toward the introduction of this method, and other alternative methods of contracting, started some years before the start of SLRP- II. In 2010, a consultancy contract for the Preparation of a Framework for the Design and Implementation of Concessions (Contracts) for Performance-Based Road Maintenance commenced under the Third East-West Highway Improvement Project (TEWHIP). This Framework was utilized by SLRP-II, which developed technical and contractual documents specifically for the Kakheti OPRC pilot. This pilot project was designed so that 200 km of paved and unpaved roads, in various conditions, would be improved to specified levels and then maintained for the remainder of a five-year contract period. However, when bids were invited for the OPRC pilot in May 2014, the lowest bid was three times the cost estimates. This led to the cancellation of the procurement process and a review and revision of the pilot’s scope. The RD’s review established bidders had priced in several risks which substantially affected the bid prices. These risks included uncertainty about the cost of maintaining gravel roads, bridge repair costs, and unspecified risks related to the novel type of contract. With the support of the Bank, the RD revised the scope of the work to suit the available budget and reduce some risks. The revision reduced the length of paved roads to be rehabilitated to 37.5 km, excluded gravel roads and bridges, and reduced the total maintained road length to 117 km. 41. The revised OPRC pilot was rebid in August 2015 for the rehabilitation of 37.5 km and maintenance of 117 km of road. A local Joint Venture contractor was awarded the contract in March 2016 and commenced work shortly after. Progress was slow at the start mainly due to the novel aspects of the pilot. The Bank worked closely with the RD to move the contract forward and through a series of contract amendments, the performance indicators and the timing of deliverables were refined to ensure outcomes were achievable. At the close of the project, all the rehabilitation work was completed with routine and winter maintenance ongoing and now funded from the state budget. 42. Household surveys in the Kakheti OPRC area reported 74 percent of households considered roads in their settlement had improved over the last five years. Residents also perceived a reduction in journey time but were concerned about vehicles being driven at high speeds. The Project also supported additional road safety measures on the OPRC sections after the joint inspection was carried out by the RD, Police, Contractor and Monitoring Consultant, which helped reduce the risk of accidents. General satisfaction with the road use increased from 27 percent in 2015 to 63 percent in 2018, an increase significantly higher than satisfaction levels measured in a control group. Page 14 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) “It is so comfortable to move from the village… As the roads are comfortable, many people go the center for work. They go to Kvareli for work… It takes us less time to move”. [Woman, Sighnaghi Municipality] 43. The experience and lessons learned by the RD from managing the OPRC contract are being carried forward. Once operational, the OPRC approach was well accepted by the RD and they are now developing a hybrid OPRC for 240 km of roads in the Guria Region, incorporating lessons learned from the Kakheti pilot. The Guria OPRC is also supported by the Bank through the SRAMP and should commence in 2020. A further performance-based contract is under preparation for secondary roads supported by the Asian Development Bank (ADB). The experience and lessons learned, identified by the supervising engineer8 and in a World Bank report9 on Georgia’s OPRC experience, are summarized in Annex 6.4. 44. Ensuring road asset management systems were in place and operational was key to strengthening the capacity of the RD. The World Bank has been supporting the introduction of road asset management for more than a decade through a series of different projects. SLRP-II continued this support through consultancy services, the development of technical specifications, and the procurement of a survey vehicle and equipment (including GPS). The RD now operates the RAMS and prepares annual reports for the RD and maintains a 5-year rolling program. The ongoing use of RAMS by the RD demonstrates a real strengthening of their capacity to manage the network and represents some of the best practice in the Region. 45. Two impact evaluations were completed covering nine of the rehabilitated roads and the OPRC area in Kakheti. These studies assessed the contribution of SLRP-II to socioeconomic conditions of the rural population as a measure of the success of road investments in Georgia. In particular, the study in the Kakheti region is an integral part of the pilot project and data from the recently completed survey will inform the government’s future policy about the management of its network. As an indicator of the RD’s commitment to the use of impact evaluations as a tool, they undertook the final survey for the OPRC area from the state budget after the project closed. 46. The project introduced the concept of standalone small-scale road safety schemes for rural roads. The RD, through two project financed consultancies, identified 22 village locations in five regions to benefit from road safety improvements. The improvements, mainly installing better signs, road markings, guard rails and safety islands, was tendered and awarded in three contracts - all completed during 2018 and early 2019. The RD, through its Road Safety Division, plans to continue with standalone small-scale road safety schemes using the state budget. Post evaluation of the effectiveness of the schemes can be undertaken by the RD once new traffic patterns and driver behaviors are fully established. Figure 3 Road Safety Improvements 8 OPRC Final Report, February 2019, Eptisa/Temelsu 9 Georgia Output- and Performance-based Contracting Experience 2018, Janusz Sobieniak, World Bank Page 15 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 47. In addition to implementing road safety schemes, the increased capacity of the Road Safety Division10 enabled independent road safety audits to be carried out during the design of all project roads. This approach allowed improvements to be introduced at an early stage, before completion of the normal design process. The Road Safety Division went on to carry out road safety audits on later Bank financed projects utilizing its own in-house capacity rather than relying on international consultants. At entry, an intermediate results indicator targeted a rather conservative total of five audits. This target was amended at restructuring to 20, reflecting the increased capacity of the Road Safety Division. 48. The project supported the RD’s use of the electronic government procurement system to improve transparency and efficiency in the procurement process and the utilization of public funds. Two project financed contracts developed an application program interface (API) and web-based visualization tools, allowing users to access and analyze electronic procurement data. In 2015, the RD became one of the pilot government agencies to use the Government of Georgia’s e - procurement system for all National Competitive Bidding (NCB). By that time, all civil works contracts were already awarded or even completed under SLRP-II. Now the use of e-procurement is mandatory for the procurement of goods and services financed regardless the source of financing, be it the state budget or IFIs. The e-procurement system has minimized the risk of collusion either between civil servants and bidders, or between bidders, and it has increased the level of competition11. That is opposite to previous practice of using paper-based procurement where the implementing agency knows which bidders submitted or intend to submit bids. Use of e-procurement system has since helped maintain good levels of competition in later Bank financed projects. 49. Outcome 2 was monitored by one PDO indicator, Indicator 4: Roads managed under alternative contracting methodology. Unit of Original End Revised End Reported Indicator Name Baseline Measure Target Target Result Roads managed under alternative Value 0.00 200.00 137.00 135.27 Kilometers contracting methodology Date 31-Mar-2012 30-Jun-2017 30-Jun-2019 30-Jun-2019 50. The methodology for PDO Indicator 4 measured the length of the roads rehabilitated under the project’s two design-build contracts and those managed under the OPRC pilot. This was used as a proxy for measuring the strengthened capacity of the RD to manage the road network. 51. The Revised End Target for Indicator 4, roads managed under alternative contracting methodology, was achieved. The Result of 135.27 km is slightly less than the revised end target (137 km) and comprises 18.27 km under design-build contracts and 117.0 km under OPRC. This shortfall is due to a slightly shorter length of road being included in the OPRC bidding documents than anticipated at restructuring. While the revised target was achieved, when the result is compared to the original end target Indicator 4 using the split rating methodology, it was only partially achieved as the length of roads managed under alternative contracting was almost one-third less than the 200 km planned. 52. PDO Indicator 4 encompassed only part of the institutional strengthening undertaken by the project. It was therefore necessary to check that the outcomes of institutional strengthening (as described in the preceding paragraphs) 10 The status of the Road Safety Unit was upgraded to the status of the Division in 2016, which demonstrated the RD’s top manage ment recognition of the importance of road safety and important contribution of its Road safety Unit (now Division) to the National Road Safety Strategy. At the time of the NRSS, that Unit played the lead role in coordinating the efforts of all stakeholder agencies 11 There were 1.9 bidders per each NCB contract in SLRP-II versus 4.5 bidders per NCB contract in SLRP-III. One of the findings of the Enhanced Procurement Review of the World Bank-financed Secondary Roads Projects. Report prepared for the Roads Department by Global Procurement Consultants, Ltd. July 2018. Page 16 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) aligned with the expected interventions and outcomes inferred from the legal agreements and PAD12. This additional analysis is included in Annex 6.2 where a Results Matrix confirms the close alignment of the actual outcomes with those intended. 53. SLRP-II achieved intended PDO Outcome 2, Strengthened capacity of the RD to manage the road network, as measured by the revised end targets. The capacity of the RD to manage roads under alternative methodology, OPRC and design-build, was shown by PDO Indicator 4, while the achievement of other aspects of strengthened capacity was demonstrated by fulfilling related inferred outcomes derived from the project documents. PDO Outcome 2 was partly achieved when measured using the original end target set by Indicator 4 as the actual length of road managed by alternative methodologies was considerably less than originally planned. Justification of Overall Efficacy Rating 54. The overall efficacy of the project across the entire project time considers the original and revised outcomes. Each outcome is given equal weight. The original and revised efficacy ratings are shown below. See Overall Justification of Outcome Rating (Section D) below for more detailed analysis of split rating weighted by disbursement. Table 4 Efficacy Outcome 1: Improved local Outcome 2: Strengthened connectivity and travel time EFFICACY capacity of the RD to manage Rating for Overall Efficacy for the selected secondary the road network and local roads Original outcome targets High Modest Substantial Revised outcome targets High High High C. EFFICIENCY Assessment of Efficiency and Rating Road Rehabilitation Works 55. An economic evaluation of some of the civil works was undertaken at the time of project appraisal and presented in the PAD13. This analysis covered six of the nineteen roads totaling 61 km. Separate economic analyses of the remaining roads, and the OPRC pilot, were undertaken during the implementation of the project. In each case, the evaluation used the World Bank’s Highway Development and Management Model (HDM-4) over a 20-year evaluation period at a 12.0 percent discount rate. 56. The analyses concluded the rehabilitated of all roads was economically justified with an Economic Internal Rate of Return (EIRR) of 18.5 percent. Sensitivity analysis also demonstrated that most of the roads maintained an EIRR>12 percent if the costs were to rise by 15 percent at the same time as a 15 percent reduction in benefits. However, the evaluations were prepared by four different consultants during 2011 and 2012 using different assumptions so the rate of returns for individual roads may not be directly comparable. A comparison of the economic models is included in Annex 4, Efficiency Analysis. 12 Project Appraisal Document (PAD) February 21, 2012, Section III, A, paras. 22-27, and Annex 2. 13 Project Appraisal Document (PAD) February 21, 2012, Appendix 1 to Annex 2 Page 17 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 57. For post-project analysis, a new model was created to validate the actual benefits from the investments using known growth rates and actual construction costs but keeping other values constant where applicable. As before, a 20- year evaluation period was used with a 12.0 percent discount rate. The analysis confirmed that the investment in rehabilitating the secondary and local roads was economically justified with an average Economic Internal Rate of Return of 14.9 percent. Table 5 Summary of Economic Evaluation Results Financial Cost Total NPV @ 12% Average EIRR 19 Road Rehabilitations in SLRP-II (US$) (US$) (weighted by cost) % At Design Stage 66.31 million 33.43 million 18.5 At Completion 56..36 million 11.11 million 14.9 58. The project remains economically justified, but the overall economic benefits are assessed to be less than expected at the project appraisal. The apparent lower return is in part due to lower economic growth than forecasted and differences between the economic models used. Further details of the post-project economic analysis are provided in Annex 4. Output and Performance-based Road Contracting (OPRC) 59. Technical and economic analysis14 in 2012 indicated that 117 km of paved roads in the project area required some type of rehabilitation, and 79 km of unpaved roads required resurfacing or regraveling for a total investment US$15.93 million. The calculated internal rates of return, calculated for the paved road sections, varied from 13.1 percent to 42.8 percent. 60. Post completion analysis shows OPRC road rehabilitation work was economically justified with an Economic Internal Rate of Return of 15.3 percent, based on the 37.5 km of paved road included in the OPRC pilot. Table 6 Summary of Economic Evaluation OPRC rehabilitation Financial Cost Total NPV @ 12% OPRC Rehabilitation (paved road section only) Average EIRR % (US$) (US$) 0.01 million to At Design Stage (117 km, 8 paved roads) 12.95 million 13.1% to 42.8% 15.55 million At completion (37.5 km of SH039) 11.65 million 2.25 million 15.3 61. The cost of rehabilitating each kilometer of road under OPRC was comparable with costs for the other roads in SLRP-II. The rehabilitation cost, which excludes the maintenance component of OPRC, was US$0.31 million per km. This compares well with the average of US$0.28 million per km for the roads rehabilitated under SLRP-II using traditional BoQ contracts. The slightly higher cost per km is may be attributable to the additional risks associated with OPRC and the cost of design. 62. The World Bank study15 on Georgia’s OPRC experience found that the cost of the maintenance component of the Kakheti OPRC was a little higher than the average for nine other RD maintenance contracts (GEL 12,481/km/year compared with GEL 11,100/km/year). However, unlike the other contracts, the Kakheti OPRC price included clearing the 14 OPRC Technical Report, Appendix C, October 2012, Roughton International 15 Georgia Output- and Performance-based Contracting Experience 2018, Janusz Sobieniak, World Bank Page 18 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) maintenance backlog and the risk of penalties for not meeting service levels, so a slightly higher cost per kilometer is to be expected. Component Costs Table 7 Component Costs Component Estimated Resource Actual Resource Allocation at Comments Allocation at Start16 End Component 1: Rehabilitation US$81.3 million total US$86 million total, and Improvement of Selected US$61.1 million Bank US$69 million Bank financing US$3.5 million was returned Secondary and Local Roads financing to the Bank due to Component 2: Institutional US$1.5 million total, US$1.8 million total misprocurement on two Strengthening and Project US$1.2 million Bank financing US$1.4 million Bank financing contracts from Component 1. Management US$3.4 remained undisbursed US$87.5 million total, US$83.1 million total at project closing. Project Total US$70.0 million Bank US$62.5 million Bank financing financing 63. The total disbursed by the Bank was US$7.5 million less than the value of the loan and credit at entry. The difference is partly due to fluctuations in the US$/SDR exchange rate over the project’s duration which reduced the dollar equivalent of the credit, and by the US$3.5 million repaid after misprocurement was declared. The available amount undisbursed at project closing was about US$3.4 million owing to the devaluation of the Georgian Lari (from late 2014 onwards) and the remaining two years of the OPRC being financed wholly from the state budget. 64. In 2019, towards the end of the project, an attempt was made to use the undisbursed funds for selected periodic maintenance works. However, the procurement process had to be canceled as the bid prices exceeded the available funds. Administrative Costs 65. Administrative costs were higher than expected at entry but did not significantly impact the overall efficiency of the project. At entry, 96 staff-weeks of supervision input by the Bank’s project team over five years was planned, as derived from the Implementation Support Plan in the PAD. At the close of the project, supervision had been ongoing for seven years and required 171 staff-weeks. The additional supervision was necessary to ensure the successful implementation of the pilot design-build and OPRC contracts which, by their more complex nature, required closer supervision, a factor that may not have been fully appreciated at entry. Irregularities identified with procurement also necessitated intensified supervision and an increased presence by the Bank’s project team over the second half of the project. The accompanying administrative review by the Bank, which led to a declaration of misprocurement, also increased the Bank’s costs. Implementation Delays 66. The duration of SLRP-II was extended once from five years to seven years to allow the project to support the OPRC pilot through retendering and implementation. In addition to higher administrative costs for the RD and the Bank, the delay meant benefits from part of the investment were not realized as early as expected. As the time extension was 16 Stated in PAD. Page 19 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) not associated with additional outcomes, overall efficiency was therefore slightly reduced in the short term. However, the additional time has allowed the RD to realize longer-term benefits from being able to manage its network using alternative contracting methodologies. As a result of the RD’s Kakheti OPRC experience and additional time that the World Bank put into helping the RD to draw lessons and identify appropriate solutions, the RD was able to take those lessons on board and develop a new (hybrid) OPRC model which is in the bidding process and will be financed by SRAMP. Moreover, the Kakheti OPRC experience influenced the RD’s approach in ensuring sustainability of its valuable secondary road assets and prompted to revisit its approach in the preservation of the secondary roads network. Inspirited by this Kakheti OPRC experience and without any IFIs support, the RD carried out the internal reform (with the State Budget) of moving from one-year routine maintenance contracts to three-year periodic and routine maintenance contracts covering the entire secondary roads network. It was also vital that the OPRC had time to succeed, as failure would have significantly hampered institutional strengthening of the RD. Rating for Efficiency 67. The overall efficiency of the project is rated Substantial. The PDO results were efficiently achieved with the overall economic returns arising from rehabilitating the roads as expected in this sector. Cost control of the civil works was good. While completion of the project was somewhat delayed, it allowed the successful implementation of the OPRC pilot and achievement of the PDO. D. JUSTIFICATION OF OVERALL OUTCOME RATING 68. SLRP-II directly improved transport links and reduced travel times for project beneficiaries in rural regions of Georgia bringing lower transport costs, and better access to markets and social services including education and healthcare. The PDO was relevant to Georgia’s development priorities and the investments economically sustainable. The capacity of the RD to manage the country’s road network has been strengthened by piloting alternative contracting methods, road safety improvements, and enhanced road asset management and planning. 69. Restructuring and a change in scope was necessary to ensure there was sufficient time to complete the OPRC process. While the PDO remained unaltered, the expected outcome targets were amended. The Overall Outcome Rating17 is therefore assessed using the split rating method. Unrevised Revised Targets (Original) Targets Relevance of PDO Substantial Efficacy (PDO) Substantial High PDO Outcome 1 High High PDO Outcome 2 Modest High Efficiency Substantial 1 Outcome ratings Moderately Satisfactory Satisfactory 2 Numerical value of the outcome rating* 4 5 3 Disbursement US$51 million US$19 million 4 Share of disbursement 73% 27% 5 Weighted value of the outcome rating (Row 2 x Row 4) 2.92 1.35 Moderately Satisfactory 6 Final Outcome Rating (2.92 + 1.35 = 4.27 rounded to 4.0) *Note: Highly Unsatisfactory (1); Unsatisfactory (2); Moderately Unsatisfactory (3); Moderately Satisfactory (4); Satisfactory (5); Highly Satisfactory (6). 17 As derived from the Overall Outcome Rating matrix in ICR Guidance Appendix H and methodology given in Appendix I. Page 20 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 70. The Final Outcome Rating of the project is rated Moderately Satisfactory, as calculated using the “split rating” method. The project rating considering only the revised targets was Satisfactory, which is consistent with the project’s rating reported in the final ISR. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 71. Under SLRP-II, the Bank opened a dialogue with the RD on women’s employment in the road sector, a sector in which women are particularly underrepresented in the workforce. This dialogue represented a significantly positive impact in itself and helped to develop a new gender equality policy in the roads sector. The introduction of new forms of contracting, such as OPRC, provided opportunities for the project to support the involvement of women in the sector. New jobs were created with the potential of long term local employment, thereby allowing women to combine work and family responsibilities. 72. The first steps towards improving employment opportunities for women were with the project’s Kakheti OPRC. Despite initial skepticism and an environment where women were not encouraged to consider this kind of work, women living along the OPRC roads were employed by the OPRC contractor. As of February 2019, 19 of 174 employees of the main OPRC contractor were female (11 percent), with nine of the women in professional positions. While still too low, this participation is higher than the 7 percent sector average. Also significant was the fact the RD started regularly monitoring the level of women’s participation within OPRC. 73. The Bank continues to work with the RD using the Kakheti OPRC pilot as a valuable source of information and resource for the preparation of future schemes. Surveys and interviews with Kakheti workers helped identify challenges to be addressed in future contracts (such as absence of personal hygiene facilities at worksites and resistance from some community members) as well as benefits (economic, and an elevated position in their households and wider communities). Building on this, the Bank prepared recommendations that the MRDI and RD could consider implementing in order to improve employment outcomes for the women and men working in road construction, rehabilitation and maintenance in Georgia. While initially applicable to performance-based road contracting model, the recommendations are applicable to broader road works. Institutional Strengthening 74. SLRP-II strengthened the institutional capacity of the RD through actions designed to achieve PDO Outcome 2. SLRP-II developed the capacity of the RD in the use of alternative contracting methods, impact assessments, road asset management, road safety schemes, and safety audits. These results are described in detail in the preceding Section B: Achievement of PDOs (Efficacy). In particular, the new OPRC concept was well accepted and internalized by the RD. While the RD understands the outcomes of the still ongoing OPRC pilot are not yet fully realized, it recognizes this approach as a way to provide satisfactory levels of maintenance over an extended period, and potentially reduce long-term costs. The RD is taking advantage of Bank financed projects to test different approaches and refine OPRC for use in different regions and conditions, including a new OPRC in Guria under the Bank financed SRAMP. Mobilizing Private Sector Financing 75. Private sector finance was not sought or obtained during SLRP-II. However, the start of output and performance- based contracting establishes a principle of longer-term Public-Private Partnerships between contractors and the Government, in which both parties have long-term commitments. It also opens the way to more sophisticated investment and maintenance arrangements in the future, including elements of private sector financing. Page 21 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Poverty Reduction and Shared Prosperity 76. A Socio-Economic Impact Evaluation, commissioned by the RD as part of the project, found that the SLRP-II had significant positive impacts on households, agricultural productivity and small businesses connected by rehabilitated roads. According to the impact evaluation, household transport costs were reduced while the frequency of bus and minibus services increased, which is important to households with children who travel some distance to school. The report also identified a significant decrease in the cost of crop production, particularly for poorer households. Despite these positive impacts, residents did not report a major change in their living standards. The impacts, which are also discussed in Section II, Outcome, are presented in further detail in Annex 6.5. Other Unintended Outcomes and Impacts 77. Kakheti OPRC also influenced the RD’s asset management strategy . One of the key benefits of OPRC is its contribution to the sustainability of the road network as road maintenance is planned and implemented over extended periods of time (five years in the case of Kakheti OPRC). As OPRC was introduced under SLRP-II, the RD started to develop plans to contract-out routine, periodic, and winter maintenance for an extended three-year period on framework contracts, as opposed to the previous practice of one-year contracts. The new maintenance framework contracts were first introduced in late 2016. 78. Women’s participation in the road sector was an issue not explicitly included in the design of the pro ject. Notwithstanding this, the Bank’s task team used the OPRC pilot as a vehicle to open a constructive dialogue with the RD on women’s employment aimed at improving opportunities for women in a sector in which they are very under- represented. The outcome and impact of this dialogue is described in more detail in the above section on gender. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 79. The design of the SLRP-II benefited from experience gained during the first Secondary Local Road Project, which was close to completion at the time of project preparation. Selection of the project roads for rehabilitation followed the same systematic approach as previously used, with a list of roads prepared using HDM-4 modeling. Project preparation also benefited from the established long-term relationship between the Government and the Bank in the road sector. 80. SLRP-II incorporated lessons learned from SLRP and other road sector projects. Specifically, the project (a) provided additional staff and consultant support to the RD to help supervise a large number of small contracts, (b) required improved road safety measures to be incorporated at the design stage and independently audited, and (c) introduced better design solutions based on economic evaluations to improve efficiency. 81. Among other things, SLRP-II was designed to assess the usefulness, efficiency, and acceptability of alternative contracting methods for road rehabilitation and maintenance. Earlier analysis18 by the Bank identified potential benefits from alternative approaches which led to the inclusion of three pilot contracts in the project, two design-build and one OPRC. The OPRC pilot also sought to demonstrate the broader concept of performance based contracts in the public sector. 18 Improving the Sustainability of Road Management and Financing in Georgia, World Bank, June 2011 Page 22 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) B. KEY FACTORS DURING IMPLEMENTATION 82. The project benefited from the Borrower’s commitment to the objectives of the project. The resources necessary for project implementation were provided through the RD as needed. Government co-financing for the project represented 20 percent of the total project costs, which was made available as per the project implementation arrangements and legal documents. 83. The bankruptcy of one contractor led to three rehabilitation works contracts being terminated. One civil works contractor underperformed and eventually became bankrupt while rehabilitating 29 km of the project’s roads. Fortunately, the three contracts were awarded in the first year of the project so sufficient time remained for the RD to terminate and rebid each contract. The three road sections were successfully completed by new contractors. The RD also successfully handled the failure of a contractor working in a joint venture partnership on another contract. 84. Delays in contracting the OPRC pilot led to the project being restructured and extended by two years . The preparation of a framework for performance based contracting started19 before SLRP-II became effective and continued through mid-2013 when procurement was due to commence. The concept of OPRC then encountered some resistance to change. Concerns included making payments without a Bill of Quantities and the capacity of the RD to measure the contractor’s performance. By Spring 2014, the RD was comfortable to proceed so bids were invited on May 1, 2014. 85. The lowest bid received for OPRC pilot was three times the cost estimate, leading to the cancellation of the bidding process as it exceeded the available budget. Limited competition, contractors’ lack of experience with performance based contracts, and the allocation of risk contributed to the high bids. Failure to timely award the OPRC placed the achievement of the PDO at risk so, following the mid-term review in February 2015, the project was restructured. The OPRC pilot was eventually retendered with a revised bidding document reflecting the lessons learned and the total length of road to be maintained reduced from about 200 km to 117 km. The successful bidder commenced work on March 28, 2016. 86. Weaknesses with procurement affected implementation, but not outcomes. At the start of the project, the impact of procurement risk was assessed as significant, but its likelihood was low. The RD’s procurement track record was satisfactory and the procurement unit had experience with World Bank financed procurement. Despite this, from the first ISR in December 2012, procurement was rated as only moderately satisfactory as competition had been assessed as poor for the first round of rehabilitation contracts. Three-quarters20 of SLRP-II works contracts received fewer than three bids, and nearly half the contracts received just a single bid. The assessment of sub-optimal performance was compounded by delays in procuring OPRC and the need to terminate some of the works contracts. However, by February 2015, the RD had strengthened its procurement team, although this was not reflected in the ISR ratings which remained moderately satisfactory. 87. The Bank declared misprocurement under SLRP-II on August 21, 2018 for two civil works contracts21 awarded in 2014. An administrative review by the Bank found indicators of procurement irregularities and sanctions were imposed for collusion on two contractors and one individual. Following the declaration of misprocurement, 19 TEWHIP/Roughton International. 20 Enhanced Procurement Review Revised Final Report, April 2018 21 Mtskheta-Shio Mgvime Monastery km5-km11 (DB) - SLRPII/CW/NCB-15 and Chiatura-Perevisa-Sveri-Tvalueti-Gezruli km8-km22 (DB) - SLRPII/CW/NCB-16 Page 23 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) procurement risk was assessed as high and procurement performance rating reduced to moderately unsatisfactory. The Borrower reimbursed US$3.5 million to the Bank related to these two contracts. 88. Misprocurement involved the two design-build contracts, which contributed directly to PDO Indicator “Roads managed under alternative contracting methodology.” These contracts also contributed to fulfilling PDO Indicators “Roads in good and fair condition as a share of total classified roads” and “Average decrease in travel time.” The project’s task team consulted with Development Effectiveness Unit of the Bank on the handling of this issue and confirmed that it was not necessary to downgrade the project’s ratings just because misprocurement was declared and the resulting funds canceled. The reasoning was that the misprocurement was declared only after each contract was successfully completed. The misprocurement declaration did not undo the milestones achieved through the completion of the activities. The contracts were completed within the project, financed by the state rather than the loan. 89. When the procurement shortcomings became known, the RD set about strengthening procurement capacity, long before the Bank formally declared misprocurement in 2018. By spring 2017, the RD had appointed a new Deputy Chairman to take charge of procurement and had started the procurement of an Enhanced Procurement Review (EPR) to cover all the Bank financed secondary road projects. The RD’s willingness to finance the EPR through the project was seen as a serious commitment to addressing weak procurement practices. The EPR was delivered in April 2018 and, based on its findings and recommendations, a procurement Improvement Plan (PIP) was prepared. The recommendations of the PIP included using an indicative fraud & corruption checklist, studying the reasons for low participation, tightening joint-venture and subcontracting procedures, and developing the procurement capacity of the RD in electronic tendering and data analysis. These recommendations are now being implemented by the International Procurement Division of the RD. 90. To further support the RD’s efforts to strengthen procurement, the project financed two important institutional strengthening activities under Component 2 to improve the RD’s procurement capacity. This commitment of the RD to finance and implement these two activities (Enhanced Procurement Review and enhancement of electronic tendering) was a strong signal to the Bank that the RD took the Bank’s recommendations seriously and they were committed to further improving their procurement processes. The e-procurement activities were successfully completed and are discussed in more detail in Section B. Outcomes. 91. The actions taken by the RD have led to substantially improved procurement capacity of the agency and a recent post-review of the ongoing SRAMP and SLRP-III projects did not identify any major non-conformity with agreed procedures. The procurement rating for both projects was moderately satisfactory, although this partially reflected the RD’s misprocurement under SLRP-II rather than current performance on the two ongoing projects. The Bank’s procedures proved robust enough to identify and address the misprocurement, and to provide the necessary specialist support and guidance to strengthen the institution. The Bank’s internal administrative review considered the collusion could not have been reasonably foreseen or prevented by the Bank’s team. However, the incident highlighted the consequences of inadequate bid participation and raised the question of how to address this during the design of a project rather than during implementation. For the future, projects introducing new methods and approaches could benefit from market analysis at an early stage. The findings would then be available to identify sector weaknesses, guide procurement strategy, and tailor bidding documents for the market. Electronic procurement may have helped mitigate risks of collusion, but at the time, the Government’s electronic procurement system was not rolled out yet. Page 24 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 92. The PDO indicators and the intermediate results indicators were intended to measure and relate the project’s progress towards the achievement of the PDO. As the PDO had two expected outcomes, separate indicators were included to measure each outcome. According to the PAD, two PDO indicators were to measure the first PDO outcome (connectivity and travel times) and one to measure the second PDO outcome (strengthening capacity). A fourth (core) PDO indicator, measuring the size of the total classified network, was also included in the Results Framework. 93. The Results Framework included two core indicators as PDO indicators - PDO Indicator 1 measured the percentage of “Roads in good and fair condition of a share of total classified roads,” while PDO Indicator 2 measured the “Size of the total classified network” in kilometers. Although these indicators refer to the “total” classified network, the baseline, targets, and results for these indicators captured only the 5446 km of the secondary road22 network, thus creating a discontinuity between the descriptions of the core indicators and what was measured and reported. 94. PDO Indicator 1 also only captured the improvements directly attributable to the project, not the total improvement to the total classified network. While this was useful for monitoring the project, the data from this core indicator may be used incorrectly if used for other purposes. The RD reports 66 percent of all secondary roads are now in a good or fair condition when improvements made through all state and IFI funded projects are considered. 95. The indicators used for the first PDO outcome captured progress well, measuring road condition, travel times and rehabilitated road length. These indicators were broad, covering all aspects of the expected outcome, and incremental thereby allowing for the timely identification of delays in implementation. The PDO indicator measuring travel time had the additional advantage of being a direct measurement of one of the expected benefits captured in the PDO. 96. The design of indicators for the second PDO outcome was not as simple as for the first outcome. The second outcome included many elements, including project management, road safety measures, impact evaluations, improvement of asset management systems, and support for financial management. During project appraisal, it was decided to measure the increased capacity of the RD using just a single PDO indicator covering “Roads managed under alternative contracting methodology.” While this indicator measured an important element of the second PDO outcome, it was not fully representative of the wide range of activities under this outcome. 97. The targets for two indicators were substantially raised at restructuring . PDO indicator 3 was the “Average decrease in travel time” and had an end target of 20 percent at entry. The 2015 Level 2 project restructuring adjusted the end target to 44 percent reflecting the 40 percent reduction already achieved. The target for the intermediate indicator measuring the number of safety audits conducted was also revised from 5 to 20. At the time of restructuring 19 audits had been carried out. While the achievements are commendable, the extent to which the original end targets were exceeded suggests some lack of ambition at entry. 22PAD para 5, “It has a total road network of 20,930 km of which …… 5,466 km are secondary” & “70 percent of secondary roads and large parts of local roads require significant improvement” Page 25 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) M&E Implementation 98. Monitoring and evaluation remained satisfactory for the duration of the project. A small dedicated monitoring and evaluation unit within the RD was responsible for collecting information related to the project from different RD units, the TRRC, and supervising consultants. The information covering progress, payments, procurement and safeguard compliance, formed the basis of the RD’s semi-annual progress reports. The quality of the information provided by the RD was good, allowing progress towards the achievement of the PDO and overall implementation progress to be accurately assessed, as well as compliance with safeguards. M&E Utilization 99. Throughout the project lifetime, all indicators included in the Results Framework were routinely updated and reported in the six-monthly Implementation Status and Results (ISR) reports. Problems with the implementation of the OPRC pilot were evidenced by the reported indicators and allowed remedial action to be designed, which led to Level 2 project restructuring in 2015. Restructuring adjusted the Results Framework, revising end dates, and the end targets of two PDO and two intermediate results indicators. No new indicators were necessary, and none were removed. 100. One of the intermediate results indicators that remained unchanged at restructuring was the core indicator enumerating the Direct Project Beneficiaries (150,338 persons). Restructuring offered the opportunity to revise this figure downwards to account for the reduced size of the Kakheti OPRC area, but this was not taken. Similarly, monitoring after restructuring continued to reflect the population of the original impact area (of the whole project). The RD has estimated in the Borrower’s completion report the number of beneficiaries was actually reduced to 122,042 by the omission of part of the OPRC roads. Justification of Overall Rating of Quality of M&E 101. The quality of monitoring and evaluation was Substantial. Implementation and utilization of M&E had no significant shortcomings, but the design of the indicators could have better represented the project’s activities towards the second PDO outcome. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 102. The project triggered Environmental Assessment (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12), and was assessed as Environmental Category B. Environmental safeguards 103. The project’s Category B classification reflected that the anticipated negative impacts were of small to medium scale, site-specific and predominantly confined to the construction phase. The RD prepared an Environmental and Social Management Framework (ESMF), which was disclosed in January 2012. Based on its principles, site-specific environmental management plans (EMPs) were developed, discussed with stakeholders, and approved for each road section. These EMPs were developed using an EMP checklist template for small-scale road construction or rehabilitation. The primary responsibility for monitoring compliance was assigned to a specialized environmental unit within the RD. 104. Environmental safeguard compliance was satisfactory at the end of the project as evidenced by the Borrower’s semi-annual reporting and oversight by the Bank. The major environmental issue to arise was poor control Page 26 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) of the disposal of excess material and construction waste. Although disposal was addressed by the EMPs, several contractors struggled to apply due mitigation measures and minimize damage. This was compounded by weaknesses in the regulatory framework and country-wide deficiency of waste management infrastructure. The issue of disposal largely led to a Moderately Satisfactory rating for more than half the project’s duration. The Bank’s team raised the waste disposal issue with the RD, and it was largely addressed through better control by the supervising consultants which included oversight on the obtaining of waste disposal agreements by contractors. Social safeguards 105. Social safeguard compliance was overall satisfactory, a rating maintained for the duration of the project. The project was prepared on the assumption that it would not finance any new roads and all the rehabilitation and maintenance works would be carried out within existing road alignments. However, a lesson learned23 from the original SLRP was the possibility that land acquisition may be required to improve geometric standards or to add road safety solutions such as sidewalks and safety barriers. Consequently, the project triggered the Involuntary Resettlement Safeguard to enable the project to handle land acquisition, if such a need arose. The Resettlement Policy Framework (RPF) was prepared by the RD, approved, and disclosed in January 2012. In the end, no land acquisition or physical displacement was necessary as it was possible to design all the works to be carried out within the existing rights of way. Procurement and Financial Management 106. Financial management remained satisfactory for the duration of the project. The project benefited from having financial management arrangements already in place for the original SLRP and other Bank funded projects. However, these arrangements were strengthened by increasing the capacity of the Transport Reform and Rehabilitation Centre (TRRC), the unit responsible for financial management, with consulting services provided under other Bank financed projects. 107. Procurement was rated satisfactory at entry but was downgraded to moderately satisfactory after six months because of poor participation of bidders in the procurement of the first round of civil works. This rating was maintained for most of the project duration until December 2018 when it was lowered further to moderately unsatisfactory because of a declaration of misprocurement relating to two SLRP-II works contracts. By the time of the formal declaration of misprocurement, remedial actions had been taken and procurement performance had already improved. Nevertheless, the procurement rating was maintained at moderately unsatisfactory level until the end of the project. Other project ratings were unaffected by the misprocurement as both work contracts were already successfully completed under the project. Details of the misprocurement and the efforts by the RD to strengthen procurement practices are provided above in Section III. Key Factors That Affected Implementation and Outcome. C. BANK PERFORMANCE Quality at Entry 108. SLRP-II was built on previous secondary and local road projects in Georgia (particularly the original SLRP) and use of existing implementation arrangements. The project started with financial management arrangements mainly in place and unchanged from those of the ongoing SLRP. Similarly, the unit responsible for procurement in the RD was familiar with the Bank guidelines and had gained considerable experience with other transport projects financed by the 23 PID Appraisal Stage February 2012, and PAD Page 27 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Bank. Safeguard arrangements were also in place as evidenced by the timely disclosure of the Resettlement Policy Framework and the Environmental and Social Management Framework in January 2012. 109. At entry, detailed designs for six of the road sections had been completed and draft bidding documents prepared. Preparations for the design for a further road sections were advancing. The selection of the supervision consultant had also started. The level of preparedness was evidenced by the ability of the RD to start the procurement of eighteen NCB and ICB works contracts within six months of the project becoming effective. Preparations for the design and implementation of the OPRC were also underway at entry, as a consultant was already appointed in March 2011 under the Third East West Highway Improvement Project. 110. The use of alternative contracting methods, including OPRC, was clearly important for the development of the road sector when the project was designed. The PAD makes explicit references to a 200 km OPRC pilot and includes a detailed rationale for introducing design-build and OPRC in Georgia’s road sector. Alternative contracting methods were also captured by a PDO Indicator and an Intermediate Indicator. However, alternative contracting methods were not explicitly mentioned in key project documents such as the loan and financing agreements. Quality of Supervision 111. The wide geographical spread of the project’s roads (the furthest two being 500 km apart by road) presented challenges for both the Bank’s supervision team, and the RD’s team, but these were overcome and the sites regularly monitored. Environmental and social safeguards monitoring was effective and properly raised issues such as the disposal of waste material, for proper action. The Bank’s supervision team and management correctly identified the risk the failed OPRC tender posed to fulfilling the PDO and properly addressed it through restructuring. 112. The World Bank task team actively engaged with the RD to assist them in properly fulfilling all aspects of the project. The team also worked closely with the RD to promote and move forward the OPRC pilot which faced some resistance at times. While presenting opportunities, changes to key RD management during the project’s implementation also meant the Bank’s team had to build new relationships and repeatedly communicate the project’s objectives. The Bank’s team was also proactive in timely addressing the procurement and contractor performance challenges encountered throughout the project’s duration. 113. The supervision of SLRP-II was not undertaken in isolation and the team ensured the benefits of the project could be capitalized on by other Bank operations within the RD. Early practical experience from SLRP-II contributed to the design and implementation of SLRP-III, and the Secondary Road Asset Management Project (SRAMP), particularly in the areas of output and performance-based road contracting, design-build contracts, road asset management, road safety, and employment opportunities for women. 114. Upon completion of the project, arrangements were in place for the RD to continue to manage, monitor and finance the remaining two years of the OPRC in the Kakheti Region. The same contractor continues to maintain the Kakheti OPRC roads and the RD has engaged a local consultant to monitor the contract. Justification of Overall Rating of Bank Performance 115. The overall rating of the Bank’s performance is rated as Satisfactory . At entry, preparations for the key project activities, including for the OPRC and the rehabilitation of roads, were advanced while the key project documents had some minor shortcomings. Overall, the project was proactively and carefully supervised by the Bank's team and successfully completed. Page 28 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) D. RISK TO DEVELOPMENT OUTCOME 116. At completion, the risk to the sustainability of the achieved development outcomes is low, but not insignificant. The RD maintains its roads by outsourcing maintenance to the private sector based on longer-term combined routine and periodic maintenance contracts, an approach partly informed by Bank financed projects. Funding for maintenance is not currently a significant issue, but the ability of the Roads Department to properly maintain the roads rehabilitated by the project depends on the continued availability of financing from the state budget, which is susceptible to policy or economic changes. Unlike the roads rehabilitated using convention contracts, separate maintenance provisions are not necessary for the OPRC roads, at least in the early years, as the contractor is required to maintain the roads for an initial period (5 years in the case of Kakheti OPRC). In the case of Kakheti OPRC, no issues have arisen with funding since the RD took over full responsibility for the contract in July 2019. 117. Four of the rehabilitated roads belong to local municipalities and therefore do not come under the RD’s maintenance program. There is a risk that the maintenance of these roads will be less than optimal, but the roads are currently being maintained with the municipalities outsourcing maintenance on an annual basis. The Bank financed SLRP project included a component for building the capacity of the municipalities to maintain roads under their responsibility, but in the future the use of OPRC could relieve municipalities from organizing maintenance, at least in the early years following rehabilitation or repair. V. LESSONS AND RECOMMENDATIONS 118. The OPRC activity demonstrated the need for flexibility when undertaking a pilot project. The ability to adjust and improve the pilot, both before and during contract execution, is essential. While one of the objectives of the pilot was to introduce a shift in the allocation of risk, the shift moved too much risk to the contractor and the allocation had to be substantially revised after the failure of the first bidding process. The lessons learned from this paved the way to a successful second bid. Even after award, the maintenance performance indicators were refined as some of them were irrelevant or difficult to measure. This demonstrates the need to adapt the standard OPRC model provided in the Bank’s standard bidding documents to local context and circumstances. 119. The key pilot activity under the project was technically complex and introduced new concepts to the sector . The first attempt to bid the pilot contract was not successful and caused delay to the whole project. While the RD benefited from consultancy services at the design and supervision stages, they may have further benefited from having highly specialized third party technical support available at key stages of the pilot. The consultant could also provide targeted training for key RD staff. The design of future projects could benefit from ensuring the availability of such independent expertise supporting the RD throughout the process. 120. The experience of the project highlights the ownership of a pilot activity by the client as a critical factor in successful implementation, and the necessity of proactive work with the client by Bank teams during management transitions. Shortly after the start of the OPRC contract, changes in the management of the RD, and at higher levels of the government, required substantial time for the new leadership to better understand and appreciate the benefits of the pilot. This learning process stalled the implementation of the OPRC for several months and it only resumed when the new management came on board following efforts of the Bank team. 121. Special attention and careful market analysis are necessary when preparing bidding documents and cost estimates for activities that are new in the sector . It would have probably been useful if a market-assessment had Page 29 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) been carried out prior to the piloting OPRC under the SLRP2. The lowest bid received for OPRC pilot was three times the cost estimate, leading to the cancellation of the bidding process and delaying the implementation of the project. The allocation of risk must be carefully described in Bidding Documents and priced for in the cost estimates. Dialogue with potential bidders during preparation should also help contractors understand new forms of contracts and price their bids correctly. More widely, the design of projects that introduce new methods and approaches should include specific requirements for market analysis. The findings of such analyses could then be used to guide procurement strategy and improve the risk allocation in the bidding documents. 122. The ultimate impact of piloting the OPRC cannot be fully detected yet as is takes some time for it to be implemented correctly and absorbed. The Kakheti pilot remains to be completed and the Guria hybrid OPRC pilot is only just about to commence. Only then can its real impact be objectively assessed at a nationwide level. 123. The project demonstrated the need to minimize the increased risk of irregularities when many small scale civil works contracts are procured over a short duration of time. Various mechanisms to reduce this risk, and already in use by the RD, include using an e-procurement system, packaging in lots, and careful scheduling especially if procurement of similar activities is ongoing on other parallel projects. 124. The participation of more bidders should be proactively encouraged for all procurement processes. The success of the project depended on awarding a relatively large number of small works contracts in geographically diverse regions. While the contracts were packaged to be attractive to local contractors and build the capacity in the sector, participation in the bids was very low. Future projects could consider encouraging wider participation by measures such as targeted advertising, workshops, and extended bid periods. A good understanding of local market capacity is also vital, as is the use of electronic procurement, which was introduced under SLRP-II and adopted by subsequent projects. 125. The early identification of lessons learned and knowledge transfer helps shape contemporary projects. Two closely related Bank financed projects (SLRP-III and SRAMP) were designed and commenced while SLRP-II was ongoing. Lessons learned, sometimes at an early stage, were applied to the new projects allowing them to capitalize on experience gained under SLRP-II. For example, experience gained with the Kakheti OPRC pilot was used to design a hybrid Guria OPRC pilot with more conventional payment procedures for the rehabilitation phase, while confidence gained using design-build allowed for the inclusion of six design-build contracts in SRAMP. Lessons learned from the Enhanced Procurement Review were quickly and widely applied within the RD before the end of SLRP-II, and steps towards improving employment opportunities for women are already being built on as part of the Guria OPRC pilot. Page 30 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) . BLANK Page 31 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improved conditions on selected secondary and local roads Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Roads in good and fair Percentage 30.00 34.00 34.00 condition as a share of total classified roads 31-Mar-2012 26-Jun-2019 26-Jun-2019 Size of the total classified Kilometers 5446.00 5446.00 5446.00 network Comments (achievements against targets): This target for roads in good and fair condition was achieved, although the target and result only took account of the impact of secondary roads rehabilitated by the project. If improvements made through all IFI and state-funded projects are considered, the RD reports 66 percent of all secondary roads are now in good or fair condition. The size of the total classified network remained the same so the subsidiary target was achieved. Page 32 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Average decrease in travel Percentage 0.00 20.00 44.00 44.00 time 31-Mar-2012 26-Jun-2019 09-Jul-2015 26-Jun-2019 Comments (achievements against targets): This target was revised upwards and achieved based on the average car journey time along the 19 rehabilitated roads. Objective/Outcome: Increased capacity of the RD to manage the road network Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Roads managed under Kilometers 0.00 200.00 137.00 135.27 alternative contracting methodology 31-Mar-2012 26-Jun-2019 26-Jun-2019 Comments (achievements against targets): The revised target was achieved although the Result of 135.27 km is slightly less than the revised end target (137 km) due to a shorter length of road being included in the OPRC bidding documents than anticipated at restructuring. The total comprises 18.27 km under design-build contracts and 117.0 km under OPRC. The original target was revised downwards to reduce cost. Page 33 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) A.2 Intermediate Results Indicators Component: Rehabilitation and Improvement of Selected Secondary and Local Roads Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Roads rehabilitated, Rural Kilometers 0.00 225.00 200.50 200.60 31-Mar-2012 26-Jun-2019 09-Jul-2015 26-Jun-2019 Comments (achievements against targets): The revised target was achieved. All 19 of the roads selected roads were rehabilitated so the length of road added as a contingency in the original target was not needed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 150338.00 150338.00 122042.00 31-Mar-2012 26-Jun-2019 26-Jun-2019 Female beneficiaries Percentage 52.40 52.40 52.40 21-Jun-2018 Comments (achievements against targets): The number of direct project beneficiaries was less than anticipated at entry due to the reduction in the length of roads covered by the OPRC pilot. The percentage of female beneficiaries remained unchanged based on national census data. Page 34 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Component: Institutional Strengthening and Project Management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of safety audits Number 0.00 5.00 20.00 20.00 conducted 31-Mar-2012 26-Jun-2019 09-Jul-2015 26-Jun-2019 Comments (achievements against targets): This target was revised upwards and achieved as the capacity of the Road’s Department to undertake this task strengthened. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Implementing pilot Number 0.00 2.00 2.00 alternative contracting methodology 31-Mar-2012 26-Jun-2019 26-Jun-2019 Comments (achievements against targets): Two alternative contracting methods were implemented, design-build, and OPRC, thereby achieving the target. Page 35 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 Improved local connectivity and travel time for the selected secondary and local roads; 1. Roads in good and fair condition as a share of total classified roads Outcome Indicators 2. Size of the total classified network 3. Average decrease in travel time Intermediate Results Indicators 1. Roads rehabilitated, Rural 1. 238 km of secondary and local roads rehabilitated Key Outputs by Component 2. 44% decrease in average decrease in travel time (linked to the achievement of the Objective/Outcome 1) 3. 122,042 persons directly benefiting from the project Objective/Outcome 2 Strengthened the capacity of the Roads Department to manage the road network Outcome Indicators 1. Roads managed under alternative contracting methodology. 1. Number of safety audits conducted Intermediate Results Indicators 2. Implementing pilot alternative contracting methodology 1. 20 road safety audits completed 2. 135.3 km of roads managed under alternative contracting methodology 3. Two pilot contracts undertaken using alternative contracting methods Key Outputs by Component 4. Road asset management system strengthened. (linked to the achievement of the Objective/Outcome 2) 5. Three standalone rural road safety schemes designed and completed in five regions 6. Two impact evaluations completed (each with studies before and after) 7. Project implemented, including financial management supported Page 36 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Joseph Melitauri Task Team Leader(s) Deepal Fernando Procurement Specialist(s) Galina Alagardova Financial Management Specialist Darejan Kapanadze Social Specialist Natalya Stankevich Team Member Joanna Peace De Berry Social Specialist Supervision/ICR Nijat Valiyev, Aymen Ahmed Osman Ali Task Team Leader(s) Sandro Nozadze Procurement Specialist(s) Djamshid Iriskulov Financial Management Specialist Graciela A. Tejeda Team Member Deepal Fernando Procurement Team Darejan Kapanadze Environmental Specialist Giang Thanh Huong Le Team Member Vusala Mamed Asadova Procurement Team Militsa Khoshtaria Team Member Sophia V. Georgieva Social Specialist David Jijelava Team Member Jonathan Tregenza ICR Contributor Page 37 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY12 34.829 93,898.86 Total 34.83 93,898.86 Supervision/ICR FY12 .300 1,538.72 FY13 47.548 124,899.05 FY14 25.548 94,433.52 FY15 27.843 74,261.25 FY16 13.649 44,239.39 FY17 22.597 100,154.05 FY18 16.174 86,614.33 FY19 17.592 105,350.53 FY20 1.470 32,312.74 Total 172.72 663,803.58 Page 38 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Rehabilitation and Improvement of Selected 86.00 81.27 94.5% Secondary and Local Roads Institutional Strengthening 1.50 1.79 119.3% and Project Management Total 87.50 83.06 94.9% Page 39 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 4. EFFICIENCY ANALYSIS Post Completion Economic Analysis Post completion economic analysis, prepared for the ICR, considered actual construction costs and known economic growth rate data. Road roughness (IRI) and Year 1 AADT traffic numbers and distribution were taken from the original economic analyses and design reports. Other variables, such as Vehicle Fleet Costs, were kept constant (where applicable). Each of the 19 rehabilitated roads were analyzed separately plus the road rehabilitated under OPRC (comprised of four sections). Post Completion Economic Analysis - by road section. ICR ICR Final Cost Length of Cost per Economic Total AADT Economic Ref. Road Section (million the road km (million Initial IRI Evaluation (Y1) Evaluation US$ equiv.) (km) US$/km) NPV (million EIRR US$) Rehabilitation Using Conventional Contracts Kveda Saqara-Sazano road 1 km1-km14 (re-advertised) (local 4.140 13.39 0.309 12 445 0.547 14.2% road) Chiatura-Usakhelo-Korbouli 2 6.100 18.40 0.332 12 845 2.757 19.0% km6-km24 Kutaisi-Alpana-Mamisoni pass 3 4.490 9.59 0.468 12 649 0.004 12.0% km116-km125 Ulianovka-Bodbe-Gamarjveba 4 4.620 21.35 0.216 8 970 0.276 12.9% km1-km21.3 Natanebi-Shroma-Ureki km0- 5 4.820 17.90 0.269 16 309 1.526 17.3% km17 (local road) Kutaisi-Tskaltubo-Lentekhi- 6 4.800 13.99 0.343 13 522 0.923 15.1% Lasdili km50-km63 Shulaveri-Red Bridge road km 7 1.340 5.12 0.262 12 2488 1.383 27.3% 7-km11 Chokhatauri-Bakhmaro road 8 0.840 3.11 0.270 12 405 0.05 13.0% km4-km6 Didi Dmanisi-Dmanisi-Bediani 9 road km17-km26 (orginal and 2.660 10.00 0.266 12 968 3.234 29.9% second contracts) Chalaubani-Signagi-Anaga km 10 1.860 6.44 0.289 8 303 -1.035 0.0% 16-km 22 (re-advertised) Tbilisi -Tianeti Road Section 11 1.840 12.68 0.145 17 224 1.586 25.4% km39-km 51 Ingiri-Shamgona km1-km7 12 1.750 7.53 0.232 16 231 0.03 12.3% (local road) Kutaisi-Alpana-Mamisoni km77- 13 1.230 4.98 0.247 9 550 -0.165 9.7% km82 Nosiri-Gejeti-Noqalaqevi km6- 14 1.080 5.00 0.216 16 249 0.453 18.9% km11 Noqalaqevi-Ledzadzame-Didi 15 5.20 16.44 0.322 20 288 0.313 13.2% Chkoni km7-km23.5 Igoeti-Lamiskana-Akhmaji km 1- 16 1.740 8.93 0.195 16 242 0.551 17.3% km9.5 (re-advertised) Page 40 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Post Completion Economic Analysis - by road section. ICR ICR Final Cost Length of Cost per Economic Total AADT Economic Ref. Road Section (million the road km (million Initial IRI Evaluation (Y1) Evaluation US$ equiv.) (km) US$/km) NPV (million EIRR US$) Zomleti-Khikhadziri km5-km11 17 road section (re-advertised) 2.610 7.00 0.373 16 298 -0.207 10.6% (local road) Rehabilitation using D&B Mtskheta-Shio Mgvime 18 2.510 8.24 0.305 20 73 -1.285 0.5% Monastery km5-km11 (DB) Chiatura-Perevisa-Sveri- 19 2.640 10.03 0.263 20 134 0.164 13.1% Tvalueti-Gezruli km8-km22 (DB) Rehabilitation using OPRC Tsnori Dedoplistskaro Kvemo 20 10.587 37.50 0.282 10.3 723 2.253 15.3% Kedi (4 Sections under OPRC) Repair and Road Safety Work Final Cost (million US$ equiv.) 21 Rehabilitation of the Tekhuri Riverbank Washouts, Road Sections of the "Noqalakevi - Didi Chkoni Road" 1.120 Restoration works for road sections damaged in 2015 by heavy rainfalls and landslide activity on the local 22 0.450 road Natanebi Shroma Ureki(km1-km18) 23 Small-scale Road Safety Improvements in Kvemo Kartli Region 0.430 24 Small-scale Road Safety Improvements in Imereti, Guria and Samegrelo-Zemo Svaneti Regions 1.150 25 Small-scale Road Safety improvements along Sec-30 Tbilisi (Gldani) -Tianeti road in Mtskheta Mtianeti 1.090 Total and Weighted Averages ICR ICR Final Cost Length of Cost per Economic Total AADT Economic Ref. Road Section (million the road km (million Initial IRI Evaluation (Y1) Evaluation US$ equiv.) (km) US$/km) NPV (million EIRR US$) Total/average for 17 roads rehabilitated under 51.406 182.33 0.282 13.4 9,986 12.226 15.6% conventional contracts Total/average for 2 roads rehabilitated using D&B 4.421 18.27 0.282 20.0 207 -1.121 7.0% contracts Total/average for the above 19 56.360 200.60 0.289 33.4 10,193 11.105 14.9% non-OPRC roads rehabilitated Total/average for 4 road sections rehabilitated using 10.587 37.50 0.282 10.3 723 2.253 15.3% OPRC Page 41 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Total and Weighted Averages ICR ICR Final Cost Length of Cost per Economic Total AADT Economic Ref. Road Section (million the road km (million Initial IRI Evaluation (Y1) Evaluation US$ equiv.) (km) US$/km) NPV (million EIRR US$) Total/average for all 66.947 237.62 0.282 13.4 10,916 13.358 14.9% rehabilitated road sections Total for repair and road safety 4.240 work Total for all road construction 71.187 work Vehicle Fleet Data used in post completion economic analysis Economic or Financial Unit Costs ($) Passenger Non- Annual Interest Lubricating Oil Working Time Working Time Maintenance New Vehicle Crew Wages Cargo Time ($/vehicle) Passenger Overhead New Tire ($/hour) ($/hour) ($/hour) ($/hour) ($/hour) ($/ liter) ($/year) ($/liter) ($/tire) Annual Labor Vehicle Fuel (%) Description Car Medium 13,253 36.00 1.23 6.02 0.96 2.17 386.0 12.0 2.17 0.48 0.00 Truck Medium 42,169 120.00 1.23 6.02 1.81 1.16 771.0 12.0 0.00 0.00 1.26 Truck 120,482 163.00 1.23 6.02 1.81 1.57 771.0 12.0 0.00 0.00 1.86 Articulated Bus Light 21,084 54.00 1.23 6.02 1.39 2.17 386.0 12.0 2.17 0.48 0.00 Basic Vehicle Fleet Characteristics Annual Working Passenger Trips Private use (%) Passengers (#) Work Related Hours (hours) Gross Vehicle Driven (km) Service Life Number of Annual km Weight (t) (years) Vehicle (%) Description Car Medium 23,000 550 10 100 2 75 1.2 Truck 40,000 1,200 12 0 0 0 7.5 Medium Truck 86,000 2,050 14 0 0 0 28 Articulated Bus Light 30,000 750 8 0 12 75 2.2 Comparison of Economic Models The original economic evaluations were prepared by four different consultants during 2011 and 2012. The assumptions and variables used by each consultant differed so EIRRs for individual roads may not be directly comparable. Page 42 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Table 8 Comparison of Economic Model Assumptions and Actual Case. OPRC D&B roads Y2 and Y3 roads rehabilitated Y1 roads (prepared by Indicator (prepared by roads Actual (from PAD) Kocks Consult Transproject Ltd) (prepared by GmbH) Roughton) Traffic Growth, based on the following factors: 6% reducing to 4.6% (GDP Average annual growth rate 6% 4.4% 5% after 5 years average 2011 TO 5.4% (average) Y1 to Y10 (Based on GDP) (average) 2018)24 Average annual growth rate 5% 2.85% Growth of 4% to 4.3% (average) 4% Y11 to Y20 (Based on GDP) (average) 5% predicted 15% used in Generated Traffic +15% 0 +52% 0 model 15.93 56.36 (19 roads) Construction cost (US$ 66.31 (for 19 roads) (117 km paved + + 10.59 (OPRC) = million) 80 km unpaved 68.52 million Over the period from 2011 to 2018, GDP rose by an average of 4.6% per year which is less than the 6% per year assumed for Y1 roads. Consequently, after the seven years to 2018, traffic (AADT) may have grown 9% less than assumed at the time of the economic analysis. However, construction costs were less, which would reduce the effect of reduced traffic on the EIRR. An alternative data point for traffic growth is provided by national statistics on the country’s vehicle fleet. The fleet grew by an average of 7.6% per year over the same period for all type of vehicle. However, the same data indicates the number of trucks and buses on the road grew by only 3.5% so the economic benefits of the growth will be considerably less than the 7.6% total. 24 World Bank DataBank Page 43 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 44 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) ANNEX 6. SUPPORTING DOCUMENTS Annex 6 Sub-sections 6.1 Document List 6.2 Alignment of SLRP-II with the Country Partnership Framework for Georgia, FY19 - FY22 6.3 Outcome 2, Inferred Outcomes and Results Matrix 6.4 OPRC Lessons Learned 6.5 Summary of the Findings of the Socio-economic Impact Evaluation 6.5 Project Map Page 45 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.1 Document List Documents Referred to in the Report • Strategic Plan of Development. Ten Point Plan of Modernization and Employment, 2011-2015. Government of Georgia, October 2011 • Project Information Document (PID) Concept Stage October 26, 2011 • Project Information Document (PID) Appraisal Stage February 01, 2012 • Project Appraisal Document (PAD) February 21, 2012 • Loan Agreement, March 22, 2012. • Financing Agreement, March 22, 2012. • Country Partnership Strategy for Georgia for the Period FY2014 - FY2017, April 9, 2014 • Country Partnership Framework for Georgia for the Period FY19 - FY22, April 25, 2018. • Roads Department, Socio-economic Impact Evaluation of the Second Secondary and Local Roads Project (SLRP-II), December 2015. • World Bank Data https://data.worldbank.org/country/georgia • Bank Guidance for Implementation Completion and Results Report (ICR) for Investment Project Financing (IPF) Operations, World Bank, September 27,2018 • Enhanced Procurement Review Revised Final Report, April 2018 • Procurement Improvement Plan for The Roads Department of Georgia, November 12, 2018 • Roads Department, Road Rehabilitation Impact Assessment, July 2019. • OPRC Report, February 2019, Eptisa/Temelsu • Georgia Output- and Performance-based Contracting Experience 2018, Janusz Sobieniak, World Bank, June 2018 Attribution Photographs and maps were provided by the Roads Department Page 46 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.2 Alignment of SLRP-II with the Country Partnership Framework for Georgia, FY19 - FY22. The CPF envisages support to Georgia across three areas: (i) enhancing inclusive growth and competitiveness, (ii) investing in human capital, and (iii) building resilience. The extent of this alignment is examined below, particularly with the first area. Table 9 Alignment of SLRP-II Objectives with current CPF Objectives CPF CPF Objective element Alignment Reference Objective 1.1 Support agriculture modernization and Aligned. Better connectivity and reduced travel times on rural roads access to markets. improve access to markets. Objective Enabling private sector participation in the Aligned. The OPRC and design-build pilots engage the private sector 1.2, Para.61 roads sector will be important for its in advanced contracting methodologies, which better allocate risks sustainability. associated with road rehabilitation and maintenance. For example, one of the bidders that did not win either of its two SLRP-II bids still learned through bidding process about performance-based approach and eventually won a bid in another project in SLRP-III. Objective Institutional capacity building to better Aligned. Strengthening the capacity of the Roads Department to 1.2, Para.64 plan and manage infrastructure initiatives. manage the road network through improved contracting, accident reduction, and asset management. Objective 1.4 Increase economic participation in the Aligned. Rehabilitating and maintaining regional and local roads regions. improved connectivity and reduced travel times thereby facilitating economic growth in the regions. Objective Continued funding for the improvement of Partially Aligned. Improved secondary and local roads provide part of 1.2, Para.58 the East-West Highway Corridor (EWHC) the necessary feeder and distribution network for the country’s main & 59 and Focus increasingly on “soft” highways and freight transit corridors. connectivity challenges Page 47 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.3 Outcome 2, Inferred Outcomes and Results Matrix The institutional strengthening undertaken by the project was only partially captured by PDO Indicator 4: Roads managed under alternative contracting methodology. The following table establishes whether the actual results of institutional strengthening are aligned with the expected interventions and outcomes inferred from the legal agreements and PAD25. Table 10 Outcome 2, Institutional Strengthening and Project Management, Results Matrix Result as of Project Degree of Alignment Between Activities Derived from Inferred Outcome Completion Inferred Outcome and Project Loan/Credit Agreements June 30, 2019 Results Strengthening the capacity of Successful management and 200.5 km of rehabilitated Good RD and FPU in Project completion of civil works, roads using civil works management and goods, and services contracts, contracts, including two implementation including pilot design-build design-build contracts. contracts and OPRC. 37.5 km of rehabilitated roads and 117 km of roads being maintained under one OPRC pilot. Development of an API and visualization tools to enable the RD to interface the Government electronic procurement system. An Enhanced Procurement Review. Strengthening the capacity of Road safety measures Three small-scale road safety Good RD and FPU in identifying, identified and implemented improvement schemes developing and implementing on secondary and local roads. completed. road safety measures on Safety Audits carried out at 20 road safety audits were secondary and local roads the design stage. carried out. This is recorded by an Intermediate Indicator, which was revised upwards at restructuring. Strengthening the capacity of Completion of impact Impact evaluation and follow- Good RD and FPU in carrying out evaluations. up evaluations were impact evaluations; through undertaken, using consultants, the provision of goods, for nine of the rehabilitated consultants' services, and roads and the OPRC roads. Training 25Project Appraisal Document (PAD) February 21, 2012, Section III, A, paras. 22-27, and Annex 2. The Project Description in the PAD is consistent with Schedule 1 of the agreements but gives more details. Unlike the agreements, the PAD does refer to pilot performance-based and design-build contracts. However, these references are found under Component 1, not Component 2 (Institutional Strengthening). Page 48 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Result as of Project Degree of Alignment Between Activities Derived from Inferred Outcome Completion Inferred Outcome and Project Loan/Credit Agreements June 30, 2019 Results Consultancy services and Specialist support for road Technical specifications were Good developing technical asset management through developed, then a vehicle and specifications for road asset consultancy services. equipment were procured. management equipment. Data is now collected, and the Acquisition of the equipment RAMS is being used for 5-year Acquisition of equipment required for the road rolling programming and required for developing the management system. annual planning. A road data new road management management consultant system. appointed to run RAMS and provide regular support to RD planning. Support to TRRC related to Consulting services and TRRC had sufficient capacity to Good Project financial management operational costs supporting financially manage the project and audits through the the TRRC and project audits in a satisfactory manner. provision of consultants' Consultancy services were services. financed from other Bank supported projects. Project financial audits were undertaken. Page 49 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.4 OPRC Lessons Learned The Kakheti OPRC pilot generated specific practical outcomes as lessons for future projects. These outcomes, identified by the monitoring consultant and in a World Bank report on Georgia’s OPRC experience are summarized below: • The pure OPRC model as envisaged in the Bank’s standard bid document is not appropriate where the scope of work involves rehabilitation, periodic maintenance and minor repairs followed by routine maintenance where contract roads are in different condition. A more flexible model should be considered such as a hybrid OPRC using a more traditional BoQ methods for the payment of the rehabilitation component. • Bidding Documents should include the detailed inventory and condition data including schedules identifying road assets, defects and repairs required by chainage. • Initial repairs, supported by the schedules of defects and repairs, should be a separate paid item. It could take the form of a lump sum, or input- or output- based rates, or some combination thereof. The contractor should be given a reasonable grace period at the start of the contract to complete the necessary repairs. • Where rehabilitation works are not straightforward, it may be more practical for the RD to prepare the necessary designs, drawings, quantities and construction specifications and include these in the bid document. This allows the RD to make trade-offs between standards and cost. • If the rehabilitation works are to be designed by the contractor, the bid documents should include a requirement that the contractor prepares a full set of design documents including drawings, technical specifications, BoQs, road safety audit and design calculations. • Reliance on ride comfort and durability performance measures such as International Roughness Index (IRI) and deflections is insufficient for quality management. The contract should require compliance with materials and construction specifications that can be inspected and enforced by the Employer through its project manager or monitoring consultant. • The number of performance indicators should be kept to the minimum needed to measure road asset conditions and monitor if essential maintenance is done. • The Contractor cannot be held responsible for long term pavement durability on road sections that it did not rehabilitate under the same OPRC. While service levels related to routine maintenance activities (e.g. patching, cleaning drainage, vegetation control) are appropriate, IRI should not be included as a performance indicator. • Incentives are needed to ensure contractors achieve at least 90% of the required maintenance performance. It may be necessary to increase the ratio of maintenance to rehabilitation prices or introduce a bonus scheme as well in addition to the application of penalties. • The Kakheti OPRC arguably transferred too much risk to the Contractor, especially for a pilot project of this nature. It would be prudent to reduce the Contractor’s risk and have all risks and their assignment clearly defined in the bid document. • Detailed technical specifications and standard drawings be included in the bid documents for maintenance activities to control the quality of workmanship and materials. • Regular and frequent meetings involving the client, contractor and monitoring consultant is vital as OPRC is a more complex contract than conventional admeasure contracts that most parties are used to. Page 50 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.5 Summary of the Findings of the Socio-economic Impact Evaluation Summary of the Findings of the Socio-economic Impact Evaluation of the Second Secondary and Local Roads Project (SLRP- II) prepared by consulting company ACT for the Roads Department of the Ministry of Regional Development and Infrastructure of Georgia, December 2015 Expected Sectors Impact Detailed Findings Outcomes • The average time spent per kilometer decreased significantly —by 0.94 minutes—in the treatment group for the whole sample. The effect is slightly higher for the B40 group, where the average time spent per kilometer decreased significantly by 0.97 minutes. • Impact evaluation also shows that cost per kilometer declined by -0.04 GEL on average and by -1.9 GEL for households with disabled persons in Reduced travel the treatment group. The impact was weakly statistically significant in time both cases. • The share of the household transportation expenditure in total Reduced vehicle household consumption also declined by 17 percent. operating costs • One interesting result revealed that the frequency of visits of households (VOC) with disabilities to medical facilities statistically increased by 9 percent. It seems that better roads made access to medical facilities easier. Reduced travel • The frequency of bus use increased for most destinations (schools, Connectivity costs 1 and Mobility + municipal centers, regional centers, the capital, hospitals, pharmacies, and main roads). On average, the frequency of bus use to these places increased by about 4 percent. • The frequency of minibus use also increased for traveling to schools by Shift in about 10 percent. transportation modes – less • The frequency of walking decreased for visiting markets, municipal travel by foot centers, pharmacies, and schools. • As evident from the respondents’ quotations, road rehabilitation has simplified their travel and made it easier for residents of the settlement to reach their desired destinations (37 percent). A reduction in travel time is also stressed by 25 percent of the survey participants. Among other positive effects of the road rehabilitation, better transport schedule (10 percent), lower expense on fuel (5 percent), and increased communication between the settlement and the city (1 percent) were named. • In the qualitative assessment of the road rehabilitation outcomes, local business as well as local government representatives report an improvement of distribution services in the treatment settlements, Simplified export which has resulted in better supply with goods and lower expenses of and import of local businesses correspondingly. Local Market goods • As local government representatives report, the import of raw materials 2 Development + has become easier for local businesses. The export of produced products Improved labor to outside markets has also been simplified. market dynamics • The data suggests the possible case in which women stayed in the settlement to work instead of men, who probably went to work elsewhere as a consequence of better roads. In fact, the increase of 18 percent of women working primarily on the household farm combined Page 51 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Expected Sectors Impact Detailed Findings Outcomes with a decrease of 15 percent of women working outside of the settlement seems to corroborate this scenario, even though there is no statistically significant evidence of an increase in men working outside the settlement. • It can be seen that after the road rehabilitation, there was a significant decrease in crop production expenses for the whole sample as well as for the Poor and B40 groups of households in the treatment settlements as compared to the control settlements. • It can be seen that after the road rehabilitation, there was a significant decrease in crop production expenses for the whole sample (by 370 GEL) as well as for the Poor (by 408 GEL) and B40 (by 1150 GEL) groups of households in the treatment settlements as compared to the control settlements. • The total income from renting the land increased significantly in general Agricultural Improved farming (by 158 GEL) and for households having a member with disability (by 120 3 Productivity activities + GEL). It can be suggested that this indicates an increased demand on agricultural land in the territories where road rehabilitation was conducted, benefitting the households in those areas. • As interviews with local government representatives show, road rehabilitation has assisted the farming activities of the local population. Farmers are able to start land ploughing and all other necessary activities for land cultivation in a timely manner. This development can be considered especially important, given the large amount of subsistence farmers in rural areas of Georgia. The area of cultivated land has also increased in villages, and some new farming activities have emerged (e.g. cultivation of nuts). • Regarding the impact of road rehabilitation on the availability of services and facilities (Cable TV, Home Cable Internet, Electricity, Municipal Water, Landline Phone), difference-in-difference regression results show that availability26 increased significantly for most services (municipal water (by 6 %), home cable internet (by 14%), Landline Phone (by 17%)), which most probably reflects a positive impact of road rehabilitation on the economic development of treatment territories. Household • The impact of road rehabilitation on non-agricultural income was a 4 Welfare Improved self- -/+ positive change solely in the case of income from selling vehicles and assessment of HH savings. However, these effects are very small in magnitude (95 GEL at welfare most) and only weakly significant. • Among different components of HH welfare, roads, means of transportation, access to credit and appliance purchase are the items in case of which respondents report improvement in 2014 as compared to 2012. • The overall subjective self-assessment of HH’s living standards remained almost the same. The majority of respondents assess their living 26The household survey respondents marked one in case the specific service is available to them and they assigned zero in case the service is not available yet. The level of availability is measured as the frequency of the assurances among the households with access to the specific service. Page 52 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Expected Sectors Impact Detailed Findings Outcomes standards as “regular” or “poor” before (2012) as well as after (2014) the road rehabilitation. • The impact of road rehabilitation on transportation costs is reported to Improved have increased the distance driven by all means of transportation 27 by productivity of about 12 percent. local businesses • The impact of road rehabilitation was observed for maintenance costs of (Decrease in vehicles used by business. The effect was marginally significant and the transportation / portion of maintenance cost declined significantly after the road maintenance rehabilitation. Small Business costs) 5 Development -/+ • As difference in differences analysis shows, the improved road infrastructure did not lead to any significant increases in the number of organized businesses in the treatment regions. This finding was double Increased number checked with official data obtained through GEOSTAT, which shows an of small increase in the number of businesses in treatment as well as control businesses localities, importantly larger increase in control area. It can be suggested that this is the reason why DID technique could not determine any positive impact that can be attributed to the project. • The improved roads are associated with increased likelihood (by 33 percent) better speed limit regulations by local governments. • The most important result is that the number of fatal car accidents and the number of people who died from car accidents were significantly reduced by about 15 percent in treatment territories. • Road safety is considered to have improved in the opinion of the majority of residents in treatment communities. Retrospective perceived evaluation of the road safety on a 10 point scale, with “1” indicating “low road safety” and “10” indicating “high road safety,” resulted in the Reduced number average evaluation moving from 4.6 points in 2012 to 7.7 points in 2014. of fatal accidents The net change for treatment areas equals 3.1 points. • In single cases, local government representatives noted that road Increased road rehabilitation has positively affected the safety of local pedestrians as safety evaluated well as drivers. In this case, the road rehabilitation involved making the 6 Road Safety by (1) local + road wider and arranging simple sidewalks for pedestrians. Such population and arrangements have positively affected road safety. (2) local • Some respondents, however, (29 percent of the respondents in businesses treatment settlements) expressed at least slight concern regarding road safety. The majority of respondents in treatment localities provided the following reasons for concern: reckless drivers along the road (71 percent), poor traffic signs (19 percent), and poor visibility and lighting (16 percent). Naturally, in control settlements the share of respondents expressing similar concerns was larger (45 percent). • Also, even though only in single cases, local government representatives did mention the increased risk of road accidents due to speeding cars. Some measures have been applied in order to reduce this negative consequence of road rehabilitation; e.g. additional traffic signs and speed reduction ramps were installed in some settlements. However, 27 The portion of the distance driven by all means of transportation used by businesses. Page 53 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) Expected Sectors Impact Detailed Findings Outcomes some LG representatives emphasize the need for such arrangements (traffic signs, traffic barriers, speed reduction ramps) in their settlements, especially near schools. Improved road quality evaluated • The average evaluation of road quality on a 10 point scale by the local by (1) local population in treatment areas increased from 3 to 8.2 points. The net 7 Road Quality population and + change for the treatment group is 5.3 points. As for the evaluation of (2) local local businesses, the net change is even higher and equals 6.2 points. businesses • The vast majority of the residents in treatment localities (91 percent) state that their access to various services has “considerably improved” Improved access or “improved”. The same results apply to the evaluation provided by to services such as local enterprises – 95 percent of business representatives in treatment banks, schools, areas state that access to services has improved for them. hospitals, etc. • The average score for satisfaction with access to services also shows an Access to + increasing trend. The net change for treatment areas for the local 8 Services Higher satisfaction population equals 4.2 points. For local business it is even higher and with access to equals 4.6 points. various services • Qualitative findings from the local government study also confirm an improvement in access. Among the benefits of road rehabilitation, faster delivery of various emergency services such as ambulance, fire rescue, etc. and better access to a larger variety of goods are named by the local population. Page 54 of 55 The World Bank Second Secondary and Local Roads Project (SLRP II) (P122204) 6.6 Project Map Page 55 of 55