PERU Decentralized Subnational Roads Management Report No. 138265 JUNE 21, 2019 © 2019 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2019. Peru—Decentralized accuracy of the data included in this work. The Any queries on rights and licenses, including Subnational Roads Management. Independent boundaries, colors, denominations, and other subsidiary rights, should be addressed to Evaluation Group, Project Performance information shown on any map in this work do World Bank Publications, The World Bank Assessment Report 138265. Washington, DC: not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC World Bank. World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Report No.: 138265 PROJECT PERFORMANCE ASSESSMENT REPORT PERU REGIONAL TRANSPORT INFRASTRUCTURE DECENTRALIZATION PROJECT (LOAN NO. 73220) DECENTRALIZED RURAL TRANSPORT PROJECT (LOAN NO. 74230) June 21, 2019 Financial, Private Sector, and Sustainable Development Independent Evaluation Group Currency Equivalents (annual averages) Currency Unit = Peruvian Nuevos Soles, S/. 2006 $1.00 S/. 3.27 2007 $1.00 S/. 3.13 2008 $1.00 S/. 2.92 2009 $1.00 S/. 2.99 2010 $1.00 S/. 2.81 2011 $1.00 S/. 2.76 2012 $1.00 S/. 2.63 2013 $1.00 S/. 2.70 2014 $1.00 S/. 2.87 Abbreviations AADT annual average daily traffic LDW local development window CAS Country Assistance Strategy M&E monitoring and evaluation CBA cost-benefit analysis MEF Ministry of Economy and Finance CEA cost-efficiency analysis MTC Ministry of Transport and CPS Country Partnership Strategy Communications ERR economic rate of return NMT nonmotorized transport FONIE Fondo para la Inclusión Económica NPV net present value de Zonas Rurales PAD project appraisal document FY fiscal year PDO project development objective IBRD International Bank for PII provincial infrastructure institute Reconstruction and Development PPAR Project Performance Assessment ICR Implementation Completion and Report Results Report PRI provincial road institute IDB Inter-American Development Bank Provias Provias Descentralizado IEG Independent Evaluation Group All dollar amounts are U.S. dollars unless otherwise indicated. Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation Ms. Alison Evans Director, Financial, Private Sector, and Sustainable Development Mr. José C. Carbajo Martínez Manager, Sustainable Development Ms. Midori Makino Task Manager Ms. Elisabeth Goller ii Contents Preface ...................................................................................................................................................................... viii Summary .................................................................................................................................................................... ix 1. Background and Context ..................................................................................................................................1 World Bank Support for the Road Network in Peru ................................................................................2 2. Regional Transport Infrastructure Decentralization Project ................................................................4 Cost, Financing, and Key Dates.......................................................................................................................4 Relevance of the Objectives and Design .....................................................................................................5 Objectives.............................................................................................................................................................................. 5 Relevance of the Objectives ........................................................................................................................................... 5 Design..................................................................................................................................................................................... 6 Relevance of the Design .................................................................................................................................................. 8 Implementation ....................................................................................................................................................9 Safeguards Compliance ................................................................................................................................................. 10 Financial Management and Procurement................................................................................................................. 11 Achievement of the Objectives ..................................................................................................................... 12 Objective 1 .......................................................................................................................................................................... 13 Objective 2 .......................................................................................................................................................................... 15 Objective 3 .......................................................................................................................................................................... 17 Efficiency.............................................................................................................................................................. 20 Outcome ............................................................................................................................................................. 22 Risk to Development Outcome ................................................................................................................... 22 Institutional Risks ............................................................................................................................................................. 22 Infrastructure Maintenance Risks ............................................................................................................................... 23 Bank Performance ............................................................................................................................................ 23 Quality at Entry ................................................................................................................................................................. 23 Quality of Supervision .................................................................................................................................................... 24 Borrower Performance ................................................................................................................................... 25 Government Performance ............................................................................................................................................ 25 Implementing Agency Performance ......................................................................................................................... 26 Monitoring and Evaluation ............................................................................................................................ 27 Design.................................................................................................................................................................................. 27 iii Implementation ................................................................................................................................................................ 28 Use ........................................................................................................................................................................................ 28 3. Decentralized Rural Transport Project...................................................................................................... 28 Cost, Financing, and Key Dates.................................................................................................................... 28 Relevance of the Objectives and Design .................................................................................................. 29 Objectives........................................................................................................................................................................... 29 Relevance of the Objectives ........................................................................................................................................ 29 Design.................................................................................................................................................................................. 30 Relevance of the Design ................................................................................................................................................ 31 Implementation ................................................................................................................................................. 32 Safeguards Compliance ................................................................................................................................................ 34 Financial Management and Procurement............................................................................................................... 34 Achievement of the Objective...................................................................................................................... 35 Outputs ............................................................................................................................................................................... 35 Outcomes ........................................................................................................................................................................... 38 Efficiency............................................................................................................................................................... 41 Outcome ............................................................................................................................................................. 43 Risk to the Development Outcome ............................................................................................................ 44 Infrastructure Maintenance Risks ............................................................................................................................... 44 Economic and Livelihood Risks ................................................................................................................................... 44 Bank Performance ............................................................................................................................................ 44 Quality at Entry ................................................................................................................................................................. 44 Quality of Supervision .................................................................................................................................................... 45 Borrower Performance ................................................................................................................................... 46 Government Performance ............................................................................................................................................ 46 Implementing Agency Performance ......................................................................................................................... 46 Monitoring and Evaluation ............................................................................................................................ 47 Design.................................................................................................................................................................................. 47 Implementation ................................................................................................................................................................ 47 Use ........................................................................................................................................................................................ 48 4. Lessons ................................................................................................................................................................. 48 References................................................................................................................................................................ 51 iv Boxes Box 2.1. Regional Project Components .............................................................................................................. 7 Box 3.1. Rural Project Components ................................................................................................................... 31 Box 3.2. Observations from IEG’s Field Visits to Ayacucho and Cajamarca ......................................... 41 Table Table 2.1. Regional Project Ratings .................................................................................................................. 22 Appendixes Appendix A. Basic Data Sheet .......................................................................................................................... 53 Appendix B. Fieldwork Methodology ............................................................................................................ 59 Appendix C. Adequacy of Road Maintenance Expenditures ................................................................ 65 Appendix D. Evaluation of the Quality of the Impact Assessment ..................................................... 70 Appendix E. Field Visits ....................................................................................................................................... 83 Appendix F. List of Persons Met ....................................................................................................................100 Appendix G. Borrower Comments ................................................................................................................103 This report was prepared by Elisabeth Goller, who assessed the project in April 2019. The report was peer-reviewed by Satoshi Ogita and panel-reviewed by Lauren Kelly. Ana Silvia Aguilera and Doruk Yarin Kiroglu, consultants, provided research assistance, and Richard Kraus and Alexandra Sears provided administrative support. v Principal Ratings Regional Transport Infrastructure Decentralization Project Indicator ICR ICR Review PPAR Outcome Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Risk to development Significant Significant Significant outcome Bank performance Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Borrower performance Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Note: The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. PPAR = Project Performance Assessment Report. Decentralized Rural Transport Project Indicator ICR ICR Review PPAR Outcome Highly satisfactory Satisfactory Satisfactory Risk to development Moderate Significant Significant outcome Bank performance Satisfactory Moderately satisfactory Satisfactory Borrower performance Satisfactory Satisfactory Satisfactory Note: The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. PPAR = Project Performance Assessment Report. Key Staff Responsible Regional Transport Infrastructure Decentralization Project Management Appraisal Completion Project Team Leader Aurelio Menendez Anna R. Okola Sector Manager or Practice Manager Jose Luis Irigoyen Aurelio Menendez Sector Director or Senior Global Practice n.a. n.a. Director Country Director Marcelo Giugale Alberto Rodriguez Note: n.a. = not applicable. Decentralized Rural Transport Project Management Appraisal Completion Project Team Leader Nicolas Peltier-Thiberge Maria Margarita Nunez Sector Manager or Practice Manager Jose Luis Irigoyen Aurelio Menendez Sector Director or Senior Global Practice n.a. n.a. Director Country Director Marcelo Giugale Livia M. Benavides Note: n.a. = not applicable. vi IEG Mission: Improving World Bank Group development results through excellence in independent evaluation. About This Report The Independent Evaluation Group (IEG) assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the World Bank’s self-evaluation process and to verify that the World Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20–25 percent of the World Bank’s lending operations through fieldwork. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or World Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, interview World Bank staff and other donor agency staff both at headquarters and in local offices as appropriate, and apply other evaluative methods as needed. Each PPAR is subject to technical peer review, internal IEG panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible World Bank Country Management Unit. The PPAR is also sent to the borrower for review. IEG incorporates both World Bank and borrower comments as appropriate, and the borrowers’ comments are attached to the document that is sent to the World Bank’s Board of Executive Directors. After an assessment report has been sent to the Board, it is d isclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrumen t, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldbankgroup.org). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current World Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, sector strategy papers, and operational policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared with alternatives. The efficiency dimension is not applied to development policy operations, which provide general budget support. Possible ratings for outcome: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for risk to development outcome: high, significant, moderate, negligible to low, and not evaluable. Bank Performance: The extent to which services provided by the World Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan or credit closing, toward the achievement of development outcomes). The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible Ratings for borrower performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. vii Preface This is a Project Performance Assessment Report (PPAR) by the Independent Evaluation Group (IEG) of the World Bank Group on the Peru Regional Transport Infrastructure Decentralization Project (P078813) and the Decentralized Rural Transport Project (P095570). The projects were selected for a PPAR as part of a cluster of assessments on subnational roads projects with strong institutional components. The projects formed part of a 25-year World Bank engagement on subnational roads in Peru. The Regional Transport Infrastructure Decentralization Project was approved on July 12, 2005, became effective on April 10, 2006, and closed on June 30, 2014. The total cost at appraisal was $200 million. The actual cost was $167 million because not all activities were carried out. The Decentralized Rural Transport Project was approved on December 19, 2006, became effective on July 12, 2007, and closed on December 31, 2013. The total cost at appraisal was $150 million. The actual cost at completion was $160 million. The project was restructured in 2010, and this restructuring lowered the project’s level of ambition. Therefore, the PPAR applies a split rating, assessing the outcome across the entire project period against both the original and the revised project targets. The Regional Transport Infrastructure Decentralization Project was designed to replicate the success of previous rural roads projects in Peru and to take them to scale, regionally. The Decentralized Rural Transport Project was also designed to improve and expand previously piloted activities in the country. Both projects were implemented within the context of the 2002 postdecentralization process whereby formal responsibility for subnational roads was devolved to local governments. This PPAR is based on a review of the World Bank’s project documentation and analytical studies, combined with a field mission to Peru carried out from March 25 to April 5, 2019. IEG conducted interviews with a range of stakeholders linked to the projects, including government officials at national and subnational levels, staff of the implementing agency, World Bank staff, other development partners, private sector and civil society representatives, and project beneficiaries. IEG is grateful to the government of Peru and the World Bank country office staff for facilitating access to the projects and their associated sites. IEG also thanks the many national, regional, provincial, and district officials for their generous time and attention. IEG would like to thank especially Sonia Pezo and Miguel Castro (from the project implementation agency) and Alexandra Sears (from the Peru country office) for their wonderful support. Following standard IEG procedure, a copy of the draft PPAR was shared with relevant government officials for their review and comments, which appear in appendix G. viii Summary This is a Project Performance Assessment Report (PPAR) on the Peru Regional Transport Infrastructure Decentralization Project (P078813) (henceforth Regional Project) and the Decentralized Rural Transport Project (P095570) (henceforth Rural Project). Project Objective, Implementation Period, and Financing The project development objective (PDO) of the Regional Project was “to improve through decentralization at the regional level the prioritization, efficiency, and effectiveness of regional transport interventions to contribute to regional development and poverty alleviation by enhancing transport conditions in the borrower’s territory.” It was implemented between 2006 and 2014 mostly at the regional level, with overall coordination by the national government. The PDO of the Rural Project was “to contribute to territorial development and the fight against rural poverty in the borrower’s territory by improving access of rural households and entrepreneurs to goods, social services, and income-generating opportunities through reduced transport costs and better rural transport infrastructure.” It was implemented between 2007 and 2013 at the national level, but it gradually decentralized activities to local governments. Project Performance of the Peru Regional Transport Infrastructure Decentralization Project The relevance of the PDO was and remains high. At appraisal, Peru’s regional system had recently been created, the capacity of the regional administrations to manage their road network was limited, and only 15 percent of this road network was in good condition. Regional roads critically connect Peru’s main production areas to its logistics corridors. The relevance of having better regional roads and improving sector management is reflected in the Country Assistance Strategy, Country Partnership Strategies, and government policy documents. The relevance of design is rated modest before and after project restructuring. Although the project components and activities were relatively well aligned with the PDO and followed a logical causal chain, the project scope was overly ambitious, as it would have been difficult for all 24 regions to implement major institutional and behavioral changes within less than five years. The project design also lacked incentives for the regions to implement these changes. More inputs focused on road works than on institutional activities even though the institutional aspects prevailed in the PDO statement. The results framework was not explicitly laid out, and the PDO level indicators were insufficient to tie together the elements of the results framework. Efficacy is rated negligible before the 2010 project restructuring and modest afterward. Before and after its restructuring, the project partially achieved the first objective of improving the prioritization of regional transport interventions. Regional participation in the preparation of road plans was less than expected, and the subsequent lack of ownership led to the plans not ix being used outside the project context. The lack of regional engagement undermined the project’s aim to build capacity as part of the decentralization process. However, the plans achieved a degree of enhanced prioritization by guiding the selection of 1,562 kilometers of regional roads for project purposes. The achievement of the second objective—improving the efficiency of regional transport—is negligible before and after the project restructuring. The project focused too much on works and not enough on institutional strengthening. Although the regions were responsible for most project implementation activities, the central-level agencies eventually carried out many activities for them. The project had no regional buy-in to carry out institutional reforms and enable capacity building in line with the country’s decentralization aims, and it was undermined by frequent staff rotations. Efficiency was improved only with respect to the contracting of road works and maintenance to the private sector rather than carrying them out inefficiently in house. The project achieved the third objective of improving the effectiveness of regional transport only to a negligible extent before, but substantially after, its restructuring. At project restructuring, the condition of the regional road network was the same as at appraisal, and there had been no progress in implementing a sustainable road maintenance model. By project end, the percentage of the road network in good condition was nearly in line with the revised outcome target. Although Peru lacks road condition data for the subnational road network, several factors support the likelihood that the project roads are being maintained, and funds to maintain part of the regional road network are available. Peru has also recently started a program to pave and maintain over half of the regional network. Efficiency of the project is rated modest. The ex post economic rate of return is below the appraisal estimate. The cost per beneficiary (per kilometer) for very low-volume roads is significantly higher than the ex ante estimate. The project experienced significant administrative and operational inefficiencies that contributed to major implementation delays. There were also technical inefficiencies; for example, the Peruvian government paved several roads improved under the project shortly after project completion. This increased the overall investment cost. The overall project outcome is rated moderately unsatisfactory because of the high relevance of the PDO before and after restructuring, the overall modest relevance of design, the negligible efficacy before the project restructuring and the modest efficacy afterward, and overall modest efficiency. Bank performance is rated moderately unsatisfactory. At the design stage, the project did not include discussions on the suitability of the rural roads solutions for the regional context. This omission led to insufficient choices and flexibility, which later influenced implementation adversely. Another critical shortcoming was the inadequate involvement of the regions in project design and preparation. The subsequent lack of the regions’ buy-in and commitment, x which eventually caused the project to only partially achieve its intended outcomes, was not anticipated, and the choices regarding project implementation arrangements contributed to significant delays. The monitoring and evaluation design was weak. Although intensive because of problems and delays, the World Bank’s implementation support was not sufficiently strong on the institutional dimensions. These dimensions were essential for the project to achieve its intended outcomes. Safeguards supervision should have been more intense, and the World Bank supervision team should have restructured the project again to reflect the regions’ lack of interest. Borrower performance is rated moderately unsatisfactory. Although strongly committed to the project, the national government did not involve the regions in project design. It also delayed the hiring of consultants to make up for the weaknesses at regional level. Elections caused staff changes and delayed the project. The national government has not yet started the process to transform the implementation agency, and decentralization in the road sector has largely stalled. The regions had a limited interest in the project. The implementation agency was experienced and played a much stronger role in project implementation than originally envisaged. Insufficient attention was paid to institutional activities. Training for regional officials mainly focused on operational skills. Safeguard supervision and reporting on institutional indicators were weak. Project Performance of the Peru Decentralized Rural Transport Project The relevance of the PDO was and remains high. At appraisal, 74 percent of Peru’s rural population was poor, and although rural poverty has declined significantly, it continues to be a challenge. Rural roads were and still are in poor condition. The relevance of territorial development and rural poverty alleviation, mainly through improved access, is reflected in the Country Assistance Strategy, Country Partnership Strategies, and government policy documents. The relevance of design was substantial. Territorial development and poverty reduction are complex issues that require a multidisciplinary approach. Rural road interventions can contribute to those aims if they are well prioritized and executed, and the roads subsequently maintained. The project design was innovative and experimented with different road-related elements designed to directly contribute to poverty reduction, including road maintenance job creation and enterprise development. The project logic was sound. The results framework adequately linked planned activities with intended project aims, but PDO outcome indicators were not explicit in the project appraisal document. This was not a substantial problem because the project commissioned an external impact assessment (Macroconsult-Cuanto 2014). Efficacy is rated substantial. This was one of the few World Bank transport projects with an impact assessment. In addition to decreased travel times and costs, increased access, and enhanced agriculture production, the impact assessment showed that the rural road xi improvements contributed to decrease extreme poverty and increase consumption (Macroconsult-Cuanto 2014). However, the impact assessment exercise was challenged by a significant level of attrition and the contamination of the control group. It had to use small and unbalanced samples, which raises questions about the reliability of the reported impacts, especially the poverty reduction impact as attributed to the rural roads. Despite the identified weaknesses of the impact assessment, juxtaposing project achievements (decreased travel times and costs, enhanced access to goods and services) against findings from the literature and contextual factors suggests that the rural road improvements contributed to rural poverty reduction, although the extent of the contribution is not known. Efficiency of the project is rated substantial. Although the project suffered from administrative inefficiencies, the ex post economic rate of return is significantly higher than that at appraisal, and the cost per beneficiary per kilometer for road improvements is within a range considered reasonable in Peru. The overall outcome of the project is rated satisfactory based on the high relevance of the PDO, and the significant relevance of design, efficacy, and efficiency. Bank performance is rated satisfactory. The project design reflected the experience gained over time and corrected past shortcomings. Most risks were adequately anticipated, and mitigation measures were satisfactory. The monitoring and evaluation design was largely adequate. The gender action plan was a vanguard initiative. The complexity of the rural infrastructure pilot was underestimated, and the small subcomponent on bridges lacked the necessary background studies. Supervision was more balanced than under the Regional Project, with considerable attention given to institutional and territorial development activities. A timely restructuring was undertaken, which could also have been used to rescale the rural infrastructure pilot. Safeguard supervision should have been more intense. Borrower performance is considered satisfactory. The national government fully supported this ambitious project, which went beyond “road building,” and it continued to increase the delegation of responsibilities to local governments. The government financed the impact assessment. Collaboration with local governments was easier than under the Regional Project. Initially, the implementation agency focused on the physical road works, but then caught up with the institutional activities. The agency was intensively involved in the decentralization process, was strongly engaged in solving the road maintenance funding shortages, and anticipated the election-related interruptions. Fiduciary aspects were handled adequately, and reporting was timely. There were shortcomings in safeguard supervision. Lessons Subnational governments need to own their road planning instruments to ensure their use. Under both projects, consultants hired by the national government produced comprehensive, xii technically sound, and well-prepared road plans. Although the subnational governments were formally in charge of plan preparation, their involvement was limited, and the national government drove the initiative. The plans guided the selection of the project priorities, but due to their limited buy-in, subnational governments no longer used those plans for setting priorities once the project ended. Ways to sustain the community-based microenterprises model for rural road maintenance should be identified. In both projects, the implementation agency supported the recruitment of members of community-based microenterprises, assisted them in formalizing the microenterprise, and provided technical and managerial skills. In this way, the projects achieved the dual aim of contributing to employment creation and maintaining the roads. However, once the projects and the support ended, this model was not sustained, and most community-based microenterprises have disappeared. Thus, it will be important to find ways to sustain the model with less support and hand-holding or to pass the responsibility for this support together with the necessary resources to the local level. Road maintenance is essential all year round, and funding and bidding schedules need to be adjusted accordingly. Since 2008, the Ministry of Economy and Finance (MEF) has transferred resources for road maintenance to subnational governments, but these resources have arrived late in the fiscal year and in several installments. Despite the MEF making the full resources available at the beginning of January 2019, it has taken the local governments until end of March to sign the necessary agreements and start the bidding processes. Because the rainy season in Peru lasts until April, maintenance is most needed in the first months of the year. Given that the budget process and timeline cannot be changed, multiyear maintenance contracts that start in midyear could ensure maintenance all year round. Otherwise, the bidding processes for maintenance could start before the end of the year to ensure that the contracts can be signed in January. Poverty impacts of rural roads projects are difficult to attribute. Rural roads projects should aim for more than just enhancing road conditions and reducing travel times and costs. The Rural Project is to be commended for trying to show its impact on poverty reduction and for the comprehensive impact assessment itself. However, it was impossible to attribute the poverty impact to the project. Thus, as desirable as it is for roads projects to measure benefits other than time and vehicle cost savings, it is better to limit the ambitions of the measurement to the type of project impacts and geographical scope that can be reliably attributed to the project. If the road agency carries out activities that are outside its core responsibilities, it needs to involve the ministries and government agencies responsible for these activities to ensure sustainability. The Rural Project supported a local development window (LDW) that financed activities to help local governments and producers’ associations enhance economic development along improved roads. The road agency outsourced the implementation of the LDW activities, and this was relatively successful. However, once those contracts finished, there xiii was a lack of organizational clarity about how the activities would be supported to ensure continuity. Multiple ministries and government agencies share the responsibility of helping small producers and local governments to foster economic development. They should be involved in the design and implementation of the LDW activities, which need to strongly focus on sustainability. Transferring successful solutions from one government level to another requires a careful contextual analysis and the subnational governments’ participation in decision making from the outset. In this case, transferring successful solutions from the rural to the regional context did not work because the success factors for the Rural Project and the differences in the regional context were not adequately analyzed. In addition, key aspects of project design (such as how to identify road priorities, select road management solutions, and identify institutional changes to be implemented by the regions) rested with the central authorities. This did not create sufficiently flexible solutions, empower the subnational governments, or create the ownership and capacity required to successfully implement the project. José Carbajo Martínez Director, Financial, Private Sector, and Sustainable Development Department Independent Evaluation Group xiv 1. Background and Context 1.1 This report is the Project Performance Assessment Report (PPAR) for the Peru Regional Transport Infrastructure Decentralization Project (P078813), henceforth the Regional Project, and the Decentralized Rural Transport Project (P095570), henceforth the Rural Project. The Regional Project became effective in April 2006 and closed in June 2014. The Rural Project became effective in July 2007 and closed in December 2013. 1.2 Peru has made great strides in reducing poverty over the past decade, including in rural areas. Poverty and extreme poverty declined from 42.4 percent and 11.2 percent, respectively, in 2007 to 21.7 percent and 3.8 percent in 2017, respectively.1 Poverty has mostly been concentrated in rural areas; in 2007, 74 percent of the rural population was poor, and one-third of this population was extremely poor. A decade later, rural poverty had declined to 44.4 percent overall. 1.3 Poverty has also declined in Peru’s remote and mountainous regions, which are the poorest parts of Peru, but levels remain high. At the beginning of the project periods, poverty in mountainous regions ranged from 70 to 89 percent. A decade later, poverty in these areas had been reduced to 48.7 percent, with levels ranging between 33 and 37 percent in Amazonas, Apurimac, Ayacucho, Huancavelica, Huánuco, Loreto, Pasco, and Puno, and a level of 52 percent in Cajamarca. 1.4 Among a host of factors enabling rural poverty reduction in Peru is access to transport infrastructure and services, especially in mountainous and remote areas with low population density. There is substantial evidence that improvements in rural transport increase accessibility to public services (Bell and van Dillen 2012; Khandker, Bakht, and Koolwal 2006; Levy 2004). There is also evidence that improved transport conditions positively affect rural productivity and that access to employment opportunities, education, health care, and other services positively affects poverty alleviation efforts (Abdulkadir, Adefila, and Musa 2013; Asher and Novosad 2019; World Bank 2005). 1.5 Peru has an extensive subnational road network, which is only partly maintained. As of 2015, Peru’s regional road network—the network under the responsibility of its 24 regions—consisted of 3,907 kilometers of paved and 20,675 kilometers of unpaved roads (MTC 2015). As of 2018, its rural road network—the network under the responsibility of provinces and districts—was officially registered as consisting of 113,857 kilometers of road, of which about 1 percent was paved.2 Transport consultants interviewed by the Independent Evaluation Group (IEG) mentioned that the rural network may be 20 to 30 percent longer. According to the counterpart, only 20 to 1 25 percent of the rural road network is regularly maintained, and this share comprises mainly roads improved under World Bank–financed projects. 1.6 The management of the subnational network has been the responsibility of the regions, provinces, and districts since the enactment of two major decentralization laws in 2002, which transferred political power to these jurisdictions. Decentralization was intended to better tailor infrastructure and services to local needs, improve the efficiency of public spending, and regionally balance economic growth and improve socioeconomic conditions in lagging regions (World Bank 2017a). 1.7 Managing such a vast road network at the subnational level and ensuring that people in the remotest areas have access to markets and services is a gigantic effort. This is especially true considering that before the mid-1990s, the subnational road network in Peru had been abandoned due to bad construction, climate factors, lack of maintenance, and resources and financing (Guerra Garcia Picasso 2016). World Bank Support for the Road Network in Peru 1.8 The World Bank has been supporting the subnational road sector in Peru for 25 years. A strong feature of this support has been to pilot and test new institutional and managerial initiatives as part of decentralized road management and territorial development. The World Bank has helped the government test these efforts on a small scale, adapt the successful experiences, and scale them nationally. In every case, the World Bank has partnered with the Inter-American Development Bank (IDB) to ensure consistency in the sector. 1.9 Between 1995 and 2000, the World Bank launched its first Rural Roads Rehabilitation and Maintenance Project (P037047), which focused on road rehabilitation and maintenance in Peru’s 12 poorest regions. The project supported road planning, road management, and the creation of community-based microenterprises, comprising residents living along the roads targeted for maintenance. The project rehabilitated and maintained approximately 12,000 kilometers of rural roads and 3,000 kilometers of nonmotorized transport (NMT) tracks. 1.10 The second Rural Roads Rehabilitation and Maintenance Project (P044601), implemented between 2001 and 2006, rehabilitated 12,350 kilometers of rural roads, 2,700 kilometers of secondary roads, and 6,300 kilometers of NMT tracks. It also provided periodic and routine maintenance. The project improved the approach to road planning and maintenance and expanded the geographical reach to 19 regions. It piloted new initiatives, which included the provincial road institutes (PRIs) and the local development window (LDW): 2 • In Peru, a PRI is a decentralized public body created by the provincial and district municipalities to plan and manage all roads in a province. It has a provincial road board, which is chaired by the provincial mayor and includes all district mayors. The interesting feature of the PRI is that it represents both the provincial and the district governments; hence it can carry out road works and maintenance on all rural roads in a province, whereas each province and district can work only within its own jurisdiction. • The LDW consisted of activities to stimulate local economic development. The concept was based on the idea of initiating territorial development through good roads and complementing them with initiatives to foster productive activities along these roads. The LDW helped local governments to (i) prepare economic development plans, (ii) identify local economic development projects and initiatives, (iii) issue favorable policy measures, and (iv) create networks of stakeholders for strategic products. The LDW activities also helped associations of producers (i) formalize, (ii) prepare business plans, (iii) strengthen management capacity, and (iv) access financing. 1.11 The Rural Project, which is part of this assessment, further expanded the previous initiatives. It expanded coverage to all 24 regions in Peru, and it planned to gradually transfer more implementation responsibilities to local governments. It also aimed at going beyond roads improvements and included the preparation of provincial economic infrastructure plans and the creation of provincial infrastructure institutes (PIIs). These were expected to manage all infrastructure in a province in an integrated manner, as the PRIs did for roads; eventually the PRIs were expected to be integrated into the PIIs. The project rehabilitated 3,277 kilometers of road, provided periodic maintenance to 7,806 kilometers of road, and rehabilitated 2,356 kilometers of NMT tracks. 1.12 The Regional Project, also assessed in this report, not only carried out road rehabilitation and maintenance, but also attempted to transfer the rural road management model, including the planning and maintenance approach, to the regional road network. This project focused on road planning and management and did not include regional territorial development activities. It rehabilitated 1,562 kilometers of road and provided periodic maintenance to 3,541 kilometers of road. It also provided routine maintenance. 1.13 Under the ongoing Support to the Subnational Transport Program Project (P132515), the focus has partially shifted to the improvement of subnational logistics corridors. The project also aims at strengthening and perfecting (i) decentralized road management, (ii) the microenterprise maintenance model, (iii) the LDW, and (iv) the 3 planning approach. It is expected to rehabilitate 2,200 kilometers and maintain 5,000 kilometers of road, and preparations to improve its impact evaluation are under way. 2. Regional Transport Infrastructure Decentralization Project Cost, Financing, and Key Dates 2.1 The total estimated project cost at appraisal, including contingencies, was $200 million. The actual cost at completion was $167.07 million, 83.53 percent of the appraisal estimate. Despite road construction cost increases during project implementation, the actual cost was lower because not all project activities were carried out as planned. 2.2 The financing came from an International Bank for Reconstruction and Development (IBRD) loan in the amount of $41.60 million and parallel IDB cofinancing of $45.97 million. Each bank originally committed $50 million. The remaining balance was covered by the borrower. 2.3 The project was approved on July 12, 2005, became effective on April 10, 2006, and closed on June 30, 2014. The loan closing date was extended three times, for four years, due to project delays. 2.4 The first extension of 24 months, from June 30, 2010, to June 30, 2012, occurred because the loan became effective later than expected, and because implementation was delayed by external factors, such as the availability of alternative funding resources, which reduced the regions’ interest in the project, and a saturated construction industry. 2.5 The second extension of 18 months, to December 31, 2013, was required because of (i) staff changes after the regional and national elections in 2010 and 2011, (ii) the limited interest of the regions, and (iii) a prolonged rainy season. The final extension of six months, to June 30, 2014, was necessary to complete ongoing work contracts and other project activities. This brought the implementation period from five years to nine years. 2.6 On July 4, 2013, the financing percentages under the loan were increased; through a level-two restructuring approved on July 10, 2010, nearly all indicator targets, including the target for one project development objective (PDO) indicator, were reduced. These changes occurred because of road construction cost increases caused by Peru’s construction boom during the project implementation period. At that time, $9.26 million, or 22 percent of the loan amount, was disbursed. This restructuring 4 lowered the project’s level of ambition and warrants a PPAR split rating, one that assesses the outcome for the entire project period against both the original and revised project targets. Relevance of the Objectives and Design Objectives 2.7 The PDO of the Regional Project, as stated in the legal agreement, was “to improve through decentralization at the regional level the prioritization, efficiency, and effectiveness of regional transport interventions to contribute to regional development and poverty alleviation by enhancing transport conditions in the borrower’s territory.” The objective stated in the project appraisal document (PAD) is nearly identical: “to improve—through decentralization at the regional level—the prioritization, efficiency, and effectiveness of regional transport interventions and hence their contribution to regional development and poverty reduction in Peru” (World Bank 2005b, 6). Relevance of the Objectives 2.8 The relevance of the PDO was and remains high. At appraisal, the 24 Peruvian regions had recently been established, the capacity of the regional administrations to manage their road network was limited, and only 15 percent of this road network was in good condition. Regional roads are also key feeders to link agriculture production areas to Peru’s logistics corridors. 2.9 The objective was and remained well aligned with the Country Assistance Strategy (CAS) and Country Partnership Strategies (CPSs). The importance of the PDO for Peru was reflected in the fiscal year (FY)03–06 CAS, which identified the need to “strengthen the subnational management capacity” by promoting decentralization reforms at the regional level. It also focused on “investment programs that have a direct impact on the productive life of the poor” (World Bank 2002). The FY07–11 CPS envisaged under the economic growth pillar extending “infrastructure to rural areas with the highest concentration of poverty” and identified transport as one of the sectors with the greatest infrastructure deficits. Under the state modernization pillar, “effective decentralization to deliver services locally was key” (World Bank 2006a). The two strategic objectives of the FY12–16 CPS focused on “connecting the poor to services and markets” through better transport and infrastructure to reduce inequality, and seeking “improved public sector performance for greater inclusion” by strengthening the subnational management capacity (World Bank 2012). 2.10 The relevance of the PDO to the government’s past and current policies was and remains high. The priorities of the 2006–11 Government Plan,3 among others, were to 5 develop rural areas by expanding infrastructure services within the context of the decentralization reforms. For the road sector, this plan envisaged the transfer of road sector management responsibilities from national to subnational levels, which would be supported by increased budgetary transfers and institutional strengthening. 2.11 Several recent government plans and programs demonstrate the continued relevance of the objective. The 2012–16 Strategic Sector Plan of the Ministry of Transport and Telecommunications (MTC) focused on establishing logistics platforms and corridors to facilitate trade and meet local and internal demand. In 2019, the MTC prepared the Development Plan for Subnational Logistics to support investments in transport infrastructure, logistic services, and capacity development at the subnational level. It is to be implemented through the Pro Región Program, which is expected to finance the upgrading and paving of 15,000 kilometers of subnational roads. Decentralization and coordination between different government entities are key pillars under the National Policy to Modernize the Public Sector to 2021, while the General Government Policy to 2021 calls for effective decentralization for development through the institutionalization of territorial coordination in national policies, the improvement of decentralized and sustainable public and private investments, and the reduction of poverty. 2.12 The PDO was realistic because improvements in regional transport are under the control of national and regional governments, and it is plausible that such improvements eventually contribute to the higher-level objective of regional development and poverty alleviation. Design 2.13 The project was designed with five components as shown in box 2.1. 6 Box 2.1. Regional Project Components Component 1: Preparation of participatory regional road planning (estimated cost $10.9 million; actual cost $9.44 million). This component (i) helped prepare, finalize, or update participatory regional road plans, aligned with the existing regional development plans, using a prioritization methodology that combined both economic potential and poverty-level criteria (subcomponent 1A); and (ii) financed feasibility and technical studies for the road segments prioritized through participatory planning exercise (subcomponent 1B). Component 2: Upgrading of regional roads (estimated cost $138.84 million; actual cost $130.29 million). This component was originally to (i) rehabilitate approximately 2,200 kilometers of regional roads prioritized through participatory planning under component 1; and (ii) perform the periodic maintenance of 2,706 kilometers of regional roads rehabilitated by Provias Descentralizado, the project implementation agency under the MTC, and transferred to the regional governments. This component was revised in July 2010 to reduce the rehabilitation target to 1,781 kilometers and the periodic maintenance target to 2,202 kilometers. This change was necessary because of road construction cost increases caused by the economic boom in Peru that saturated the market. Component 3: Routine maintenance of regional roads (estimated cost $26.12 million; actual cost $13.85 million). This component was originally to finance (i) routine maintenance and supervision of 4,900 kilometers of regional roads rehabilitated or having received periodic maintenance under the previous component; and (ii) specific road maintenance interventions, such as annual mechanized maintenance carried out once a year after the rainy season. For the same reason and on the same date as mentioned under component 2, the routine maintenance target was reduced to 4,219 kilometers. Component 4: Institutional capacity building (estimated cost $11.01 million; actual cost $5.01 million). This component was originally to (i) rationalize the current institutional framework of regional governments, in particular restructure or possibly merge the Regional Roads Departments (previously with MTC) and the Regional Infrastructure Management Units (newly created as part of the organizational structure of the regional governments); (ii) support a transition from direct administration of road maintenance or rehabilitation to contracting this function out to the private sector; (iii) train regional and the national governments in safeguards management; (iv) clarify and assign regulatory responsibilities, such as those related to road safety and the regulation of transport services; (v) evaluate the financial situation of the regions and define a strategy for road sector financing; (vi) carry out the restructuring of Provias Departamental (the predecessor of Provias Descentralizado); and (vii) handle monitoring, auditing, and evaluation. The July 2010 restructuring narrowed the focus for technical assistance and capacity building to 17 regions that had shown commitment to the project. This also implied a reduction in the targets for microenterprise creation and the proportion of regions that successfully implemented reforms. Many activities under this component were not implemented as planned. Component 5: Project administration (estimated cost $6.13 million; actual cost $8.39 million). This component supported project management. 7 Relevance of the Design 2.14 This assessment rates the relevance of design as modest before and after project restructuring. The July 2010 restructuring lowered the level of ambition of the project because of construction cost overruns and a lack of interest in the project by the regions. It did not change the essence of the project design. 2.15 Before restructuring, the project was expected to cover 2,200 kilometers of rehabilitation, 2,700 kilometers of periodic maintenance, and 4,900 kilometers of routine maintenance, and also to provide technical assistance and capacity strengthening to all 24 regions. The restructuring reduced the works to 1,781 kilometers of rehabilitation, 2,202 kilometers of periodic maintenance, and 4,219 kilometers of routine maintenance. It reduced the coverage for technical assistance and capacity strengthening to 17 regions. 2.16 The project components and activities before and after restructuring were relatively well aligned with the PDO. Participatory regional road plans using appropriate planning tools were expected to produce more relevant road investment priorities in line with local needs to improve investment prioritization. The rationalization of the institutional framework at the regional level, and changes in the management of the road sector, together with strong capacity strengthening efforts, were expected to increase the regions’ road management capacity and thus sector efficiency. The upgrading of road infrastructure and the focus on road maintenance were expected to improve road quality and instill a maintenance culture, which was considered a synonym for sector effectiveness. The project was to contribute to decentralization through capacity strengthening and the implementation of key components at the regional level. 2.17 However, the project scope was overly ambitious because it would have been difficult for all 24 regions to implement major institutional and behavioral changes within less than five years. The project design also lacked incentives for the regions to implement these changes. More inputs focused on road works than on institutional activities, even though the institutional aspects prevailed in the PDO statement. 2.18 The PAD did not lay out a results framework that linked inputs to outputs, intermediate indicators, and outcomes. The intended framework can be reconstructed through elements of the PDO and text in the PAD, including the description of the project activities (World Bank 2005b, 6). However, the relevance of the design is undermined by the absence of outcome indicators for most parts of the PDO that would weave the elements of the results framework together and link them to the achievement of the PDO. 8 Implementation 2.19 The Regional Project was implemented in a largely decentralized manner in most parts of Peru. The overall responsibility was with Provias Descentralizado (henceforth Provias), a national agency that is part of the MTC. Provias centrally managed (i) components 4 and 5, (ii) the project’s overall monitoring and evaluation (M&E), and (iii) coordination across regions and the MTC. As part of the decentralization aspect of the project, the regions were responsible for preparing the participatory regional road plans and carrying out the investments under components 2 and 3. 2.20 Project implementation was beset by several issues that nearly doubled the implementation period. As a result of these issues, not all activities were carried out, the full loan was not disbursed, and outcomes were lukewarm. Issues included the following: • Delays in the preparation and start of the Regional Project. The counterpart informed IEG that the Regional Project constituted the first collaboration between Provias and the MTC and the regions. The project was originally conceived in 2002, when Peru’s economy was still struggling and the regions had limited resources. There was a lengthy process to bring the regions on board. It also took nine months to declare the Regional Project effective because of delays in securing counterpart funds. As a result of these delays, several regions lost their enthusiasm for participating, especially because they got access to alternative sources of financing in the meantime. • Availability of attractive alternative funding. The improvement in Peru’s fiscal and financial situations lifted the budgetary restrictions existing during project preparation and increased transfers to subnational governments. Several regions received a substantial amount in mining royalties. In addition, the Ministry of Economy and Finance (MEF) created FONIPREL (Fondo de Promoción a la Inversión Pública Regional y Local), a fund for infrastructure, which provided 90 percent of the funding compared with 50 percent under the Regional Project. Thus, the project lost its initial appeal, particularly for the more affluent regions, and a lot of convincing had to be done by Provias to get regions to participate. • Disagreement on technical solutions. During project implementation, the MTC was starting to pave more roads using low-cost pavement options, and several regions with abundant resources wanted paved roads. In most cases, these roads were not economically justified. There were lengthy discussions on the topic, which further delayed project implementation. Provias and the banks agreed to pilot different low-cost pavement options. Local resources financed the pilots. 9 The pilots had no impact on the technical solutions used under the Regional Project, which financed only unpaved roads. • Reclassification of the road network. During implementation, Peru’s new road classification went into force, and several roads to be rehabilitated under the Regional Project were reclassified as national; hence substitutes had to be found. Lower-level roads were reclassified from rural to regional. Many of these roads had already been rehabilitated and needed only periodic maintenance, hence the higher target achieved for this item. • Inexperience of regional staff with World Bank projects, staff rotation, and change in priorities. Regional staff had no experience with World Bank rules and procedures. In addition, there was generally a limited interest in the Regional Project. Therefore, Provias carried out a good deal of training and had to hire consultants for the regions to carry out project implementation. The situation worsened after the regional elections in 2006 and 2010, which led to changes in high-level officials and turnover of 70 percent of the technical staff. Professionals also left the public sector for more attractive positions in the private sector because of the economic boom. New teams needed to be trained and had different priorities. Provias was also affected by several management changes and substantial rotation of technical staff. • Restructuring of Provias. During the first two years of project implementation, there was an ongoing restructuring process in Provias (for details see the Objective 2 section in this chapter). This negatively affected the pace of project implementation. • Climatic factors. Some delays during the last year of project implementation were due to longer rainy seasons than in the past. Safeguards Compliance 2.21 The project was classified as category B for environmental assessment purposes, and the following safeguard policies were triggered: Environmental Assessment (OP4.01), Cultural Property (OP4.11), Involuntary Resettlement (OP4.12), and Indigenous Peoples (OP 4.10). A strategic environmental assessment for the regional roads sector, an environmental evaluation of the first road packages, and a conceptual framework for the project’s social and environmental management, including a training plan, were prepared (World Bank 2005b, 98–99). The World Bank considered the project in compliance with the World Bank’s safeguard policies at appraisal. 10 2.22 The project did not involve major negative social and environmental impacts because it rehabilitated and maintained only existing roads within the original rights of way. According to the aide-mémoire of September 2011, the project did not intervene in areas with indigenous peoples. 2.23 However, World Bank staff informed IEG that safeguards arrangements and supervision were not fully satisfactory. Safeguards implementation was the responsibility of the regions, which were not familiar with World Bank safeguard procedures and frequently changed staff. Safeguards trainings had to be repeated. Provias had only one staff member in charge of reviewing and supervising the safeguard aspects of both the Regional Project and the Rural Project, and it delayed the hiring of additional social and environmental consultants. 2.24 During the 2009 midterm review, several shortcomings became apparent. Neither Provias nor the regions were using the conceptual framework for social and environmental management, and in one case the environmental management plan was marginally applied. Weaknesses in safeguard management and supervision persisted during the lifetime of the project. 2.25 According to World Bank staff, reporting on safeguards compliance had shortcomings, too. In 2011 (aide-mémoire of September 2011), Provias agreed to prepare an overview of the social safeguards compliance under the project. It is not clear if this overview was prepared. 2.26 By project closure, the safeguard performance was rated moderately satisfactory. Financial Management and Procurement 2.27 The financial management arrangements under the project were complex. They involved multiple agencies at the national level and within the regions. Several trainings took place in the regions because they had limited capacity. Financial management was affected by high staff turnover in the regions, which required constant retraining and support from Provias and the World Bank. Provias had a strong team in charge of financial management, which was key for the project’s success in overcoming the regions’ weaknesses and challenges. 2.28 No significant financial management issues occurred under the Regional Project, and it was carried out according to World Bank policies on internal controls and monitoring, reporting of project activities, information systems, and the systematic processing and monitoring of transfers to the regions. 11 2.29 Interim financial reports and financial audit reports were submitted in a timely manner and provided reliable information. All audit reports had unqualified options. 2.30 The regions procured the civil works, but they were not familiar with World Bank rules and procedures. This situation worsened with the extensive rotation of the procurement staff. Although Provias had an experienced procurement team that provided training and follow-up, procurement was a constant bottleneck in the first years of project implementation. 2.31 Two main procurement issues contributed to the slow project implementation: (i) participation of consultant firms and individual consultants was limited due to excessive eligibility requirements, and (ii) many bidding processes were declared nonresponsive, either because the lowest evaluated bid exceeded the estimated budget or because no bidder met the specified qualification criteria. The project had two Integrity Vice Presidency investigations, which substantiated allegations of fraudulent practices. Provias took measures to address the issues, including enhanced procurement supervision and due diligence checks. 2.32 Despite these problems, IEG’s interviews with World Bank staff indicated that with the high number of procurement processes in this project, the overall procurement performance was similar to that of other projects with less complex implementation arrangements. 2.33 Information on procurement is found in most if not all aide-mémoire. The World Bank carried out seven ex post procurement reviews between November 2007 and March 2014. The reviews included recommendations to improve the management of procurement processes. Achievement of the Objectives 2.34 This assessment uses the following three outcomes embedded in the PDO statement, which jointly were expected to improve access and hence contribute to the higher-level objective of regional development and poverty reduction in Peru: • To improve, through decentralization at the regional level, the prioritization of regional transport interventions • To improve, through decentralization at the regional level, the efficiency of regional transport interventions • To improve, through decentralization at the regional level, the effectiveness of regional transport interventions 12 Objective 1 2.35 Objective 1 was to improve, through decentralization at the regional level, the prioritization of regional transport interventions. According to the PAD (World Bank 2005b, 6), this means better aligning transport investments to local needs as identified by participatory regional development plans and appropriate planning and evaluation tools. 2.36 The efficacy is rated modest before and after the 2010 restructuring, which slightly expanded the output target and project scope related to this objective. Outputs 2.37 The project financed the preparation and updating of participatory road plans for regions without plans or with outdated plans. The counterpart pointed out, and IEG confirmed through the review of the planning methodology and procedures, that “participatory” referred to the participation of regional officials and representatives of local governments and certain trades. It was not meant to involve communities. Decentralization was to be achieved by having the regions lead the preparation of the plans rather than carrying them out at the national level. 2.38 The preparation of the plans followed a sound methodological approach and a detailed procedure elaborated centrally by Provias. The methodology focused on an analysis of the regional development context and its economic potential, a diagnostic of the road network and the available transport services, and the elaboration of proposals to put roads at the service of economic and social development. Roads were prioritized through a multicriteria analysis. 2.39 Consultants hired by Provias and a technical team of regional officials jointly prepared the plans. A regional road planning commission, including representatives from entities such as municipalities, universities, professional organizations, and the chamber of commerce, oversaw the preparation. The regions determined the composition of the technical team and the road planning commission, so they varied from region to region. 2.40 Although the Implementation Completion and Results Report (ICR, World Bank 2014b) indicates that by project end, all 24 regions had approved participatory regional road plans, in one of two regions visited the plan had not been approved. The counterpart clarified that the approval did not happen because of the region’s lack of interest. 13 Outcomes 2.41 IEG’s assessment found that participation in the preparation of the regional plans was not carried out as envisioned and that the consequent lack of engagement and ownership of the regions led to the plans not being used or updated outside of the project context. As learned in interviews with regional officials, the regions now use a range of approaches to prioritizing road interventions, including local demand and other ad hoc criteria. The prioritization obtained through the plans did guide the selection of the 1,562 kilometers of roads rehabilitated under the project; IEG’s field visits to two regions and the review of their plans showed that all five rehabilitation works carried out in these regions were in line with the plans’ priorities. 2.42 However, discussions revealed that the involvement of the regions (that is, the decentralization and participatory elements of the process) was weak. Consultants carried out most work, and neither the regional Economic Development Divisions nor local producer representatives were consulted, although they are key stakeholders. In one region visited, the road planning commission also had less representation than expected because the region’s Infrastructure Management Division was not part of the process. An interview with a member of the region’s road planning commission indicated that its participation was “informed” rather than “involved.” The commission participated in two to three meetings that were mainly informative, providing the members with an overview of the activities to be carried out, preliminary conclusions, and the justification for the selection of priorities. 2.43 This weakness in the participatory process not only limited the usefulness of the plans but also undermined the project’s contribution to capacity building at the regional level. IEG found that the planning exercise did not strengthen the capacity of the regions to plan road interventions or provide them with lasting planning tools. This was validated in discussions with the two regional governments interviewed, who attested to this weakness. 2.44 At present, Peru’s road sector shows a new impetus for planning. Although interviews revealed that Peru’s planning culture remains weak and that its track record in plan compliance is poor, road planning is recognized as important and requires mechanisms for enforcement. As an example, under the ongoing World Bank–financed Support to the Subnational Transport Program Project, Provias is updating the regional road planning methodology to include an emphasis on logistics corridors and better integrate the plans at different government levels. Provias might finance pilots to test the methodology, but generally it will be up to the regions to carry out and finance their plans. 14 2.45 In addition, the MEF’s investment planning system, Inverte.pe, has recently been reformed and requires that all investment proposals follow prioritization criteria, which for roads are linked to logistics corridors. There is also a requirement that provinces or regions submitting road investment proposals for financing have updated participatory road plans. Objective 2 2.46 Objective 2 was to improve, through decentralization at the regional level, the efficiency of regional transport interventions. According to the PAD (World Bank 2005b, 6) this is to be understood as strengthening the institutional framework to achieve the appropriate management of transport interventions at the regional level, with due consideration to environmental and social issues, including issues related to the indigenous peoples. 2.47 The efficacy is rated negligible before and after the project restructuring, which reduced the output indicator and project scope related to this objective. Outputs 2.48 With 46 percent of the allocated resources spent, the following outputs were obtained: • Training of 6,938 local officials in fiduciary aspects, technical issues, environmental and social safeguards, and M&E. Most trainings focused on aspects related to World Bank project implementation and, as confirmed during IEG’s field visits, most regional officials have changed since project completion. Therefore, it is unlikely that the capacity created by the project is still used at the regional level. • Training for 32 microenterprises in technical aspects of road maintenance. Provias stopped collecting information on microenterprises after project closure. In the two regions visited by IEG, none of the community-based microenterprises were still active. • Acquisition of 171 units of equipment to manage regional road assets, such as computer hardware and software. IEG was not able to verify if this material still exists and is used. • Standardized bidding documents for periodic maintenance. These are still used. • Signature, but marginal implementation, by 17 of 24 regions of institutional agreements with Provias that stated the region’s obligation to implement organizational changes and road management improvements envisaged under 15 the Regional Project. These included (i) rationalizing the institutional framework for road transport management, (ii) transitioning from force account to contracting out of road works and maintenance, and (iii) improving the implementation capacity, including fiduciary, safeguards, and M&E aspects (the original indicator target had all regions implementing the agreements). Outcomes 2.49 The project only marginally contributed to achieving an appropriate management of transport interventions at the regional level. The counterpart and regional officials mentioned that (i) the Regional Project did not contribute to institutional strengthening, but focused on works; (ii) the regions had no interest in it; (iii) it was difficult to work with the regions, whose staffs often changed (as described in the Implementation section in this chapter); and (iv) Provias’s consultants or local offices, called zonal divisions, carried out all day-to-day implementation activities, so the project did not strengthen the local capacity. 2.50 The Regional Project had a likely impact on moving toward increased outsourcing of road activities, which has been shown to be more effective than force account.4 At appraisal, the regions carried out most maintenance activities through force account. The counterpart and IDB staff mentioned that the project likely helped make outsourcing more common because (i) the regions contracted out the work and activities under the Regional Project, (ii) Provias provided them with standard bidding documents and procurement training, and (iii) the annual maintenance funds from the MEF needed to be executed through contracts. In one of two regions visited, maintenance and other road works are contracted out. In the other, both outsourcing and force account are used, and the new regional government plans to use more force account for works (not maintenance) given recent negative experiences with contractors. During project implementation, Provias focused on instilling a maintenance culture; since 2011, it has also assisted the regions in obtaining maintenance funds from the MEF. 2.51 The rationalization of the institutional road management framework has not happened. The project did not prepare institutional diagnostics and action plans. According to the counterpart, discussion on the topic took place during project implementation, but nothing advanced. IEG was informed by sector specialists and the counterpart that, in most regions, two entities still remain in charge of road management. The Regional Transport Department oversees (i) road maintenance carried out with funds from the MEF, (ii) transport services and licenses, and (iii) inspections; the Infrastructure Management Division carries out road investments and other activities financed with regional funds. This model is used in the two regions visited by IEG. In these regions, both entities exchange information, but there is limited 16 collaboration and they work rather independently. Regulation in the road sector is shared between the Regional Transport Department and the MTC. The MTC has been opening customer service offices in regional capitals and risks infringing on the regions’ area of competence. 2.52 Safeguards supervision and compliance had shortcomings. Furthermore, given frequent staff rotations and trainings mainly tailored to meet World Bank requirements, the Regional Project succeeded only minimally in developing regional capacity to manage social and environmental issues. This is also the case for project and road sector management in general. 2.53 Provias Departamental and Provias Rural were merged into Provias Descentralizado (or Provias) in 2006, but this change was not affected by the Regional Project. All equipment of Provias Departamental was transferred to the regions and Provias reduced its staff, especially in the zonal divisions. However, Provias has not yet taken steps to become the small, agile, and well-trained regulatory agency for subnational roads in charge of policy, technical assistance, and M&E envisaged at appraisal. National government officials mentioned to IEG that Provias is still implementing road works on behalf of subnational governments, and that with the Pro Región Program, Provias is likely to go backward in terms of decentralization, at least initially. Objective 3 2.54 Objective 3 was to improve, through decentralization at the regional level, the effectiveness of regional transport interventions. According to the PAD (World Bank 2005b, 6), this means upgrading the quality of regional transport infrastructure and developing sustainable maintenance mechanisms to improve regional mobility, which can ultimately foster growth and reduce poverty. 2.55 The efficacy is rated negligible before and substantial after the restructuring, which reduced the ambition of the project related to this objective. Outputs 2.56 The following outputs were obtained: • Feasibility studies and designs for the works implemented under the project. • Rehabilitation of 1,562 kilometers of regional roads prioritized through the participatory planning process compared with the revised target of 1,781 kilometers and original target of 2,200 kilometers. 17 • Periodic maintenance of 3,541 kilometers of regional roads compared with the revised target of 2,202 kilometers and original target of 2,700 kilometers. • Routine maintenance of 2,570 kilometers of regional roads, or 60.9 percent of the revised target of 4,219 kilometers and 64.3 percent of the original target of 4,900 kilometers. • Creation of 110 microenterprises by closing compared with the revised target of 169 and original target of 180 microenterprises. As previously mentioned, Provias stopped collecting information on microenterprises after project closure, and in the two regions visited by IEG no community-based microenterprise was active any longer. Outcomes 2.57 The project had two outcome indicators: (i) the reduction of transport tariffs along upgraded regional roads, and (ii) the percentage of regional road network in good condition. As noted in the Monitoring and Evaluation section in this chapter, the former was not adequate and was not measured. 2.58 By project restructuring in July 2010, the condition of the regional network was the same as at appraisal. By project restructuring, the regions had carried out 70 kilometers of road rehabilitation and 1,713 kilometers of periodic maintenance in a decentralized way. Despite this, the condition of the network was the same as at appraisal, with 15 percent in good condition. This percentage is based on the original network length. If the network length at restructuring is considered—that is, if the nearly 70 percent increase in the network is taken into account—only 9 percent of the regional road network was in good condition (World Bank 2010). 2.59 By project end, the percentage of the regional network in good condition was almost in line with the revised indicator target. At that time, nearly 16 percent of the network, corresponding to 3,796 kilometers, was in good condition compared with the revised target of 17 percent. 2.60 Peru has not collected data on the regional road condition since the Regional Project closed. IEG expects the quality of the regional network to have remained largely the same since project end based on the following findings: • Annual maintenance expenditures increased on average by 10 percent between 2014 and 2018, and, based on a rough calculation by IEG, they are enough to keep the 3,796 kilometers of regional roads dealt with under the project in good condition. However, they are far too low for the network. Average routine maintenance expenditures since 2014 would have been enough for 18 approximately 7,882 kilometers of unpaved roads per year, or 38 percent of the network. Periodic maintenance expenditures, on average, would enable intervention in 10 percent of the unpaved and 3 percent of the paved network annually. With this allocation, it would take 10 years to complete the periodic maintenance on the unpaved network compared with the required five years (see appendix C). • Transfers from the MEF for road maintenance were always available late in the year, well after the rainy season was over, which is when maintenance is most needed. In the two regions visited, there was no allocation yet for maintenance in 2019. • Overall expenditures for regional roads stayed relatively stable during and after the project, although the network length increased and greater emphasis has been placed on paved roads, which are more expensive. On average, road expenditures increased by 6 percent and were approximately $337 million over the period from 2010 to 2018. Expenditures increased significantly in 2014 and 2105, but then went back to nearly pre-2014 levels (see appendix C). The road network increased from 23,740 kilometers by project end to 24,582 kilometers in 2015 (4 percent). At that time, an expansion of 4,981 kilometers was planned, which was to bring the network to 29,563 kilometers.5 • The site visits to two regions provided further contextual evidence. In these regions, the unpaved roads visited were in fair to good condition, which means that they have slightly deteriorated since they were rehabilitated. No routine maintenance has taken place in 2019 because of delays in the allocation of the budget by the MEF. When asked about the network condition, officials in the regional governments interviewed cited an expansion but a lack of maintenance. Some also mentioned that the network condition since 2013 has remained stable or slightly worsened. • The 2017 Peru Public Expenditure Review (World Bank 2017b) found that maintenance of subnational roads is not sufficient to maintain even the recently rehabilitated roads. Subnational governments spend relatively little on maintenance. At the regional level, it was 22 percent of their transport expenditures compared with 26 percent at national level. The Public Expenditure Review noticed that there was an increase in expenditures for regional roads between 2012 and 2015. 2.61 As mentioned under objective 2, the Regional Project helped regions advance in terms of maintenance, but the regions still must find a fully sustainable model. 19 Outsourcing of maintenance is common, but the private sector does not always deliver high-quality maintenance according to the counterpart and sector specialists. In addition, although maintenance funding has increased, it is not enough for the whole network, and budget allocations depend on annual negotiations with the MEF that come late in the year, well after the end of the rainy season. Efficiency 2.62 The efficiency of the project is considered modest. The PAD (World Bank 2005b, 89) specifies that a cost-benefit analysis (CBA) was to be carried out for roads with traffic volumes higher than 40 annual average daily traffic (AADT).6 These roads were expected to produce benefits not only in alleviating isolation but also in reducing transport costs for the long-distance traffic they carry. A cost-efficiency analysis (CEA) was to be used for very low-volume roads, with daily traffic of less than 40 AADT. For these roads, the primary justification came from social considerations, such as providing accessibility to people from isolated communities. 2.63 At appraisal, two CBAs were carried out because most project investments were not yet identified; they were to be selected through the participatory regional road plans. The CBAs used the Roads Economic Decision model for the economic evaluation of low- volume roads. This model applies the consumer surplus approach to estimate project benefits that consist of road user cost savings. 2.64 One CBA was for a program of 51 tentative road sections to be rehabilitated, totaling 2,230 kilometers, with a cost of $113.24 million, traffic ranging from 50 to 317 AADT, and an average percentage of trucks and buses of 40 percent. The other was for the first-year rehabilitation program consisting of eight sections totaling 251 kilometers, with a cost of $11.70 million, traffic ranging from 71 to 188 AADT, and an average percentage of trucks and buses of 31 percent. For the first-year rehabilitation program, the cost efficiency was calculated for all roads, independently of their traffic, by dividing investment costs per kilometer by the beneficiaries per kilometer. 2.65 Sensitivity analyses were conducted. In both cases, the analysis period was 15 years and the discount rate was 14 percent, which was the norm in Peru at that time. The assumptions and cost data used are reasonable. 2.66 The CBA for the program of 51 tentative sections yielded an economic rate of return (ERR) of 26 percent and a net present value (NPV) of $63.4 million. The CBA of the first-year rehabilitation program generated an ERR of 25 percent and an NPV of $5.58 million. The CEA yielded an average investment cost per beneficiary per kilometer of $39.60. 20 2.67 At completion, the CBA used the Roads Economic Decision model and, with two exceptions, followed the methodology of the ex ante evaluation, but with actual data on costs and benefits and updated data on traffic composition and growth. The CBA differed from the ex ante evaluation in two ways: (i) it used the same specifications for all rehabilitation works, reflecting what was actually the case, whereas the ex ante evaluation had predicted multiple rehabilitation alternatives; and (ii) it classified roads into four categories based on the traffic volume instead of the two used in the ex ante analysis, and thus provided more granularity. Both changes are justified. 2.68 According to the ICR (World Bank 2014b, 17), this analysis resulted in an ERR of 19 percent for the whole program. The NPV is not given. The economic analysis report carried out by a consultant, on which the ICR write-up is based, showed an ERR of 16.13 percent and an NPV of $10,230. The difference seems due to different investment costs used. The ICR calculation used $102.36 million, which includes only civil works in line with the ex ante analysis, whereas the consultant used $140.89 million, which also includes supervision. In any case, both assessments show an ERR well below the ex ante ERRs of 25 percent and 26 percent due to road construction cost increases. 2.69 The ICR (World Bank 2014b, 37) also includes a CEA for the roads with very low traffic (below 50 AADT) by calculating the investment costs per beneficiary per kilometer. It reports a cost per beneficiary per kilometer of $148, which is considerably higher than the ex ante estimate of $39.60 and exceeds the cost per beneficiary of $100, which was the rule of thumb in Peru for the feasibility of a road investment. This analysis contained an error, and the recalculation by IEG showed a cost per beneficiary per kilometer of $78.63. 2.70 The project had significant administrative and operational inefficiencies that contributed to major implementation delays. Project effectiveness took place nine months after Board approval. The implementation period nearly doubled, and only $41.60 million of the $50.00 million World Bank loan was disbursed by loan closing. Nevertheless, the revised physical targets were complied with, whereas the loan fell short in terms of institutional strengthening. 2.71 IEG also found that several roads rehabilitated under the project were paved shortly after the project’s completion without changing the structure or alignment, which created smoother-surfaced, faster, but less safe roads. This significantly increased the overall investment cost, even if low-cost pavements were used. Such cost can be outweighed by user cost savings and other benefits such as dust reduction. However, the road needs to have a reasonable number of users. As a yardstick, paving is considered economically justified when the traffic is about 400 AADT, but other considerations also come into play. The low-cost payment pilots carried out under this 21 project showed that paving might be reasonable when the road reaches approximately 200 AADT. By project end, the average traffic on the rehabilitated roads was 151 AADT. Outcome 2.72 The overall outcome of the Regional Project is rated moderately unsatisfactory, as shown in table 2.1. Table 2.1. Regional Project Ratings Rating Dimension Ratings without Restructuring Ratings after Restructuring Relevance of objective High High Relevance of design Modest Modest Overall efficacy Negligible Modest Objective 1 Modest Modest Objective 2 Negligible Negligible Objective 3 Negligible Substantial Efficiency Modest Modest Outcome rating Unsatisfactory Moderately unsatisfactory Outcome rating value 2 3 Amount disbursed ($, millions) 9.26 157.81 Disbursement (percent) 0.06 0.94 Weight value 0.12 2.82 Total weight 2.94 (rounds up to 3) Overall outcome rating Moderately unsatisfactory Risk to Development Outcome 2.73 The risk to development outcome is rated as substantial mainly because of the limited achievement in road sector management and the lack of budget for the road sector. Institutional Risks 2.74 Although prioritization of road investments improved only at the project level, there is a new emphasis on planning and a recognition that it is important. Provias is updating the road planning methodology, and the regions are responsible for implementing and financing their plans. MEF requires that road investment proposals put forward for financing are based on clear prioritization criteria and that the proposing subnational government has an updated road plan. 2.75 IEG did not carry out a capacity assessment, but it is expected that the regions, having been in existence for nearly 20 years, have improved their capacity to manage the 22 road sector. Based on visits to two regions, it does not appear there is much will to rationalize the regional road sector institutional arrangements, and duplication of efforts and limited coordination and communication still exist. In terms of safeguards, the bidding documents used by the regions include environmental and social requirements in line with Peruvian law, but IEG was not able to collect information on safeguard supervision and compliance. Infrastructure Maintenance Risks 2.76 The budget allocations for regional road maintenance increased on average by 27 percent between 2013 and 2018, even if the commodity price boom and windfall of mining royalties had ended. However, the current allocation is only enough to maintain less than half of the network. No binding legal framework for budgetary transfers to regions exists. As a result, the regions depend on budget negotiations, with no assurance of outcomes year by year. 2.77 The Pro Región Program, which has just started, is expected to pave and maintain 15,000 kilometers of mostly regional roads under five-year performance-based contracts by 2024. This is more than half of the current regional network. This program could significantly increase the quality of the network; if carried out with strong regional involvement and adequate capacity strengthening, it could also highly improve the regions’ road management capacity. However, the targets are ambitious, and there is not yet clarity on the funding for the whole program. In addition, the level of regional involvement in the implementation of the program is uncertain. Bank Performance 2.78 Bank performance is rated moderately unsatisfactory. Quality at Entry 2.79 Bank performance at entry is rated moderately unsatisfactory. The Regional Project built on the successful experience of the two previous rural roads projects in Peru. It aimed at transferring significant parts of this experience to the regional context. This aim was relevant considering the condition of the regional road network, and it was in line with the country’s decentralization efforts. 2.80 However, no discussion on the suitability of the rural roads solutions in the regional context had taken place. Such a discussion would have been justified because regional roads generally carry more traffic than rural roads and hence might require different interventions. The discussion could have avoided the lack of choices and flexibility in the types of road works,7 construction budgets, and private sector solutions for maintenance,8 which later complicated project implementation. 23 2.81 A key shortcoming in project design and preparation was the lack of adequate involvement of the regions, which were represented only through the National Decentralization Council. With the benefit of hindsight, regions should have actively participated in design decisions, considering that they were expected to strengthen their road management capacity and implement important institutional changes, both essential for the achievement of the project objectives. They were also expected to own the project outcomes. 2.82 The project design did not sufficiently consider that the political power and influence of the regions was stronger than that of provincial and district governments. Although the PAD (World Bank 2005b, 180) points out that “the relation between the national government and the regions was often charged by political pressure,” the regions’ lack of buy-in and commitment to this project was not anticipated; this was what eventually caused the project to only partially achieve its outcome. 2.83 The choices related to implementation arrangements contributed to the significant project delays. The rural roads projects were initially fully centrally implemented, and the national government transferred certain responsibilities to local governments only later. This gradual decentralized implementation proved successful and was identified as a success factor for decentralization in one of the relevant lessons from previous projects (World Bank 2005b, 10). 2.84 The performance related to risk identification and mitigation was mixed. Although most risks that could have been foreseen by project preparation were correctly identified, including a substantial risk related to the insufficient regional capacity to assume road management responsibilities, the risk mitigation measures were generally weak. 2.85 The fiduciary and safeguards considerations were considered adequate at appraisal, but as seen in paragraph 2.23, the safeguards supervision arrangements had shortcomings. The M&E design was weak (as discussed in the Monitoring and Evaluation section in this chapter). The financial and economic analysis was thorough and sound. Quality of Supervision 2.86 Bank performance at supervision is rated moderately satisfactory. This was a complex project to implement given the rigidities in the design and limited interest of the regions. Consequently, it had many delays and problems that required significant attention from the World Bank team, who also played an important role in coordinating alternative financing for road works and who contributed to the dialogue and pilot on road paving. 24 2.87 Interviews with the counterpart confirmed that the World Bank was readily available to provide support and that its focus on outcomes and timelines, and its flexibility in agreeing to extensions and project changes, were appreciated. In its completion report, the borrower recognized the World Bank team’s contribution in coming up with proposals to solve problems and implementation issues. 2.88 The aide-mémoire of the World Bank supervision missions indicate that the focus of the World Bank implementation support was more on civil works than on the institutional aspects of the project. The World Bank supervision team regularly insisted on the institutional dimension, which was critical for the achievement of the project objectives. However, the aide-mémoire show hardly any discussion on rationalizing or improving the institutional framework for transport management at regional level. It is true that the regions had a limited interest in the project and that Provias was reluctant to carry out the institutional activities under it; several planned activities were not launched or were launched close to project end. However, more could have been done to move ahead with a few interested regions or change the project scope. Anecdotal evidence suggests that the indicator to measure the progress in institutional terms reported on the number of institutional agreements signed between Provias and the regions and not on the implementation of reforms detailed in these agreements, as envisaged. 2.89 Given the importance of the institutional aspects and the obvious lack of interest in these aspects by at least certain regions, the World Bank team could also have been more proactive in restructuring the project again to focus on fewer, poorer regions, or on the activities that were key to project success and sustainability. The World Bank team also did not cancel part of the loan when it became clear that it could not be fully disbursed. 2.90 Finally, although the World Bank team paid due attention to the fiduciary aspects, safeguard supervision should have been stronger given the borrower’s weaknesses (according to the aide-mémoire, an environment specialist did not participate in all supervision missions, and field visits were limited). Borrower Performance 2.91 The borrower performance is rated moderately unsatisfactory. Government Performance 2.92 The government performance is rated moderately unsatisfactory. The national government was interested in finding new approaches to transport management and set up a multisector commission, including the MEF, MTC, National Decentralization 25 Council (which also represented the regions), and Provias. This multisector commission played an important role in project preparation (World Bank 2005b, 5, 11). However, the regions, which were the main beneficiaries and implementers of this project, were absent in the discussions about project design. 2.93 The national government, through the MEF, was actively involved in project implementation and supervision, but initially it was reluctant to hire additional consultants. These consultants were to support Provias to make up for the lack of technical staff and the unfamiliarity with World Bank rules and procedures at the regional level. 2.94 When it became obvious that the project design might not be most appropriate, and that interest from the regions was limited, the national government was not proactive in proposing changes to the project. Moreover, due to national elections, management and staff at Provias changed, which also slowed down implementation. 2.95 Finally, the national government has not yet started the process to transform Provias into a smaller and more agile regulatory body for the subnational road sector as expected at appraisal, and the decentralization progress in the road sector has largely stalled. 2.96 Although certain regions might have been committed to the civil works, IEG’s key informant interviews with the counterpart and regional officials showed that the regions were not interested in this project and the available technical solutions. The regions heavily relied on Provias for project implementation. Some of the regions also paved the rehabilitated roads shortly after project completion, thus wasting resources. Implementing Agency Performance 2.97 The performance of the implementation agency is rated moderately satisfactory. Provias already had experience in implementing World Bank projects through the previous rural roads projects. Provias played a stronger role in project implementation than originally envisaged since it and its consultants carried out a significant part of the implementation activities for the regions. To accelerate project implementation, for instance, Provias recentralized the implementation of design studies. In addition, Provias consultants played a critical role in the day-to-day project management at the regional level. 2.98 Provias spent a lot of time and effort in convincing regions to participate. Provias also carried out frequent trainings for regional officials, who kept on changing as staff turned over. However, these trainings were mostly operational. Although the regions had limited interest in the project, Provias could have been more proactive and 26 could have provided capacity strengthening in road management, including technical aspects, at least to the most committed regions. 2.99 Fiduciary compliance was largely adequate, and the role of Provias was essential. Provias was particularly proactive in helping regions expedite procurement processes and attract more competition. However, as indicated in the Safeguards Compliance section in this chapter, the safeguards supervision had shortcomings. 2.100 Provias constantly monitored project implementation and provided detailed project progress reports in a timely fashion, but the reporting on the institutional indicators was weak. Monitoring and Evaluation 2.101 M&E is rated modest. Design 2.102 M&E was designed to be implemented at two levels: (i) the continuous review of project performance and annual plans by Provias, and (ii) periodic performance audits, participatory evaluation exercises involving the regions, and impact assessment studies carried out by independent firms and specialized nongovernmental organizations. These activities were to be financed from the resources allocated to component 4. 2.103 The results framework had two outcome indicators (as described in the Objective 3 section in this chapter). These were (i) the reduction of transport tariffs along upgraded regional roads, and (ii) the percentage of regional road network in good condition. The former was not defined, had no methodology to collect information, and was missing a baseline. It was not easily measurable because of the wide geographical dispersion of the project interventions. It was also not especially meaningful because transport tariffs depend not only on good roads, but also (and mostly) on demand and supply. The second indicator was readily measurable and reliable. 2.104 Although improved planning and institutional strengthening were part of the PDO statement, these aspects were to be measured through output indicators. For the planning aspect, the indicator was designed to report on the number of participatory regional road plans approved; for the institutional dimension, the indicator measured the percentage of institutional agreements successfully implemented. For the latter, there was no description of what successful implementation of the institutional agreements meant. 27 Implementation 2.105 Provias carried out regular project progress monitoring and reporting, including on the progress on most indicators, and adequately covered technical, fiduciary, and safeguards issues. For the indicator on successfully implemented institutional agreements, however, Provias simply reported on the number of agreements signed. For the indicator on microenterprise creation, several years of data were missing. Correctly, no data were collected on transport tariffs. 2.106 The participatory evaluation exercises and the impact assessment studies by independent firms and specialized nongovernmental organizations did not materialize, except for six case studies by project end to assess how communities along the road benefited from the project. Use 2.107 The M&E information was used for project purposes. For instance, the information on project progress and the achievement of the indicator targets during the midterm review was essential to prepare the project restructuring. The information on project progress was also important to decide on loan extensions and the reallocation of loan proceeds. After project completion, even information such as the condition of the regional road network and the creation and operation of community-based microenterprises was not collected any longer. 3. Decentralized Rural Transport Project Cost, Financing, and Key Dates 3.1 The total estimated project cost at appraisal, including contingencies, was $150 million. The actual cost at completion was $160 million, 6.77 percent more than the appraisal estimate. This cost overrun was caused by the increases in road construction costs in Peru during project implementation. 3.2 The financing came from an IBRD loan in the amount of $49.95 million and from parallel IDB cofinancing in the amount of $50 million. The original commitment of each banks was $50 million. The borrower’s counterpart funding was $60.05 million compared with the appraisal estimate of $50 million. 3.3 The project was approved on December 19, 2006, became effective on July 12, 2007, and closed on December 31, 2013. The loan closing date was extended twice. The first extension, from March 31, 2012, to March 31, 2013, was due to implementation delays caused by the regional and national elections in 2010 and 2011. The second 28 extension of nine months to December 31, 2013, was granted to complete the institutional activities and the civil works for periodic maintenance and NMT tracks. The civil works were delayed by an unusually heavy rainy season and the lack of archeological monitoring plans in bidding documents for touristic NMT tracks. Together with this loan closing date extension, a reallocation of loan proceeds was processed. 3.4 Through a level-two restructuring that became effective on July 23, 2010, funds were reallocated among categories and the performance indicator targets were revised. Although the targets for periodic maintenance, NMT tracks, and bridge improvements were revised downward, the targets for rehabilitation were increased despite cost overruns. Rehabilitation emerged as a government priority in the wake of the global economic crisis. The restructuring did not substantially change the project’s ambitions, and no split rating is warranted. 3.5 A final restructuring took place on December 30, 2013, to reallocate funds. Relevance of the Objectives and Design Objectives 3.6 The objective of the project, as stated in the loan agreement and the PAD (World Bank 2006b, para. 24), was “to contribute to territorial development and to the fight against rural poverty in the borrower’s territory by improving access of rural households and entrepreneurs to goods, social services, and income-generating opportunities through reduced transport costs and better rural transport infrastructure.” Relevance of the Objectives 3.7 The relevance of the PDO was and remains high. At appraisal, 74 percent of the rural population was poor, and rural poverty continues to be an important challenge. In addition, rural roads were—and many still are—in very poor condition. 3.8 The PDO’s relevance is shown through its alignment with the CAS and CPSs. The FY03–06 CAS (World Bank 2002) highlights the need for competitiveness, economic management, and private sector development by integrating poor rural areas with transport networks. It also aimed at providing employment opportunities in rural areas. The PDO remained in line with the first pillar of the FY07–11 CPS (World Bank 2006a), which envisaged (i) extending infrastructure to rural areas with the highest concentration of poverty, (ii) meeting the basic needs of the poor, and (iii) decentralizing effectively to deliver services locally. The PDO is in line with the FY12–16 CPS (World Bank 2012), which centered on (i) connecting the poor to services and markets through 29 better transport, and (ii) improving public sector performance through greater inclusion of subnational systems. 3.9 Rural poverty reduction through rural road development was and is relevant to the government's strategy on decentralization and the transfer of public responsibilities to local governments. The 2006–11 Government's Development Plan prioritized (i) the fight against poverty as part of its human development strategy, and (ii) the development of rural areas by expanding infrastructure services and continuing the decentralization reforms. The 2012 National Strategy for Development and Social Inclusion focused on (i) intersectoral and intergovernmental coordination to help households in poor areas increase the capacity to generate incomes, and (ii) infrastructure, including rural roads, to connect these people to markets. The General Government Policy to 2021 calls for effective decentralization for development through (i) institutionalization of territorial coordination in national policies, (ii) improvement of decentralized and sustainable public and private investments, and (iii) reduction of rural and urban poverty. Finally, the new Policy on Development and Social Inclusion to 2030 also emphasizes the need to provide poor and vulnerable people with basic infrastructure, including roads. 3.10 The PDO was realistic since the literature (described in chapter 1) shows that rural road improvements enhance access to markets and social services and this, in turn, is expected to contribute to economic development and poverty reduction. Nevertheless, the impacts on territorial development and poverty reduction are difficult to attribute and measure. Design 3.11 The project had four components as shown in box 3.1. 30 Box 3.1. Rural Project Components Component 1: Improvement of rural transport infrastructure (estimated cost $99.6 million; actual cost $129.8 million). This component financed (i) the rehabilitation of 3,277 kilometers of rural roads prioritized through the participatory provincial road plans instead of 3,000 kilometers as planned, (ii) the implementation of one bridge compared with a target of 50, (iii) periodic maintenance on 7,806 kilometers of rural roads instead of 11,200 kilometers as planned, (iv) the improvement of 2,356 kilometers of nonmotorized transport tracks compared with a target of 2,650 kilometers, and (v) pilot studies on slope stabilization. No other transport infrastructure was improved. The cost overrun of $30 million under this component was due to road construction cost increases during project implementation. Component 2: Institutional development (estimated cost $14.7 million; actual cost $12.2 million). This component was expected to finance institutional strengthening for the Ministry of Transport and Communications (MTC) and provincial and district municipalities. Although the support to the MTC did not materialize as planned (and this had no significant impact on project outcomes), this component supported local governments to (i) prepare or update participatory provincial road plans; (ii) improve the routine maintenance system with microenterprises; (iii) enhance local capacity to handle social, cultural, and environmental safeguards; (iv) scale up the geographic information system; and (v) build capacity for municipalities, provincial road institutes, and provincial infrastructure institutes (see chapter 1). Component 3: Transport for territorial development (estimated cost $11.1 million; actual cost $2.0 million). This component aimed at enhancing the impact of improved transport conditions on rural development by fostering complementarities with other investments and promoting productive activities. It scaled up the local development window (described in chapter 1) to accelerate the emergence of productive activities in the areas where transport conditions improved (estimated cost $1.3 million, actual cost $1.2 million). It also experimented with the rural infrastructure pilot in 15 provinces (estimated cost $9.8 million, actual cost $800,000). Although the rural infrastructure plans were prepared, and works on several rural roads prioritized under the plans were implemented, the PIIs (described in chapter 1) were not established. During the life of the project, no other sector implemented the complementary rural investments identified in the plans. Component 4: Project management (estimated cost $17.4 million; actual cost $16.0 million). This component supported the day-to-day management of the project through (i) technical advisory services for monitoring, evaluation, and audits, including the (i) updating or expansion of Provias’s financial management system SIGAT and related training, (ii) midterm and final impact evaluation studies, and (iii) technical, operational, environmental, and social auditing activities; and (ii) administrative and operational support. Relevance of the Design 3.12 This assessment rates the relevance of design as substantial for the following reasons: 3.13 Territorial development and poverty reduction are complex issues, which require a multidisciplinary approach. Although this predominantly transport-related 31 project was designed to contribute positively to these issues, it was not expected to solve them. Through an innovative design, which was the result of fine-tuning activities tried out in previous projects and adding more sophistication over time, the project tried to enhance the contribution of better rural transport to territorial development and poverty alleviation. 3.14 The links between the project activities, their outputs, and the intended outcomes are logical and clear but not explicitly laid out in the PAD. The civil works, together with the project’s institutional and capacity strengthening activities in the road sector, were expected to provide better rural transport infrastructure. Improving the right roads under the provincial road and infrastructure plans was expected to reduce transport times and costs. Rural households and entrepreneurs would then have better access to goods and social services, and eventually contribute to increased economic activities and the development of a territory. The latter, in turn, can be expected to reduce poverty. The effect of improved rural transport infrastructure on territorial development and poverty reduction was to be enhanced by the promotion of productive activities on a relatively small scale through the LDW and the provision of other rural infrastructure on a pilot basis. 3.15 However, measurability was a challenge because the project had no outcome indicators capable of measuring the higher-level objective of territorial development and poverty reduction. Although the results framework lacked these causal connections, the project commissioned an impact assessment, as in the previous rural roads projects, to assess attributable outcomes of the project. 3.16 For the rural infrastructure pilot, the activities were insufficient and implementation arrangements inadequate. Although the literature encourages the integrated implementation of infrastructure to enhance sustainability (United Nations 2019), the complexity of implementing multisector infrastructure was underestimated. Nevertheless, this was a laudable attempt by a road project to innovate on an experimental basis. Implementation 3.17 The responsibility for project implementation and coordination was with Provias. The PRIs were to slowly take over project implementation and fiduciary activities, including planning, feasibility studies, routine maintenance, and—depending on their capacity—small works. The zonal divisions of Provias were to provide technical assistance to the PRIs and ensure monitoring in the field. Special implementation arrangements were envisaged for the rural infrastructure pilot, for which participating provinces had to set up a technical secretariat to prepare the rural infrastructure plan. 32 The different infrastructure sector agencies at the national level signed a memorandum of understanding to implement the plan and formed a committee of executive directors. It was envisaged that after the 2006 national election, a commission with sector representatives would be set up at vice minister level. 3.18 Provias had gained considerable experience in project implementation in the previous rural roads projects. Therefore, apart from some delays, project implementation was relatively smooth, but the following implementation issues are worth highlighting: • Delays in project implementation. As in the Regional Project, the ongoing restructuring process in Provias delayed the initial project implementation. This caused civil works and institutional activities to lag. During the first year of implementation, the rural infrastructure plans under the rural infrastructure pilot, the monitoring of the PRIs, the LDW activities, the gender action plan, and the contracting of the impact assessment were delayed. By midterm in 2010, progress was achieved for several of these activities, but delays in the rural infrastructure pilot, PRI monitoring, and the gender action plan continued throughout 2011 and 2012. Periodic maintenance also moved slower than expected because some part of it was decentralized. • Insufficient routine maintenance. By midterm, only a third of the rehabilitated roads had regular routine maintenance because the Rural Project did not finance it. The problem was eventually resolved when Provias agreed with the MEF that the MEF would transfer funds to local authorities to finance two-thirds of routine maintenance. The remaining part had to come from the local authorities. Provias also intensified its support for the creation and strengthening of community- based maintenance microenterprises. • Shift of focus under the project from periodic maintenance to rehabilitation. The Peruvian government requested prioritizing of rehabilitation works to stimulate the economy after the 2008 global recession. This left roads that had been rehabilitated a certain number of years previously without the periodic maintenance needed to maintain their expected life spans. It also increased the need for routine road maintenance because the newly rehabilitated roads had to be maintained. • High staff turnover due to elections. In the local election, 83 percent of local mayors were changed, 47 percent of PRI managers were replaced, and 24 percent of PRIs had no manager for at least two years. The new managers often were political appointees with little experience in rural roads, causing a significant 33 capacity loss for PRIs. This delayed the transfer of responsibilities to the PRIs and required a loan closing date extension. Safeguards Compliance 3.19 The project was classified as category B for environmental assessment purposes. As in the Regional Project, the following safeguard policies were triggered: Environmental Assessment (OP4.01), Cultural Property (OP4.11), Involuntary Resettlement (OP4.12), and Indigenous Peoples (OP4.10). 3.20 An environmental and social assessment report, an environmental and social management framework, a resettlement policy framework, an indigenous peoples planning framework, and a plan to strengthen social and environmental management were prepared. With these, the project was considered in safeguard compliance at appraisal (World Bank 2006b, 32). The necessary documents and processes to manage archeological findings for touristic NMT tracks were not in place at appraisal and delayed implementation later. 3.21 The Rural Project did not have major social and environmental impacts because it intervened only on the existing rights of way. The involuntary resettlement and indigenous peoples planning frameworks did not have to be used. 3.22 The shortcomings in the safeguard arrangements and supervision were similar to those in the Regional Project (see the Safeguards Compliance section in chapter 2). The PRIs did not have social and environmental specialists, and the recommendations of the plan to strengthen social and environmental management were gradually but not fully implemented by the provinces due to low capacity. 3.23 The safeguards performance was rated satisfactory at project closure. Financial Management and Procurement 3.24 Provias had a strong team in charge of financial management, which made up for the weaknesses at the provincial level. There were no significant financial management issues, and Provias’s performance was satisfactory throughout project implementation. Provias had adequate internal controls and used an information module to track transfers to and execution by the provinces. Financial management reporting was timely, and financial audit reports had unqualified opinions. 3.25 Most procurement activities were carried out by Provias, which had an experienced procurement team. The provinces had limited capacity, and there were many staff rotations. However, the provinces procured only small maintenance contracts, and all provinces received regular support and training from Provias. 34 3.26 No major procurement issues occurred. The World Bank carried out six ex post procurement reviews and an independent procurement review. Provias’s procurement management was highly praised in the independent procurement review. Provias largely complied with the recommendations of the reviews. The aide-mémoire and progress reports included adequate information on procurement. Achievement of the Objective 3.27 The objective of the Rural Project was to contribute to territorial development and the fight against rural poverty. 3.28 The efficacy of this objective is rated substantial. Outputs 3.29 The Rural Project obtained the following outputs: • For rural roads prioritized through the participatory planning process, 3,277 kilometers were rehabilitated compared with the revised and original targets of 3,358 and 3,000 kilometers, respectively; and 7,806 kilometers of rural roads received periodic maintenance compared with the revised and original targets of 7,506 and 11,200 kilometers, respectively. Three of four rural roads visited by IEG were in fair to good condition. One road was in bad condition. It is necessary to point out that the visit took place toward the end of the rainy season and no maintenance had yet taken place in 2019. • For NMT tracks, 2,356 kilometers were rehabilitated compared with the revised and original targets of 2,515 and 2,650 kilometers, respectively. The two NMT tracks visited by IEG had lost their importance because the villages previously accessed by these tracks were now reachable by road. However, IEG was informed by sector specialists that the condition of NMT tracks is different in different parts of Peru and that in many places people still depend on these tracks. One of the tracks visited by the IEG mission was in fair to good condition; IEG could not ascertain the condition of the other due to rain. People living near the NMT tracks reported to IEG that maintenance was carried out once a year by the municipality or the community. Sector specialists told IEG that “nobody cares anymore about NMT tracks,” and the ongoing Support to the Subnational Transport Program Project does not include NMT improvements. • The project rehabilitated one bridge in line with the revised target, but this was significantly less than the original target of 50. The counterpart mentioned that bridges were included in the project without knowing where they were needed. 35 In the absence of a reliable bridge inventory, Provias carried out a bridge condition study; when it finished, it was too late to implement the bridge works. Shortly before project closure, the national government acquired 1,000 modular bridges, and several were installed on the rural network. • Pilot studies looked at bioengineered solutions to stabilize slopes along the Llapay–Laraos–Lima road and tested chemical stabilization additives in Madre de Dios. Provias did not implement the recommendations or perform any further activities on slope management even though IEG observed that slope stabilization is critical for many of the rural roads that were visited. • In 2008, through the active involvement of Provias under the Rural Project, the MEF began transferring funds for rural road maintenance to local governments. The ICR (World Bank 2014a, 15) indicates that the amount transferred in 2013 was $250 million compared with the target of $75 million, and that this amount covered 90 percent of the road maintenance needs. The data provided to IEG during the mission showed that the MEF transfers for maintenance in 2013 were approximately $29 million; transfers increased over the years and reached $77 million in 2019 (between 2010 and 2019, the transfers were $51 million a year on average). • Using the same rough calculation as for the Regional Project (see appendix C), these transfers from 2013 to 2018 would be enough to maintain annually about 55 percent of the core network improved since 1995 under World Bank projects. • The overall budget spent for rural roads between 2010 and 2018 was $570 million a year on average. According to the 2017 Public Expenditure Review for Peru (World Bank 2017b), maintenance expenditures for rural roads were far too low, local authorities spent only 14 percent of their budget on maintenance, and only 12 percent of the overall rural roads network of approximately 100,000 to 120,000 kilometers was in good condition. The amount derived using the same rough calculation—14 percent of the overall annual rural roads expenditures spent on annual maintenance—would be enough to maintain 85 percent of the core network improved since 1995 (see appendix C). • For providing routine maintenance, 325 microenterprises were created compared with the original and revised targets of 120 and 324, respectively. These community-based microenterprises have nearly disappeared due to lack of technical assistance and capacity strengthening and have been replaced by small companies.9 IEG was informed by sector specialists and the counterpart that the quality of the work of the small companies is generally worse than that provided 36 by community-based microenterprises. Companies received less oversight by PRIs, and Provias does not contract road monitors anymore. These were young civil engineering graduates hired by Provias to support the community-based microenterprises and supervise their activities. • The microenterprises created 3,250 one-year equivalent permanent jobs compared with the revised and original targets of 1,560 and 1,400 jobs, respectively. IEG was not able to get updated information on job creation by microenterprises. • Under the project, 188 PRIs were established, covering nearly all provinces compared with the target of 150. Reportedly, by project end, the capacity of PRIs was strengthened beyond what had been expected at appraisal, and all PRIs had become involved in road investment planning, studies, routine maintenance, and supervision activities. The idea was for PRIs to eventually take over all road management activities in a province. • IEG learned from sector specialists and the counterpart that most PRIs are still active but have different levels of performance. The two PRIs visited were responsible for routine maintenance. The PRIs had a manager, operational staff, and a secretary. The municipalities covered staffing and operating costs and transferred the MEF resources for routine maintenance to the PRIs. By the end of March 2019, they had begun the biddings for routine maintenance. Some interviewees still referred to the PRI as an entity of the national government because PRIs were created with the support of Provias. Based on assessment findings, it seems unlikely that local governments will transfer the responsibility for all road management activities to the PRIs. • A total of 188 participatory provincial road plans were prepared and approved, which is significantly more than the target of 150. The planning procedure and methodology were nearly identical to the ones for regional road plans presented under Objective 1 in chapter 2, and they had similar merits and shortcomings (discussed in the same place). IEG’s review of the Contumazá Province plan and the field visits showed that the prioritized roads connected larger villages to bigger centers and were economically important. The review also showed that the plan included all the roads that the project intervened in. • The ICR (World Bank 2014a, 34) mentioned that a geographic information system was provided to 164 PRIs to develop road network inventories, but the two PRIs visited were not aware of this. 37 • LDW activities were carried out in 28 provinces. As a result, many local governments set up Economic Development Divisions, prepared local economic development plans (30 plans compared with a target of 40), passed favorable regulations, and supported rural producers and associations. The counterpart mentioned that the Economic Development Divisions still exist generally, but they have not continued with the activities supported by the LDW. • Under the LDW support to local producers, the project identified and assessed 210 productive activities compared with the target of 200. The LDW financed 52 business plans for rural producers' associations, which were used to request funding for activities identified in the plans. A total of 46 of the identified activities received financing from public or private entities compared with the target of 50. In the two regions visited, most activities were still ongoing. • As part of the rural infrastructure pilot, 15 multisectoral rural infrastructure plans, including roads, energy, water, telecommunications, and irrigation, were prepared and approved by the municipalities, which is in line with the target. Despite Provias’s notable efforts, the plans were not implemented because the necessary cross-sector cooperation did not materialize; but Provias rehabilitated 19 prioritized road sections with local funds. Due to the lack of cross-sector cooperation, the 15 planned PIIs were also not set up. • The ICR (World Bank 2014a, 35) points out that the basic underlying principle of the rural infrastructure pilot—tackling rural development through cross-sector cooperation and bundling of infrastructure—was adopted by the Ministry of Development and Social Inclusion’s Fund for the Inclusion of Rural Areas (Fondo para la Inclusión Económica de Zonas Rurales, FONIE). FONIE’s function was to promote rural development through integrated infrastructure investments in the poorest areas. FONIE financed several infrastructure projects, but the expected integrated provision did not happen because FONIE had no mechanism to ensure the necessary cooperation. FONIE was dissolved in the beginning of 2019. Outcomes 3.30 This project is one of the few road projects in the World Bank that had a comprehensive impact assessment using a difference-in-differences approach (Macroconsult-Cuanto 2014). The impact assessment carried out in 2013 looked at the impact of rural road rehabilitation and maintenance and NMT track improvements, but it did not cover the other project activities. Previous rural roads projects that received assessment had similar assessments. 38 3.31 For rural roads rehabilitation and maintenance, the 2013 impact assessment showed a positive and statistically robust impact on decreasing extreme poverty and increasing consumption. It also found a statistically significant positive impact on structural poverty, measured through the degree of “unmet basic needs,” but no change in total poverty. This could mean that the poorest people benefited the most from the project. No statistically relevant impact was seen on income, which was to be expected given the increase in consumption. 3.32 The impact assessment also showed that the project contributed to territorial development. The cultivated land surface increased and the fallow surface decreased in a statistically significant manner. This change most likely reflects increased agriculture productivity. The increase in livestock production also had a robust statistical significance. 3.33 For the more direct impacts of road improvements, the assessment found statistically significant travel time reductions to points of sale, farms, and schools, but travel times to markets went up. No statistically relevant reduction in travel time to health centers was observed. Microbus fares decreased, but freight tariffs increased. The increase in transport services by car was statistically relevant. Statistically relevant increases were also observed for school attendance of students between 12 and 18 years and for medical consultations, which could be the result of better transport conditions. 3.34 For NMT tracks, the impact assessment did not find statistically relevant impacts on poverty, income, or expenditures. It also did not find positive changes in land use patterns. However, it observed statistically relevant increases in the number of public and private institutions in villages and the percentage of health centers, radio stations, and television channels. These could indicate an increase in economic and social activities. 3.35 As for rural roads, there were statistically relevant reductions in travel times to certain destinations, such as to work or district capitals. However, no statistically relevant reductions in travel time to markets, schools, or health centers were found. Despite this, the number of health consultations increased. No statistically relevant increase in school attendance was seen. A summary of the full results of the 2013 impact assessment is included in appendix D. 3.36 The impact assessment was challenged by a significant level of attrition and the contamination of the control group. To increase the number of available observations, the impact assessment used the data from the Rural Project and from the 2006 impact assessment of the second Rural Roads Rehabilitation and Maintenance Project. With this 39 combination of data sets, the impacts found are attributable to both projects, and thus it is not possible to isolate the effects of the Rural Project. 3.37 In addition, the small and unbalanced samples used call into question the reliability of the impacts, such as poverty reduction, which are far down the direct causal pathway of road improvements and therefore more sensitive and difficult to measure (see appendix D). 3.38 Although the impact assessment has shortcomings, it is plausible that the project contributed to the significant poverty reduction and territorial development effects in rural Peru between 2007 and 2017 (as described in chapter 1). This is due to the fact that the road investments (i) reduced travel times, (ii) decreased travel costs, (iii) increased household access to goods and services, (iv) increased access to points of sale, and (v) contributed to the ability to increase agricultural activities. Road investments had these four effects even if these results are not uniquely attributable to the project. 3.39 The findings are in line with the literature, particularly a recent study on connectivity and rural economic development in Peru (Webb 2013), which included a literature review, field visits, interviews in rural areas, an econometric analysis, and an estimation of the historical evolution of Peru’s rural economy. Although none of the findings in this study directly proved that rural connectivity was responsible for the recent economic growth in rural Peru, all pointed in this same direction. 3.40 The conclusion that the Rural Project contributed to territorial development and poverty reduction is supported by further explanatory factors, gathered by IEG from observations and interviews with project beneficiaries and other stakeholders, which provide context about this contribution (see box 3.2). 3.41 Finally, it is worth mentioning that the Rural Project was a champion in terms of gender issues in roads projects. The achievements under the project’s gender action plan, among others, included (i) creating awareness within transport and other government institutions about the importance of gender equity in all rural roads management activities, (ii) opening employment opportunities for women, (iii) providing training to local governments, PRIs, and the LDW operators on gender equity, and (iv) monitoring gender indicators. 40 Box 3.2. Observations from IEG’s Field Visits to Ayacucho and Cajamarca IEG’s field visit to two of the poorest regions in Peru, where the project had invested in rural roads, provided some observational context. In the provinces of Cangallo and Contumazá, for instance, population centers with several thousand inhabitants had in the shops a wide variety of goods secured from other regions. The markets were bustling, and traffic was relatively heavy on the main roads, with loaded trucks and new vehicles. These population centers also had several banks. Although the field visits were not used to gather quantifiable evidence, they provided some context to better understand the nature and needs of Peru’s rural market towns and villages from a transport perspective. Many persons interviewed by the Independent Evaluation Group, chosen because they reside or their production activities are located along the improved roads, pointed out that the improvements facilitated the transport of their products to market. There was a general consensus among residents interviewed in the two regions that accessibility and motorized travel in Peru had significantly increased since mid-2000, and that this has led to reduced travel time and costs, improved market access, an increased number of teachers in schools, and improved access to health services. For example, IEG observed ambulance services in a town visited, which was in an isolated and remote mountain area. In the visited areas, IEG also confirmed that transport services were available on a regular basis, although some interviewees complained about high transport costs. The IEG mission also met with members of two producer organizations supported by the local development window. Several farmers credited the local development window for helping them increase the value of their livestock and milk production. Efficiency 3.42 The efficiency of the project is considered substantial. 3.43 At appraisal, a CBA using the Roads Economic Decision model was carried out for a representative sample of roads with traffic volumes higher than 15 AADT and NMT tracks to be rehabilitated. The total investment cost was $27.1 million, which corresponded to 18 percent of the overall project cost. As in the Regional Project, the CBA used a discount rate of 14 percent, but an analysis period of only 10 years. The methodology and assumptions were reasonable. The shorter analysis period compared with the Regional Project is due to different technical solutions. 3.44 A CBA was also conducted for the roads with traffic volumes higher than 15 AADT and the NMT tracks included in the first-year work program, using the same methodology and assumptions as for the representative sample. The total investment cost was $770,000. 3.45 Given that lower-volumen roads mostly aim to provide accessibility, roads in the first-year program with traffic volumes of less than 15 AADT (total investment cost of 41 $860,000) were the subject of a CEA, which calculated the ratio between the costs and the beneficiary population. 3.46 The CBA for the representative sample of roads and NMT tracks yielded an NPV of $13.91 million and an ERR of 29.2 percent. The CBA for the first-year program showed an NPV of $380,000 and an ERR of 29 percent. Sensitivity analysis was carried out for the representative sample, which showed satisfactory results for all hypotheses tested. The CEA showed an average investment cost per beneficiary per kilometer of $57.70. 3.47 The ex post CBA used the same methodology and assumptions as the appraisal analysis, with updated figures on costs, benefits, traffic, and traffic growth. According to the ICR (World Bank 2014a, 42), the total investment cost was $77.2 million, which corresponded to 48 percent of the total project cost. The economic analysis report, on which the ICR was based, indicated a total investment cost of $87.8 million, but this included roads with an AADT of less than 15. 3.48 For roads with an AADT of less than 15, the CEA was repeated. Although the ICR indicated a total investment cost of $10.6 million for these roads (7 percent of the total investment cost), the economic analysis report based the analysis on a cost of $16.7 million, which included maintenance (10 percent of the project cost). 3.49 The ICR (World Bank 2014a, 42) and the economic analysis report mention that the CBA at completion showed an NPV of $108.3 million and an ERR of 54.6 percent. This is a huge increase compared with the appraisal ERR of 29 percent. In addition, such high rates of return are uncommon for the rehabilitation of low-volume roads. 3.50 The ICR and the economic analysis report attribute the high return to considerably higher cost savings for transport and travel time than anticipated, which is reasonable considering Peru’s strong economic growth during project implementation. However, IEG found that the investment costs per kilometer used in the ex post analysis also nearly doubled compared with the ex ante analysis. 3.51 In addition, the Regional Project had significantly larger traffic volumes and used nearly the same parameters to calculate the vehicle operating costs, which determine the benefits together with traffic volumes and time savings. However, it yielded a return only slightly above the discount rate. The difference in the rate of return could depend on the proportionally higher investment costs or greatly different road condition assumptions under the Regional Project. It could also indicate an error in the CBA calculation. With the available data, IEG was not able to determine the reason for the difference”. 42 3.52 The ICR (World Bank 2014a, 42) reports that the CEA showed an average investment cost per beneficiary per kilometer of $26.90 or $30.20, depending on whether the results figure in the table or text is used. IEG recalculated this cost with the numbers in the ICR using the total investment cost and not the NPV, which was not given. This approach showed an average investment cost per beneficiary per kilometer of $80.98 compared with the appraisal estimate of $57.70. This is in line with the economic analysis report, which shows an average investment cost per beneficiary per kilometer of $82.72. Although the results are higher than the ex ante estimate of $57.70, they are still below $100, which is considered reasonable in Peru based on previous project experience. 3.53 In terms of administrative efficiency, the project experienced cost and time overruns. The overall project cost was 6.77 percent higher than at appraisal, but for the infrastructure improvement works costs increased by 30 percent. This is explained mostly by the consequences of the economic boom in Peru. The project closing date was extended by 21 months. However, the project management costs were only 92 percent of the appraisal estimate. 3.54 Efficiency was also negatively affected by the shift of the project focus from periodic maintenance to rehabilitation. As in the Regional Project, IEG found that one of four rural roads visited during the mission was paved shortly after completion of the project and reclassified as a national road. This increased the overall investment cost (see the Efficiency section in chapter 2). Outcome 3.55 The overall outcome of the Rural Project is rated satisfactory. The objective of contributing to territorial development and poverty reduction in rural Peru was and remains highly relevant. This is reflected in national policy documents and initiatives and in the CAS and CPSs. The relevance of the design is rated substantial. Although the design was innovative, and the project’s ambitions increased compared with previous rural roads projects, the project had shortcomings in the measurability of the PDOs. These were overcome through the impact assessment. Efficacy is rated substantial. Despite the weaknesses of the impact assessment, juxtaposing project achievements (decreased travel times and costs, enhanced access) against findings from the literature and contextual factors suggests that the rural road improvements contributed to rural poverty reduction, although the extent of the contribution is not known. The efficiency of the project is rated substantial. Although the project suffered from administrative inefficiencies, the ex post ERR was significantly higher than at appraisal, and the cost per beneficiary per kilometer for roads with an AADT of less than 15 was still within a range considered reasonable in Peru. 43 Risk to the Development Outcome 3.56 The risk to development outcome is rated substantial. Infrastructure Maintenance Risks 3.57 The main risks are associated with road and NMT track maintenance. Although the MEF has continued to provide transfers for road maintenance, and these transfers have been increasing, they are not enough to maintain all rural roads rehabilitated by previous World Bank projects. In addition, the effectiveness of road maintenance has decreased with the exit of many community-based microenterprises and the reduced support to and control from the PRIs. There is also no guarantee that the transfers from the MEF will continue even if they are already well established and Provias is a strong advocate for them. There is no legal framework that obliges the MEF to carry out these transfers, and they depend on the annual budget negotiations. The local authorities also have no self-financing mechanisms for rural roads maintenance, and the allocations from their budget are minimal. Provias, under the ongoing Support to the Subnational Transport Program Project, is committed to create, support, and strengthen community- based microenterprises, and it will continue to lobby for road maintenance funds. Economic and Livelihood Risks 3.58 In terms of the impact of the LDW, although most activities promoted in the local governments did not materialize or survive beyond project completion, the support to producer associations has shown positive results, which are likely to be sustained. However, not all activities were successful. Under the ongoing Support to Subnational Transport Program Project, Provias is currently preparing an additional phase of the LDW that will correct some shortcomings of the past and aim for greater sustainability. Bank Performance 3.59 The Bank performance is rated satisfactory. Quality at Entry 3.60 The Bank performance at entry is rated satisfactory. As mentioned in chapter 1, the World Bank’s involvement in rural roads in Peru dates to 1995. This long-standing engagement provided a comprehensive understanding of the sector and a solid analytical underpinning for project preparation. In addition, the cofinancing arrangement with IDB in all subnational road projects in Peru enriched the dialogue by bringing different experiences and a second view to the table. 44 3.61 The project design reflected the experience gained over time. Elements tested in previous projects were scaled up and improved, the project objective became more ambitious, and the PAD indicates a real intention to be held accountable for a contribution to territorial development and poverty reduction (World Bank 2006b). However, the bridge subcomponent lacked the necessary background studies at appraisal and had to be scaled down later. 3.62 Arrangements were made to correct past shortcomings and ensure smooth implementation, such as capacity strengthening for PRIs to manage fiduciary responsibilities. The financial and economic analysis was sound. The safeguards considerations at appraisal were considered satisfactory, but the safeguards supervision arrangements had shortcomings. 3.63 Most risks were adequately anticipated, and the mitigation measures were satisfactory. However, the complexity of the task undertaken by the rural infrastructure pilot—implementing rural infrastructure investment in a coordinated way—was underestimated given the significant budget implications for sectors involved. The respective mitigation measure was insufficient. 3.64 The M&E design was adequate, except for indicators to directly measure poverty reduction and territorial development. This shortcoming was corrected through the impact assessment. The gender action plan was a vanguard initiative in the road sector. Quality of Supervision 3.65 The Bank performance at supervision is rated satisfactory. As in the Regional Project, the World Bank team used a problem-solving approach and was quick in making decisions and taking remedial measures when needed. Supervision focused on monitoring project outputs and outcomes. The team provided support, advice, and warnings. The team was praised for ensuring that project timelines were followed as much as possible and outcomes were achieved. The implementation agency also appreciated the World Bank’s accessibility, the quick support it offered, and its flexibility in extending the loan or reassigning resources. 3.66 Project supervision was more balanced than under the Regional Project, with considerable attention to institutional and territorial development activities even with delays at the beginning. The World Bank team played a strong role in the implementation of the gender action plan and provided support for the impact assessment. 45 3.67 The technical composition of the team was appropriate, but, as in the Regional Project, safeguards supervision should have been stronger. The World Bank team also played a strong role in supervising procurement and ensuring fiduciary compliance. 3.68 A timely restructuring was undertaken after the midterm review in 2010 to accommodate cost overruns. This restructuring should have rescaled the rural infrastructure pilot because the problems involved in implementing it as planned were already evident. Borrower Performance 3.69 The borrower performance is rated satisfactory. Government Performance 3.70 The government performance is rated satisfactory. The national government fully supported this ambitious rural road project, which went well beyond “road building,” and it continued to increase the delegation of responsibilities to local governments. The government financed the comprehensive impact assessment. The government also provided adequate counterpart funding and, over time, increased the funding to maintain the roads financed under this and previous projects. 3.71 Collaboration with local governments was easier than with the regions, at least partially due to the long-standing engagement. Local governments were generally committed to road-related and complementary activities. They showed interest in the LDW. They set up the Economic Development Divisions, passed territorial development regulations, and took responsibility for promoting rural producers’ associations. However, these efforts mostly stopped after project closure, which was followed by a change in administration. Implementing Agency Performance 3.72 The performance of the implementation agency is rated satisfactory. Provias had experience in implementing World Bank projects through the previous rural road projects, including in promoting collaboration between national and local governments. The Rural Project was complex, had a broad scope, and included many activities outside Provias’s core strengths. Provias was intensively involved in the decentralization process, which required extensive training and adequate monitoring and follow-up. 3.73 At project implementation, Provias focused on the physical works, and most institutional activities started late. However, Provias later invested considerable energy into many of them through committed professionals. 46 3.74 Provias was also strongly engaged in solving the fund shortages for road maintenance. It managed to obtain regular funds transfers from the MEF for that purpose. These still continue, but they have to be negotiated on an annual basis, a process in which Provias is heavily involved. Provias adapted to the unexpected increase in the price of inputs by providing incentives for more firms to compete. Finally, Provias anticipated election-related interruptions in project implementation and put mitigation measures in place. 3.75 Provias had strong fiduciary teams. It constantly monitored project implementation and provided detailed progress reports in a timely fashion. M&E implementation was adequate, but there were some shortcomings in safeguards supervision. Monitoring and Evaluation 3.76 M&E is rated substantial. Design 3.77 Using data collected by its zonal divisions, the PRIs, and the PIIs, Provias was to monitor and evaluate project progress, assess the achievement of indicators, and carry out financial and operational monitoring. Based on the experience in the previous two rural road projects, an impact assessment of the Rural Project focused on rural living conditions was to be carried out. Evaluations were also to be done for the LDW and the rural infrastructure pilot. 3.78 There were shortcomings in the design of the results framework, which were mitigated through the impact assessment. Although the results framework included intermediate outcomes, it lacked indicators to measure the achievement of the objective (that is, contributing to territorial development and the fight against rural poverty). This was not a major issue since the variables to be tested in the impact assessment included these types of indicators. All intermediate outcome indicators were also expected to be measured through the impact evaluation. The Rural Project had a significant number of well-defined output indicators. Implementation 3.79 Provias carried out regular monitoring and reporting of the project’s progress, adequately covering results, as in the Regional Project. 3.80 A baseline for the impact assessment was developed in 2006 and the endline in 2013. Such an impact assessment is a rare effort in road projects, and the borrower should be recognized for it. This assessment was comprehensive and provided the 47 information for all impact-related indicators. As seen in appendix D, the exercise was not exempt from difficulties. 3.81 Provias evaluated the LDW. This evaluation focused mainly on the activities carried out and the financing resources leveraged. It did not investigate the impacts of the LDW on territorial development and poverty reduction. The evaluation of the rural infrastructure pilot was not done because the pilot did not materialize as planned. Use 3.82 The information from the impact assessment was cited in successive studies and was used to design the ongoing Support to the Subnational Transport Program Project. The information on the output indicators collected as part of the project’s M&E system was mostly used for project purposes, including for the midterm review and the project restructurings. After project completion, the information on the operation of community-based microenterprises and PRIs was no longer collected. However, preparations are under way to monitor and evaluate these aspects again within the ongoing project. 4. Lessons 4.1 Subnational governments need to own their road planning instruments to ensure their use. Under both projects, consultants hired by the national government produced comprehensive, technically sound, and well-prepared road plans. Although the subnational governments were formally in charge of plan preparation, their involvement was limited, and the national government drove the initiative. The plans guided the selection of the project priorities, but due to their limited buy-in, subnational governments no longer used those plans for setting priorities once the project ended. 4.2 Ways to sustain the community-based microenterprises model for rural road maintenance should be identified. In both projects, the implementation agency supported the recruitment of members of community-based microenterprises, assisted them in formalizing the microenterprise, and provided technical and managerial skills. In this way, the projects achieved the dual aim of contributing to employment creation and maintaining the roads. However, once the projects and the support ended, this model was not sustained, and most community-based microenterprises have disappeared. It is important to find ways to sustain the model with less support and hand-holding, or pass the responsibility for this support together with the necessary resources to the local level. 4.3 Road maintenance is essential all year round, and funding and bidding schedules need to be adjusted accordingly. Since 2008, the MEF has transferred 48 resources for road maintenance to subnational governments, but these resources have arrived late in the fiscal year and in several installments. Despite the MEF making the full resources available at the beginning of January 2019, it has taken the local governments until end of March to sign the necessary agreements and start the bidding processes. Because the rainy season in Peru lasts until April, maintenance is most needed in the first months of the year. Given that the budget process and timeline cannot be changed, multiyear maintenance contracts that start in midyear could ensure maintenance all year round. Otherwise, the bidding processes for maintenance could start before the end of the year to ensure that the contracts can be signed in January. 4.4 Poverty impacts of rural roads projects are difficult to attribute. Rural roads projects should aim for more than just enhancing road conditions and reducing travel times and costs. The Rural Project is to be commended for trying to show its impact on poverty reduction and for the comprehensive impact assessment itself. However, it was impossible to attribute the poverty impact to the project. As desirable as it is in roads projects to measure benefits other than just time and vehicle cost savings, it is better to limit the ambitions of the measurement to the type of project impacts and geographical scope that can be reliably attributed to the project. 4.5 If the road agency carries out activities that are outside its core responsibilities, it needs to involve the other ministries and government agencies responsible for these activities to ensure sustainability. The Rural Project supported an LDW that financed activities to help local governments and producers’ associations enhance economic development along improved roads. The road agency outsourced the implementation of the LDW activities, and this was relatively successful. However, once those contracts finished, there was a lack of organizational clarity about how the activities would be supported to ensure continuity. Multiple ministries and government agencies share the responsibility of helping small producers and local governments to foster economic development. They should be involved in the design and implementation of LDW activities, which need to focus strongly on sustainability. 4.6 Transferring successful solutions from one government level to another requires a careful contextual analysis and the subnational governments’ participation in decision making from the outset. In this case, transferring solutions from the rural to regional context did not work because the success factors for the Rural Project and the differences in the regional context were not adequately analyzed. In addition, key aspects of project design (such as how to identify road priorities, select road management solutions, and identify institutional changes to be implemented by the regions) rested with the central authorities. This did not create sufficiently flexible solutions, empower the subnational governments, or create the ownership and capacity required to successfully implement the project. 49 1Poverty data are from Peru National Statistics Bureau (INEI). Poverty means lack of access to or inability to afford basic foods and services; extreme poverty means lack of access to or inability to afford basic foods. 2The task team of the ongoing Support to the Subnational Transport Program Project provided this information. Peru has no updated data on the full length and condition of the complete rural road network. 3Partido Aprista Peruano, Plan de Gobierno 2006 a 2011 del Presidente Alan Garcia, http://www2.congreso.gob.pe/Sicr/Comisiones/2009/comvirahaya.nsf/3D7A3BA8901FF3FF052578 3A00706D87/$FILE/plan_de_gobierno_partido_aprista_peruano_(pacto_etico_electoral).pdf. 4Under an outsourcing arrangement, maintenance activities are contracted out to the private sector, including community-based microenterprises, whereas under force account, road maintenance activities are carried out in-house by the road agency with its own staff and equipment. 5According to Inventario de Caracter Basico de la Red Vial Departamental (MTC 2015), 20,675 kilometers of the existing network were unpaved and 3,907 kilometers were paved in 2015. 6AADT is a measure used primarily in transport planning and engineering. It is the total volume of vehicle traffic on a road for a year divided by 365 days. 7 This mainly refers to the flexibility to use paved or unpaved road treatments, as appropriate. 8This mainly refers to the possibility of using small private firms in addition to the community- based microenterprises for routine maintenance. 9 The ongoing Support to the Subnational Transport Program Project plans to again support community-based microenterprises and work on the sustainability of the model mainly through training and professional certification of maintenance workers. 50 References Abdulkadir, Usman, J. O. Adefila, and J. Musa. 2013. “Impact of Rural Road Transport on Agricultural Production in Kwara State.” Nigerian Journal of Agriculture, Food and Environment 9 (2): 20–25. Asher, Sam, and Paul Novosad. 2019. “Rural Roads and Local Economic Development.” Policy Research Working Paper 8466, World Bank, Washington, DC. Bell, Clive, and Susanne van Dillen. 2012. “How Does India's Rural Roads Program Affect the Grassroots? Findings from a Survey in Orissa.” Policy Research Working Paper 6167, World Bank, Washington, DC. Guerra Garcia Picasso, Gustavo. 2016. “Diagnostico y Propuesta de Politica para el Mantenimiento de la Red Vial Vecinal.” Peru. INEI (Instituto Nacional de Estadística e Informática). 2018. Evolucion de la Pobreza Monetaria 2007–2017. Lima, Peru: Instituto Nacional de Estadística e Informática. Khandker, Shahidur R., Zaid Bakht, and Gayatri B. Koolwal. 2006. “The Poverty Impact of Rural Roads: Evidence from Bangladesh.” Policy Research Working Paper 3875, World Bank, Washington, DC. http://documents.worldbank.org/curated/en/568171468003911910/The- poverty-impact-of-rural-roads-evidence-from-Bangladesh. Levy, Hernan. 2004. “Rural Roads and Poverty Alleviation in Morocco.” World Bank, Washington, DC. http://documents.worldbank.org/curated/en/420301468756995092/Rural-roads-and- poverty-alleviation-in-Morocco. Macroconsult-Cuanto. 2014. Elaboración de la Evaluación de Impacto y Ampliación de la Línea Base del Programa de Transporte Rural Descentralizado Informe Final. MTC (Ministry of Transport and Telecommunication). 2015. Inventario de Caracter Basico de la Red Vial Departamental. Peru. United Nations. 2019. “Inter-Agency Statement: How Integrated Approaches Can Help Deliver the 2030 Agenda? A Contribution to the G20 DWG.” https://www.greengrowthknowledge.org/sites/default/files/downloads/resource/Interage ncy%20Statement%20On%20Sustainable%20Infrastructure_UN%20Environment.pdf. Webb, Richard. 2013. Conexion y Despegue Rural. Lima, Peru: Universidad de San Martin de Porres and Instituto del Perú. World Bank. 2002. Peru—Country Assistance Strategy. Washington, DC: World Bank. ———. 2005a. Peru—Opportunities for All: Peru Poverty Assessment. Washington, DC: World Bank. 51 ———. 2005b. “Peru—Regional Transport Infrastructure Decentralization Project.” Project Appraisal Document 32223-PE, World Bank, Washington, DC. ———. 2006a. Peru—Country Partnership Strategy, FY07–FY11. Washington, DC: World Bank. ———. 2006b. “Peru—Decentralized Rural Transport Project.” Project Appraisal Document 36484-PE, World Bank, Washington, DC. ———. 2010. “Restructuring Paper for a Decentralized Rural Transport Project. ” Report 54496- PE, World Bank, Washington, DC. ———. 2012. Peru—Country Partnership Strategy, FY12–FY16. Washington, DC: World Bank. ———. 2014a. Implementation Completion and Results Report for a Decentralized Rural Transport Project. Washington, DC: World Bank. ———. 2014b. Implementation Completion and Results Report for a Reginal Transport Infrastructure Decentralization Project. Washington, DC: World Bank. ———. 2017a. Peru: Building a More Efficient and Equitable Fiscal Decentralization System. Washington, DC: World Bank. ———. 2017b. Peru Public Expenditure Review. Washington, DC: World Bank. 52 Appendix A. Basic Data Sheet Regional Transport Infrastructure Decentralization Project (Loan IBRD- 73220; P078813) Table A.1. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percent of Financing ($, millions) ($, millions) Appraisal Estimate Total project costs 200.00 167.07 83.53 Loan amount 50.00 41.60 83.20 Cofinancing 150.00 45.97 83.64 Borrower 100.00 79.50 79.50 IDB 50 45.97 91.94 Cancellation — — — Table A.2. Cumulative Estimated and Actual Disbursements Disbursements FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 Appraisal estimate ($, millions) 0 1.695 9.718 22.5 38.580 48.162 50.00 50.00 50.00 Actual ($, millions) 0 .125 .657 2.975 6.139 15.131 20.531 30.031 40.531 Actual as percent of appraisal 0 7.37 6.76 13.5 15.9 31.4 41 60.0 81.06 Date of final disbursement n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. 12/2/2013 Table A.3. Project Dates Event Original Actual Concept review 4/14/2004 4/14/2004 Board approval 7/12/2005 7/12/2005 Effectiveness 4/10/2006 4/10/2006 Closing date 6/30/2010 6/30/2014 53 Table A.4. Staff Time and Cost World Bank Budget Only Staff time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY03 — 38.26 FY04 — 55.02 FY05 — 154.93 FY06 — 4.24 FY07 — 0.00 FY08 — 0.00 Total — 252.45 Supervision or ICR FY03 — 0.00 FY04 — 0.00 FY05 — 0.00 FY06 — 29.44 FY07 — 51.65 FY08 — 93.95 Total — 175.04 Note: ICR = Implementation Completion and Results Report. a. Including travel and consultant costs. Table A.5. Task Team Members Responsibility or Name Titlea Unit Specialty Lending Keisgner De Jesus Alfaro Senior Procurement Specialist LCSPT Procurement Sally L. Burningham Sector Manager LCSDE n.a. Maria Elizabeth Dasso Senior Social Development & Civil LCSSO Social Tatiana S. Daza Senior Executive Assistant TWI Assistant Mohammed D. E. Feghoul Lead Municipal Engineer MNSSD Urban Maria Emilia Freire Senior Adviser UDR n.a Patricia Mc Kenzie Sector Manager AFTME n.a Aurelio Menendez Sector Manager LCSTR Transport Isabella Micali Drossos Senior Counsel LEGAM Legal Xiomara A. Morel Senior Financial Management Specialist LCSFM Financial Management Alexandra P. Orellana Bonilla Program Assistant CPALS Assistant Nicolas Peltier-Thiberge Assistant to the President EXC Transport 54 Fernando Rojas Lead Public Sector Management LCSPS Public Sector Management Marco Antonio Zambrano Chavez Consultant LCSEN Environment Alonso Zarzar Casis Senior Social Scientist LCSSO Social Supervision or ICR Maria Margarita Nunez Senior Highway Engineer—TTL(supervision) GTIDR Transport Nicolas Peltier-Thiberge Assistant to the President—TTL (entry /early EXC Transport supervision) Ana Lucia Jimenez Nieto Financial Management Specialist LCSFM n.a. Selene del Rocio La Vera Procurement Specialist LCSPT n.a. Maria Virginia Hormazabal Finance Officer CTRDM n.a. Anna R. Okola Transport. Engineer—ICR TTL GTIDR n.a. Aracelly Woodall Senior Program Assistant GTIDR n.a. Sebastian Elias Guerrero Consultant GTIDR n.a. Mirtha Pokorny Consultant GTIDR n.a. Pablo Gonzalez Rueda Consultant GTIDR n.a. Karla Dominguez Gonzalez Consultant GTIDR n.a. Jorge Minaya Osorio Temporary GTIDR n.a. Raul Tolmos Environmental Specialist GENDR n.a. Rodrigo Archondo-Callao Senior Highway Engineer ECSTR n.a. Maria Luz Caballero Alonso Consultant SEGES n.a. Maria Elizabeth Dasso Senior Social Development & Civil LCSSO n.a. Nicolas Drossos E T Consultant LCSFM n.a. Joseph Paul Formoso Senior Finance Officer CTRLA n.a. Nelly Ikeda Financial Management Specialist LCSFM n.a. Patricia Mc Kenzie Sector Manager AFTME n.a. Isabella Micali Drossos Senior Counsel LEGAM n.a. Xiomara A. Morel Senior Financial Management Specialist LCSFM n.a. Pedro Olinto Senior Economist PRMPR n.a. Keisgner De Jesus Alfaro Senior Procurement Specialist LCSPT n.a. Pierre-Antoine Picand Temporary LCSTR n.a. Juan D. Quintero Senior Environmental Engineer EASER n.a. Francisco Rodriguez Procurement Specialist LCSPT n.a. Nicolas Serrie Junior Professional Associate LCSTR n.a. Tomas Socias Senior Procurement Specialist LCSPT n.a. Luis Tineo Senior Operations Officer GFDRR n.a. Evelyn Villatoro Senior Procurement Specialist EASR1 n.a. Alonso Zarzar Casis Sr Social Scientist LCSSO n.a. Luz A. Zeron Financial Management Specialist LCSFM n.a. Note: ICR = Implementation Completion and Results Report; TTL = task team leader. n.a. = Not Available. a. At time of appraisal and closure, respectively. 55 Decentralized Rural Transport Project (Loan IBRD-74230; P095570) Table A.6. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percent of Financing ($, millions) ($, millions) Appraisal Estimate Total project costs 150.00 160.11 106.74 Loan amount 50.00 49.95 99.99 Cofinancing 100.00 110.16 110.16 Borrower 50.00 60.16 120.32 IDB 50.00 50.00 100.00 Cancellation n.a. n.a. n.a. Table A.7. Cumulative Estimated and Actual Disbursements Disbursements FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 Appraisal estimate ($, 0.50 10.00 20.00 30.00 40.00 50.00 50.00 50.00 millions) Actual ($, millions) 0 0.50 4.079 29.802 37.802 42.802 49.00 49.946 Actual as percent of 0 5 20.39 99.3 94.5 85.6 98.0 99.89 appraisal Date of final n.a. n.a. n.a. n.a. n.a. n.a. n.a. 09/5/2013 disbursement Table A.8. Project Dates Event Original Actual Concept review 4/11/2006 4/11/2006 Board approval 12/19/2006 12/19/2006 Effectiveness 07/12/2007 07/12/2007 Closing date 03/31/2012 12/31/2013 56 Table A.9. Staff Time and Cost World Bank Budget Only Staff time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY06 16.18 74.35 FY07 28.27 129.05 FY08 1.80 10.43 FY09 1.33 5.08 FY10 1.10 4.40 Total 48.68 223.31 Supervision or ICR FY07 1.36 8.53 FY08 16.96 90.40 FY09 15.71 68.38 FY10 24.12 114.91 FY11 17.03 117.29 FY12 22.78 143.80 FY13 11.13 109.29 FY14 7.13 110.27 Total 116.22 762.87 Note: FY = fiscal year; ICR = Implementation Completion and Results Report. a. Including travel and consultant costs. Table A.10. Task Team Members Responsibility or Name Titlea Unit Specialty Lending Nicolas Peltier-Thiberge Assistant to the President–TTL EXC Transport Alonso Zarzar Casis Senior Social Scientist LCSSO Social Demetrios Papathanasiou Sector Leader AFTSN n.a. Evelyn Villatoro Senior Procurement Specialist EASR1 Procurement Isabella Micali Drossos Senior Counsel LEGAM Legal Juan D. Quintero Consultant Environmental EASDE Environment Keisgner De Jesus Alfaro Senior Procurement Specialist LCSPT Procurement Luis M. Schwarz Senior Finance Officer CRTLA Financial Management Luis Tineo Senior Operations Officer GFDRR Operations Luz A. Zeron Financial Management Specialist LCSFM n.a. Maria Angelica Sotomayor Sector Leader LCSSD n.a. Maria Elizabeth Dasso Consultant Social Development LCSTR n.a. 57 Maria Luz Caballero Alonso Consultant SEGES n.a. Marco Zambrano Consultant Environmental AFTG1 n.a. Melanie Glass Consultant LCSTR n.a. Nicolas Drossos Consultant Financial Management AFTSW n.a. Nicolas Serrie Junior Professional Associate LCSTR n.a. Pedro Olinto Senior Economist PRMPR n.a. Rodrigo Archondo-Callao Senior Highway Engineer ECSTR n.a. Rossana Polastri Senior Economist PRMVP n.a. Susan Bogash Consultant Energy Economist LCSEG n.a. Tomas Socias Senior Procurement Specialist LCSPT n.a. Xiomara A. Morel Senior Financial Management LCSFM n.a. Specialist Supervision or ICR Maria Margarita Nunez Senior Highway Engineer–TTL LCSTR n.a. Nicolas Peltier-Thiberge Assistant to the President–TTL EXC n.a. Ana Lucia Jimenez Nieto Financial Management Specialist LCSFM n.a. Andrea Monje Silva Consultant Gender LCSTR n.a. Alonso Zarzar Casis Senior Social Scientist LCSSO n.a. Anna R. Okola Transport. Engineer LCSTR n.a. Aracelly Woodall Senior Program Assistant LCSTR n.a. Elizabeth Huaman Team Assistant LCC6C n.a. Francisco Rodriguez Procurement Specialist LCSPT n.a. Julie Chretien Temporary LCSTR n.a. Licette Moncayo Program Assistant LCSTR n.a. Mara La Rosa Program Assistant LCC6C n.a. Maria Elizabeth Dasso Consultant LCSTR n.a. Maria Jose Sala Pelufo Consultant EASIS n.a. Marco Zambrano Consultant Environmental AFTG1 n.a. Omar Wahab E T Consultant MNSTI n.a. Oswaldo Patino Consultant LCSTR n.a. Pablo Riestra Temporary ECSTR n.a. Pierre-Antoine Picand Temporary LCSTR n.a. Rafael Bernardo Romeo Temporary ECSTR n.a. Raul Tolmos Environmental Specialist LCSEN n.a. Sebastian Elias Guerrero Consultant LCSTR Selene La Vera Procurement Specialist LCSPT Note: ICR = Implementation Completion and Results Report; TTL = task team leader. n.a. = Not Available. a. At time of appraisal and closure, respectively. 58 Appendix B. Fieldwork Methodology The Peru Regional Transport Infrastructure Decentralization Project (P078813) (henceforth Regional Project) benefited regions all over Peru. The objective was to improve, through decentralization at the regional level, the prioritization, efficiency, and effectiveness of regional transport interventions to contribute to regional development and poverty alleviation by enhancing transport conditions in the borrower’s territory. The main project activities included the following: • Preparation of participatory regional road plans • Road rehabilitation • Periodic maintenance • Routine maintenance • Capacity strengthening in all regions and for Provias The Decentralized Rural Transport (P095570) Project (henceforth Rural Project) benefited provincial and district governments all over Peru. The objective was to contribute to territorial development and the fight against rural poverty in the borrower’s territory by improving the access of rural households and entrepreneurs to goods, social services, and income-generating opportunities through reduced transport costs and better rural transport infrastructure. The main project activities included the following: • Road rehabilitation • Periodic maintenance • Nonmotorized transport (NMT) track improvement • Preparation of participatory regional road plans • Capacity strengthening of provincial road institutes (PRIs) • Capacity strengthen of community-based microenterprises • Activities to foster local economic developments in the framework of the local development window (LDW) • Preparation of rural infrastructure plans 59 Overall Data Collection Strategy The data collection strategy consisted of premission interviews with World Bank staff, the implementation agency, and other key stakeholders to better understand the project context, prepare the mission agenda, and collect data. This was followed by the collection of key project documents and other literature before the mission, individual and group interviews and site visits during the mission, review of project outputs, and additional data collection and individual follow-up interviews after the mission. Sample Selection for Field Visits For the selection of the places to be visited in addition to Lima, spreadsheets were made with the names of all road and NMT track sections financed under the two projects. Because the Project Performance Assessment Report (PPAR) mission was only two weeks in Peru, it was decided to visit only two regions and two provinces and two districts within these regions. Given that the objective of the Rural Project aimed at poverty reduction, it was decided to visit two of the poorest regions in Peru that combined all or most of the project activities listed previously for both projects. According to the 2018 National Statistics Office (INEI) Report on Evolucion de la Pobreza Monetaria 2007-2017, the poorest region in Peru is Cajamarca, with poverty levels over 36.8 percent, followed by Huánuco, Puno, Huancavelica, Amazonas, Apurimac, Loreto, Pasco, and Ayacucho, with poverty levels between 33.3 and 36.8 percent. As shown in table B.1 used for the selection of the regions, these are also the regions with the lowest gross domestic product per capita, except for Apurimac. The final choice among these regions was made based on the largest number of road sections and networks and NMT tracks and networks connecting markets and social services and other project activities to be visited. 60 Table B.1. Data Used for the Selection of the Regions to Visit Regional Local Number of Number of Number of road GDP per Rural Roads Length of Regional Total number of Number of Number of Roads Development Provincial projects projects capita at Rural Roads Periodic Tracks Roads Rural Projects Provincial Periodic Window in Road prioritized prioritized and constant 2007 rehab (Km) Maintenance rehab rehab Microenterprise Financed by Infrastructure Maintenance Existence Institutes in PIP implemented in prices (2016) (Km) (Km) (Km) s (n=185) LDW (n=44) Plans (n=15) (Km) (n=30) (n=188) (n=629) PIP (n=19) Peruvian Sol Región Amazonas 85.0 - 191.9 156.9 80.8 8 2 6 7 - - - 6,566 Áncash 143.5 762.7 173.8 - - 10 - - 20 1 76 5 15,907 Apurímac 128.2 671.7 - 92.0 272.5 5 2 - 7 1 23 - 13,777 Arequipa 262.9 51.6 126.1 - 103.7 15 2 8 8 1 43 - 22,823 Ayacucho 479.5 638.3 156.7 89.9 1,029.7 21 4 5 11 2 92 9 7,406 Cajamarca 335.4 1,141.2 - 79.4 120.4 29 4 1 13 1 17 - 6,902 Cuzco 262.2 603.7 142.9 114.0 276.5 21 2 5 13 2 87 1 16,482 Huancavelica 133.6 997.4 - 126.9 338.1 7 2 3 7 1 32 1 6,444 Huánuco 141.8 838.7 311.0 19.2 298.8 7 2 - 11 2 133 - 6,138 Ica 36.6 - 19.9 63.2 165.5 1 2 2 5 - - - 19,201 Junín 91.2 712.6 94.2 89.8 160.8 2 2 4 9 - - - 10,501 La Libertad 311.9 - 367.1 338.0 269.9 10 - - 11 1 29 - 10,859 Lambayeque 20.3 - 76.3 - 71.4 4 - - 3 - - - 8,727 Lima 35.1 - 77.5 - - - - - 7 - - - 19,987 Loreto - - 84.0 - - - - - 5 - - - 7,144 Madre de Dios 18.2 16.5 - - - 1 - - 3 - - - 18,972 Moquegua - 72.3 - - - - - - 3 - - - 47,465 Pasco 31.0 251.9 85.5 115.4 196.4 6 - - 3 1 21 1 17,413 Piura 55.9 - 98.6 74.5 - 7 2 4 8 1 27 - 10,258 Puno 271.3 341.1 92.1 162.8 108.1 6 2 6 13 1 49 2 6,346 San Martín 202.6 578.7 134.6 86.5 114.0 20 2 - 10 - - - 6,557 Tacna 20.6 - - - - - - - 4 - - - 18,842 Tumbes - - - 42.9 - - - - 3 - - - 10,423 Ucayali 15.7 - 48.9 - - 5 - - 4 - - - 8,308 Total 3,082.3 7,678.4 2,281.1 1,651.4 3,606.5 185 30 44 188 15 629 19 323,448 Source: Elaboration by IEG based on project information and data from the Peruvian National Statistics Office The provinces and districts to be visited in these regions were also selected to include the largest amount of project activities. To minimize the amount of time wasted in driving to intervened road and NMT track sections and to cover the largest amount of project interventions, the road and track sections to be visited were randomly selected based on a map with all roads within one to two hours driving distance from a district, provincial, or regional capital. The final sample of places to be visited was discussed with the project implementation unit to ensure that it was feasible and minor changes were made. Evaluation and Interview Questions The main evaluation questions for the Rural Project included the following: • What was the context of this project? • Did the project positively contribute to territorial development and the fight against rural poverty in the borrower’s territory, and does this impact persist? • What are the impacts of rural roads improvements under the project? • What are the impacts of NMT track improvements? • Who is currently in charge of NMT tracks? 61 • What has happened to the bridges not implemented under the Rural Project, and what impact did it have on the project outcome? • What has happened with the slope management pilot, and what impact did it have on the project outcome? • What were the impacts of rural roads improvements in the two visited regions in general? • How was the impact assessment done? • Has road planning at provincial level been improved compared with the situation at appraisal and did the intervened road sections have the potential to contribute to territorial development and hence reduce poverty? • How is maintenance carried out and financed? • Is road maintenance sufficient? • What are the road conditions of rural roads? • What was and is the role of the road monitors? • What do the zonal divisions of Provias? • How well do the provincial road institutes work? • How well was rural road planning done and what is happening with it? • How was the participation of subnational governments and other stakeholders in road planning? • Do provinces use the georeferenced information systems from the road plans? • Are the priorities in the road plans the right ones and have provinces complied with them? • What has happened to the provincial (economic) infrastructure plans? • What is happening in Peru in terms of territorial development and poverty reduction? • Has out-migration from rural areas happened, and what are the lessons for future rural road improvements? • Are there other problems to rural development in Peru not taken into account by the project? 62 • What was the impact of the Rural Project on gender? • How did the LDW work and which activities were carried out through it? • Have the productive activities supported through the LDW been sustained and how important are they for the local economy? • What was the motivation for implementing the LDW? • Are the Economic Development Divisions able to take over the activities supported by the LDW? • What would it take to roll it out at scale? • What is the link between the production activities implemented and the road improvements? • What was and is the role of the PRIs? • What is the Support to the Subnational Transport Project Program doing? • What were the main issues faced by the Rural Project in terms of design, implementation, safeguard, and fiduciary aspects? • How was the World Bank and borrower performance? The main evaluation questions for the Regional Project included the following: • Has the regional capacity to plan and prioritize roads been improved and is planning a priority for the current government? • How do regional governments prioritize their road interventions? • Did the Regions use the participatory regional road plans? • Were the investments in the project carried out based on the priorities identified in the plans? • Was the participatory process adequate? • Did the Regions get new planning tools? • Did the planning exercise under the project adequately capture local needs? • Has the road sector institutional framework at the regional level been rationalized? • Is outsourcing of road maintenance and rehabilitation commonplace at the regional level? 63 • Has the project helped improve safeguard management and monitoring and evaluation at the regional level? • Did the project strengthen the transport management capacity of the regions? • Has Provias/zonal division continued to strengthen the transport management capacity of the regions and support them? • Has the restructuring of Provias taken place as planned? • Have road rehabilitation and periodic maintenance been adequate? • Are the roads upgraded under the project in good condition? • Have these roads been maintained, what is the current maintenance model at the regional level, and how sustainable is it? • What were the main issues faced by the Regional Project in terms of design, implementation, safeguard, and fiduciary aspects? • How was the World Bank and borrower performance? These questions were adopted for each interviewee group. 64 Appendix C. Adequacy of Road Maintenance Expenditures Total Regional Road and Maintenance Expenditures As seen in table C.1, the average annual maintenance expenditures for regional roads between 2014 and 2018 were 216,489,542 nuevo soles (S/.), which is approximately equivalent to $72 million. The expenditures increased on average by 10 percent. Table C.1. Maintenance Expenditures for Regional Roads, 2014–18 Annual Maintenance 2014 2015 2016 2017 2018 Average Routine maintenance unpaved 69,330,184 78,959,714 42,176,878 84,911,391 96,191,643 74,313,962 (S/.) Periodic maintenance unpaved 76,263,675 136,673,568 110,578,251 117,104,925 89,702,846 106,064,653 (S/.) Routine maintenance paved (S/.) 10,297,363 11,504,140 21,046,140 15,116,163 11,402,266 13,873,214 Periodic maintenance paved (S/.) 11,318,404 8,828,955 31,412,340 34,194,454 25,434,408 22,237,712 Total routine and periodic 167,211,640 235,968,392 205,215,625 251,328,950 222,733,181 216,489,542 maintenance (S/.) Annual increase/decrease (%) 41 −13 22 −11 10 Source: Elaboration by IEG based on Siaf Consulta Amigable. Note: S/. = nuevos soles. Overall expenditures for regional roads stayed relatively stable during and after the Regional Project. As seen in table C.2, on average, regional road expenditures increased by 6 percent and were about S/. 1 billion (approximately $337 million) from 2010 to 2018. They increased strongly in 2014 and 2105 but then went back to nearly pre-2014 levels. Table C.2. Total Regional Road Expenditures, 2010–18 Expenditures and Annual Annual Change 2010 2011 2012 2013 2014 2015 2016 2017 2018 Average Total regional road 1,025.45 932.30 723.64 819.75 1,251.38 1,556.83 760.02 908.81 1,121.91 1,011.12 expenditure (S/., millions) Annual −9 −22 13 53 24 −51 20 23 6 increase/decrease (%) Source: Elaboration by IEG based on Siaf Consulta Amigable. Note: S/. = nuevos soles. 65 Adequacy of Maintenance Expenditures on the Regional Network Based on a rough calculation by the Independent Evaluation Group (IEG), maintenance expenditures would be enough to keep the 3,796 kilometers of regional roads rehabilitated under the Regional Project in good condition. However, they are far too low for the network as a whole. As shown in the table C.3, average routine maintenance expenditures since 2014 would be enough to maintain about 7,882 kilometers of unpaved roads per year, which is 38 percent of the network. Periodic maintenance expenditures, on average, would enable interventions on 10 percent of the unpaved and 3 percent of the paved network annually. With this allocation, it would take 10 years to complete the periodic maintenance on the unpaved network versus the required five years. Table C.3. Network Receiving Routine and Periodic Maintenance Given Current Expenditures (S/.) Kilometers Average Intervened Years to Annual Average Cost Annually Complete Road and Expenditures Kilometers of of Routine Given Periodic Maintenance 2014–18 Regional Maintenance Available Percent of Maintenance Types (S/.) Network (S/.) Expenditures Network on Network Unpaved 10 Routine 74,313,962 20,675 9,429 7,882 38 maintenance Periodic 106,064,653 20,675 51,667 2,053 10 maintenance Paved 33 Routine 13,873,214 3,907 9,429 1,471 38 maintenance Periodic 22,237,712 3,907 210,000 105 3 maintenance Source: Elaboration by IEG based on Siaf Consulta Amigable; Diagnostico y Propuesta de Politica para el Mantenimiento de la Red Vial Vecinal; Peru Public Expenditure Review; Indonesia Public Expenditure Review. Note: In 2015, the regional network measured 24,582 kilometers. S/. = nuevos soles. The cost per kilometer for routine maintenance on the unpaved network of S/. 9,429 (approximately $3,000) comes from the 2016 Report on “Diagnostico y Propuesta de Politica para el Mantenimiento de la Red Vial Vecinal” (54 and 55). This report contains a detailed analysis of rural roads routine maintenance costs in Peru. In the absence of cost data for routine maintenance on the regional unpaved network, it was assumed that the cost is the same for the rural network even if the regional network carries higher traffic volumes, hence costs might be higher. 66 The cost per kilometer for periodic maintenance on the unpaved network of S/. 51,667 (approximately $16,000) comes from the 2017 Peru Public Expenditure Review (PER). Due to a lack of cost data for routine maintenance on the paved network, the same figure of S/. 9,429 (approximately $3,000) for the unpaved network was used because the type of work is similar. This is in line with the 2012 Indonesia Road Sector PER, which gives an amount of $2,000 to $5,000 per kilometer for routine maintenance on the paved network as an international comparator. Similarly, for the cost per kilometer of periodic maintenance on the paved network, the 2012 Indonesia Road Sector PER gives a range of $70,000 to $100,000 per kilometer as an international comparator. The lower range of $70,000 was used for this analysis. Total Rural Road and Maintenance Expenditures As shown in table C.4, the transfers from the Ministry of Economy and Finance (MEF) to local governments for rural roads maintenance by project end in 2013 were S/. 79.10 million (approximately $29 million). They increased over the years to reach S/. 256.46 million (approximately $77 million) in 2019. On average, between 2013 and 2019, the transfers were $51 million. Table C.4. Annual Transfers for Road Maintenance from MEF to Subnational Governments in S/. 2013 2014 2015 2016 2017 2018 2019 Annual TRANSFERS average FROM MEF IN S/. Transfer To Local To Local To Local To Local Transfer To Local Transfer To Local To Local Transfer type Transfer type Transfer type Transfer type transfer type Governments Governments Governments Governments type Governments type Governments Governments PIA 57,000,000 PIA 57,000,000 PIA 57,000,000 PIA 57,000,000 PIA 57,000,000 PIA 57,000,000 PIA 182,000,000 D.S. 214- DS Nº 009- 2013-MTC 22,100,400 LEY 30191 71,882,827 D.S. Nº 116-2015 76,913,091 DS Nº 048-2016 6,941,384 2017 14,771,716 D.U. Nº 2018 51,894,095 ROUTINE DS Nº 004- 0 DS Nº 047-2015 22,292,397 DS Nº 064-2016 904,401 2017 111,011,923 MAINTENANCE 0 DS Nº 059-2015 355,860 DS Nº 332-2016 32,304,950 DS Nº 023, 047, 0 066, 116-2015 DS Nº 047-2016 Total Routine Maintenance 79,100,400 Total 2014 128,882,827 Total 2015 156,561,348 Total 2016 97,150,735 Total 2017 182,783,639 Total 2018 108,894,095 Total 2018 182,000,000 133,624,721 (A) PIA 0 PIA PIA PIA PIA 74,455,690 PERIODIC DS Nº 004- 0 D.S Nº 116-2015 DS Nº 332-2016 98,748,953 D.U. Nº 2018 18,845,963 MAINTENANCE 2017 DS Nº 009- 0 DS Nº 047-2016 2017 Total Periodic Maintenance 0 Total 2014 0 Total 2015 0 Total 2016 0 Total 2017 98,748,953 Total 2018 18,845,963 Total 2018 74,455,690 64,016,869 (B) TOTAL (A + B) IN S/. 79,100,400 Total (A+B) 128,882,827 Total (A+B) 156,561,348 Total (A+B) 97,150,735 Total (A+B) 281,532,592 Total (A+B) 127,740,058 Total (A+B) 256,455,690 161,060,521 TOTAL (A + B) IN $ 29,296,444 Total (A+B) 44,906,908 Total (A+B) 49,078,792 Total (A+B) 28,742,821 Total (A+B) 86,359,691 Total (A+B) 38,826,765 Total (A+B) 77,245,690 50,636,730 Source: Provias Descentralizado Note: PIA = Presupuesto Inicial de Apertura (annual initial budget); DS = Decreto Supremo (supreme decree); DU = Decreto de Urgencia (urgency decree) Table C.5 shows that the overall budget spent on rural roads annually between 2010 and 2018 was on average S/. 1,709.39 million (approximately $570 million). According to the 2017 Peru PER, local governments spent 14 percent of their budget on maintenance. If this percentage is applied to the total rural roads expenditures from 2013 to 2018, the average annual amount 67 spent on maintenance of rural roads would have been S/. 282.62 million, which is 95 percent more than the annual transfers from MEF of S/. 145.16 million in the same period. Table C.5. Total Rural Road Expenditures from 2010 to 2018 and Estimated Maintenance Expenditures as Portion of These Expenditures, 2013–18 Annual Year 2010 2011 2012 2013 2014 2015 2016 2017 2018 Average Total road 1,473.47 1,182.05 616.79 1,459.70 1,917.39 1,748.18 1,921.62 2,138.37 2,926.91 1,709.39 expenditures (S/., millions) Increase/decrease −20 −48 137 31 −9 10 11 37 in total road expenditures (%) Estimated 204.36 268.43 244.75 269.03 299.37 409.77 282.62 maintenance (14% of total road expenditures) (S/., millions) Maintenance 79.10 128.88 156.56 97.15 281.53 127.74 145.16 transfers from MEF (S/., millions) Percent 95 additional to transfers from MEF Source: Elaboration by IEG based on Siaf Consulta Amigable. Note: MEF = Ministry of Economy and Finance; S/. = nuevos soles. Adequacy of Maintenance Expenditures on the Rural Network IEG used the same type of rough calculations and cost data for the Regional Project to assess the adequacy of the maintenance on the rural network. As seen in table C.6, this calculation shows that the transfers from MEF since 2013 would be enough to maintain annually about 55 percent of the core network of 24,000 kilometers improved since 1995 under World Bank projects. In terms of periodic maintenance, it would take 20 years to cover the network of 24,000 kilometers instead of the desirable five years. 68 Table C.6. Network Receiving Routine and Periodic Maintenance Given MEF Transfers, 2013– 18 (S/.) Kilometers of Kilometers Years to Average Annual Core Network Average Cost Intervened Complete Road and MEF Transfers Upgraded Mainly of Routine Annually Percent Maintenance Maintenance 2013 to 2018 by World Bank Maintenance Given MEF of on Core Types (S/.) Projects (S/.) Transfers Network Network Routine 125,562,174 24,000 9,429 13,317.00 55 Periodic 58,797,458 24,000 51,667 1,138 5 20 Source: Elaboration by IEG based on Siaf Consulta Amigable; Diagnostico y Propuesta de Politica para el Mantenimiento de la Red Vial Vecinal; Peru 2017 Public Expenditure Review. Note: The rural network improved mostly under World Bank–financed projects measures 24,000 kilometers. S/. = nuevos soles. Table C.7 shows that, if it is assumed that 14 percent of the overall rural roads expenditures are invested in maintenance on the core network of 24,000 kilometers and the ratio between periodic and routine maintenance is the same as for the transfers from MEF, the expenditures would be enough to annually maintain 85 percent of this network. Periodic maintenance could be completed within 6.5 years instead of five years. Table C.7. Network Receiving Routine and Periodic Maintenance Based on Allocation of 14 Percent of Total Rural Roads Expenditures, 2013–18 (S/.) 14 Percent Kilometers of Total of Core Years to Rural Roads Network Average Complete Expenditures Upgraded Cost of Kilometers Periodic (Average Mainly by Routine to be Percent Maintenance Unpaved 2013–18) World Bank Maintenance Intervened of on Core Maintenance (S/.) Projects (S/.) Annually Network Network Routine maintenance 192,482,694 24,000 9,429 20,414.52 85 unpaved Periodic maintenance 90,134,574 24,000 51,667 3,755.61 16 6.25 unpaved Source: Elaboration by IEG based on Siaf Consulta Amigable; Diagnostico y Propuesta de Politica para el Mantenimiento de la Red Vial Vecinal; Peru 2017 Public Expenditure Review. Note: S/. = nuevos soles. In both cases, the maintenance expenditures are far too low to maintain the overall rural roads network of approximately 100,000 to 120,000 kilometers. 69 Appendix D. Evaluation of the Quality of the Impact Assessment Introduction The 2013 impact assessment (the collection of the endline and the evaluation of the results) for the Decentralized Rural Transport Project, the subject of this performance assessment, was carried out by Macroconsult-Cuanto 2013. The methodology and results are described in the 2014 final report Elaboración de la Evaluación de Impacto y Ampliación de la Línea Base del Programa de Transporte Rural Descentralizado Informe Final (henceforth Macroconsult Final Report). However, the description of the methodology is not comprehensive and Independent Evaluation Group (IEG) had several discussions and exchanges with Marcroconsult. As seen subsequently, these still left some doubt and uncertainties. The 2013 impact assessment covers rural road rehabilitation and maintenance and nonmotorized transport (NMT) track improvements. It does not cover the activities under the local development window. The 2013 impact assessments looked separately at rural roads and NMT tracks. Impact assessments had also been conducted for the First Rural Roads Project from 1995 to 2001 and the Second Rural Roads Project from 2002 to 2006, both cofinanced by the World Bank. These projects were similar to the Decentralized Rural Transport Project, the latter being a continuation and expansion of the formers. For the First Rural Roads Project, an impact assessment was conducted in 2000 using a cross- sectional methodology that only compared households near control and treatment roads after the interventions because an ex ante baseline had not been collected. In 2004, the households surveyed in 2000 were surveyed again to quantify longer-term effects. In 2006, a new impact assessment used the ex ante household data collected in 2004 and the ex post data collected in 2006 to evaluate the short-term impacts of the Second Rural Roads Project. It also resurveyed the households that took part in the 2004 impact assessment for the First Rural Roads Project, households first surveyed in 2000, to assess the sustainability of the earlier results. Finally, it collected the baseline for the Decentralized Rural Transport Project. These previous impact assessments were carried out by different consultants. The baseline for the 2013 impact assessment of the Decentralized Rural Transport Project was collected by the firm GRADE in 2006. The respective information is included in the 2006 final report Elaboracion de la Evaluacion de Impacto Economico, Social, Institucional y Ambiental del Program de Caminos Rurales, Informe Final, Linea de Base (henceforth GRADE Final Report). 70 IEG’s evaluation of the 2013 impact assessment described in the Macroconsult Final Report aims at ascertaining the reliability of the results. Methodology Used The 2013 impact assessment consisted of resurveying the households that had originally been surveyed in 2004 and 2006 to collect household and village-level data. It used the difference-in- differences methodology, which compares how an impact variable changes after a road or track improvement in the treatment group compared with the control group. The adequacy of this methodology critically depends on how well the control group represents the counterfactual changes that would have taken place in the treatment group without the improvement. The 2013 impact assessment of the rural roads and NMT improvements looked at many dependent variables to measure the impact at the household and village levels, which included transport-related data (for example, travel times and cost), access to health and education, access to other services, income and employment, productive activities, expenditures and poverty, and social capital. These independent variables are in line with the basic theory of change and the expected project outcomes. Selection of Treatment and Control Groups for the 2013 Impact Assessment The Macroconsult Final Report does not provide details on the selection of the control and treatment groups. It only mentions that the selection of the control groups in 2004 and 2006 used a careful process consistent with the search for similarities in terms of road and NMT track sections and population centers in the treatment group. The GRADE Final Report with the 2006 baseline for the 2013 impact assessment clarifies that the 2006 baseline for both the control and treatment groups were selected ex ante, but not simultaneously. Treatment Group Similar to the Macroconsult Final Report, the GRADE Final Report using the 2006 baseline does not mention how the roads in the treatment group were selected and calls these roads the “prioritized roads.” This could indicate that the roads were prioritized according to some criteria and were not randomly selected. Macroconsult told IEG that they had been informed that the treatment sections were selected randomly from a list of priority roads that needed interventions. The randomized assignment of the treatment generates the most accurate and unbiased results. However, this is unusual in rural roads projects because there is nearly always a political dimension in the road selection at 71 play. For the impact assessment of the ongoing Peru Support to the Subnational Transport Program Project, the project implementation agency rejected this option. IEG came across a 2007 research paper (“Connecting Rural Communities for Development: An Impact Evaluation of a Rural Roads Program in Peru”) by Paola Vargas (2007), who was part of the 2006 baseline study. This study mentions that for each round of surveys (referring to 2000, 2004, and 2006), a set of rural roads and NMT tracks to be treated were randomly selected to be included in the evaluation study. The random selection was multistaged and stratified by region. However, it is not explicitly stated if this applies to the impact assessments of the previous two rural road projects or also to the baseline for the Decentralized Rural Transport Project collected in 2006. Control Group For the control groups, roads and NMT tracks with similar characteristics to the ones in the treatment group were chosen in terms of observable variables. This was done separately for roads and NMT tracks. When selecting the control groups, the control roads and NMT track sections were identified first. This was followed by the selection of the population centers and the identification of the households to be interviewed along these roads or tracks. The control group selection followed the methodology used in the previous impact studies. The roads and tracks in the control group needed to comply at least with the following: (i) the road or track was not part of the rural road program and hence was not planned to be intervened, (ii) it was the same type of road (rural road or NMT track), and (iii) there was no intersection with a treated road or track to minimize the probability that benefits for treated population centers spill over to control population centers. In addition, the roads or tracks needed to have similar characteristics, such as (i) the length, (ii) population centers to be connected (for example, district or provincial capital, population size, access to basic public services and infrastructure, such as schools), (iii) agroclimatic conditions (for example, altitude, type of land, humidity, principal type of land use), (iv) function of the road or track (for example, connects a big center to markets, connects poor and remote areas with a smaller center, connects two bigger cities), (v) size and socioeconomic characteristics of the population, and (vi) public investments to be carried out in the area. The consultant used the following main steps to select the roads in the control group: (i) prepare a provincial map with all roads and tracks and the population centers they connect, (ii) identify the treatment roads and tracks on the map and the size of the population centers connected, (iii) identify on the map all the roads and tracks that have the same pavement type, (iv) identify clusters of roads and tracks in a province with the same pavement type that connect the same type of population center as the roads and tracks in the treatment group, (v) eliminate the sections that intersect with sections in the treatment group, (vi) identify sections that have a 72 similar length as the sections in the treatment group, and (vii) make sure the control sections have the same function. This procedure required a few adjustments. For instance, in some cases, it was not possible to find an equivalent control section in the same province as the treatment province. In this case, a control section was chosen from a neighboring province. In the GRADE Final Report and other documents reviewed by IEG, it was not mentioned how the population centers and the households to be interviewed along the roads and tracks were chosen, and how comparability between control and treatment groups was ensured. Subsequently, GRADE checked if the households and population centers associated with identified roads and tracks in the control group were similar in terms of socioeconomic characteristics (for example, size and composition of household, education, native language, assets) and the other relevant indicators as the ones in the treatment group. This was done by calculating pretreatment means for the treatment and control groups. For many sociodemographic and transport variables, the treatment and control groups seemed similar, but there were dimensions with significant differences. No significant differences were observed between the treatment and control groups in terms of variables such as the age composition of the population, the level of schooling, the percentage of households with women as head of household, or the percentage of head of households who spoke a native language. There were also no significant differences in the level of monthly income per capita, although there were differences in the per capita expenditures and the poverty rate, which was 6 percent higher in the control group than in the treatment group. For education and health, there were small, but statistically significant, differences in the school attendance rate and the rate of adult illness. In the case of the school attendance rate, the attendance was higher in the control group than in the treatment group, especially among girls aged between 12 and 18 years. In that category, the control group had an attendance rate of 83 percent, whereas in the treatment group it was 77 percent. For health, the rate of chronic diseases was also slightly higher in the treatment group (17 percent) compared with the control group (14 percent). There were also significant differences within the GRADE Final Report between households and population centers in each of the different regions and recommends for the future impact analysis to take the interregional differences into account when analyzing the results. Finally, GRADE compared the treatment groups of 2004 and 2006 to make sure there were no substantial differences between them. Among others, this showed that the baseline of 2006 had a larger proportion of NMT tracks. For the 2004 baseline, the proportion of NMT tracks to rural roads was 1:3; for the 2006 baseline, it was 1:2. This is because the Decentralized Rural Transport Project included a larger proportion of NMT tracks than previous projects. 73 Sample Size for 2013 Impact Assessment The sample size of the 2013 impact assessment was affected by attrition and the contamination of control groups. Combination of Data Sets Since attrition significantly reduced the sample size for both rural roads and NMT, the impact assessment combined the baselines and endlines of the Decentralized Rural Transport Project (assessed by IEG) and the Second Rural Roads Rehabilitation and Maintenance Project. The baseline and endline for the Second Rural Roads Project were collected in 2004 and 2006, respectively. The baseline for the Decentralized Rural Transport Project was collected in 2006. The endline for this project was carried out by Marcoconsult in 2013, by interviewing all households not affected by attrition and those previously interviewed for the Second Rural Roads Project and the 2006 baseline for the Decentralized Rural Transport Project. Macroconsult pointed out in the meetings with IEG that the combination of the two data sets was reasonable because the characteristics of both projects were similar, with one project being a continuation of the other. Due to the similar characteristics, geographical distribution of the works across the country, and the same survey design for the impact assessment, Macroconsult claimed that the results were valid for both projects. Attrition Attrition at the time of the 2006 endline survey was 47 percent compared with 2004. For the 2013 endline survey, it affected 40 percent of the sample of the 2006 survey. IEG reproduced graphs from the Macroconsult Final Report (24 and 25) correcting a mistake in the 2013 sample sizes (which was confirmed by Marcroconsult) (see table D.1). Based on this reproduction, after attrition, for rural roads the total households available in 2013 appeared to be 744 (486 from the 2004 and 2006 surveys + 258 from the 2006 surveys) in the treatment group and 742 (488 from the 2004 and 2006 surveys + 254 from the 2006 surveys) in the control group. Table D.1. Households Surveyed for Rural Roads in 2004, 2006, and 2013 Source: IEG, adapted from Marcroconsult Final Report Note: Red = sample of households that only appears in 2004; green = sample that appears in 2004 and 2006; yellow = sample that appears in 2004, 2006, and 2013; orange = sample that appears in 2006 and 2013; blue = sample that appears only in 2006. 74 For NMT track (table D.2), the total households available in 2013 appeared to be 442 (71 + 164) in the treatment group and 399 (162 + 237) in the control group. Table D.2. Households Surveyed for NMT Tracks in 2004, 2006, and 2013 Households NMT tracks Total Treatment Total Control 2004 Survey 2006 Survey 2013 Survey 2004 Survey 2006 Survey 2013 Survey 2004 Survey 2006 Survey 2013 Survey 153 73 80 342 342 71 71 71 71 326 326 326 164 164 164 162 162 162 516 516 279 279 237 237 322 153 169 Source: IEG, adapted from Marcroconsult Final Report Note: Red = sample of households that only appears in 2004; green = sample that appears in 2004 and 2006; yellow = sample that appears in 2004, 2006, and 2013; orange = sample that appears in 2006 and 2013; blue = sample that appears only in 2006. According to the Macroconsult Final Report, the color coding has the following meaning: red is the sample that only appears in 2004; green is the sample that appears in 2004 and 2006; yellow is the sample that appears in 2004, 2006, and 2013; orange is the sample that appears in 2006 and 2013; and blue is the sample that appears only in 2006. However, this color coding is not clear, and there is still uncertainty about the number of households available for rural roads and NMT tracks in 2013. Potential reasons for the high level of attrition include (i) the long period between baseline and endline (seven and nine years); (ii) lack of survey protocols or adherence to them (Macroconsult mentioned that the addresses, which were not formal street addresses but references to geographic points (for example, the second house next to a river in a certain area), were generally well recorded and that they checked with neighbors if they had difficulties in finding households); (iii) interviewee fatigue (some interviewees were surveyed three times), and (iii) migration. According to Macroconsult, the level of attrition was not abnormal for Peru in periods of seven and nine years, but that it caught their attention and therefore they further analyze it given the reduction in sample size and other potential consequences. The results of the analysis, summarized in tables 2 and 3 within the Marcoconsult Final Report (27 and 28) showed that the attrition rate was not statistically different between treatment and control households. There was some statistical significance for the 2004 data (27), showing that attrition was correlated with poverty and household size, and to a lesser extent with the available resources, but this was not confirmed in the 2006 data (page 28). The report mentions that “these initial results are important because they seem to show that the exit of households from the sample due to attrition is not systematically correlated with the treatment status, even though it is with some other variables” (26). In other words, attrition does not seem to be systematically affected by the treatment. 75 When looking at variables correlated with attrition separately for the control and treatment groups at baseline (30 and 31), Macroconsult found that for rural roads, in the 2004 baseline, households in the treatment group affected by attrition were not as poor as the households in the control group. This was not the case with the 2006 baseline, for which households in the treatment group affected by attrition seemed to have a greater chance of being from an area with productive forests and greater investments in transport and rural roads than households in the control group. For NMT tracks (31), households in the treatment group from the 2004 baseline unable to be located seemed to have less income per capita than those in the control group. They also seemed to have fewer investments in health, transport, and water and electricity in the area where they lived than the households in the control group unable to be located. For the 2006 baseline, households in the treatment group unable to be located seemed to possess more productive and transport-related assets and lived in a place with an existing agricultural program. Based on these findings, the Macroconsult Final Report concludes (4, 28, 29, and 91) that the correlation of pretreatment variables with attrition, especially for rural roads, indicates that at baseline households unable to be located in the treatment group were systematically less poor than those unable to be located in the control group. They also concluded that these households resided in places with more natural resources and investments than the ones unable to be located in the control group. The Macroconsult Final Report also mentions that if this is interpreted as evidence that the households that were lost due to attrition were those with the greatest capacity to generate wealth, then it is likely that the impacts of the project would be biased downward. To control for the differences generated by these characteristics, Macroconsult introduced them as covariates in the regressions (Macroconsult Final Report, 26). Contamination of Control Groups for Rural Roads The other problem encountered during the analysis of the results of the 2013 surveys was that rural roads in the control group were improved outside of the project by the national and local governments and by the private sector. Therefore, an engineering survey was carried out, but only for a subsample because of time and resource constraints. The subsample was obtained by randomly selecting 42 provinces and then randomly selecting a control road and a treatment road for each province. This led to a subsample of 82 roads—42 in the control group and 40 in the treatment group—which represented nearly 50 percent of the total sample. The control and treatment roads were similarly distributed between the 2004 and 2006 baselines. 76 The engineering survey found that 50 percent of the road sections in the control group had received substantial interventions (eight had been improved to higher standards than simple rehabilitation, ten had been rehabilitated, and three had received other interventions). To mitigate the inadequate control issue, the intervened controls were dropped. This left 21 road sections in the control group that had not been intervened at all or only to a minor extent. Had all roads that received some type of maintenance been taken out, it would have left only 16 road sections in the control group. To make the most out of the remaining data and correct for biases in the assignment of controls, propensity score matching was used to find the most adequate controls in the difference-in- differences framework. This approach consisted of assigning each observation (control and treated) with a probability of having been intervened that was modeled based on observable characteristics (the propensity score) and weighting the control group using these probabilities. As a result, the analysis assigned a greater weight to those controls that were the most similar to the treated group (higher propensity score) and forced those in the control or treated groups with very high or very low probabilities of treatment to be dropped from the data set. This left a final “clean” subsample of 18 control road sections with 36 population centers and 205 households in the control group and 79 treatment road sections with 152 population centers and 1,133 households in the treatment group (Macroconsult Final Report, 49). IEG requested a clarification from Macroconsult on how the total number of households in this clean subsample in the treatment group could be 1,133 if, according to the table above, the number of households in the treatment group available in 2013 was only 744. The explanation from Macroconsult was not clear, but they confirmed that the final number of households in the treatment group was 1,133 and that they had not combined the data sets for rural roads and NMT tracks. Macroconsult used the clean subsample to estimate the impact of rural roads instead of the full sample considering its greater reliability. For NMT tracks, Macroconsult assumed that they were not improved outside the project since nobody except the implementation agency was responsible for these tracks. No engineering survey was carried out, and IEG assumed that the full sample of 443 households belonging to the treatment group and 399 households belonging to the control group was used for the impact analysis. Considering Macroconsult discovered the problem with the controls late in the process, the impact analysis with the clean subsample for rural roads was not as extensive as the analysis for NMT tracks. 77 Results of 2013 Impact Assessment To understand the results of the impact assessment is challenging because they are presented in different ways in different places of the Macroconsult Final Report. In addition, two analyses were carried out and presented for rural roads: one for the complete sample with the contaminated control groups and one for the clean subsample. For rural roads, the results of the analysis for the “clean” subsample are presented in the summary table (6 and 7) and on pages 50 to 55 of the Marcoconsult Final Report. For these tables, only pretreatment average values of the treatment and control groups, the estimated coefficient, and the standard error are given. Macroconsult informed IEG that because the analysis was carried out late in the process, they only had this information and they did not have the full data. The complete analysis for rural roads using the full sample with the contaminated control groups is in annexes 3 and 4. Due to the control group contamination, this information is not reliable and hence useless. For the NMT tracks, the results are presented in the summary table (6 and 7), on 56 to 86, and in Annex 4 of the Macroconsult Report. The presentation differs slightly in each place. Other problems encountered in results reporting include different units of measurement (for example, hours and minutes) in different tables, no specification of the unit of measurement, and a lack of definition of the different impact variables tested. All this seems to indicate that the report was put together in a hurry. Table D.3 presents a summary of the most significant results of the impact assessment for both rural roads and NMT tracks based on the following simplified causal chain: road and NMT track improvements are expected to produce travel cost and time reductions, which in turn are to increase people’s access to social services, workplaces, and market opportunities. This should then be reflected in enhanced territorial development by affecting land use, production patterns, and other aspects, and increased income and expenditures, and ultimately less poverty. Table D.3. Results of the Impact Assessment Rural Road Rehabilitation NMT Track Improvements Estimated Standard Estimated Standard Impact Coefficient Error Coefficient Error Travel time and cost Travel time to the nearest regional market 1.16 (0.63)* −8.90 (12.43) (hours) Travel time to district capital (hours) 1.27 (0.50)** −1.27 (0.46)*** Travel time to work (min) −0.74 (1.97) −13.80 (6.55)** Travel time to point of sale (unit n.a.) −26.27 (10.18)*** n.a. n.a. Travel time to farm (unit n.a.) −78.82 (25.87)*** n.a. n.a. 78 Rural Road Rehabilitation NMT Track Improvements Estimated Standard Estimated Standard Impact Coefficient Error Coefficient Error Travel time to school for students between −5.60 (3.57)* 7.30 (6.19) 12 and 18 years (min) Travel time to health center (min) 4.18 (8.85) −5.80 (3.67) Travel time to health center by vehicle (min) −22.21 (23.19) n.a. n.a. Fare of microbus (small bus) −10.72 (2.57)*** n.a. n.a. Freight tariff 5.54 (3.25)* n.a. n.a. Access Frequency of service by car (no. of cars per 17.97 (8.54)** n.a. n.a. week) Days of road closure per year −21.25 (13.17)* 6.50 (6.18) School attendance for students between 12 0.16 (0.04)*** 0.02 (0.04) and 18 years (percent) Medical consultation (percent) 0.10 (0.06)* 0.19 (0.05)*** Land use and production patterns Hours worked per week in principal 2.83 (1.46)* 1.58 (1.71) occupation Cultivated land (ha) 0.36 (0.11)*** −0.24 (0.13)* Cultivated land (percent) 0.04 (0.04) 0.04 (0.05) Fallow land (ha) −0.42 (0.22)* −0.11 (1.17) Fallow land (percent) −0.14 (0.02)*** −0.03 (0.03) Forest surface (ha) 1.36 (0.57)** −0.54 (0.35) Forest surface (percent) 0.05 (0.02)** −0.01 (0.02) Other land (percent) −0.02 (0.01)** 0.00 (0.01) Bovine production (number, average) 0.17 (0.04)*** 0.06 (0.08) Bird production (number, average) 0.12 (0.04)*** −0.06 (0.08) Rabbit production (number, average) 0.11 (0.04)*** 0.03 (0.08) Public institutions (no.) n.a. n.a. 0.57 (0.25)** Private institutions (no.) n.a. n.a. 0.51 (0.27)* Health centers (percent of villages) n.a. n.a. 0.19 (0.07)*** Radio stations (percent of villages) n.a. n.a. 0.30 (0.11)*** TV channels (percent) n.a. n.a. 0.27 (0.11)*** Income and Expenditures Percentage of agriculture production sold −0.05 (0.03)* n.a. n.a. Percentage of agriculture production sold in −0.14 (0.04)*** −0.03 (0.05) market Percentage of agriculture production sold at 0.10 (0.05)* n.a. n.a. the farm or to intermediary Price of wheat (S/. per kg) 0.56 (0.20)*** 0.73 (4.84) Price of potatoes (S/. per kg) 0.03 (0.15) 0.10 (0.05)** 79 Rural Road Rehabilitation NMT Track Improvements Estimated Standard Estimated Standard Impact Coefficient Error Coefficient Error Price of peas (S/. per kg) 0.91 (0.16)*** Sales of agriculture, fishery and forest 337.87 (503.89) −73.0 (276.89) products (S/. per year) Farming income (S/. per year) 379.37 (378.31) 0.10 (256.70) Households with access to credit (percent) 0.00 (0.02) −0.03 (0.03) Monthly income per capita (2012 nuevos 9.46 (18.31) 17.96 (27.87) soles) Expenditures per capita 9.34 (12.30) −6.30 (11.91) Logarithm of expenditures per capita 0.16 (0.08)** n.a. n.a. Poverty Total poverty (percent) 0.02 (0.04) 0.05 (0.05) Extreme poverty (percent) −0.14 (0.04)*** −0.05 (0.06) Structural poverty (NBI) (percent) −0.07 (0.04)* 0.05 (0.05) Source: Elaboration by IEG based on Marcroconsult Final Report Note: Key results not as expected are marked in bold and italic. ha = hectare; n.a. = not applicable; S/. = nuevos soles; NBI = Necessidades Básicas Insatisfeitas (unmet basic needs). *10% significance **5% significance ***1% significance The results reported in the table are the results obtained from the simple fixed effect model, without control variables. These results were also used by Macroconsult in their summary reporting. Most statistically significant variables were significant across the different specifications of the model, which confirms their robustness. For rural roads, the results seem to largely confirm the simplified causal chain mentioned previously and indicate a contribution of the project to territorial development and poverty reduction, but there were also some unexpected results. There is statistically significant evidence that travel times to certain destinations, such as point of sale, farm, and school, decreased. However, travel times to the nearest market increased. No statistically relevant reduction in travel time to health centers was observed. Microbus fares also decreased, but freight tariffs increased. The increase in transport services by car is also statistically relevant. Statistically relevant increases in school attendance of students aged between 12 and 18 years and in medical consultations were observed, which could be the result of better transport conditions. Similarly, in terms of production patterns, the increase in land surface under cultivation, and the decrease of fallow land, were also statistically relevant. A statistically relevant increase in cattle and other farm animals was also observed. 80 The percentage of agriculture production sold and sold in the market went down, whereas production sold at the farm or to an intermediary went up. Prices for most agricultural products also showed statistically relevant increases. Although the logarithm of expenditures per capita increase was statistically significant, there was no increase in expenditures per capita, farming income, or monthly income. Finally, the reduction in both extreme and structural poverty was also statistically relevant. For NMT tracks, no statistically relevant difference was found for most variables, including poverty reduction. As for rural roads, there was a statistically relevant reduction in some travel times to certain destinations, such as work or district capitals. However, no statistically relevant reduction in travel time to markets, schools, or health centers was found. Despite this, the number of health consultations increased. No statistically relevant increase in school attendance was found. Land use patterns also did not change, except for a reduction in the cultivated land, which was statistically relevant. There were statistically relevant increases in the number of public and private institutions in villages and the percentage of health centers, radio stations, and television channels. There was no statistically relevant increase in animal production, and only the price for potatoes increased in a statistically relevant way. In line with the pattern of the overall results, there was no statistically relevant increase in income and expenditures, and no impact on poverty. Reliability of the Results of the 2013 Impact Assessment The 2013 and the previous impact assessments are laudable efforts to demonstrate the impact of rural roads and NMT track improvements on poverty reduction. Considerable efforts went into making sure that the same methodology was followed over nearly 15 years, but the process was not exempt from serious problems. The Marcoconsult Final Report does not comment on the impact of these problems and the necessary methodological changes and shortcuts on the reliability of the findings. This Report is also not comprehensive in the description of the methodology used and leaves several doubts and uncertainties. IEGs believes that the selection of the treatment and control groups was sound. It seems that the treatment groups for rural roads and NMT tracks were randomly selected even if IEG could not confirm it with certainty. To use random selection would be good practice. The selection of the control groups followed the methodology of the previous impact assessments, which seems sound. The pretreatment means showed, in general, that households and population centers associated with identified roads or tracks in the control groups were similar in terms of socioeconomic characteristics and other relevant indicators to the ones in the 81 treatment groups. However, there were some significant differences, especially in terms of extreme poverty. This is not considered a problem because homogeneity is not essential with the difference-in-differences approach. Comparing the treatment groups of 2004 and 2006 showed that the baseline of 2006 had a larger proportion of NMT tracks. Again, since the analyses for rural roads and NMT tracks were done separately, this is not an issue. In terms of sample size, given the large number of households lost due to attrition, combining the 2004 and 2006 data set of the Second Rural Roads Project with the 2006 and 2013 data set of the Decentralized Rural Transport Project seems reasonable. The two projects are identical in terms of interventions, and the combined data sets considerably increase the number of observations and, consequently, the chances of finding statistically significant results. However, the results are valid for both projects combined; therefore, it is not possible to isolate the results of the Decentralized Rural Transport Project, which is only subject of this performance assessment. Attrition was not found to be different between the control and treatment groups. When looking at variables correlated with attrition separately for the control and treatment groups at baseline, it seems that, especially for rural roads, households that could not be located were systematically less poor in the treatment group than in the control group. The households in the treatment group not located also resided in places with more natural resources and investments than the ones not located in the control groups. The Marcroconsult Final Report argues that if this is interpreted as evidence that the households that were lost due to attrition are those with the greatest capacity to generate wealth, the results of the impact assessment could be biased downward. The reasoning behind this argument is not fully clear. The biggest challenge encountered in the 2013 impact assessment seems to be the contamination of the control group for rural roads. After efforts to make up for the loss in households in the control group were made, including propensity score matching, according to Marcoconsult, the sample for analysis consisted of 205 control households in the control group and 1,133 households in the treatment group. This indicates a significant imbalance between the control and treatment groups and a very small sample size in the control group for a project with nationwide geographic scope and aimed at picking up very sensitive impacts far down the direct causal path of rural roads improvements. Therefore, IEG believes that the previously mentioned sample sizes, especially for the control group for rural roads improvement, were too small to reliably affirm that the projects contributed to poverty reduction. This is an impact far down the direct causal pathway, hence more sensitive and difficult to measure. 82 Appendix E. Field Visits Rural Project: Roads Huallchanca–Chuschi, Cajamarca (31.4 kilometers) Interventions carried out: Section rehabilitated as nonpaved road under the Rural Project in 2010. In 2014, approximately half of the section was paved under the “Proyecto Peru,” an initiative of the Peruvian government, under which they paved several subnational links without changing the road structure or alignment. This was a first step toward a paved road with the recommended width and characteristics when the traffic was enough to justify it. Current road classification: The paved section was reclassified as a national road; the unpaved section as a regional road. Road dimension: Single lane (average 4.5 to 5 meters). Pavement condition: The asphalt pavement was generally in good condition with some minor potholes and rutting. There were problems with asphalt pavement edges. On the unpaved part of the section, the pavement was also in relatively good condition. Maintenance: The asphalt pavement showed signs of repairs. The drainage was not well maintained. 83 Traffic: During the visit, there was light traffic: one car, an interprovincial bus, a minibus, some motorcycles, and people walking. Average speed: About 60 kph on the paved section and about 40 kph on the unpaved part. Safety: Traffic speeds on the paved part increased. Since the road alignment was not changed, the road is now more dangerous. 84 Divisa La Cocha–La Cocha, Contumazá (5.5 kilometers) Interventions carried out: Section received periodic maintenance as nonpaved road under the Rural Project in 2013. Current road classification: The road is classified as a rural road. Road dimension: Single lane (average 4.5 meters). Pavement condition: The pavement was generally in good Chilete condition, except for the sections where the water from the mountain crosses the road, and some small landslides have occurred. Maintenance: The 2019 maintenance contracts have not yet started. People living along the road confirmed that the road Contumaza received regular maintenance in the past. One farmer pointed out that his wife was part of the maintenance crew. Traffic: During the visit, there was minimal traffic: one car, a donkey, and some people walking. Average speed: About 30 to 35 kph. Safety: Since the road is unpaved, speeding is not a problem. 85 86 Contumazá–Las Totorillas–Catán, Contumazá (48.2 kilometers) Interventions carried out: Section received periodic maintenance as nonpaved road under the Rural Project in 2013. Current road classification: The road is classified as a rural road. Road dimension: Single lane (average 4 to 4.5 meters). Pavement condition: The pavement in several parts was in bad condition with potholes and mud. Due to heavy rains, the road was not passable after approximately 4 kilometers. Maintenance: The 2019 maintenance contracts have not yet started. People living along the road confirmed that the road received routine maintenance last year. Traffic: During the visit, there was minimal traffic: one car, a motorcycle, and some people walking. Average speed: About 30 kph. Safety: Since the road is unpaved, speeding is not a problem. 87 88 San Juan–Cachilgon–Ogoriz, Cajamarca (10.5 kilometers) Interventions carried out: Section rehabilitated as nonpaved road under the Rural Project 2012. Current road classification: The section is still classified as rural road. Road dimension: One lane only (average 4.5 meters). Pavement conditions: The pavement was in fair condition with problems where the water crosses the road. Maintenance: This year’s maintenance contracts have not yet started; especially the drainage system was in urgent need of cleaning. In a few selected locations, there were signs that somebody had cleaned the drainage. Local people confirmed that last year the road received manual routine maintenance and that about two years ago, it received mechanized routine maintenance. Traffic: During the visit of about 20 to 30 minutes, there was one minibus and several people walked along the road. Average speed: About 30 to 35 kph. Safety: Traffic speeds are low, and there is limited traffic, so road safety seemed no issue. 89 Rural Project: NMT Tracks Huaccanccasa–Urihuana–Tucsen–Paucayocc–Puente Sol de Oro, Cangallo (13.6 kilometers) 90 Interventions carried out: The track was improved under the Rural Project in 2013. IEG discussed the use of the section from Tancarpampa to Urihuana with local person living next to the track. Condition of the track: IEG was not able to visit the track because of the rain. Use of the track: Urihuana is now reachable by road, hence the track is only used for the works in the nearby field and by people who are not willing or able to pay the fare for the minibus. Maintenance: The community meets once a year to maintain and improve the track. The public sector does not take care of it. 91 Cajamarca Granja Porcon and Accesses (34.5 kilometers) 92 Interventions carried out: The track was improved under the Rural Project. IEG visited part of the track that is now interrupted about every 100 meters by a national road and discussed its use with local people. Condition of the track: The track was in good to fair condition. Use of the track: People use the track mainly to reach the main road to catch a minibus and to bring their children to the kindergarten. Maintenance: The municipality carries out some maintenance activities once a year. 93 Regional Project: Roads Cangallo–Huancapi, Cangallo (22.2 kilometers) Interventions carried out: Section rehabilitated as nonpaved road under the Regional Project between 2009 and 2010. In 2014, it was paved under the “Proyecto Perú,” an initiative of the Peruvian government, under which several subnational roads were reclassified as national roads and paved without changing their structure or alignment. Five-year contracts for low-cost paving and maintenance, using a performance-based approach, were used. Current road classification: The road was reclassified as national road. Road dimension: One lane (average 4.5 to 5 meters). Pavement condition: The pavement was generally still in good condition without potholes. However, there was some accumulation of water on the pavement surface and the shoulders. There were problems with the pavement edges. The pavement also had problems in the areas were the water from the mountain crosses the road (no cross drainage). Slopes: Some very steep slopes and no protection measures. Maintenance: The pavement showed signs of well-done repairs, and signs were mostly intact. The drainage was not well maintained. 94 Traffic: During the visit, there was light car and truck traffic. Average speed: About 60 kph. Safety: Traffic speed has increased due to the paving of the road. However, since the alignment was not changed, it is more likely for accidents to happen. 95 Chilete–Contumazá, Contumazá (40 kilometers) Interventions carried out: Section rehabilitated as nonpaved road under the Regional Project in 2010. In 2014, it was paved under the “Proyecto Perú,” an initiative of the Peruvian government, under which they paved several subnational links without changing the road structure or alignment as a first step toward a paved road with the recommended width and characteristics when the traffic was enough to justify it. Current road classification: The road was reclassified as national road. Road dimension: One lane (on average 4.5 to 5 meters). Pavement condition: The pavement was generally in fair to good condition. However, there were problems in some areas were the water from the mountain crosses the road (limited cross drainage). Slopes: Many very steep slopes, no protection measures, and lots of landslides (three crews were working to keep the road open). Maintenance: The pavement showed signs of recent repairs, and road signs were still relatively intact. Traffic: During the visit of about an hour there was light traffic (three cars, three motorcycles, a small truck, and a minibus). Average speed: About 50 to 60 kph in very mountainous terrain. 96 Safety: Traffic speed has increased due to the paving of the road. However, since the alignment was not changed, and the road is in steep, mountainous terrain, the accident risk increased. Guzmango–San Benito–Limón–L. D. La Libertad (Ascope), Section: Guzmango–San Benito, Contumazá (25.7 kilometers) 97 Interventions carried out: Section rehabilitated as nonpaved road under the Regional Project in 2010. Current road classification: The road is classified as regional road. Road dimension: One lane (average 4.5 to 5 meters). Pavement condition: The pavement was in fair to good condition with some potholes and small landslides. Slopes: Some steep slopes and no protection measures. Maintenance: Lack of maintenance, especially as far as drainage systems and the removal of landslide material are concerned. A sign showed that the regional government was about to start periodic maintenance on the section. Traffic: During the visit of about 50 minutes there was light traffic (one minibus, three motorcycles, one small bus, and two bicycles). Average speed: About 30 kph since it was very mountainous. Safety: Traffic speeds are low, and there were some road safety measures. 98 99 Appendix F. List of Persons Met Name Title Organization Miguel Castro Engineer Provias Descentralizado Sonia Pezo Local Development Window Specialist Provias Descentralizado Aurelio Menendez Practice Manager World Bank Jorge Malvila Planning Specialist Provias Descentralizado Jose Carlos Palomino Provincial Road Institute Specialist Provias Descentralizado Abraham Vilca Monitoring and Evaluation Specialist Provias Descentralizado Nelson Loarte Microenterprises Specialist Provias Descentralizado Marta Huamanga Local Development Window Specialist Provias Descentralizado Jose Inciso Work Management Division Provias Descentralizado Griselle Felicita Vega Senior Agriculture Specialist World Bank Juan Tapia Transport Consultant Cidatt Consulting Hugo Brousset Chaman Social Protection Specialist World Bank Daniel Francisco Barco Rondan Lead Economist World Bank Guerra Garcia Picasso, Gustavo Consultant DEE Consultores Benjamin Gutierez Leguia General Coordinator Fondo Para La Inclusión Económica de Zonas Rurales Fabiola Caballero n.a. Ministry of Transport and Communication Juan Cardenas Responsible for Decentralization Ministry of Transport and Communication Luis Cabezas Maintenance Division Provias Descentralizado Eddy Arones Works Division Provias Descentralizado Carols Daniel Figueroa Consultant n.a. Richard Webb Professor n.a. Raffael Capristan Transport Specialist Inter-American Development Bank Pompeo Mallma Morales Regional Director Program for Productive Rural Development Ayacucho, Ministry of Agriculture Messias Julca Trisolin Coordinator of Zonal Division Provias Descentralizado Ayacucho Jony Antonio Quispe Poma Director Direccion Regional de Transporte Ayacucho Moises Gutierrez Submanager Gerencia de Infrasestructura de Ayacucho Guido Diaz Martinez Manager Gerencia de Planeamento Vilcashuaman Wilhelm Gilberto Ore Chipana Manager Gerencia de Infrasestructura de Ayacucho Mario Cueto Cardenas Engineer Member of Engineering College 100 Eleodoro Tenorio Prado Manager Provincial Road Institute Cangallo Abeleno Guamani Perina Manager Maintenance Company Cangallo Elvis Galindo Tenolio Manager Maintenance Company Cangallo Fernando Anchona Quqi Manager Maintenance Company Cangallo Walter Hugo Huaman Lillagares Manager Gerencia de Infrasestructura de Cangallo Daniel Roca Silca Mayor Municipality of Cangallo Edwin Ruben Manager Gerencia de Desarrollo Económico de Cangallo Eduardo Flores Chanco Submanager Gerencia de Desarrollo Social de Cangallo Edgar Pomasonco Rodriguez Manager Gerencia de Planeamento de Cangallo Carlos Santos Member Asociacion Agrolactea Los Morrochucos Hector Martinez Consultant Local Development Window Operator in Ayacucho Raul Polanco Coordinator of Zonal Division Provias Descentralizado Cajamarca Victor Abel Rodriguez Arana Manager Gerencia de Infrasestructura Regional de Cajamarca Luis Alberto Vallejos Portal Manager Gerencia de Planeamento Regional de Cajamarca Fernando Hernandez Director Proregion Cajamarca Messias Guevara Governor Cajamarca Region Manuel Lorenzo Romero Zarate Director Direccion Regional de Transporte Cajamarca Manuel Carlos Local Development Window Operator Centro de Estudios Desarrollo y Acción Social CEDAS Joaquin Izquierdo Manager Provincial Road Institute Contumazá Gladis Julca Paredes Manager Gerencia de Planeamento de Contumazá Noel Becerra Rojas Manager Gerencia de Desarrollo Social de Contumazá Wilfredo Avila Santiago Manager Gerencia de Desarrollo Económico de Contumazá Martin Plasencia Director Agencia Agraria Contumazá Segundo Gregorio Biuza Santos Manager Gerencia de Infraestructura de Contumazá Juana Kuramoto Undersecretary for Territorial Presidencia del Consejo de Ministros Development Alberto Rodriguez Country Director World Bank Ana Lucia Jimenez Financial Management Specialist World Bank Ximena Herbas Environmental Specialist World Bank 101 Raul Torres Former Coordinator of Provias, Consultant Elisabeth Dasso Social Safeguard Specialist World Bank Raul Tolmos Former World Bank Environmental Specialist Francesco Rodriguez Procurement Specialist World Bank Maria Margarita Nunez Former Task Team Leader Javier Morales Sarriera Economist World Bank Yohnny Campana Consultant Marcoconsult Alvaro Monge Consultant Marcoconsult 102 Appendix G. Borrower Comments From: Uriel Gonzalo Cisneros Castro Sent: Tuesday, June 18, 2019 5:54 PM To: Elisabeth Goller, Midori Makino, jmavila@proviasdes.gob.pe; 'America Masias' ; rguerrero@mef.gob.pe; ETOLEDO@MEF.GOB.PE Cc: 'MICHELL LUIS LEON ALARCON' ; crevilla@proviasdes.gob.pe; serv_de_09@proviasdes.gob.pe; mcastro@proviasdes.gob.pe [External] Buenas Tardes Sra. Elizabeth Goller (The World Bank) Asunto : Comentarios a Borrador de Informe de Evaluación Independiente Referencia : Informe de Evaluación del Desempeño de los Proyectos: Proyecto de Caminos Departamentales (IBRD-7423) y Proyecto de Transporte Rural Descentralizado (IBRD 7322). De acuerdo al Borrador de Evaluación independiente, no se tiene comentarios a lo ejecutado en los proyectos de la referencia, sin embargo, mencionamos lo siguiente como lecciones aprendidas para el Programa de Apoyo al Transporte Subnacional - PATS: Sobre Planes Viales Provinciales Participativos-PVPP, El PATS en su componente 3, considera el fortalecimiento de la gestión vial descentralizada hacia los gobiernos locales, y para el logro de los “PVPP“ como instrumentos de planificación vial para garantizar su sustentabilidad, ahora considera una interrelación (Transversal) entre el GL-MTC-MEF, para garantizar el desarrollo, implementación y mantenimiento de Proyectos de Infraestructura Vial, con el propósito de mejorar las condiciones de vialidad, mediante la integración de redes viales a los corredores logísticos y que sirvan para el traslado de bienes y personas en función de los intereses de la población. Por otra parte dentro del proceso de descentralización, se ha elevado al Congreso de la Republica un proyecto de Ley para la transferencia de Partidas Presupuestarias para que el GL con soporte del PVD elabore sus planes viales provinciales participativos y sea útil en la planificación del plan de desarrollo provincial y regional. Sobre el modelo de microempresas basadas en la comunidad para el mantenimiento rutinarios de caminos vecinales, dentro del proceso de descentralización ahora se realiza en cumplimiento 103 de las normas en las adquisiciones públicas del Estado Peruano (El Organismo Supervisor de las Contrataciones del Estado (OSCE) es la entidad encargada de velar por su cumplimiento). En el marco del PATS se viene fortaleciendo a los GL mediante el IVP, para que los beneficiarios a lo largo de la trayectoria intervenida participen en los procesos mediante la certificación aprobada por el Ministerio de Trabajo – MINTRA a los trabajadores en mantenimiento de caminos vecinales, y lograr que exista sostenibilidad en el mantenimiento de las vías, con implementación de nuevas tecnologías de soluciones básicas (Afirmado estabilizado y con Tratamiento superficial). Sobre el mantenimiento rutinario de los caminos vecinales, se viene proponiendo llevar adelante los procesos de convocatoria en el tercer trimestre del año para que inicie el servicio el primer día del siguiente año y así garantizar una vida útil al menos de 3 a 5 años. El PATS, también considera una evaluación de Impacto, en los caminos vecinales, con el fin de medir la incidencia en la reducción de la pobreza. En el marco del PATS, también se prevé ventana de desarrollo local (VDL) con un plan de acción y su implementación, y una guía metodológica para planes de desarrollo económico, liderados por los gobiernos locales. Atte. CPC. Uriel Gonzalo Cisneros Castro Tel. Central 5145300/1160 -Cel: 949354381 MTC- Provias Descentralizado Jr. Camaná 678 - Lima 1 piso 07- PATS 104