Document of The World Bank Report No: ICR00003027 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-76220) ON A LOAN IN THE AMOUNT OF US$4.4 MILLION TO THE UNITED MEXICAN STATES FOR A RESULTS-BASED MANAGEMENT AND BUDGETING PROJECT June 30, 2014 Poverty Reduction and Economic Management Colombia and Mexico Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective April 10, 2014) Currency Unit=Mexican Pesos US$ 1.00 = MXP $13.05 MXP $1 = US$ 0.076 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CONEVAL National Council for the Evaluation of Social Development Policy (Consejo Nacional de Evaluación de la Política de Desarrollo Social) GoM Government of Mexico IADB Inter-American Development Bank IEG Independent Evaluation Group IFAI Federal Institute for Access to Public Information and Data Protection (Instituto Federal de Acceso a la Información y Protección de Datos) IT Information Technology M&E Monitoring and Evaluation NAFIN Development Banking Institution (Financial Agent: Nacional Financiera, S.N.C.) NDP/PND National Development Plan (Plan Nacional de Desarollo) OECD Organization for Economic Co-operation and Development PBR Results Based-Budgeting (Presupuesto Basado en Resultados) PFM Public Financial Management PGCM Close and Modern Government Program (Programa de Gobierno Cercano y Moderno) PI Performance Information PMG Management Improvement Program (Programa de Mejora de la Gestión) PSBR Public Sector Borrowing Requirements RBB Results-based budgeting SE Undersecretary of Expenditures (Subsecretaría de Egresos) SED Performance Evaluation System (Sistema de Evaluacion del Desempeno) SFP Ministry of Public Administration (Secretaría de la Función Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) SIDAFF Integrated Federal Financial Management System (Sistema Integral de Administración Financiera Federal) SIGAIF Harmonized Governmental Financial Information System (Sistema Gubernamental Armonizado de Información Financiera) SISED Performance Evaluation Program Information System (Sistema de Información para el Sistema de Evaluación del Desempeño) Vice President: Jorge Familiar Country Director: Gloria M. Grandolini Sector Manager: Arturo Herrera Gutierrez Project Team Leader: Pedro Arizti ICR Team Leader: Pedro Arizti MEXICO Results-based Management and Budgeting Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 6 3. Assessment of Outcomes .......................................................................................... 11 4. Assessment of Risk to Development Outcome......................................................... 15 5. Assessment of Bank and Borrower Performance ..................................................... 15 6. Lessons Learned ....................................................................................................... 18 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19 Annex 1. Project Costs and Financing .......................................................................... 22 Annex 2. Outputs by Component ................................................................................. 23 Annex 3. Economic and Financial Analysis ................................................................. 26 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 28 Annex 5. Beneficiary Survey Results ........................................................................... 30 Annex 6. Stakeholder Workshop Report and Results................................................... 31 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 32 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 33 Annex 9. List of Supporting Documents ...................................................................... 34 MAP A. Basic Information MX Results-based Country: Mexico Project Name: Management and Budgeting Project ID: P106528 L/C/TF Number(s): IBRD-76220 ICR Date: 06/30/2014 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: TAL Borrower: MEXICO Original Total US$ 17.2M Disbursed Amount: US$ 0.37M Commitment: Revised Amount: US$ 4.4M Environmental Category: C Implementing Agencies: Secretaría de Hacienda y Crédito Publico (SHCP) Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/20/2007 Effectiveness: 03/19/2010 03/19/2010 Appraisal: 10/02/2008 Restructuring(s): 11/15/2012 Approval: 12/11/2008 Mid-term Review: 02/06/2012 Closing: 12/31/2013 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Moderately Quality at Entry: Government: Unsatisfactory Unsatisfactory Moderately Implementing Moderately Quality of Supervision: Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Moderately Overall Borrower Moderately Performance: Unsatisfactory Performance: Unsatisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Unsatisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 100 100 Theme Code (as % of total Bank financing) Managing for development results 50 50 Public expenditure, financial management and 50 50 procurement E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Gloria M. Grandolini Axel van Trotsenburg Sector Manager: Arturo Herrera Gutierrez Nicholas Paul Manning Project Team Leader: Pedro Arizti Roberto Adrian Senderowitsch ICR Team Leader: Pedro Arizti ICR Primary Author: Azul Del Villar F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective of the project is to establish that federal departments and agencies in Mexico provide decision makers and the public with rigorous, timely, user-friendly information on the efficiency and effectiveness of government organizations and program expenditures. Revised Project Development Objectives (as approved by original approving authority) The objective of the project is to strengthen the Ministry of Finance’s (Secretaría de Hacienda y Crédito Público, SHCP) capacity to (i) allow the use of standardized performance information of priority public programs during budget preparation; (ii) generate harmonized subnational fiscal information; and (iii) disseminate budget performance information to the Borrower's citizens on a regular basis. (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Performance information using a performance framework is generated and used Indicator 1 : during budget request discussions between line secretariats and SHCP. Performance information (PI) is used by SHCP during Performance budget information (PI) Performance is not widely preparation in presented in a used by line secretariats conversation timely fashion to in their budget requests. with line Congress in a Budget preparation ministries. format that allows Value discussions are mainly SHCP interpretation and (quantitative or focused on inputs and are uses meets their needs. qualitative) not based on standardized standardized Content includes performance information performance outcome and output that allow to measure information measures. Line relevance, efficiency of that allow to ministries send programs. measure budget requests to relevance, SHCP with PI. efficiency of effectiveness of programs. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Achieved achievement) An active network and a portal to inform civil society at large on a regular basis Indicator 2 : about budget performance increasing transparency and accountability to citizens. There is a portal containing Portal containing PI financial and Budget performance on budget non-financial information is readily performance is Value performance available to civil society available to citizens quantitative or information via the new portal but has + Citizen’s Budget Qualitative) about the not been expanded to all + Citizen’s Public budget and its levels of Government. Accounts (Cuenta process Pública) available for citizens, and the information has been expanded to all the levels of Government. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Achieved, the portal site address is: www.transparenciapresupuestaria.gob.mx achievement) Indicator 3 : Percent of public programs with improved strategic design and reporting The majority of the programs have a For the 2013 more robust design budget across key preparation the dimensions In 2008 the percentage of percentage of Value (objectives, public programs using programs quantitative or indicators, performance information using Qualitative) baselines, links to was 28%. performance strategies) enabling information is better reporting and set to be over evaluation. (832 75%. programs out of 984) Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Achieved at 84.5% achievement) Percent of municipalities that have a simplified financial management system Indicator 4 : installed and working. Around 80% of The Accounting approximately Unit changed Value 1,200 small priorities to focus quantitative or 0 municipalities on developing the Qualitative) (of total Government approximately Accounting System 2,500 (SCG) first. municipalities) Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Number of municipalities that have been trained in the use of the new municipal Indicator 1 : simplified FMIS. Pilot of Harmonized Governmental Financial Information System (SIGAIF) installed Around 80% in some Value of municipalities, not (quantitative approximately operational though. or Qualitative) 1,200 small The Accounting municipalities Unit changed priorities to focus on developing the Government Accounting System (SCG) first. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Functional design developed for the amplified FMIS that will cover the Indicator 2 : decentralized institutions in the Government. The Accounting SIDAFF has a Unit changed Value module priorities to focus (quantitative Non existent designed for on developing the or Qualitative) decentralized Government institutions. Accounting System first. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Design of a conceptual model for integration of financial information in Indicator 3 : municipalities, states and federation. SIDAFF has a The Accounting module Unit changed Value designed for priorities to focus (quantitative Non existent integrating all on developing the or Qualitative) three levels of Government Government. Accounting System first. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Design and dissemination of a number of strategic evaluations (e.g. security Indicator 4 : policy, water policy, education). Strategic social and Three strategic non-social evaluations were Value evaluations published in the (quantitative Non existent carried out and Budget or Qualitative) disseminated Transparency Portal through the in April 2014. portal. Date achieved 10/03/2012 12/31/2013 04/30/2014 Comments (incl. % Partially achieved. achievement) Indicator 5 : Design of terms of reference for non-social program evaluations. SHCP has advised on adapting ToR to conduct program Terms of evaluations other Reference Value than social, and on (ToR) for non- (quantitative Non existent revising the ToR of social program or Qualitative) the Consistency and evaluations Results evaluations available. as well as the Cost Effectiveness for non social programs Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Development of a methodology and guidelines for better social program Indicator 6 : focalization. Methodology and guidelines Value for better (quantitative Non existent Not achieved social program or Qualitative) focalization available. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Public programs portfolio review to identify program delays and development of Indicator 7 : methodology for ex ante screening. Public programs portfolio Value review carried (quantitative Non existent out and Not achieved or Qualitative) methodology for ex ante screening available. Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) Indicator 8 : Design and dissemination of an evaluation for the SED SED Important changes evaluation will take place Value carried out and fostered by the (quantitative Non existent disseminated PND, regarding the or Qualitative) through the SED’s scope (i.e. portal. PGCM) Date achieved 10/03/2012 12/31/2013 12/31/2013 Comments (incl. % Not achieved achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (US$ millions) 1 01/27/2009 Satisfactory Satisfactory 0.00 Moderately 2 09/18/2009 Moderately Satisfactory 0.00 Unsatisfactory Moderately 3 06/11/2010 Moderately Satisfactory 0.00 Unsatisfactory Moderately 4 12/14/2010 Moderately Satisfactory 0.00 Unsatisfactory Moderately 5 04/26/2011 Moderately Satisfactory 0.00 Unsatisfactory Moderately 6 11/02/2011 Moderately Satisfactory 0.00 Unsatisfactory Moderately 7 05/16/2012 Moderately Satisfactory 0.00 Unsatisfactory Moderately 8 01/07/2013 Moderately Satisfactory 0.02 Unsatisfactory Moderately 9 04/01/2013 Moderately Satisfactory 0.02 Unsatisfactory 10 12/07/2013 Moderately Unsatisfactory 0.21 Unsatisfactory H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in US$ millions The restructuring aligned the 11/15/2012 Y MS MU 0.00 PDO to a reduced Project scope and partial loan cancellation. If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below: Outcome Ratings Against Original PDO/Targets Moderately Unsatisfactory Against Formally Revised PDO/Targets Moderately Unsatisfactory Overall (weighted) rating Moderately Unsatisfactory I. Disbursement Profile 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. During the 2004-2007 period, the Mexican economy experienced a balanced and broad-based economic expansion with an average annual gross domestic product (GDP) growth rate of 3.8 percent. A vigorous global economic expansion contributed to the enhanced growth performance, as the value of exports increased at a double-digit annual average rate. Price stability also contributed to a healthy growth of domestic demand, with improved purchasing power of wages and salaries, and an expansion of domestic credit and consumer lending. 2. In this context, the fiscal reform approved in September 2007 marked a crucial step in strengthening the country’s fiscal accounts. This fiscal reform enabled the Government of Mexico (GoM) to raise tax collection rates, strengthen fiscal federalism, and decrease tax evasion. Building on the improved macroeconomic and fiscal framework, the GoM focused on enhancing the quality of public expenditures throughout the budget cycle, from planning to execution and evaluation. As a result, the Government established a results-based budgeting initiative, the Performance Evaluation System (Sistema de Evaluación del Desempeño, SED), for which it sought Bank support. 3. Furthermore, the 2008-2013 Country Partnership Strategy (CPS) for Mexico called for strengthening institutions to improve their functioning and enhancing citizen perceptions of the public sector through a set of initiatives that included a move towards results-based budgeting. The higher-level objective of the Project—improving the quality of public expenditures in Mexico—was therefore fully consistent with the FY08- 13 CPS. The Project was also aligned with two of the primary objectives of the 2007- 2012 National Development Plan (NDP), which aimed to “improve regulation, management, the processes and results of Federal Public Administration to satisfy citizen needs in relation to the provision of public goods and services;” and “promote and guarantee transparency, accountability, access to information, and the protection of personal information in all spheres of government.”1 1.2 Original Project Development Objectives (PDO) and Key Indicators 4. The original objective of the Project was to strengthen the Borrower’s capacity (through its federal departments and agencies) to provide decision makers and the public with rigorous, timely, user-friendly information on the efficiency and effectiveness of government organizations and their program expenditures.2 1 Mexico Country Partnership Strategy FY 2008-2013, March 4, 2008, Report No. 42846-MX. 2 There is a slight difference between the Original PDO in the PAD and the Loan Agreement (LA). The one used in the ICR is that of the LA. 1 5. The original PDO indicators were: (i) Performance information presented in budget requests from line ministries to SHCP and in budget submitted to Congress; (ii) Satisfaction of Congress and civil society organizations with the performance information made available via government websites (as measured through user surveys); (iii) Percent of programs with improved strategic design and reporting 3 ; (iv) Progression of organizations through the steps (levels) established for each Management Improvement Program (Programa de Mejora de la Gestión, PMG), marking increased levels of administrative capacity and efficiency. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The Project had to be restructured (level 1) in November 2012 as a result of changes in national priorities and most importantly, due to the fact that some activities had been financed and implemented using local funds, in light of the loan’s start up delays and lengthy process towards effectiveness. Therefore, the Government and the Bank agreed on reducing the Project scope, partially cancelling the loan and redesigning activities to better align them with Government priorities. 7. In order to align them with a reduced Project scope and partial loan cancellation, the PDO and its key indicators and the results framework were revised. The revised PDO is to strengthen SHCP’s capacity to: (i) allow the use of standardized performance information of priority public programs during budget preparation; (ii) generate harmonized sub-national fiscal information; and (iii) disseminate budget performance information to the Borrower’s citizens on a regular basis. The table below illustrates the change in key indicators. Table 1. Changes in PDO indicators Original PDO-Level Indicator Revised PDO-Level Indicator (restructuring (PAD 2008) 2012) 1. Performance information presented in budget (Revised) Performance information using a requests from line ministries to SHCP and in performance framework is generated and used budget submitted to Congress. during budget request discussions between line secretariats and SHCP. 2. Satisfaction of Congress and civil society (Revised) An active network and a portal to organizations with the performance inform society at large on a regular basis about information made available via government budget performance, increasing transparency websites (as measured through user surveys). and accountability to citizens. 3. Percent of programs with improved strategic (Revised) Percent of public programs for which design and reporting. there is performance information using a standardized performance framework. 3 There is evidence from previous analytic work supported by the Bank (Technical assistance to the SHCP on October 2007, corroborated by subsequent CONEVAL analysis) that the quality of strategic design of public programs is weak along several different dimensions (objectives, indicators, link to strategies). Using equivalent dimensions, the UED unit was supposed to be able to measure overall quality over time, using that previous work as a baseline. 2 4. Progression of organizations through the (Dropped original and replaced with new) steps (levels) established for each PMG Percent of municipalities that have a simplified system, marking increased levels of financial management system installed and administrative capacity and efficiency. working. Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan 68464-MX, 2012. 1.4 Main Beneficiaries 8. The primary target groups of the Project were the users of performance information, including line ministries, Congress and the general public. These main beneficiaries did not change when the Project was restructured. 1.5 Original Components (as approved)4 Component 1.Management Improvement Program. 9. Provision of technical assistance and training for the design and implementation of selected management improvement and austerity systems, including, inter alia: (a) assistance in the design of the institutional module of the Borrower’s Programa Especial de la Mejora de la Gestión en la Administración Pública Federal 2008-2012; (b) assistance for technical groups established and to be established with respect to each of said management improvement and austerity systems; and (c) capacity building for actors responsible for the implementation of said management improvement and austerity systems. Component 2. Results-Based Management and Budgeting Integrated Information Systems 10. Provision of technical assistance and training to support the development of an integrated information system, including, inter alia, the development of the necessary institutional arrangements, procedures, and an associated electronic database to: (a) receive budget and other expenditure information from different sources; (b) organize and analyze said information; and (c) produce abridged, user-friendly reports with key messages to inform various stakeholders and improve transparency of budget execution. Component 3.Financial Management Strengthening for Results-Based Budgeting 11. Provision of technical assistance and training to strengthen the quality and integrity of financial data, including, inter alia: (a) modernization of the Borrower’s public accounts (Cuenta Pública) and the underlying government accounting systems to strengthen the usefulness, integrity and transparency 4 The components are presented as per the Loan Agreement. For a more detailed description of the original components see the Project Appraisal Document Report No: 46271-MX. 3 of public accounting records and related accounting management systems for management decision making and public accountability; and (b) strengthening of the quality and timeliness of the Borrower’s budget execution reporting. Component 4. Results-Based Management and Evaluation System Consolidation 12. Provision of technical assistance and training to: (a) address government’s capacity limitations to carry out, contract and use evaluations of government programs; and (b) consolidate a system to evaluate the effectiveness of federal policies and programs through, inter alia: (i) (A) creation and consolidation of a Sistema de Evaluación del Desempeño in the Borrower’s government; and (B) strengthening of the alignment of the Borrower’s National System for Democratic Planning components, including, Vision 2030, National Development Plan, sector strategies and budgetary programs and the strengthening of a selected set of strategic indicators based on sharper program design, and ownership building of these indicators among the Borrower’s line secretariats; (ii) strengthening the generation of performance data for the analysis of the Programas Federales; and (iii) implementation of results-based management approaches at the state level. Component 5. Monitoring and Evaluation 13. Provision of technical assistance and training to SFP and SHCP to develop the necessary organizational and evaluation capacity to monitor the progress and to evaluate the implementation of the Borrower’s Sistema de Evaluación del Desempeño and related institutional reforms. 1.6 Revised Components 14. The November 2012 restructuring (level 1) introduced the following changes to the components: (i) Components 1 (Management Improvement Program) and 2 (Results-Based Management and Budgeting Integrated Information Systems) were cancelled as a result of changes in Government priorities and use of own resources; (ii) Component 3 (Financial Management Strengthening for Results-Based Budgeting) continued as originally designed with some changes in activities that focused on modernizing Government accounting and public accounts, as well as harmonizing and integrating budget execution reporting; (iii) Component 4 (Results-Based Management and Evaluation System Consolidation) was revised to update Project activities with respect to consolidating the 4 Government evaluation system and to include a fourth sub-component according to the following description5: “the provision of support to the Borrower’s fiscal transparency initiatives including: (a) the participatory design and implementation of an integrated fiscal transparency web portal, and (b) the design of guidelines and regulations to foster fiscal transparency in the Borrower’s federal, state and municipal levels of government”; (iv) Component 5 (Monitoring and Evaluation) would continue with some changes in activities including cancellation of activities to be originally implemented by SFP. The revised description6 is as follows: “Provision of technical assistance and training to SHCP to develop the necessary organizational and evaluation capacity to monitor the progress and to evaluate the implementation of the Borrower's Sistema de Evaluación del Desempeño and related institutional reforms.” 1.7 Other significant changes 15. The November 2012 restructuring (level 1) introduced these additional changes: (i) the restructured Project was to be implemented solely by the SHCP through its Subscretaría de Egresos (SE) eliminating the participation of the Public Administration Ministry or Secretaría de la Función Pública, SFP; (ii) the Project financing reflected the cancellation of US$12.756 million of the loan amount (from US$17.2 million to US$4.4 million) and revisions to the costs by component, financing plan and expenditure categories; (iii) the procurement arrangements were changed to apply the Consultant and Procurement Guidelines of May 2004 as revised October 1, 2006 and May 1, 2010 to the restructured Project; this was to enable application of the Bank Guidelines that are harmonized with the then currently authorized policies and procedures for externally funded projects in the country; and (iv) the procurement plan and implementation schedule was revised as a result of the reduction in Project scope and partial loan cancellation. 16. With respect to Project financing changes, the total Project cost was reduced from US$31.9 million to US$6.7 million (excluding front-end fee)7. In terms of the Project’s financing plan, the reduction in total Project cost reduced both the counterpart fund contribution as well as the required IBRD loan amount (from US$17.2 million to US$4.4 million including the front-end fee). See Table 2 for details. 5 Amendment to the Loan Agreement dated November 29, 2012. 6 Ibid. 7 See Annex 1 for more details. 5 Table 2: Summary of Project Total Costs and Estimated Financing Plan (US$ million) As approved (PAD) After restructuring Components/Activities Local IBRD Total Local IBRD Total 1. Management 0 7.600 7.600 0 0 0 improvement program (PMG) 2. Development of an 2.100 1.900 4.000 0 0 0 Integrated Information System for Results-Based Budgeting and Management (SISED) 3. Strengthening Financial 2.075 1.925 4.000 1.850 2.925 4.775 Information for the implementation of Results Based Budgeting 4. Consolidation of the 6.050 4.000 10.050 0.440 1.131 1.571 Performance Evaluation System and Results Based Budgeting 5. Institutional 4.500 1.775 6.275 0 0.3875 .3875 Strengthening and Program Monitoring and Evaluation TOTAL 14.725 17.200 31.925 2.29 4.4435 6.7335 Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan 68464-MX, 2012. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 17. The quality at entry is viewed to have been moderately unsatisfactory. During preparation, the Project benefitted from synergies that emerged from activities organized by the Bank, such as the Performance Budgeting workshop that took place in Mexico in June 2008, which presented international experiences on the topic and became an important input for the initial design of the reforms. In addition, the two original counterparts, SFP and SHCP, also benefited from the economies of scale derived from implementing the results-oriented agenda towards a comprehensive reform–including both management and budget. However, the design of the Project was complex; the potential lack of coordination between the two main counterparts (SFP and SHCP), identified as substantial risk at appraisal, materialized during the Project’s initial stages and implementation. Moreover, it should be noted that moving from monitoring budget execution to results monitoring, and from compliance to results in public sector management via a more comprehensive program, such as the PMG, implied a big reform. Most countries in Latin America and in Asia—as highlighted by one of the peer reviewers—have struggled with this type of changes. As such, even though counterparts coordinated their efforts and defined their roles for the results-based management and budgeting reform during the Project’s preparation, the significant delay in loan effectiveness and subsequent delay in implementation suggests that the coordination mechanisms needed for the Project were underestimated. 6 18. The PAD indicated that the Project met the criteria for readiness for implementation. 8 However, there were delays between the signing of the Loan Agreement and the declaration of effectiveness. The Project was approved by the Board on December 11, 2008. The Project’s legal agreement was signed on April 21, 2009, and became effective on March 19, 2010 due to an unanticipated delay in internal GoM administrative processes – specifically, the execution of the administrative agreements (Contrato de Mandato)9 between the Borrower’s Representative (Public Credit Unit in SHCP), the Financial Agent (Nacional Financiera, NAFIN), and the implementing agencies - SFP and SHCP. 19. Overall, the long time elapsed since the loan’s approval and effectiveness (16 months), along with a challenging procurement environment involving multiple parties (i.e., in Mexico it typically involves participation of three to four parties - the implementing entity/ies, the financial agent and the Bank) and weak procurement capacities in the implementing agencies, made it impossible for the Government to start utilizing the loan proceeds until one and a half years after Board approval. 2.2 Implementation 20. The implementation of the Project is viewed to have been moderately unsatisfactory primarily due to the lack of physical progress under the Project. 21. Start-up delays, a complex project design that included a governance framework involving two implementing agencies, limited implementation capacity, as well as changes in Government counterparts10, severely hampered Project execution. Moreover, within SHCP the reform and Project activities were not fully internalized or appropriated in particular by the operational teams. SHCP areas (Budget Direction, Evaluation Direction, and International Affairs) were not fully aligned conceptually regarding Performance Based Budgeting and its priorities and targets. These resulted in the Government requesting a partial cancellation of the Project (US$12.7 million equivalent to 74% of Loan amount) as early as August 2011. As of Project restructuring in November 2012, only the front-end fee for the IBRD loan of US$43,108 had been disbursed under the loan. Moreover, lengthy procurement processes, in some cases caused by changes in Project Government leaders and operational teams, not only affected Project implementation but also posed a risk to its continuity, triggering internal 8 Readiness for implementation means that the Project ideally should become effective within four months from Board approval and is able to evidence physical implementation including initial disbursement for project activities (not just front-end fee) within the first six months following effectiveness. 9 There were two specific effectiveness conditions for this operation:(a) the duly executed Contrato de Mandato by the respective parties; and (b) related legal opinions from the respective parties of the Contrato de Mandato. 10 In the SFP, there were 5 subsecretarios between 2008 and 2012. 7 Bank discussions on a possible loan closing during the process of restructuring in 2011/2012. 22. Despite this lack of progress, the Government remained committed to the overall objective of the Project that supports a critical public sector reform, namely the introduction of results-based management and budgeting in the context of a continued effort to modernize and harmonize public financial management (PFM) rules and procedures in all levels of government in Mexico. This is evidenced by some activities being advanced and financed with local resources and which was the basis for the continued positive assessment of the achievement of the PDO 11. 23. Using its own resources, significant progress was made by the Government in pursuing the indicated PFM reform agenda particularly in the areas of results-based budgeting, performance evaluation, accounting harmonization, fiscal transparency and public procurement management. The most important achievements in these areas are presented below: 24. Results-Based budgeting and performance evaluation : The PBR-SED12 reform led by the SE has successfully advanced the design and introduction of specific guidelines and instruments to gradually adopt a results and performance orientation in the Federal Government’s budgeting process. By developing a conceptual model, the new system articulated two instruments aiming at incorporating the results and performance dimensions into the resource allocation procedures, i.e., the adoption of performance indicators for public sector programs and the implementation of systematic evaluation activities over public sector policies and programs. 25. Accounting harmonization: As part of the overall strategy to modernize public sector financial management practices in the country, the Government started an ambitious program towards the harmonization of public accounts and accounting practices across Federal and sub-national governments. Introducing results-informed management capabilities require adequate accounting, reporting and consolidation of information mechanisms as a sine qua non condition for success. Thus, based on timelines clearly established in legislation passed at the request of the Administration, efforts were deployed to modernize and harmonize accounting systems. 26. Fiscal transparency: The SE also achieved significant progress in the fiscal transparency agenda. After a broad consultation and participatory process involving specialized Non Government Organizations (NGOs) and experts, and in collaboration with the Access to Information regulatory entity – IFAI, the SE launched a fiscal transparency Portal13that provides access to detailed information on the allocation and use of federal government’s resources. The portal is now fully operational and has also included a citizens’ friendly application–Presupuesto Ciudadano–that not only facilitates 11 ISR DO rating was Moderately Satisfactory from late 2009 – early 2013. 12 PBR-SED stands for Presupuesto Basado en Resultados y Sistema de Evaluación del Desempeño . 13 For more details visit http://www.transparenciapresupuestaria.gob.mx 8 access to budget information, but also describes the budgetary process in a very accessible way for citizens. 27. The SFP had also been advancing the public sector management modernization agenda complementing efforts in areas related to the improvement of public sector management practices, streamlining internal regulatory frameworks, and modernizing the Federal Government’s public procurement system and procedures. 28. Progress achieved by both the SE and SFP using own resources was in line with the PDO and had received Bank’s feedback and guidance. In the case of the PBR-SED effort, the Bank sustained technical dialogue during Project implementation support missions for those activities that should be carried out through the Bank loan. On the other hand, the PFM harmonization work was supported through both Bank implementation support activities and a parallel advisory services activity. 29. With these advances using local funds, a Government request for a partial cancellation was received on August 8, 2011. This was followed by a request in late 2011 to reduce the Project scope to reflect the advances and redesign of activities to better align them with current priorities. In this regard, the processing of the cancellation by the Bank was delayed pending validation of the final set of activities that would need to be dropped and those that needed to be incorporated under the restructured Project to ensure viability after restructuring. Moreover, the November 2012 restructuring was finalized after a careful re-assessment by Government counterparts and the Bank that took into account the potential impact of the political transition following the Presidential elections in July 2012. Specifically, the SFP indicated its preference to cancel its activities in the Project. 30. Moreover, soon after the Project restructuring, the new Administration took place in 2013. Among the first impact on the Project was the change in the executing unit from the Budget Office of the SE to the Performance Evaluation Unit (Unidad de Evaluación del Desempeño, UED)14. This facilitated a better alignment of presidential priorities with sectorial plans and programs’ objectives/ targets. However, the political transition also resulted into a set-back in Project execution since the activities needed to be revalidated with the new counterparts. Actual Project activities implemented were minimal with total disbursements of less than US$400K. Furthermore, priority shifts in some critical results areas of the Project led to some results indicators not being achieved at all. This was particularly true for the roll-out of simplified financial management systems to small municipalities that could not be achieved because of a priority shift towards the development of the Government Accounting System first. 14 This evaluation unit reported to the Budget Office Director (UPCP). In October 2012, the internal Regulations of the SHCP were amended through a Decree; these reforms included the creation of the Performance Evaluation Unit with attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility Federal Law (LFPRH). 9 31. Another factor affecting implementation was the existence of a similar operation supported by IADB. The IADB supported Performance Based Budgeting through a loan, but there was a lack of effective coordination between the two institutions. In this context, both Banks were almost competing and not aligned in technical aspects. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 32. M&E design. The results framework presented in the PAD provided for performance indicators for the PDO and intermediate results for each component. However, these were largely qualitative in nature and with only the desired target value at end of Project identified. No target values were identified for the time period between beginning and ending which made monitoring a challenge and perhaps more susceptible to optimistic assessments as evidenced by the Moderately Satisfactory ratings for PDO achievement throughout the life of the Project except for the last ISR. Progress towards achieving the PDO could have been more meaningfully and objectively assessed if target values were identified for the implementation period and not just at exit. Moreover, with mid-course target values, mid-course corrections could have been introduced earlier. 33. M&E implementation. During the second and third years of implementation, the Bank scheduled periodic high level meetings with and between Deputy Ministers to discuss the progress of the Project, but these meetings did not occur as planned and when they did, they could not provide guidance to the Project at the political or technical level needed. The PDO and intermediate indicators were adjusted during Project restructuring (level one) in 2012 to establish a better link with Project activities. While the target values were also identified only for Project exit, the indicators have improved to allow for a more objective if not quantitative assessment during implementation post- restructuring. The Bank regularly monitored progress under the Project through periodic implementation support missions and were also included in the implementing agencies progress reports. 34. M&E utilization. The PDO and intermediate results indicators, particularly with respect to the target measures or percentages, were closely monitored by the Bank team and reported in ISRs. The counterparts provided updates to the Bank on the results indicators usually at the conclusion of every implementation support missions. Post- restructuring, the Government also used the transparency portal to monitor most of the results indicators. However, more in situ monitoring effort is done in each ministry through their planning units instead of the UED at SHCP. 2.4 Safeguard and Fiduciary Compliance 35. No safeguard policies were triggered by the Project at design, an appropriate decision, as no such issues arose during implementation. 36. The key issues and risks concerning procurement for Project implementation were identified since Project appraisal; these included the lack of experience of the two implementing agencies in applying Bank procurement guidelines. The corrective 10 measures were agreed with the executing agencies and were mainly related to keeping enough trained personnel on Bank guidelines in the implementing agencies’ procurement units. However, limited procurement capacity continued to be an issue during implementation before and after restructuring which posed a substantial risk to the Project’s success. 37. Risk for Project financial management was assessed as low and was considered to be satisfactory throughout Project implementation. Financial monitoring reports were submitted on time and were acceptable to the Bank. All minor inconsistencies found during supervision were properly addressed by the client and did not impact the controls of the Project. Annual audit reports for the Project were submitted accordingly and reviewed by the Bank, with auditors issuing their unqualified opinions. 2.5 Post-completion Operation/Next Phase 38. Continuing commitment to the development objective beyond the scope of the loan has been demonstrated through the decisions of actors including the UED to engage in related activities with their own funding. There are also positive signals that future Bank engagement with SHCP would include a focus on strengthening the UED as well as the M&E capacities of line ministries and sub-national governments. To this end, the lessons from this Project are expected to inform future engagements and could therefore help to consolidate and sustain the Close and Modern Government Program (Programa de Gobierno Cercano y Moderno, PGCM),15 as well as the reform process of SHCP in the area of performance information, integrated financial management, use of this information for improving commitments that result from evaluations and for management and budget decision making. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 39. To achieve faster growth, reduce extreme poverty and promote shared prosperity, Mexico needed to improve public service delivery. Mexico’s public service delivery is still hindered by expenditure inefficiencies at all levels of government: federal, state, and municipal. In this context, sound and informed budget decision making were critical to ensure that services were delivered efficiently, as well as in a transparent and accountable manner. 40. The GoM recognized this challenge in the 2013-18 NDP, which lays out a cross- cutting program for a “Close and Modern Government”. The PGCM aims at establishing a results-based, efficient government, as well as developing monitoring and evaluation 15 See section 3.1 for more details on this cross cutting program included in the National Development Plan 2013-2018. 11 mechanisms for improving performance and service delivery. Additionally, the program seeks to simplify procedures and regulations, allow for greater transparency and accountability, ensure an optimal use of public resources, and seize the potential of information and communication technologies. By focusing on enhancing the information on performance available for government entities and citizens, the Project’s original design and implementation are fully aligned with the GoM’s current strategy and continue to be highly relevant for Mexico. 41. As mentioned in the Project Appraisal Document (PAD), the Project was aligned with the CPS FY08-13 16 that called for strengthening institutions to improve their functioning and to enhance citizen perception of the public sector through a set of initiatives including results based-budgeting. The Project was also consistent with the World Bank thematic engagement set forth in the new CPS FY14-19, specifically with the third theme, “Strengthening Public Finances and Government Efficiency”. 17 The Bank strategy underscores the need for Mexico to improve public sector performance, including expenditure quality and equity, and aims at supporting better service delivery. Through this Project, the Bank has contributed to extending the monitoring and evaluation techniques of key public policies, strengthening the supervision of indicator achievements, and generalizing the follow up of intermediate results, establishing the foundations for a more efficient and transparent public administration. 42. As mentioned earlier, design of the Project was complex but it was key to enable the changes that needed to happen in order to pursue a results-oriented reform in budget and public sector management and effectiveness. With respect to implementation, albeit unsatisfactory in the end, it was relevant as it allowed for a continuing engagement and dialogue between the Government and the Bank on the critical public sector reform areas beyond the Project. 3.2 Achievement of Project Development Objectives 43. The overall rating for PDO achievement is Moderately Unsatisfactory. 44. Prior to the November 2012 restructuring, the achievement of outcomes is assessed in this ICR to be moderately unsatisfactory. While credit is due to the SHCP and SFP for their achievements using own resources, these did not enable the Project to sufficiently progress towards achieving its desired outcomes. Nevertheless, as discussed in Section 2, their achievements in advancing results based budgeting and performance evaluation, accounting harmonization and fiscal transparency, albeit without any loan disbursement except for the front-end-fee, were aligned with the PDO. 45. Post-restructuring, the achievement of outcomes with disbursement of approximately US$400K are assessed to be likewise moderately unsatisfactory for failure 16 Report No. 42846-MX 17 Report No. 83496-MX 12 to achieve several results indicators including the PDO indicator and related intermediate outcome indicators with respect to the roll-out of a simplified financial management system to small municipalities and the evaluation of public programs. Nevertheless, the following achievements after the restructuring are worth noting: (i) Performance information (PI) is now presented in a timely manner to Congress in a format that allows interpretation and meets their needs. The content includes outcome and output measures. (ii) The majority of the public programs (84.5%) have a robust design across key dimensions (objectives, indicators, baselines, links to strategies) enabling better reporting and evaluation. (iii) The Portal containing the PI on budget performance is available to citizens. In 2013, the UED developed more citizen friendly applications such as Cuenta Pública Ciudadana 2012 and 2013, as well as their English versions. These initiatives were mostly financed by component 4 of the loan. An improved version of the budget transparency portal is to be launched in June 2014, including new platform, features, etc. See figure below and Annex 2 for more details on outputs that contributed to achieve PDOs. Outputs. 1. Budget Transparency Portal 2. Established methodology and manuals that set the Project Development minimum technical Component 4. Objective 2. An active requirements and guidelines Consolidation of the network and a portal to inform for budget transparency and performance evaluation society at large on a regular provide orientation for system’s (Sistema de basis about budget performance indicators Evaluación del Desempeño performance, increasing – SED) institutional transparency and accountability 3. Developed Training architecture and to citizens Programs based on the strengthening the system challenges identified and on for evaluating policy and demands (i.e. logical program effectiveness framework matrix) (Presupuesto Basado en Resultados – PBR) 46. Overall, the Project supported the introduction of better management practices, processes and tools in the Mexican public administration, including better and timely performance information, enhanced reporting and evaluation of public programs, and transparent and available budget information. The adoption of the Budget Transparency Portal, which was largely funded by loan funds, resulted into tangible outcomes in terms of generating information and enabled the country to maintain its Open Budget Index standing. 18 At the same time, the Project financed the training of officials through a 18 In 2012, Mexico has been one of the countries that provide “significant budget information” per the Open Budget Index. 13 series of activities including online courses, workshops and the development of a series of manuals and methodologies to collect, monitor and evaluate budget information. This now appear to have good results as evidenced by the work being continued and specifically evidenced with SHCP’s presentation of the “Guidelines to register, revise, update, schedule and monitor the Budget Programs Results Indicators Matrix (MIR)” in the first semester of 2014. 3.3 Efficiency 47. Project efficiency is low. The reduction in scope and the limited results achieved despite the restructuring and extension suggest that overall efficiency is much lower than expectations at design and at restructuring. The long time elapsed between preparation and the Project closing with minimal physical accomplishments and limited results vis-à- vis the overall desired outcomes suggest there were opportunities missed which were partly alluded to in the Borrower’s comments on the relevance of Project activities (particularly those of SFP) having been reduced given the delays. In monetary terms, from the beginning, a cost-benefit analysis was not considered to assess the impact of these institutional reforms, as problems in the quantification of intangibles and issues of attribution make it difficult to express their impacts. Moreover, given loan amount disbursed (US$373,684) vis-à-vis the time-overrun in implementation and the resources mobilized in the Bank for preparation and provision of implementation support (US$967,977) which are more than double the loan disbursement, it could be considered that the Project failed to be efficient. Nevertheless, the operation managed to achieve significant results in terms of increased transparency, good governance and accountability, as measured by greater availability and access to budget data and performance information (see Annex for more details). Further potential efficiency gains and cost savings are expected from these types of reforms when fully implemented. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory 48. As discussed above, while the relevance of the objective remains high with the public financial management and results-based agenda at the federal and subnational levels remaining a priority, the achievement of the PDO could only be rated moderately unsatisfactory given the inefficient Project implementation and the non-achievement of significant outcomes. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 49. No direct poverty, gender or measurable impact on the population is attributable to the Project. However, the Project’s impact on public sector management areas has potential indirect impact on reducing poverty through the effect of better resource 14 allocation and improved service delivery conditions, as well as by channeling generated savings more efficiently to national priorities. (b) Institutional Change/Strengthening 50. The M&E capacity of the UED was significantly strengthened in the latter part of the Project. From Project start to November 2012, the Project was executed by the Budget Office of the SE. Beginning December 2012 (when the current administration took office) the UED19 took over Project execution. This facilitated a better alignment of presidential priorities with sectorial plans and programs’ objectives and targets. Technical assistance during implementation support missions by the Bank contributed to this, as well as feedback to the reform decisions provided by TTLs and experts members of the World Bank team. (c) Other Unintended Outcomes and Impacts (positive or negative) 51. No significant unintended outcomes or impacts were identified. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 52. No beneficiary survey or stakeholder workshops were conducted specifically for reviewing Project performance. 4. Assessment of Risk to Development Outcome Rating: Moderate 53. The general direction of the reforms supported by the Project is aligned with those of the Government’s PGCM. As such, risks to the outcomes that have been achieved to date are considered to be moderate. While in the medium to long term, further advances in effectiveness building off Project activities will likely require additional investments in technology and process redesign, all of these are considered in the PGCM. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 19 This evaluation unit reported to the Budget Office Director (or UPCP). In October 2012 through a Decree the internal Regulations of the SHCP were amended; these reforms included the creation of the Performance Evaluation Unit (UED) with attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility Federal Law (LFPRH). 15 54. The Project was intended to achieve challenging reforms in public management and results-based budgeting, building from efforts supported by the Bank during the beginning of the 2006-2012 administration and Project preparation. The Project was prepared taking into account lessons learned from international experiences and taking advantage of the synergies between SFP and SHCP who were then working together towards a results oriented agenda. The Project was consistent with the federal government’s objectives and the Bank strategy in Mexico. However, several shortcomings in Project design posed significant challenges to implementation. In particular, the failure to identify the challenges involved in coordinating amongst and with several federal government entities, and the overestimation of the counterpart’s capacity to carry out complex procurement processes (both in SFP and SHCP) caused significant risks to the successful completion of the Project. As noted by the Borrower in their comments, Project implementation could have benefited from having a focal point for coordination as well from providing a venue to collaborate among all the stakeholders during Project preparation. The delay in declaring the loan effective (16 months from Board approval) suggests a significant under-estimation of the Project’s readiness to be implemented. (b) Quality of Supervision Rating: Moderately Unsatisfactory 55. The initial implementation support phase of the Project could be characterized as overly optimistic and highly focused on technical aspects, with inadequate attention paid to operational constraints. On the positive side, in a context where the Government was moving fast with their reform agenda and the demands were changing, the Bank was able to sustain technical dialogue specifically on the PBR-SED and PFM harmonization efforts which were financed by local funds. On the negative side, the Bank failed to take advantage of the synergies in place between SFP and SHCP in the early stages of implementation. On top of that, the SFP faced high-level public officials’ turnover and the World Bank team failed to recognize these implementation risk flags or adjust to these changes in the context accurately. The Bank’s implementation support team also suffered from lack of leadership at the beginning of implementation when the TTLs were changed20. Hence, realistic assessments of Project implementation constraints early on could have facilitated a Project turnaround by addressing the changing dynamics in the reform. There were also indications that communications between the Bank team, fiduciary teams, financial agent and implementing agencies could have been better to improve Project implementation support and administrative processes. 56. By 2011, it became clear that the Project scope had to be reduced and Project activities redesigned to better align them with the incoming Government’s priorities as 20 The Project had a total of three TTLs throughout the five years of implementation, 16 was detailed in the Government requests. However, the processing of the partial cancellation by the Bank was postponed to allow for a thorough assessment of viability and commitment from the newly elected administration (at the end of 2012). During this time, the Bank flagged potential issues and took action to solve them effectively throughout the last year of implementation. The new government had just taken office and continued to align its priorities; and as noted above, even though the Bank team engaged in intensive implementation support, procurement processes lingered for extensive periods given procurement capacity constraints within the implementing agency and as such only a few contracts were executed with less than 10% of the total restructured loan funds disbursed. Nevertheless, post-restructuring, a closer collaboration between the Bank team and the counterparts were established that not only help move the Project forward as well as contributed to the institutional strengthening of the UED. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 57. Based on the Bank performance in ensuring quality at entry, and the quality of implementation support, the overall Bank performance is rated as Moderately Unsatisfactory. A number of challenges that led to delays in Project start-up and execution were not fully identified at design, and communication between the Bank and the client could have been better during the initial part of implementation. However, this ICR recognizes the Bank implementation support team’s significant efforts during the latter part of implementation to better address the procurement issues and to try to achieve the new Government’s goals. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 58. Despite the fact that Project implementation suffered delays, the areas supported under this Project were part of the Government of Mexico’s priority agenda with respect to improving results based management and budgeting as evidenced by the reform advances that took place using their own resources. However, the political commitment to the Project was weakened by the weak (if not lack of) coordination between SFP and SHCP and compounded by the turnover of government officials (5 subsecretarios held office in the SFP as the main counterparts for Components 1 and 2). Moreover, the Project was not fully internalized or appropriated in particular by the SHCP’s operational teams. SHCP areas (Budget Direction, Evaluation Direction, and International Affairs) were not fully aligned conceptually regarding Performance Based Budgeting and its priorities and targets. SFP and SHCP’s high-level commitment could have translated into proactive measures by ensuring that the Project operational units were progressing with Project implementation satisfactorily. 17 (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 59. The Project Implementation Units both at the SFP and SHCP were committed to the Project but had a low capacity in fiduciary aspects, especially procurement as identified in the risks at appraisal. During the last year of implementation, SHCP’s implementing unit (within the UED) was very organized and engaged but lacked familiarity with the Bank’s procurement guidelines that did not allow them to disburse most of the resources. Nevertheless, despite their capacity constraint, they were able to focus on limited activities that had significant contribution to the desired Project outcomes. With their limited implementation capacity, they were able to advance on the Budget Transparency Portal that enabled the Project to deliver on its objective of disseminating budget performance information to the Borrower’s citizens on a regular basis. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 60. Considering the above arguments, the overall Borrower performance is deemed moderately unsatisfactory. 6. Lessons Learned 61. The instrument was not the best alternative for what was needed for these reforms. The mitigation measures in place did not lower the risks identified at appraisal. This Project benefitted the most from the technical assistance and feedback received both during preparation and the last years of implementation. In retrospect, it would have been better to have supported the Government’s agenda with a Non-Lending Technical Assistance (NLTA) program or a Reimbursable Advisory Services (RAS) to support more effectively the type of reforms the Government was pursuing. The chosen instrument allowed the Bank to promote the technical dialogue with the authorities and amongst stakeholders, and also ensured agreements and commitments at the highest levels. However, these were not fully internalized or appropriated in particular by the SHCP’s operational teams as discussed above. 62. Assessment of Risks - Coordination mechanisms were not enough. Although there were mitigation measures in place for this risk, the fact that there were two implementing agencies and a financial agent was more complex than initially assessed. Moreover, within each of the implementing agencies, coordination was not enforced thus affecting project implementation and disbursement performance. The designation of a focal point for coordination could have helped as noted by the Borrower. 63. Demand not well read in the case of SHCP. While Components 1 and 2 were very much in line with the Government’s (SFP) agenda, both conceptual and technical; the 18 Components related to results-based budgeting (SE of SHCP) which included the Budget Office (UPCP) in the first stages and the UED in the last years were not aligned. This is largely reflected in the lack of disbursements throughout the Project. 64. Additional implementation support should be considered for projects that involve more than one implementing unit that are not familiar with large and complex procurement processes, such as some investments within this Project. For this Project, the need for additional support should have been anticipated at the beginning of Project implementation. While capacities at the Ministries were strengthened, this was not enough. Moreover, improved harmonization of procurement procedures between the Bank and GoM is a necessity as noted by the Borrower’s comments. 65. Ensure alignment between Government processing and Bank required action. As reflected in the Borrower’s comments below, timely action from the Bank with respect to issuing Bank feedback especially “no objection” is also a critical element for timely implementation. 66. Meaningful and objective results indicators are critical for monitoring complex reform initiatives. While reform initiatives do not always allow for quantifiable measures of success, this Project could have benefited from a results framework that provided for mid-course target values that could have informed counterparts and the Bank that mid-course corrections were warranted earlier. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 67. The Borrower provided very informative and pertinent comments and these have been taken into consideration in the assessment of overall performance of the Project, the Borrower and the Bank as well as in the lessons learned section. These comments are as follows: a) The initial objective of the GoM’s “Results based Management and Budgeting” Project aimed at having the departments and federal organizations of the country provide decision makers and citizens in general with rigorous, timely and easy to access information regarding the efficiency and effectiveness of public expenditure programs. b) The GoM sought Bank support for the financing of the Project through an IBRD loan, with the objective of reaching the goals proposed under the five components that integrated the Project. Given the legal attributions conferred by the Federal Public Administration Organic Law, the Government mandated the two main ministries involved in the PBR-SED initiative (SHCP and SFP) with the implementation of the Project. 19 c) Having two implementing agencies was an obstacle to the successful execution of the Project, basically due to the complexity of coordinating the work of the two ministries and the lack of clarity in the Project objectives, as well as its scope. This was accentuated by the absence of a coordination focal point and the limited cooperation amongst both ministries, the Financial Agent and the Bank. Also, the pace of the execution of the Project did not satisfy Government expectations in terms of results. d) In addition, the delay between the signing of the Legal Agreement (April 21, 2009) and effectiveness (March 19, 2010) had also a negative influence in the implementation of the Project. During the first stages of Project execution, it was not possible to hire any consultancy and advisory services due to delays in the execution of Bank administrative processes—such as “no objections”— through the Financial Agent. As a result, the first consultancy services were only provided in 2011. e) Under these circumstances, one of the executing agencies, SFP, decided not to continue with the implementation of the Project, cancelling the total amount of resources allocated to its components. As the entity responsible for the PMG, SFP was counting on the Project to help advance that agenda, including the design of documents, tools and methodologies. The initial delays reduced the relevance of the components included in the Project from the perspective of SFP. f) Consequently, the Project was restructured in order to have only one executing agency, the SHCP. The SHCP used the loan resources to: hire, inter alia, the consultancy services to analyze and evaluate the strategic indicators of the budget programs; conduct training sessions with regards to PBR-SED; and implement the mechanisms to report the federal resources transferred to state entities and municipalities. These activities had a positive impact in the PBR-SED implementation. g) Under the restructured Project, the Financial Agent acted in a timely manner, which was key for the review of the documents needed to register processes and for obtaining “no objections” from the Bank with respect to the hiring of consultancy services, as well as disbursements. This also helped produce Project-required financial and progress reports on time. h) In the view of SHCP, Bank procurement and hiring processes, especially those requiring “no objections”, could be improved. The long time elapsed between Bank approval/rejection caused delays in the processing of contracts, which greatly affected the proper implementation of Project activities and as a result had a negative impact on its overall implementation. i) In terms of areas of opportunity, greater harmonization between GoM and Bank procurement procedures could be sought, even though there are already 20 documents streamlining Federal Government and World Bank processes. It would be also advisable to establish work programs on time and in accordance to reality, including clear hiring procedures that expedite processes so the products delivered can be used to influence the final results. Finally, it would be convenient to improve project preparation by working jointly and with better coordination and communication among involved stakeholders. 68. The Borrower also provided additional editorial comments in other sections of the ICR. These have been addressed in finalizing the document. (b) Cofinanciers Not applicable (c) Other partners and stakeholders 69. IADB supported Performance Based Budgeting through a loan, but did not work in coordination with the World Bank. Based on the interviews of stakeholders of the reforms, the Banks were seen as almost competing and not aligned technically. 21 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (US$ millions) Appraisal (US$ millions) Component 1 7.6 0 0.00 Component 2 4 0 0.00 Component 3 4 0 0.00 Component 4 10.050 0.875* 8.70 Component 5 6.275 0 0.00 Total Baseline Cost 31.495 0.875 2.80 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 0.00 0.00 Front-end fee PPF 0.00 0.00 0.00 Front-end fee IBRD 0.43 0.43 0.00 Total Financing Required 31.925 0.918** 2.90 Source: Informe de avance segundo semestre de 2013, UED - SHCP *Sum of Loan Disbursement of US$330,576.52 and Counterpart Fund Contributions of US$543,391. **May not equal due to rounding off. (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (US$ millions) (US$ millions) Borrower 14.725 0.543 3.70 International Bank for Reconstruction 17.200 0.331 1.90 and Development 22 Annex 2. Outputs by Component As discussed in the main section of the ICR, the executed Project activities relate to Component 4. Below presents the outputs under said Component. Component 4 - Results-Based Management and Evaluation System Consolidation 1. The key outputs delivered by Component 4 of the Project are described below: a) Budget Transparency Portal (BTP): Since 2011, SHCP has had in place a Budget Transparency Portal (www.transparenciapresupuestaria.gob.mx) on which performance information of the budget programs can be found, as well as public financial information. The main objectives and contents of this portal are describe in the table below: Table 2.1. Objectives and contents of the BTP Objectives Content More transparency in Public  Public Finance Statistics Resources Management  Transferred resources to the local governments  Budget Information  Quarterly Reports in Open Data formats  Cuenta de la Hacienda Pública Federal Budget transparency towards citizens  Citizen Budget (friendly language)  Cuenta Pública Ciudadana or Public Accounts for Citizens  Frequently Asked Questions  Glossary Transparency by Thematic areas,  Investment Projects focused on citizens interests  Performance Information (indicators and goals)  External/Independent Evaluations  Aid and support for Natural Disasters Reconstruction Source: Final Report and Results Framework Matrix, UED, SHCP. April 21, 2014. The main strength of the portal compared to other countries is the fact that all the performance information is public, including external evaluations (more than a thousand) and MIRs (results indicators matrix) with indicators (757). There is also access to public investment information including those funded under special funds such as FONDEN (or Natural Disaster Fund). The UED has also reached out to NGOs and civil society to ensure that the information is useful for their purposes and 23 understandable for everyone. For instance, the information included in the citizen budget for 2014 explains in a friendly manner the following topics:  Growth  Total Budget  Sources of revenue and debt  Deficit  Functional Classification of Expenditures (why the Government spends?)  Administrative Classification of Expenditures (who spends?)  Economic Classification of Expenditures (on what the Government spends?)  Investment Projects  Measures taken to reduce expenditures  SED information such as percentage of expenditures with performance indicators  Links to more detailed information b) Methodology and Manuals were developed, including: - “Methodology to develop and implement budget transparency portals at the federal and local levels (technical and functional)” - “Guide to establish and monitor goals for federal entities and agencies of the Government” - “Training Manual to monitor Budget Program’s objectives based on strategic indicators” - “Improved guide to design strategic and management indicators” c) Various Training Programs have been and will continue to be conducted: - Online Training - Training for Trainers - Training based on identified challenges on MIRs (theory, practical, continued technical assistance, ad hoc workshops) - Training to improve the Single Format System or Sistema de Formato Único (SFU), i.e. information collection and registration as well as quality of information. 2. A full account of the outputs completed by the Project under component 4 is presented in the table below. These outputs were mostly financed by counterpart funds and contributed towards achieving the intermediate result indicator “Consolidation of the performance evaluation system’s (SED) institutional architecture and strengthening the system for evaluating policy and program effectiveness” by supporting the targets specified below as well. Table 2.2. Completed Project outputs by source of financing Original Executed Output Target supported Budget Budget Outputs financed with loan resources 346,000 325,884 Consultancy to prepare and facilitate the 31,000 24,259 Evaluation unit operational in SFP, 24 Project’s selection and hiring processes (1) including an institutional performance evaluation system Methodological guide for the technical and Evaluation unit operational in SFP, functional implementation of the transparency including an institutional 40,000 40,000 web portal at the federal and subnational performance evaluation system levels Development of minimum criteria and Evaluation unit operational in SFP, transparency guidelines for the federal and 32,000 32,000 including an institutional subnational governments performance evaluation system The SED unit manages the Collection of feedback on PBR program and 52,000 52,000 evaluation of non-social programs the SED successfully Set of strategic indicators has improved after support and training Training of trainers course 33,000 33,000 provided to Government officials in charge of programs Evaluation unit operational in SFP, Consultancy for the analysis of budget 18,000 18,000 including an institutional strategic indicators for the 2013 budget cycle performance evaluation system Evaluation unit operational in SFP, Consultancy for the maintenance of the 18,000 18,000 including an institutional Budget transparency portal performance evaluation system Set of strategic indicators has Consultancy for the implementation of the improved after support and training 18,000 18,000 SFU provided to Government officials in charge of programs Set of strategic indicators has Consultancy for the conceptualization and improved after support and training analysis of the information produced by the 25,000 25,000 provided to Government officials in SFU charge of programs Evaluation unit operational in SFP, Consultancy to prepare and facilitate the 52,000 37,500 including an institutional Project’s selection and hiring processes (2) performance evaluation system Consultancy for the establishment of a Evaluation unit operational in SFP, monitoring and evaluation system for the 27,000 28,125 including an institutional Project’s selection and hiring processes performance evaluation system Outputs financed with local resources 574,093 574,093 TOR for the carrying out of the activities to be conducted under components iii, iv and v of 102,241 102,241 - the Project External audit for loan 7622-MX 2,852 2,852 - Set of strategic indicators has Results-based budgeting online course improved after support and training (auditing and results-oriented indicators 469,000 469,000 provided to Government officials in modules). charge of programs 25 Annex 3. Economic and Financial Analysis 1. The overarching goals of the reforms supported by the Project aimed at increasing transparency and strengthening good governance and accountability in the Sistema de Evaluación al Desempeño (SED), which are difficult to measure. Therefore, the economic and financial analysis for this operation needs to go beyond quantitative estimations and rely on qualitative data. The assumption is that the support to the GoM’s institution building efforts provided by the Project, notably in terms of results-based management and budgeting, has contributed to a more transparent and accountable governance framework. However, the outputs of the reforms carried out by the GoM, including the Programa de Mejora de la Gestión (PMG) and Presupuesto Basado en Resultados (PBR), as well as the SED and the sectorial strategies and specific programs, are not fully attributable to the Project or a single reform initiative, and their impact may not be apparent in the short-term. 2. Overall, the Project supported the introduction of better management practices, processes and tools in the Mexican public administration, including better and timely performance information, enhanced reporting and evaluation of public programs, and transparent and available budget information. The most important instrument in this regard is the Budget Transparency Portal, which was mostly financed with the loan resources. The adoption of this portal resulted into tangible outcomes in terms of generating information, since according to the Open Budget Index (OBI), Mexico has been one of the countries that provides “significant budget information” in their portal since 2012. Today most budget information is presented in Open Data format, which allows the private sector, NGOs and civil society in general to access what is relevant for them. In fact, online visits increased to 28.5% in 2013 compared to 2012, and as of the first quarter of 2014, there was an increase of almost 50% compared to the same period of 2013 and more than doubled compared to the first quarter of 2012. 3. At the same time, the Project financed the training of officials through a series of activities, including online courses, workshops and the development of a series of manuals and methodologies. These initiatives were aimed at increasing the capacity of public officials to collect, monitor and evaluate budget information, and now appear to have achieved good results. At the end of 2013, out of the 981 budget programs included in the programmable budget (the total is 1,005), 854 programs have performance information, which is equivalent to 87% of the programmable budget. The work has continued in the first semester of 2014, when the SHCP presented the “Guidelines to register, revise, update, schedule and monitor the Budget Programs Results Indicators Matrix (MIR)”. 4. The upward trend in the use of information by the public, along with the training received by officials, also suggests a positive impact of the Project in terms of creating an institutional context more conducive to efficiency. Although it cannot be quantified, it seems likely that the combined effect of greater accountability and increased capacity 26 will produce financial savings eventually, ultimately boosting the quality of public expenditures in Mexico. 27 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialists Lending Pedro Arizti Senior Public Sector Specialist LCSPS TTL Rosa Maria Hernandez- Senior Executive Assistant LCC1C Assistant Fernandez Management and HR Mariano Lafuente Public Sector Mgmt. Spec. LCSPS Improvement Karla Soledad Lopez Flores Operations Assistant CTGBD Assistant Felix Prieto Arbelaez Senior Procurement Specialist LCSPT Jeffrey James Rinne Consultant PRMPS HR Management Fernando Rojas Consultant LCSPS Public Sector Maria Guadalupe Toscano Public Sector Mgmt. Spec. LCSPS Nicolas Manuel Antonio Vargas Lead Financial Management Spec MNAFM Madrigal Supervision/ICR Juan Carlos Alvarez Senior Counsel LEGES Pedro Arizti Senior Public Sector Specialist LCSPS TTL Financial Dmitri Gourfinkel Financial Management Specialist LCSFM Management Alberto Leyton Senior Public Sector Specialist ECSP4 TTL Senior Financial Management Xiomara A. Morel LCSFM Specialist Victor Manuel Ordonez Conde Senior Finance Officer CTRLN Jeffrey James Rinne Consultant PRMPS HR Management Tomas Socias Senior Procurement Specialist LCSPT Maria Guadalupe Toscano Public Sector Mgmt. Spec. LCSPS Nicolas Manuel Antonio Vargas Lead Financial Management Spec MNAFM Madrigal May Olalia Senior Operations Officer LSCPS Azul del Villar Baston Consultant LCSPS ICR Eguiar Lizundia Gonzalez Consultant LCSPS ICR (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$ Thousands (including No. of staff weeks travel and consultant costs) Lending FY07 2.81 18.963 FY08 52.03 301.141 FY09 27.55 131.268 Total: 82.39 451.373 28 Supervision/ICR FY09 11.19 41.890 FY10 25.14 89.238 FY11 19.33 82.145 FY12 21.75 108.016 FY13 25.39 99.577 FY14 17.20 95.645 Total: 120.00 516.604 29 Annex 5. Beneficiary Survey Results Not applicable 30 Annex 6. Stakeholder Workshop Report and Results Not applicable 31 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 32 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable 33 Annex 9. List of Supporting Documents Project Appraisal Document on a Results-based Management and Budgeting Loan (Report No: 46271-MX), 2008 Mexico Country Partnership Strategy FY14-19 Mexico Country Partnership Strategy 2008-2012 Proposed Project Restructuring of Results-based Management and Budgeting Loan (Report Number 68464-MX), 2012. Proyecto de Apoyo a la Gestión y Presupuesto para Resultados Préstamo 7622- MXBanco Internacional de Reconstrucción y Fomento. Informe de Progreso en la Ejecucion del Proyecto, Segundo Semestre 2013. SHCP, abril 21, 2014. 34 0