Document of The World Bank Report No: ICR00003114 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A LOAN IN THE AMOUNT OF US$300 MILLION TO THE STATE OF RIO DE JANEIRO, BRAZIL WITH THE GUARANTEE OF THE FEDERATIVE REPUBLIC OF BRAZIL FOR A FISCAL EFFICIENCY FOR QUALITY OF PUBLIC SERVICE DELIVERY DEVELOPMENT POLICY LOAN July 17, 2014 Governance Global Practice Brazil Country Management Unit Latin America and Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective July 17, 2014) Currency Unit = Real Brazil R$ 1.00 =0.45 US$ US$ 1.00 = R$ 2.21 FISCAL YEAR January 1 – December 31 ABBREVIATION AND ACRONYMS AGE Internal Audit Department Auditoria Geral do Estado CPS Country Partnership Strategy Estratégia de Parceria com o País DPL Development Policy Loan Empréstimo para Políticas de Desenvolvimento FY Fiscal year Ano Fiscal GDP Gross Domestic Product Produto Interno Bruto GIDE Integrated School Management Gestão Integrada da Escola GORJ Government of Rio de Janeiro Governo do Estado do Rio de Janeiro IBRD International Bank for Reconstruction Banco Internacional para Reconstrução e and Development Desenvolvimento ICMS Tax on Goods and Services Imposto sobre Circulação de Mercadorias e Prestação de Serviços IDEB Index of Development of Basic Índice de Desenvolvimento da Educação Education Básica LOA Annual Budget Law Lei de Orçamento Anual LRF Fiscal Responsibility Law Lei de Responsabilidade Fiscal PAC Growth Acceleration Program Programa de Aceleração do Crescimento PAHI Program to Support Hospitals Programa de Apoio aos Hospitais do Interior PAF Program of Fiscal Adjustment Programa de Ajuste Fiscal PFM Public Financial Management Gestão de Finanças Públicas PPA Multi-year Plan Plano Plurianual PPP Public-Private Partnerships Parcerias Público-Privadas PSF Program for Primary Health Care Programa Saúde Família REGIN Integrated Recording System for taxes Sistema de Registro Integrado SAERJ Education evaluation system of the Sistema de Avaliação da Educação no State of Rio de Janeiro Estado do Rio de Janeiro SEEDUC State Secretariat of Education Secretaria de Estado de Educação SEFAZ State Secretariat of Finance Secretaria de Estado da Fazenda ii SEPLAG State Secretariat of Planning Secretaria de Estado de Planejamento SES State Secretariat of Health Secretaria de Estado de Saúde SESDEC State Secretariat of Health and Civil Secretaria de Estado de Saúde e Defesa Defense Civil SIL Specific Investment Loan Empréstimo para Investimento Específico STN National Treasury Secretariat Secretaria do Tesouro Nacional SUS Public Health System Sistema Único de Saúde SWAP Sector Wide Approach Abordagem Setorial Ampla TAL Technical Assistance Loan Empréstimo para Assistência Técnica UPA 24 hour emergency care unit Unidade de Pronto Atendimento VAT Value Added Tax Imposto sobre o Valor Adicionado WB World Bank Banco Mundial Vice President: Jorge Familiar Country Director: Deborah L. Wetzel Senior Practice Director: Mario Marcel Cullel Practice Manager: Arturo Herrera Gutiérrez Team Leader: Roland Clarke ICR Team Leader: Lorena Viñuela ICR Lead Author Eduardo Somensatto iii CONTENTS Data Sheet ........................................................................................................................................ 1. Program Context, Development Objectives and Design ....................................................... 1 2. Key Factors Affecting Implementation and Outcomes ......................................................... 7 3. Assessment of Outcomes......................................................................................................... 10 4. Assessment of Risk to Development Outcomes .................................................................... 16 5. Assessment of Bank and Borrower Performance ................................................................ 17 6. Lessons Learned ...................................................................................................................... 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ................... 21 Annexes ........................................................................................................................................ 22 iv Data Sheet A. Basic Information RIO STATE Country: Brazil Program Name: DEVELOPMENT POLICY LOAN III Program ID: P126465 L/C/TF Number(s): IBRD-81910 ICR Date: 07/10/2014 ICR Type: Core ICR THE STATE OF RIO Lending Instrument: DPL Borrower: DE JANEIRO Original Total US$US$ 300.00M Disbursed Amount: US$US$ 300.00M Commitment: Revised Amount: US$US$ 300.00M Implementing Agencies: Secretaria de Fazenda Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 07/28/2011 Effectiveness: 11/30/2012 11/16/2012 Appraisal: 02/09/2012 Restructuring(s): Approval: 08/30/2012 Mid-term Review: Closing: 01/31/2014 01/31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Low or Negligible Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating: Performance (if any) Potential Problem Quality at Entry Program at any time No None (QEA): (Yes/No): Problem Program at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) General education sector 20 20 Health 20 20 Sub-national government administration 60 60 Theme Code (as % of total Bank financing) Education for all 20 20 Health system performance 20 20 Managing for development results 10 10 Public expenditure, financial management and 30 30 procurement Tax policy and administration 20 20 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Hasan A. Tuluy Country Director: Deborah L. Wetzel Deborah L. Wetzel Sector/Practice Manager: Arturo Herrera Gutierrez Arturo Herrera Gutierrez Program Team Leader: Roland N. Clarke Roland N. Clarke ICR Team Leader: Lorena Viñuela ICR Primary Author: Eduardo Somensatto i F. Results Framework Analysis Program Development Objectives (from Project Appraisal Document) The objective of the proposed operation is to assist the Government of Rio de Janeiro in strengthening its tax administration, improving the efficiency of public financial management, increasing the quality of public service provision in education and health and ensuring that policies adopted are both consistent with priorities of the State Government and with resources likely to be available in the medium term. Revised Program Development Objectives (if any, as approved by original approving authority) Not applicable. (a) PDO Indicator(s) Actual Value Original Target Formally Revised Achieved at Indicator Baseline Value Values (from Target Values Completion or Target approval documents) Years Indicator 1 : ICMS tax annual revenue Value (quantitative or 24.8 billion 37.7 billion 28.67 billion 30.75 billion Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 12/31/2013 Target was fully achieved. Comments The baseline and target value included in the approved Program Document were erroneous. (incl. % The original baseline for 2011 was R$32.6 billion, but official data showed that it was R$24.8 achievement) billion. The target was 15.6% increase and the result 23.99%. Indicator 2 : ITD tax annual revenue Value (quantitative or 418 million 470 million 644.6 million Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments Target was fully achieved. (incl. % The target was 12.4% increase and the result 54%. This is not a normalized indicator. achievement) Establishment of Unified General Taxpayer ledger, with access of individual taxpayers to their Indicator 3 : unified accounts Unified General Taxpayer ledger has been created, but it is Unified General still in experimental Value Taxpayer ledger, with phase for internal No Unified General (quantitative or access of individual purposes. GoRJ Taxpayer Ledger Qualitative) taxpayers to their Authorities estimate unified accounts that individual taxpayers will have access to their unified accounts in 2015. ii Date achieved 12/31/2011 12/31/2013 12/31/2013 Target was partially achieved. Comments The program required advancing through a series of sequential steps, before the ledger could (incl. % be fully operational. Those steps took longer and proved more difficult to effect than initially achievement) anticipated. Number of indicators adopted as part of the progressive implementation of performance based Indicator 4 : management in tax administration Value (quantitative or 0 indicators adopted 20 indicators adopted 20 indicators adopted Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments Target was fully achieved. The target number of indicators adopted was 10 by 2012 and 20 by (incl. % 2013. Both goals were met. achievement) Number of public investment projects estimated to cost over R$50,000,000 subjected to the Indicator 5 : new technical screening process and with consolidated information showing annual commitments published 0 prior investment 5 prior investment 0 prior investment Value projects projects projects (quantitative or shown in the annual shown in the annual shown in the annual Qualitative) commitments of commitments of commitments of investments investments investments Date achieved 12/31/2011 12/31/2013 12/31/2013 Target not achieved. Comments Three investment proposals were identified for screening. However, these were subjected to (incl. % an ex-post evaluation rather than ex-ante process and the costing information was not included achievement) in the budget. Indicator 6 : Number of major policy programs with detailed costing and key results defined and published 3 major policy Value 0 major policy 4 major policy programs with costing (quantitative or programs with costing programs with costing (ex-post) and results Qualitative) and results published and results published prepared but not published Date achieved 12/31/2011 12/31/2013 12/31/2013 Target was partially achieved. Comments With the analysis, SEFAZ was able to link the costs of the programs to the number of (incl. % beneficiaries, and begin to identify those factors, such as eligibility criteria, that could drive achievement) future expenditures. Implementation of an action plan to improve internal control system is developed on the basis Indicator 7 : of the annual report from State Audit on the performance of internal control systems Action plan to improve Action plan to improve internal controls internal controls developed on the basis published and annual Value of the annual report of report on human No plan or regular (quantitative or the State Audit agency resource reforms and reporting. Qualitative) to the State performance of Government on the internal control performance of systems corresponding internal controls to 2013 published Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments Target was fully achieved. (incl. % Additional reforms recommended by the 2013 report are under implementation. Indicators to achievement) measure and monitor the process have been developed and used since 2012. iii Indicator 8 : Number of regional directors selected using new merit-based process Value (quantitative or 0 28 35 Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments Target was fully achieved. (incl. % 18 regional supervising directors and 17 regional pedagogical directors represent a 100% of achievement) the regional positions. Indicator 9 : Number of school directors selected based on merit Value (quantitative or 0 82 164 Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments Target was fully achieved. (incl. % Additionally, 184 assistant directors were selected based on merit. achievement) Indicator 10 : Annual bonuses paid to school personnel for achieving 80% of improvement targets. Value Bonuses for 2012 paid Bonuses for 2012 paid (quantitative or No bonuses paid by June 2013 by June 2013 Qualitative) Date achieved 12/31/2011 12/31/2013 12/31/2013 Comments (incl. % Target was fully achieved. achievement) (b) Intermediate Outcome Indicator(s) Actual Value Original Target Formally Revised Achieved at Indicator Baseline Value Values (from Target Values Completion or Target approval documents) Years Indicator 1 : Annual school targets communicated to schools (before March) 2013 and 2014 annual Value No annual Annual school targets school targets (quantitative or improvement targets communicated to communicated to Qualitative) communicated schools (before March) schools (before March of the previous year) Date achieved 12/31/2011 12/31/2013 06/30/2013 Comments (incl. % Target was fully achieved. achievement) G. Ratings of Program Performance in ISRs No. Date ISR DO IP Actual iv Archived Disbursements (US$ millions) 1 11/14/2012 Satisfactory Satisfactory 0.00 2 03/20/2013 Satisfactory Satisfactory 300.00 3 02/08/2014 Satisfactory Satisfactory 300.00 H. Restructuring (if any) Not Applicable. v 1. Program Context, Development Objectives and Design 1.1 Context at Appraisal 1. The Development Policy Loan (DPL) formed part of a continuous engagement of the World Bank with the State of Rio de Janeiro. The operation was the third in a quasi- programmatic series of DPLs that started in 2009 and supported important public sector and service delivery reforms in the State. This engagement continues to date with follow-up policy based operations and targeted technical assistance. 2. The initial discussions of a possible third DPL took place in the early stages of the second term of Governor Sergio Cabral’s administration. The Governor had been re-elected at the end of 2010, and had begun his second term with the intention of accelerating the public sector reforms started during his first term (2007-2010). The new set of reforms was motivated by the realization of the need to show greater results, to effect more fundamental changes, and to consolidate earlier policy changes. 3. During the administration’s first term, and in the midst of the financial crisis, the Government of Rio de Janeiro (GORJ) embarked on a set of much needed reforms aimed at regaining control of the State’s fiscal position, improving public sector management, and increasing the efficiency in the provision of public services. The administration had inherited a difficult fiscal situation and a deficient administrative capacity in many areas. Prior to 2007, the annual budget process was essentially inoperative, with funds allocated in an ad-hoc basis, the revenue base had eroded, and planning was almost non-existent. 4. The operation was designed to assist the GORJ accelerate and deepen its reform process. The loan built on the prior two DPLs, approved in 2010 and 2011, that had supported key measures in fiscal management, competitiveness, service delivery and urban infrastructure. These areas had been identified as priorities in the Government’s Multi-year Plan 2012-2015. 5. The continued engagement by the Bank helped to build a solid long term relationship with the GORJ, which proved crucial in advancing the reform process in Rio. The Bank extended significant knowledge support during the preparation of the loan and worked closely with the administration in identifying measures that would consolidate previous reform efforts. 6. The Bank’s close engagement in the definition of the program was the result of accumulated experience in the State, such as in the health, education and financial management, and also drew on experience built up elsewhere. The Bank’s involvement with the GORJ was also strengthened by the close working relationship it had developed with the Municipality of Rio de Janeiro and its work with other states in Brazil. 7. The Bank had accumulated considerable experience with public sector reforms at the sub-national level in Brazil. It has worked with (and continues to work with) many states to help improve their fiscal management capacity and, more importantly, to pursue results-based public sector management reforms. This helped to build the Bank’s reputation as a trusted and knowledgeable partner in the area of public sector reforms. As a result, the authorities requested the operation and the continued participation of the Bank in the reform process of the State. 8. The time to prepare the operation was very compressed and a rapid preparation was possible because of the rich on-going dialogue between the Bank and the GORJ. The program encompassed several important areas of the public administration and required considerable 1 work effort by a large team from both the Bank and the State Government. Even though discussions on some topics had taken place over an extended period of time, the definition of the details of the program occurred over a short period of time, from late 2011 and early 2012. To ensure the strength of the program, it was necessary to reach quick resolutions and agreements by the involved parties. This included also establishing estimates of expected results two years hence, which are now the basis for the evaluation of the program in this ICR. 1.2 Original Project Development Objectives and Key Indicators 9. The main objective of the operation was to support the State Government to strengthen its tax administration, improve the efficiency of public financial management, and increase the quality of public service provision in education and health. The overarching goal was to promote broader changes in the administrative structure of the State Government, to improve its institutional capacity, and to consolidate and secure the advances made since 2007. 10. The key outcome indicators and expected results under each policy area are summarized below. 11. In the area of Tax Administration: The objectives were to strengthen and increase the efficiency of tax administration, to raise additional resources, and maintain a solid fiscal position. The prior actions included legal and administrative measures designed to strengthen the basis for the introduction of results based management, for establishing standards to improve tax collection, and to strengthen the tax payer registration system. The key outcome indicators and expected results were: a) an increase in the revenue of ICMS and ITD taxes by 15.6 and 12.4 percent by 2013; b) the establishment of the general tax payer ledger, allowing individual taxpayers to access their unified accounts; and c) the gradual introduction of performance indicators for tax administration, from 0 to 10 indicators by the end 2012 to 20 by the end of 2013. 12. In the area of Public Financial Management: The objectives were to build the institutional capacity and introduce the systems and procedures to improve the management and controls of public expenditures, particularly in large investments and programs. The prior actions included the adoption of a methodology and procedures for evaluating investment projects, the introduction of pilots to test methodologies for costing social and economic development programs, and the restructuring of the internal audit institutional arrangements. The key outcome indicators and expected results were: a) at least five investment projects (with costs over R$ 50 million) to be subjected to a screening process for approval, according to the adopted techniques; b) publishing the annual commitments for five large selected investment projects; c) publishing the cost and defining key results for at least four major policy programs; d) improving the internal control systems used by AGE in accordance with the 2013 Action Plan developed on the basis of its 2012 Annual Report. 13. In the areas of Public Education and Health Services: The objectives were to: first, in education, to contribute to raise the State of Rio to the top five performers in the national development index of basic education by 2013; and second, in health, to improve the management and efficiency of service delivery. The prior actions entailed formally adopting the merit based systems for selection of personnel in the education sector, and introducing the school level targets and bonus program. In the health sector, the measures included adopting the legal and administrative framework for contracting social (non-public) organizations to deliver services, and regulations governing the incentives, transfers and allocation of services by local 2 hospitals. The key outcome indicators and expected results in the education and health included: i) the merit based selection of all 28 regional directors and 164 school directors by 2013; ii) the application of school level targets for both 2013 and 2014 (established and communicated in the previous year); iii) the payments of performance bonuses in 2013 based on 2012 targets; iv) improve the participation of non-profit providers through the signing of contracts with social organizations (OS) to manage 15 health units and strategic services (e.g. lab services, imagery services) by 2012 and 22 by 2013; iv) raising the number of hospitals under the performance- based transfer PAHI1 program with ombudsman services from 44 to 63 by the end of the period; v) increasing the number of hospitals with infection evaluation committees from 46 to 66 by the end of the period; and vi) having 24 percent of the complex treatments in hospitals (compared to current 17 percent) be for patients from municipalities other than the location of the hospital. 1.3 Revised Program Development Objective and Key Indicators, and Reasons 14. Not applicable. 1.4 Original Policy Areas Supported by the Program (as approved) 15. They policy areas covered by the operation were well aligned with the priorities and objectives outlined in the Government’s Multi-year Plan (for the period 2012-2015). The main areas of the plan were economic management, infrastructure, social policies, human capital, and quality of life. The primary goal was to accelerate and deepen the reforms began in the first term and effect more fundamental changes. 16. The present operation was also consistent with and closely linked to the objectives of the 2012-2015 Bank’s Country Partnership Strategy (CPS) for Brazil, which emphasized sound macroeconomic management, fiscal consolidation, increasing the efficient public sector management, and improving quality of education and health expenditures for low income households, especially at the sub-national level, as key pillars for achieving sustainable economic development and inclusive economic growth in Brazil. 17. The selection of the components emerged from the Bank’s engagement with the State of Rio, and the joint work that had been done in the earlier phases of the reform program. While not a formal programmatic series, the sequence of loans that supported different phases of the reform process and the nature of the continuous policy dialogue essentially makes the operations a quasi-programmatic series (the main operations are summarized in Figure 1 and more details are available in Annex 6). The sequencing of the reform process and the developing priorities of the Government provided a good rationale for the content of the third DPL. These were the context in each policy area of the DPL. 18. The first State DPL focused on fiscal management and service delivery in the health and education sector. It established the basis for the current operation by supporting earlier measures to consolidate the fiscal position, improve the efficiency of tax administration, strengthen budget procedures and financial management, and raise the quality and efficiency of the education system, along with increasing access to health services. The third DPL built and deepened the work started under the first DPL by addressing the same broad areas of fiscal reform, health, and education. The operation was also accompanied by important technical assistance in the areas of public financial management, tax administration, education, and health, and by knowledge services, in both the health and fiscal areas. Technical assistance has focused on improving information systems for service delivery, strengthening the evidence base of policy, and 3 expanding performance-based management. The third DPL helped the Government launch an important new phase of reforms and initiate new ones in public investment management. Figure 1: Rio de Janeiro Quasi-Programmatic Series Note: Implementation Completion/Support Reports’ ratings: HS = Highly Satisfactory, S = Satisfactory, MS = Moderately Satisfactory, MU = Moderately Unsatisfactory 19. Policy Area 1 – Strengthening Tax Administration. The process of improving tax administration was already underway when the operation was designed and negotiated. Previously, the Secretariat of Finance had professionalized the Revenue Secretariat and adopted electronic invoices. The GORJ had seen an important increase in tax collection since 2006, primarily due to economic growth and the performance of key sectors in the economy, but also to modernization of tax administration by the Authorities. The actions included in the operation to support the on-going reform process were: (i) the introduction of performance management in the tax administration unit of the Secretariat of Finance; (ii) the integration of taxpayer information; (iii) the introduction of digital certificates for taxpayers; (iv) the development of the first stages of the taxpayer ledger; and (v) the consolidation of the collection system. 20. Policy Area 2 – Improving Public Financial Management. Improved budgetary management and administrative reforms were central to the overall program, and had been a main component of DPL I. Since 2007, the GORJ devoted considerable attention to enhance fiscal control and make budget execution more orderly. They put in place financial management systems that regularized payments and eliminated arrears. They also strengthened the institutional capacity of SEFAZ, through new staffing, policies and procedures. However, there was still much to be done to improve public investment management, to better understand the factors driving the costs of different programs, and to modernize internal controls; areas in which the Government had not until that date engaged with the Bank. 4 21. In the area of Public Investment, the institutional capacity of the SEPLAG had declined over time. There were major deficiencies in the preparation and implementation of the investment program. Even though State’s budget planning has been guided by the Multi-Annual Plan, in practice, investment decisions were fragmented among different agencies, driven by political and sectorial considerations and budgetary allocations done in an ad hoc manner. There were few formal mechanisms for filtering and prioritizing investment proposals, and hardly any feasibility studies done. While sector review committees were responsible for reviewing proposed investments, in many cases they lacked the basic planning information on costs to make an informed recommendation. They also did not have information on the operation and maintenance costs of new investments, which in some instances were not part of budgetary planning exercises. These deficiencies made imperative the need to improve the GORJ’s investment planning capacity, which became a crucial part of the reform supported by the Bank. 22. As a result, the Authorities and the Bank agreed on including key measures under the DPL in the investment area. These included the adoption of formal procedures for evaluation, selection and approval of public investment projects that had a total estimated cost of over R$50 million. In addition, the State was to carry out detailed analysis of all ongoing major investment projects to document costs and future operating and maintenance costs. The process began with five investment projects during 2012 that served as pilots.1 This was to be followed up with ex- ante analysis for other projects above the R$50 million threshold. It was recognized that it was going to be a major undertaking, since there were 26 investment projects over R$50 million in the budget for 2012, accounting for 68 percent of the R$6.8 billion State investment budget. 23. Costing of Programs. In recent years the GoRJ had improved service delivery by expanding or introducing new programs, in transport, housing, health, and income subsidies. These programs included basic cost estimates when first introduced, but these estimates did not fully capture the future fiscal implications of the programs. To improve the planning, the authorities piloted a program to improve their costing methodology. This was done prior to the preparation of the loan, and was applied to four small programs. As part of the DPL, the costing methodology was expanded and was to be applied to four large programs. The objective of the new approach was to derive methodologically consistent data for future cash outflows. 24. The development of costing methodologies for selected programs was initially produced by two working groups and served as basis for the follow up work. These methodologies included standard cost parameters, basic scenario analysis and expected costs on an annual basis for at least the next three years. The methodology was to be applied to four major programs: the Bilhete Único, Renda Melhor, Distribuição de Remédios, and Aluguel Social, during 2012. This exercise was to provide the GORJ with a baseline for evaluating the cost-effectiveness of expanding services and for increasing the predictability of budget and fiscal sustainability. 25. Auditing and Internal Controls. Finally, the authorities initiated the modernization of the internal control and audit system. Prior to the preparation of the DPL, the Auditoria Geral do Estado (AGE), the agency responsible for internal control in the State, focused mainly on ex-post reviews of accounting and procurement documents for actions already taken by a myriad of 1 Including: Arco Metropolitano (R$230 million); Ampliação da Rede de Ensino (R$222 million); Ampliação da Via Light (R$161 million); Conservação de Rodovias (R$133 million); Projeto Iguaçu (R$89 million); Projeto Saneamento (R$80 million) projects. 5 entities and satellite State organizations. Essentially all of AGE’s work was compliance based and had limited capacity to carry out risk-based, performance or value for money auditing. As part of the preparation of the DPL, the Bank team helped the agency identify the process required to modernize and shift its focus away from compliance to risk based performance audit. 26. Prior to the approval of the DPL, the Government issued a new decree governing the role of AGE and adopted associated initiatives, allowing the agency to rebuild its workforce and to undertake a more effective work program. The decree created a new management and career structure for the accounting and audit personnel of the State Government. It also established a process for AGE to develop annual audit plans and report annually on its accomplishments. This sub-component effectively helped change the focus and modus operandi of AGE. 27. Policy Area 3 – Quality and Efficiency of Public Education and Health Services. The final main component of the operation was to support important changes in the provision of public services in education and health for lower income groups. The actions represented an important change in the models for the provision of quality public services. 28. Human Capital (Education). Improving education was a key element in the GORJ’s agenda to promote social inclusion, labor productivity and growth. The Government wanted to tackle poor educational outcomes and improve student learning through accountability reforms. Rio had set an ambitious target of becoming one of Brazil’s five highest-performing states on the Federal Government’s IDEB education quality index by 2013 (in contrast to 26th position or second to last in 2009). To achieve this, the State Secretariat of Education had made some progress prior to the loan preparation by building a solid administrative technical team and developing a comprehensive reform agenda. This work was supported by the Bank through a Technical Assistance Loan and a close working relationship among the teams. The reforms were in line with global best practices and research on how to improve school systems. They started with curriculum reform and changes in the training for teachers, and eventually included wider management and governance reforms. 29. The reforms had two important and overarching goals. The first was to raise the quality and accountability of school managers. In January 2011, the State adopted legislation that established new processes for the appointment of school district managers (regional directors) and school directors. For the first time, these appointments were to be made through a competitive selection process, and not through political appointments. The same legislation also introduced the first-ever process of annual performance evaluation for regional and school directors. These politically sensitive measures were expected to increase the transparency of the selection processes and to upgrade the technical quality of the state’s top education managers. 30. The second goal was to strengthen the incentives for schools to focus on results, with the aim of improving the State’s low test scores and graduation rates. The Authorities introduced in January 2011 annual school-level improvement targets and bonus pay linked to these results. The State also adopted an annual learning assessment test, which was based upon the national test, to monitor learning outcomes more effectively. The personnel in the schools that achieved at least 90 percent of their targets received a salary bonus (of up to 25 percent of their annual salary). The targets for each year are based on a linear projection of the improvement required at the local level in order to raise the state’s overall national ranking. 31. Health. The primary focus of health policy in the State was to improve the management in the sector and increase its efficiency, while expanding the coverage and quality 6 of services. To achieve these goals and to promote better governance, the GORJ introduced new contract models based on payments for performance. The Government also sought to improve the coordination between the primary health services and hospitals in small municipalities, by setting-up better referral and counter-referral processes and by increasing the capacity of regional hospitals to respond to more complex health care needs. In addition, the Government expanded access to urgent and emergency care by extending the 24-hour emergency care unit (Unidade de Pronto Atendimento–UPA) model beyond the Rio de Janeiro Metropolitan Region. 32. The Government adopted foundational measures in the early phases of the reform program. It created the legal framework for implementing new models of contracting for health services from non-profit providers. In addition, it provided support and financial incentives for hospitals in small municipalities in the interior to improve primary health care and the quality of hospital care (PAHI1). It introduced incentives for regional hospitals to respond to medium and high complexity care needs, including for patients from outside the municipality where the hospital is located (PAHI2) 2 . Finally, it created the State Health Foundations to introduce flexibility and strengthen performance orientation and accountability in the system. 33. The phase of the program supported by current operation entailed the launching of many complementary initiatives, particularly those of subcontracting social organizations emergency local units and some public hospitals. This new management model was to provide the state with increased flexibility to expand the supply of health care services and was expected to stimulate innovation and improved efficiency in service delivery. In addition the DPL supported improvement of the system of incentives to municipal governments and hospitals under the Program to Support Hospitals in the Interior (PAHI1 and PAHI2). Specifically, new regulations established indicators, administrative systems, and other operational features of the program, with the aim to strengthen primary care and hospital services in small municipalities (less than 115,000 population), and enhance the quality, efficiency and inter-municipal coordination of medium- and high-complexity services provided by regional hospitals. 2. Key Factors Affecting Implementation and Outcomes 2.1 Program Performance 34. This was a single-tranche operation which disbursed on effectiveness. All policy actions required by the Loan Agreement and described in section 1.2 were met by GoRJ prior to negotiations. The loan was approved by the Board on August 30, 2012 and signed on October 30 of the same year. Effectiveness was declared on November 16, 2012 and the project closed according to its original schedule in January 31, 2014. 2.2 Major Factors Affecting Implementation 35. The implementation experience of this program was consistent with many similar operations and reinforced lessons already learned. As with any set of comprehensive reforms, implementation always poses a considerable challenge. This is more pronounced when the program entails building institutional capacity in challenging environments. The implementation 2 PAHI1 (Programa de Apoio aos Hospitais do Interior 1); PAHI2 (Programa de Apoio aos Hospitais do Interior 2) is dedicated to support improvements on management and infrastructure for regional state hospitals or those hospitals with medium complexity covering more than one municipality according to the health regionalization plan of the Rio de Janeiro State. 7 issues under the DPL III can be grouped into four different categories: (i) those of political leadership, commitment, and personnel capacity; (ii) those of learning by doing; design updates, and adjustment; (iii) those of unexpected developments, technical setbacks, and unrealistic timetables; and (iv) those of technical assistance, close supervision and follow up. 36. The success of the program in different areas was clearly associated with the administrative capacity of individual agencies, the political leadership within those agencies, and the degree of commitment and focus by the implementing agencies. The greatest accomplishments occurred in those areas where the responsible authorities were confident on the nature of the reforms and the goals to be attained, and were able to garner the full commitment of the involved personnel to achieve the established results, and had adequate administrative capacity. In all successful instances, the reforms were also central to the goals of the agencies. This was most evident in the case of SEFAZ, where there was a strong commitment to strengthen fiscal controls and improve tax administration, consistent with the design of the loan. The same can be said about the reforms in education, where strong leadership provided guidance to implement reforms that faced considerable political resistance. It took persistence in this case to begin to ease opposition to the reforms. 37. The most difficult area was the public investment program, where the institutional and political economy constraints were quite significant. Both the Bank and the authorities recognized this was a critical component, given the deficiencies in investment planning and cost control. The Bank team noted the importance of introducing changes in the manner investments were considered, selected, and funded over time. This was reflected in the proposal that five new large investment proposals had to be reviewed under a new set of evaluative and screening rules. The authorities agreed, but had reservations on the ability of the Government to carry out the proposed reforms. They entailed a significant change in the operations of the investment agency and the collaboration of powerful executing line agencies, as well as confronting strong political interests and institutional constraints in a short period of time. Still, the authorities agreed to developing a more coherent process, in part as a managerial strategy to promote changes in the Secretariat of Planning and the line agencies, while recognizing that those were going to be difficult to achieve. 38. The actions in the investment and costing areas proved to be even more challenging than expected. SEPLAG did not focus on implementing the new screening process, as day to day responsibilities and budget execution trumped planning. The Secretariat has a mandate to carry out investment decisions of the Government, and to ensure that proper financing be available to the chosen projects. The GORJ did not take additional measures to ensure that the Secretariat’s capacity was strengthened to properly evaluate proposed investments, which emerged from powerful executing agencies. These agencies had greater capacity to promote and obtain support for their investments. Furthermore, often the investments came with associated financing that were outside the standard budget process. All of these factors contributed to SEPLAG’s inability to achieve the expected results. In this component, clearly there was a need to focus on strengthening the institutional capacity of the agency, before establishing goals that required considerable cooperation and coordination with many other agencies in Government. 39. Another important implementation issue was the adjustments made through a process of learning by doing. In several areas of the program, there were adjustments made after the initial measures were carried out. Many of these adjustments improved the program, especially in the tax administration and health areas. Following the adoption of performance indicators in the tax 8 area, staff was able to identify a number of measurements, practices, and procedures that needed to be further reviewed. This led to an analysis of how they could be adjusted to improve tax administration. The same was in the case of health, where the experience with the UPAs’ contracts led to a constant updating and adjustments of their content, and an application of a similar approach to local hospital services, which had not been contemplated initially. Another illustration of the learning by doing process was the expansion of the criteria and indicators used to determine the incentives provided to hospitals. Initially, two basic indicators were included in the operation (the introduction of ombudsman services and infection evaluation committees). Soon after the loan was approved, the responsible staff for implementation recognized those indicators were not sufficient to bring about the expected changes in hospital management, and introduced other indicators to give financial recognition to hospitals for their efforts. 40. All areas of the program confronted some unexpected events and a few areas some technical setbacks. While not significant to derail any major part of the program, they introduced some delays in implementation. This was present mostly in the areas that required gathering information from different agencies, such as in the costing of expenditure programs and the investment planning area, and areas that required introducing technically complex processes, such as in the development of a taxpayer single account (or general ledger). The latter has been difficult to construct and is not likely to reach its final stage until a year after the original expected date. Just after the closing of the loan, the technical specifications of the proposed ledger would be tested and validated. The final stage of making the information available to the taxpayers still has to go through several steps, including a review of security considerations. The anticipated results in this area did not foresee many of these implementation issues. 41. Finally, the implementation experience clearly shows that a close engagement with the authorities, either through technical assistance or close supervision, significantly increased the chances of achieving the expected results. This is shown by the fact that in those areas where the Bank had an ongoing technical assistance program closely aligned with the DPL, results exceeded expectations. Nowhere is this more evident than in the education sector, where the technical support predated the preparation of the DPL and continued during its implementation. The same is the case, albeit to a lesser degree, with the engagement in the health sector. Whereas in those areas where the Bank followed or assisted the Government less closely with the implementation of the program, the results were mostly not satisfactory, especially in the area of public investment management. Bank staff lacked resources (budget and time) to support closely the actions by the Government and had to rely on the ability and willingness of the respective agencies to reach the expected results. The same Bank staff were also tasked with the preparation of new operations (in Rio and elsewhere), thus diverting some of their focus from implementation support. A closer engagement with the implementing agencies could have prompted earlier corrective actions and ensure better performance. 42. Still, the assessment of the program’s implementation experience is positive, since it hinges on measuring the degree of accomplishment against the expected outcomes. Most goals were achieved, and the results were essentially according to plans. The preponderance of the positive results reaching or surpassing the original goals substantiates the generally positive evaluation of the program. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 43. The monitoring and evaluation of the program was based on the expected results for each component. Those results were fairly detailed and had specific dates attached to them to 9 follow the achievements under the program. There was, however, no formal M&E arrangement for the whole operation. While the authorities formed an ad-hoc committee within SEFAZ to monitor the actions agreed under the loan, the group worked mostly to satisfy the needs of the periodic supervision missions of the Bank. There were dedicated groups that closely monitored the advances of the reforms in the Education and health sectors, but those were essentially set up for implementation purposes. In the education sector, there are ongoing impact evaluations supported by the Technical Assistance that are expected to provide additional information on the results from implementing a new management model and results-based pay in secondary education. A closer scrutiny and monitoring of the actions of those responsible for carrying out the measures agreed under the program could have helped the program performance. 2.4 Expected Next Phase/Follow-up Operation 44. The Bank is providing continued support for the implementation of reforms in the major areas of the DPLIII. This is occurring through a follow up operation, DPL IV (Enhancing Public Management for Service Delivery) and two Technical Assistance Loans. The new DPL operation is assisting the GORJ to enhance public management for the delivery of key public services for vulnerable populations by instituting new policies and regulations to improve: the medium term planning and monitoring of public expenditures (Policy Area 1); the accessibility, quality and affordability of urban mobility services for the poor (Policy Area 2); and the availability of targeted social services aimed at reducing domestic and gender based violence (Policy Area 3). 45. The Bank also has two Technical Assistance Loans in place—Public Sector Modernization and Strengthening Public Management and Integrated Territorial Development. They support the GORJ’s efforts with performance-based management, and the introduction of information technologies aimed at improving service delivery in key areas, including secondary education and hospital care. The loans are also supporting the modernization of the tax administration and public finance management. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. The objectives of the loan were and continue to be highly relevant for improving the public administration in the State of Rio. The objectives were consistent with the Government’s reform program, but also with the identified needs of the State in important areas of the public sector. The relevance is enhanced by the continuity of the program. Consistency in pursuing different phases of the reform over an extended period adds to the significance of the program. Furthermore, many of the objectives entailed fundamental changes in the operation of the sectors, in the way that resources are allocated, incentives are structured, performance is evaluated, and services are rewarded. One of the most important aspects in terms of both objectives and design was the expansion of the use of indicators and modern management methods. 47. The design of the program was also very relevant for achieving the expected results, focusing mostly on the policies and measures that were critical to the success of the Government’s reform program. It ensured continuity and furthered the transformation of key areas of public administration. Even in the investment area, where there was recognition that the program was quite ambitious, there was a need to promote reforms, given the critical role of better investment planning for the budget process. 10 48. Implementation did not match the ambition of the design, but can still be considered good, particularly when evaluated from the perspective of the achievements under the overall program. There were considerable efforts to focus actions on the most important components, and the result was, on balance, very positive. Fundamental reforms were undertaken in the education area, substantial improvements were made in tax collection, there were some gains in the efficacy of tax administration, and important performance based management tools in both the health and education sector were adopted. The major shortcoming remains in the area of public financial management, particularly as it pertains to investment decisions. 3.2 Achievement of Project Development Objectives 49. Achievement of the PDO as a whole is rated moderately satisfactory, as most of the results indicators were achieved or exceeded their targets. This is a weighted rating, which takes into consideration the importance of the achievements in key service areas, such as Education, as compared to the disappointing progress in some parts of the public financial management area. A very significant part of the reform program was successfully implemented. The operation identified 15 specific results and outcome indicators to be achieved by the end of 2013. Ten of those were fully met; three were mostly achieved; and two failed to satisfy basic expectations. The achievements were most significant in the education and health sectors, as well as in tax administration. Results failed to meet initial expectations primarily in the public investment program. 50. The positive evaluation of the program is also based on the consistency and persistence of the reforms over an extended period of time. In this context, the DPL should be seen as part of a sequence of operations supporting earlier phases of the reform program. The current operation continued these achievements by supporting deeper, and in some cases more significant, reforms. Policy Area 1 – Efficiency of Tax Administration. Rating: Satisfactory 51. The overall rating for this policy area is based on the fulfillment of the original objective, which was to strengthen and increase the efficiency of the tax administration, to raise additional resources, and maintain a solid fiscal position. The State improved on all fronts. The rating is reinforced by the fact that in two of the sub-components (revenue generation and the introduction of performance indicators) the achievements exceeded the expectations established at the time of the loan preparation. In the third sub-component (the establishment of a general tax payer ledger), delays revealed the difficulties of introducing such an extensive administrative reform, when both political and technical factors can alter the best plans. 52. Tax Revenues. Performance in this sub-component was highly satisfactory. The final revenue figures for both ICMS and ITD exceeded the 2013 targets by 8.4 and 41.8 percent points respectively. The initial targets, following a correction of the baseline, were for increases of 15.6 percent for ICMS and 12.4 percent for ITD. The good performance was due to both internal actions taken to improve tax collections and external developments, including the rise in food prices and higher vehicle sales. These developments offset the lower than expected revenues in other areas such as the electricity sector, where the authorities expanded exemptions. It is difficult at this stage to determine the extent to which the increase in revenues was due to the tax collection measures or simply better economic conditions. The actions aimed at improving the efficacy, effectiveness and efficiency of tax collection are in their early stages. It will take time for them to be fully operational and to show solid results. The initial targets for increasing ICMS 11 and ITD considered some gains in the efficacy of tax administration, but those cannot be substantiated with the current information. A proper analysis of the impact of efficiency improvements in tax collection would require knowing the initial assumptions and calculating the effects of unexpected economic developments, data that is not available for the preparation of the report. 53. General Tax Payer Ledger. The result of this prior action was expected to be the establishment of a unified general taxpayer ledger by the end of 2013 that would be available to individual taxpayers online, as the final result of a broader process of improving and upgrading the tax collection systems, called Gestão do Crédito Tributário (GCT). Its achievement can be considered moderately satisfactory. The program required advancing through a series of sequential steps before the ledger could be fully operational, which took longer and proved more difficult to effect than initially anticipated. The platform and information consolidation has been accomplished, and the authorities are now entering the stage of testing and certifying the technical adequacy of the new system. It is not yet certain when the ledger will become fully operational. The tentative date is the beginning of 2015. The ledger will first be used internally to analyze the tax situation of individual business entities. The slippage in the calendar is also explained by unanticipated events, such as changes in tax laws that demanded urgent changes in many systems and caused the pace of GCT project to be significantly slowed. 54. Tax Administration Performance Indicators. The sequenced introduction of performance indicators was carried out as envisaged. The final batch of indicators was done in March of 2014. The last phase occurred a bit later than expected, but the delay, due mostly to personnel changes, did not affect the overall thrust of the program. Hence this sub-component is considered satisfactory. Overall, SEFAZ’s experience with the indicators has been positive. Their introduction has led to a better understanding of the deficiencies in the processes and also to an adjustment in their management. The simple act of collecting information prompted the authorities to investigate the factors behind certain outcomes and to adopt measures to improve the results. However, most of the indicators are geared towards looking at the efficacy of the tax administration. They are internally oriented and tend to focus on administrative enforcement; few measure efficiency and effectiveness, particularly in terms of the quality of client services. The indicators were constructed depending on the availability of data and administrative demands, and are now being organized according to the logic of administrative procedures. The calculation and the disclosure of the indicators remain restricted to a small group within SEFAZ. There seems to be room to improve in both of these areas, by automating the calculations, and making them more broadly available to all tax administration employees. In addition, there are benefits to creating formal arrangements such as regular high level meetings to discuss the indicator outcomes, and defining appropriate actions. Policy Area 2 - Improving the Efficiency of Public Financial Management Rating: Moderately Unsatisfactory 55. The evaluation of results under this policy area is heavily influenced by the failure to achieve the original goals in one important area, not necessarily the progress that has been made in each sub-component. As a whole, there have been improvements in the public financial management of the state, in part due to actions contemplated under the loan. But those actions yielded results well below expectations, and failed to reach a key goal of the reform program. 12 The shortcomings were most pronounced in the public investment area and to a minor extent in the costing of Government programs. 56. The failure to properly establish an investment review, screening and selection process implies a major gap in the Government’s ability to improve its fiscal management. The measures in this area were critical for the broader objective of the operation, which was to solidify fiscal control. The overall achievements in this policy area are in the right direction, but they fall short of expectations and much remains to be done in this area. 57. Public Investment Management. The implementation of this sub-component was under the purview of the State Secretariat of Planning (SEPLAG), which has considerable responsibilities for administering the information of a large and diverse public investment portfolio. The reforms supported by the loan anticipated a significant change in the manner in which proposed investments, particularly large ones, were considered by the agency. The program not only attempted to change the procedures for reviewing, evaluating and selecting investments, but also introduced a pilot program, where 5 new investment proposals would be assessed under the new procedures. However, for the 2013 budget only 3 new investment proposals were identified and subjected to the proposed process. The evaluation was done only ex-post and was not included in the ordinary budget process, thereby defeating the purpose. The objective became a mere bureaucratic obligation. The proposed methodology was not incorporated into the 2014 budget cycle. SEFAZ introduced a parallel process to calculate possible future commitments emerging from maintenance and operational costs. The duplication of efforts shows the importance of improving the process and garnering greater support from SEPLAG and improving cross-agency cooperation. The achievements under this sub-component are considered unsatisfactory. 58. Costing of Programs. Most of the targets under this sub-component were achieved, but overall the effort fell short of the broader goal of analyzing the cost-effectiveness of the programs. The efforts to estimate the costs of four large public programs helped the authorities to gain a better understanding of the features of those programs. With the analysis, SEFAZ was able to link the costs of the programs to the number of beneficiaries, and begin to identify the factors, such as eligibility criteria, that could drive future expenditures. This provided important information and was a first step in developing the required budget tools for improved management of the programs. 59. Despite the achievements, however, the initiative is not yet an effective tool of budget analysis. The costing only covered actual expenditures and number of beneficiaries. There are no well-developed projections of future costs. There are also no means to undertake a cost-benefit analysis, given the lack of information on benefits provided. Officials recognized many of these shortcomings, and are hoping that the technical assistance provided by the Bank will spur advancements in the application of the methodology. Finally, there is the important issue of transparency, which was to be addressed by the publication of the estimates and analysis. The costing reports were not made available beyond the working group, due to delays in obtaining and refining the information needed to produce the report. The authorities, however, reaffirmed their commitment to make the report public after it is finalized. As a whole, therefore, the sub- component can be classified as being moderately satisfactory. 60. Internal Audit Agency and Controls. This sub-component proceeded according to plan, and its achievements were satisfactory. The measures taken included: developing an annual 13 plan of action and reporting on their accomplishments; issuing key audit regulations; training additional staff; creating an internal career plan; and hiring a new cadre of younger professional auditors. In addition, staff has been decentralized to the executing agencies. New procedures have been established to create oversight on the basis of results and ex-ante procedures, moving away from ex-post reviews. These are the first steps in a long and slow process of incorporating modern audit and accounting systems in the State Government. Policy Area 3 - Quality and Efficiency of Public Education and Health Services Rating: Satisfactory 61. Education Sector. The reforms undertaken in this area were quite significant, and all of the expected results have been fully met. Hence, the achievements under this sub-component can be considered highly satisfactory. The core elements of the program were designed to change the incentive structure for educators and school administrators, with the purpose of improving student learning performance and reducing repetition and drop-out rates. Administratively, the Education Secretariat was able to appoint all new regional directors under a competitive system, along with 15 percent of all school directors in the first three years. The Secretariat also streamlined the regional structures, consolidating 30 regional supervisor positions into 14 new districts, reducing the number of regional directors by more than half. To signal a stronger focus on school quality and student learning results, the Secretariat created a new position of regional pedagogical supervisor. On incentives, all schools received individual performance targets. More importantly, the staff in about 30 percent of the schools received bonuses for meeting their targets. The bonuses were substantial, amounting to approximately 25 percent of the annual salaries. The number of schools achieving 80 percent of their annual targets, and therefore receiving bonuses, increased by 23 percent in comparison with the first year. 62. These reforms were politically sensitive and required considerable determination to carry out. The authorities had to overcome opposition from many groups, particularly those who benefited from the previous arrangements, and had to be convincing to overcome doubts about targets and bonuses based on standardized tests. Given this environment, the achievements under the program are even more noteworthy. While it is early to assess the overall improvement in the quality of education in the State, very preliminary indications show an impressive rise in the State’s ranking for secondary education quality, going from 26th in 2009 to 15th in the latest national IDEB scale; one of the largest improvements of any state. Results have consistently improved in the annual Index of Educational Development of the State of Rio de Janeiro as well. Crucially, there are indications that governance changes are being accepted, and a new culture focused on results has taken hold among school leaders and administrators. Among other innovations, the Secretary of Education introduced meetings of the entire state leadership team (approximately 600 state and district level administrators) every six months to review progress against district goals, to highlight progress and disseminate lessons from the principals of schools that have made exceptional gains, and to develop strategies for intensive support to schools with low performance. While the teachers’ strike for a 19-percent wage increase in late 2013 are expected to have a negative effect on the State’s results in the 2013 IDEB (to be released in August 2014), they are unlikely to offset the substantial improvements registered since 2010. 63. Health Sector Reforms. The achievements in this area were also satisfactory. Essentially all but one of the expected results was achieved. The primary focus of health policy in the State was to improve the management in the sector and increase its efficiency, including 14 expanding coverage and improving services. In the area of contracting social organizations (OS) to manage emergency services, the Government exceeded the targets, and now has 30 UPAs under contract, as compared to the 22 originally expected by this time. The process of contracting service providers has dramatically changed the modus-operandi of the Secretariat of Health, which is now much more focused on contract management than oversight of direct provision of services. In fact, management contracts have now been extended to some local hospitals, which were not planned initially. Contracts still need to be improved, in areas such as reimbursement for performance targets instead of the cost for providing services. But, still significant progress has been made. The goals on referrals and coordination of treatment were also achieved. More than 25 percent of complex treatments handled by regional hospitals were for patients outside the immediate jurisdiction. 64. Finally, the Government has successfully handled the PAHI program, which provides financial support to hospitals that are managing patient care according to certain targets and indicators. Initially, the DPL operation focused on two indicators to gauge progress; the number of hospitals that adopted an ombudsman program to review patient treatment, and the number of hospitals with an infection evaluation committee. The number of ombudsman has been met, but the created committees were found not to be very effective in achieving their goals. Because they did not operate as expected, the Government expanded the indicators for this purpose, to include hygiene standards and hospital waste disposal, among others. The broader approach served to accomplish the goals initially established for the committees and a better indicator to track results than the original one. While there are preliminary indications of improvement in hospital disease management, those are not readily and consistently available for outside evaluation. A new monitoring system is under implementation as part of the first Technical Assistance Loan to address this gap. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 65. The justification for the overall rating is based on the magnitude of the program and the results achieved. As noted in section 2.1, the operation supported reforms in important areas of the public sector in the State of Rio de Janeiro. Two thirds of the target results were fully met. While the results were not uniform, the main achievements, particularly in the education sector and to some extent in health and tax administration, were significant and likely to be sustained. These results offset any shortcomings in other areas, such as public investment. 66. The operation was well designed to help launch a new phase of reforms that included some noteworthy changes in the nature of key public sector programs. While the overall approach was to strengthen results-based management, the more immediate achievement was to help personnel to focus on targets, to review processes and procedures, and to improve service delivery. More importantly, the program helped set up new incentive structures that over time could help change the culture within certain sectors. Many of the initiatives helped to build the institutional structure of the State Government and are likely to continue in the future. 67. The positive evaluation of the entire program is also based on the consistency and persistency of the reforms over an extended period of time. As noted, the operation was part of a series of loans that ensure the continuity of the reform process in Rio. Even in those areas where the outcomes were not as good as expected, such as with the public investment program, the continuous involvement and inclusion of the issues on the debate with the authorities raised their 15 profile and led to some improvements. Better management of the public investment remains a critical issue for the State Government and still need to be addressed. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 68. The evaluation of the main beneficiaries of the reforms supported by the DPL was quite extensive and thorough. It showed that those in the lower quintiles of the income distribution and the most vulnerable groups in society would be the main beneficiaries from the reforms and the improvement in the service delivery attained in education and health. It is not possible to validate these assertions ex-post, given the early stage of the reform process. The information for a deeper analysis is not available at this time, although an impact evaluation is underway in the education sector. But, it is safe to assert that the reforms should have a positive social impact, given the characteristics of the primary users of the public services being reformed. (b) Institutional Change/Strengthening 69. Unquestionably, the operation has contributed to build the institutional capacity of the GORJ. In some cases, such as in the education sector, this has been done with the close assistance of the Bank. In other cases, this has taken place through a process of learning by doing, as the executing agencies carry out the reforms aimed at achieving the targets established under the program. Furthermore some of the changes, such as in the adoption of the contracting for services in the health sector, led to a change in the focus of the agency away from direct provision of services toward oversight and enforcement of contracts. In the case of tax administration, the institutional strengthening (which was initiated in earlier operations) is taking place in part through the adoption of performance indicators. These have highlighted a number of issues and provided new insights about operational efficiency. With the new information, staff are now reviewing processes and procedures in order to improve institutional efficiency. Finally, the most important change is taking place through the incentive structures that are being introduced in the education and health sectors. They are leading to a new culture of focusing on targets, and being remunerated accordingly. Still, one of the most evident lessons from the operation is that creating institutional capacity is a slow methodical process that requires considerable persistence and close support from principals, which was not always present. 4. Assessment of Risk to Development Outcomes Rating: Moderate 70. The second term of the current administration is coming to an end in 2014 and there are some reversal risks. This is particularly the case in the area of tax and costing of programs where the policy changes were instituted through directives and have not been fully consolidated. Continued strong leadership at the executive level and focus on seeing through the reforms taking place would be critical to preserve the gains made. 71. Conditions are more positive, however, in the education and health sectors where the changes are taking root and their benefits are becoming more apparent. This is particularly the case for those involved in the service provision, who are now benefitting from additional financial incentives. The reforms have built constituencies that have an interest in the continuation of the programs. The cultural change taking place is an element that should reduce the possibilities of reversal. A similar development is taking place in the Secretariat of Health, where the new focus on oversight of contracts is seen by staff as a much better use of their 16 talents. The same can be said for the introduction of new systems, processes and procedures in tax administration and internal audit. The staff responsible for the introduction of these new tools is already developing follow up improvements. All these factors contribute to making more difficult to reverse the policies and return to the previous management practices. 5. Assessment of Bank and Borrower Performance Rating: Moderately Satisfactory 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 72. As noted in Section 2.1, the Bank worked closely with the GORJ during the preparation of the loan, providing important advisory support in the design of the program. The Bank also worked to ensure timely financing. The time-frame for the preparation of the loan was driven in great part by the financing plan of the Government. 73. The preparation of the loan and the design of the program benefitted enormously from the experience of both the Authorities and Bank staff. Both sides understood the challenges of the program and generally agreed on the follow up measures needed to accelerate the process. There were solid diagnostics of the existing conditions and very good analysis of required policies. The Bank had a group of dedicated professionals for each area, and also brought top experts to discuss the experiences of other states and countries. The advice was followed in some areas more than others. For example, the modernization of the audit agency continues to date according to initial recommendations. In the health component, an important recommendation was to develop an information system with cost standards for service providers (a dashboard with scorecards). This has yet to be done, in part because of magnitude of the task and the technical challenges. But it reveals the breadth of the advice provided during preparation. The experience with the DPL underscores the fact that programs are more likely to succeed when there are high level professional teams involved in the design stage. 74. There was also sufficient preparation time to develop a sound program. Initial discussions on the nature of a loan began in late 2010 and pre-appraisal missions formally began in the later part of 2011. These involved both sides and resulted in detailed sector background information and indicator monitoring data. In particular, the World Bank technical staff helped assess the proposed policy actions against international best practices. They also carried out a thorough review of the fiscal situation of the State and a beneficiary analysis. 75. The evaluation of the Bank’s performance during preparation is partially affected by the adoption of ambitious targets for some parts of the program. The Bank team promoted strong measures and bold goals, in part to propel the executing agencies to move expeditiously and also to strengthen the quality of the program. In most instances, such as in education, this approach was successful, whereas in others, such as in the goals for the investment reforms, it failed. The final decision to have ambitious targets fell to the central authorities, who understood the challenges that the line agencies would confront. It was based on a calculus that setting ambitious goals would drive actions. It turned out that without the proper political support, leadership, and technical assistance the goals were simply not achievable. Bank staff recognized that agencies needed more time and assistance to proceed with the basic elements of the reforms, but was not able to provide it uniformly because of staff and budget restrictions. 17 76. Nevertheless, the Bank team responded to the client’s needs, with timely financing and advice; it was an adequate and timely response to the challenges faced by the GORJ. The actions ensured continuity and the adoption of measures that will have lasting impact, particularly on the provision of and management of public services. (b) Quality of Supervision Rating: Moderately Satisfactory 77. While Bank support and involvement during the preparation was highly significant, more resources and staff time would have improved outcomes. The loan established a period of 18 months to meet the targets. This provided a short window for the Government to demonstrate results. During that time there were (two) full formal supervision missions, which were mainly geared at obtaining information on the advances being made. Only in a few areas did the Bank accompany closely the progress of the implementation of the program and assisted the authorities on a continuous basis. In many other areas, such as with tax administration and public investment, the Bank did not have a close presence. The missions were able to ascertain the progress made, but did not have the resources (time and budget) to engage closely with the authorities and help develop remedial measures or assist with the details of implementation. More could have been achieved with closer supervision. 78. The rating here is not necessarily based on the effort by Bank staff or for that matter the quality of the dialogue with the authorities. Instead, it is a reflection of the fact that supervision of a single tranche loan disbursed on the basis of prior actions does not carry the same priority to Management as other loans. The nature of the supervision per se is different, simply to monitor results. But, the needs under the loan were also to help build the institutional capacity, which required a much stronger focus on the processes and procedures, not just the results. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 79. Bank guidelines suggest that a moderately satisfactory in one of the dimensions of Bank performance, should automatically translate also into a moderately satisfactory rating for the overall Bank performance. This presupposes that each dimension, quality at entry and supervision, should carry similar weight. While the guidelines are followed here, it should be noted that in this operation, as in many other single tranche loans with short periods of execution, supervision is not a major element of the Bank’s involvement. Most of the work by the Bank in this operation took place as part of an on-going dialogue with the GORJ as part of other loans, and not so much as part of the supervision of this particular operation. Hence, when considering the total effort and involvement by the Bank, perhaps greater weight should be given to preparation and design. 80. The Bank involvement and support to the State under this operation was very sound. It assisted the State to deepen a reform process that improved many aspects of the public administration in the GORJ. The Bank has maintained a close policy dialogue with the State, and has developed follow up operations that will ensure continuity in the process. The Bank is committed with technical assistance to help build the institutional capacity of many agencies. The only aspect that detracts, from an otherwise, very good performance is the constraint faced by the public sector and fiscal teams to be even more closely engaged during the implementation. 18 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 81. The government’s performance is evaluated on the basis of the strength of the program, the breadth of the results achieved, and the role of different agencies involved in the process. The reforms entailed significant efforts to improve the managerial capacity, to build better information systems, to confront political opposition, to change the modus operandi of certain sectors, and to build a better institutional structure. Nevertheless, the Government’s performance was uneven. It varied by agency and different areas of intervention, hence the moderate rating. Still, on the whole there was an important advancement that reflects on the commitment by the Authorities to the overall objectives of the operation. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 82. While there were several agencies responsible for individual components of the program, formally SEFAZ was the coordinating agency. SEFAZ demonstrated very strong leadership and cooperation throughout preparation and implementation of the program. It administered the improvements in tax administration. It also followed the actions of the other agencies and spurred their greater involvement in the program. AGE, the audit agency, systematically pursued its modernization plan and begun to change its oversight approach. Many of the other agencies performed well, broadly in accordance with expectations, except the Secretariat of Planning, as explained in more detail in section 3.2. The Secretariat of Education exceeded expectations by assiduously executing a far reaching program that had detractors and required leadership. The Secretariat of Health embraced the program under its purview and dedicated considerable efforts to performance based management, leading to the UPA and PAHI programs meeting their targeted results. SEPLAG had challenges carrying out the expected reforms of the public investment program. The agency has remained more focused on its considerable day-to-day responsibilities of administering the investment program, and did not devote the time to develop the information and procedures expected under the reform program. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 83. The overall performance is deemed as moderately satisfactory as a result of the shortcomings in the implementation of the program. Despite the moderate rating, the achievements by the Borrower should be recognized, given the scope of the reforms pursued and the number of targets met. The operation signified an important effort to accelerate the reforms started earlier, and demonstrated a commitment to show much better results, particularly in service provision. While the execution was somewhat uneven, the aggregate effort was significant and merits recognition. 6. Lessons Learned 84. There are a number of lessons under this operation that reinforce those learned in similar programs. The implementation experience with the DPL highlights the importance of the quality of personnel capacity, of learning by doing, of design updates, dealing with technical 19 setbacks, and the need for technical assistance and close supervision. Here, only three key lessons are noted. 85. The first is the importance of considering the dynamics of the political economy of reforms in establishing performance targets, even when dealing with individual executing agencies. This was most evident in the establishment of ambitious targets for parts of the program, such as in the investment area. The Bank felt it was important for the planning secretariat to show improved results in the identification, selection and tracking of investment projects. The lead agency negotiating the loan, SEFAZ, also had a particular interest in agreeing with those targets. This approach was seen as means to induce the line agency to make greater efforts in handling the evaluation of public investments. All parties recognized, however, that without greater political leadership and without greater commitment by all involved agencies, those goals had little chance of being achieved. The decision to proceed, despite considerable evidence of the political roadblocks, proved to be over ambitious. The goals were not fulfilled and little progress has been made to date. In retrospect, it might have been better to help build the institutional capacity of the agency through other means. 86. Conversely, in those areas where there was clear leadership and commitment to the reforms, the program achieved much greater success. The greatest accomplishments occurred in those areas where the responsible authorities were able to garner the full commitment of the involved personnel to achieve the established results. In all successful instances, the reforms were also central to the goals of the agencies. 87. The second main lesson concerns building institutional capacity. Successful public sector reforms are often accompanied by measures to create stronger institutions. This includes both better organizational structures and incentives frameworks. The DPL was a good example of the importance of institution building. Those areas that saw sustained improvements, such as in education, introduced new incentive systems and more importantly new methods for improving service provision. The same occurred, but to a lesser extent in the health sector. The shift to a system of contracts for urgent care services transformed not only the service provision but also the Secretariat of Health, which is now much more focused on supervising the contracts and following up on the standards of hospital services. 88. Another example along the same lines is the modernization process that is just starting in AGE. There is a new vision on the role of the agency and the manner in which it can perform its tasks, away from ex-post compliance reviews to much more significant ex-ante risk based standard setting tasks. In general, the areas of success under the DPL program where those that saw improvements in their institutional capacity. Improvements occurred through the methodical introduction of new incentives and slow changes in behavior that helped to create a new culture in the institutions and the acceptance of modernization processes. As in most cases, implementation could have been better if there were greater focus and resources to assist in institutional strengthening, including through supervision. Achievements were more limited in those areas where institutional deficiencies were not addressed. The experience shows that greater focus and resources to assist in institutional strengthening remains a key to development, particularly as it pertains to public sector reforms. 89. Finally, one last important lesson is the value of consistency and persistency in the reform process. The DPL was the result of a long engagement with the State of Rio, as well as the city of Rio. The engagement occurred through the preparation and supervision of several 20 operations that focused generally in the same areas. This helped to build expertise and knowledge through a process of learning by doing, which proved to be critical. This knowledge not only applied to the design of the programs and the adjustment to measures over time, but equally as important to overcome the resistance to reforms. It was through a slow and methodical process of trial and error along with showing and explaining that changes are eventually being accepted. The focus on results-based management also helped to make the merits of the reforms more transparent. The continuous engagement by the Bank and the persistency of the Authorities to pursue key reforms contributed significantly to build a solid basis for future, and to make the program a relative success. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/Implementing Agencies 90. No issues were raised. The GoRJ formally agreed with the ratings and provided editorial comments that have been incorporated in the final version of the report. (b) Cofinanciers N/A (c) Other Partners and Stakeholders NA 21 Annexes Annex 1. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Responsibility/ Names Title Unit Specialty Lending Barbara Bruns Lead Education Economist LCSHE Education Andre C. Medici Sr Economist (Health) LCSHH Health Alberto Coelho Gomes Costa Senior Social Development Specialist LCSSO Social Development Pablo Fajnzylber Sector Manager, PREM AFTP2 Economic Policy Joseph Kizito Mubiru Lead Financial Management LCSFM Financial Specialist Management Yaye Seynabou Sakho Adviser MDI Economic Policy Mariana Margarita Montiel Senior Counsel LEGLE Legal Patricia Miranda Senior Counsel LEGOP Legal Magnus Lindelow Sector Leader LCSHD Human Development Roland N. Clarke Lead Economist and Sector Leader, LCSPR Public Sector PREM Governance Maria Florencia Liporaci Senior Program Assistant MNSSD Program Assistant Fanny Weiner Public Sector Mgmt. Spec. LCSPS Public Sector Governance Edith Kikoni Economist LCSPE Economic Policy Flavia Nahmias da Silva Program Assistant ECRBX Program Assistant Gomes Claudia Rocio Manrique LCSPS Program Assistant Cindy Audiguier LCSPE Economic Policy Rafael Chelles Barroso Economist LCSPE Economic Policy Ezau Pontes Senior Health Specialist LCSHH Health Rogerio Bianchi Santarrosa LCSPP Poverty Fabio Sola Bittar Research Analyst LCC5C Economic Policy Leandro Oliveira Costa Economist LCSHE Education Ana Mie Horigoshi Reis Junior Professional Associate LCSPS Public Sector Governance Supervision Barbara Bruns Lead Education Economist LCSHE Education Andre C. Medici Sr Economist (Health) LCSHH Health Mariana Margarita Montiel Senior Counsel LEGLE Legal Patricia Miranda Senior Counsel LEGOP Legal Magnus Lindelow Sector Leader LCSHD Human Development Roland N. Clarke Lead Economist and Sector Leader, LCSPR Public Sector 22 PREM Governance Maria Florencia Liporaci Senior Program Assistant MNSSD Program Assistant Fanny Weiner Public Sector Mgmt. Spec. LCSPS Public Sector Governance Edith Kikoni Economist LCSPE Economic Policy Flavia Nahmias da Silva Program Assistant ECRBX Program Assistant Gomes Claudia Rocio Manrique Program Assistant LCSPS Program Assistant Cindy Audiguier Consultant LCSPE Economic Policy Laura De Castro Zoratto Economist LCSPS Public Sector Governance Rafael Chelles Barroso Economist LCSPE Economic Policy Ezau Pontes Senior Health Specialist LCSHH E T Consultant Ana Mie Horigoshi Reis Junior Professional Associate LCSPS Public Sector Governance Angela Nieves Marques Porto E T Temporary LCSPS E T Temporary Eduardo Somensatto Consultant LCSPS Economic Policy Lorena Viñuela Public Sector Specialist LCSPS Public Sector Governance (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage US$ Thousands (including No. of staff weeks travel and consultant costs) Lending Total: 79.09 286,513.80 Supervision/ICR Total: 19.56 59,625.87 23 Annex 2. Beneficiary Survey Results Not applicable. 24 Annex 3. Stakeholder Workshop Report and Results Not applicable. 25 Annex 4. Summary of Borrower’s ICR and/or Comments on Draft ICR The GoRJ formally agreed with the ratings and provided editorial comments that have been incorporated in the final version of the report. 26 Annex 5. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 27 Annex 6: Recent Operations in Rio de Janeiro While not a formal programmatic series, the sequence of loans that supported different phases of the reform process and the nature of the continuous policy dialogue essentially makes the operations a quasi-programmatic series. Details are available in Table A.1. The first State DPL (Fiscal Sustainability, Human Development, and Competitiveness DPL– P117244) was approved in February 2010. The first DPL was done partially in response to the crisis when the Government was concerned that the fall in revenues could destabilize the overall reform agenda by jeopardizing financing for the health and education programs. It focused on the initial GORJ efforts to improve fiscal management, competitiveness, and social service delivery. The first DPL established the basis for the current operation by supporting earlier measures to consolidate the fiscal position, improve the efficiency of tax administration, strengthen budget procedures and financial management, and raise the quality and efficiency of the education system, along with increasing access to health services. The Second State DPL (Rio de Janeiro Metropolitan Urban and Housing Development Policy Loan – P122391), approved in March 2011, was in part a response to the natural disasters from heavy rains, which highlighted the issue of urban housing. The operation was designed to support the GORJ to strengthen its policies for planning and managing territorial growth in the Rio de Janeiro Metropolitan Region, and promoting the provision of affordable housing. The third DPL built and deepened the work started under the first DPL by addressing the same broad areas of fiscal reform, health, and education. The operation was also accompanied by important technical assistance, for example in the education area, and by knowledge services, in both the health and fiscal areas. This DPL helped the Government launch an important new phase of reforms. The Bank’s involvement with the GORJ was also strengthened by the close working relationship it had developed with the Municipality of Rio de Janeiro and its work with other states in Brazil. The Bank had a similar operation with the Municipality of Rio that focused on many of the areas that formed part of the State’s DPLIII. The experience of working at the municipal level revealed the importance of coordinating the reforms at different levels of government and contributed to identifying the measures that needed to be taken by the State to improve the joint provision of public services, such as in the health and education sectors. The Bank is providing continued support for the implementation of reforms in the major areas of the DPLIII. This is occurring through a follow up operation, DPL IV (Enhancing Public Management for Service Delivery) and two Technical Assistance Loans. The new DPL operation is assisting the GORJ to enhance public management for the delivery of key public services for vulnerable populations by instituting new policies and regulations to improve: the medium term planning and monitoring of public expenditures (Policy Area 1); the accessibility, quality and affordability of urban mobility services for the poor (Policy Area 2); and the availability of targeted social services aimed at reducing domestic and gender based violence (Policy Area 3). The Bank also has two Technical Assistance Loans in place (Public Sector Modernization and Strengthening Public Management and Integrated Territorial Development) to support the GORJ to support performance-based management and introduce information technologies to drive service delivery improvements in key public services, including secondary education and hospital care. As well, the loans are supporting modernization of the tax administration and public finance management. 28 Table A.1: Rio de Janeiro Operations, 2009-2013 2009 2010 2011 2012 2013 2014 CPS 2012- DPL 1 TAL (P106768) DPL 2 DPL 3 (P106465) DPL 4 (P147695) TAL II (P126735) 2015 Priority Areas Fiscal Sustainability, Public Sector Metropolitan Urban and Fiscal Efficiency for Enhancing Public Strengthening Public Human Development Modernization Housing Quality of Public Management For Management and and Competitiveness Service Delivery Service Delivery Integrated Territorial (US$ 18.6 million) (US$ 485 million) Development (US$ 485 million) (US$ 300 million) (US$ 500 million) (US$ 48 million) Fiscal  Capitalization of  Adoption of Public  Piloting of a  Treasury’s Consolidation Social Security Sector Performance methodology for operational risk and Public System for State Indicators costing public policy Financial public servants  Real Estate policy programs  Adoption of budget Management  Reorganization of Property  Restructuring of the preparation budget processes Management internal audit practices to System agency.  improve its  Social Security accountability and Management fiscal discipline System  Adoption of financial and debt management tools. Tax  Strengthening of  Strengthening of  Strengthening core Administration the Revenue synchronized taxation functions Secretariat. taxpayer  Adoption of registration system. electronic invoices.  Adoption of standards to improve collection of State taxes.  Implementation of a results-based management system for tax administration. 29 Public  Adoption of a Investment methodology and procedures for evaluating, selecting and approving proposals for large public investment projects. Private Sector  Streamlining of business registration process Education  Implementation of  Building of  Adoption of a  Developing In- learning program to SEEDUC Technical merit-based Service Teacher reduce of age-grade Capacity for selection process Training Network. distortion in basic Studies and for regional and  Mapping of education. Evidence-based school directors. demographic and  Implementation of Policy.  Establishment of infrastructure new school  School-level annual school-level needs. management Performance targets for information Targets for State improving student program Schools. learning and graduation rates and a bonus pay system for schools. Health  Adoption of  Information System  Framework to  Cost-Accounting national standard for Health Supply assess and contract Methodology for practices in 24-hour Chain Management social organizations UPAs and operating Urgent Evaluation of to manage public Hospitals. and Emergency Financial Incentive hospitals and health Care Units (UPAs). Programs for units.  Establishment of a Municipalities  Evaluation system of mechanisms to performance-based transfer bonuses transfers from the and monetary State to incentives (i) to 30 municipalities municipalities and municipal hospitals  and (ii) to regional State hospitals. Housing and  Enhancement of  Improving living Land low-income conditions in the Management housing policy and most vulnerable financing social settings mechanisms.  Strengthening land management and titling.  Enhancement of the governance of the State Housing Fund (FEHIS).  Strengthening of ITERJ’ land tenure regularization capacity  Restructuring of the Secretariat of Social Assistance and Human Rights. Urban  Strengthening  Strengthening Management metropolitan metropolitan management management through improved through integration integration and and coordination in coordination in urban development, urban development housing, transport, and transport. environment and  Establishment of a disaster risk Directive management Committee for Metropolitan Strategy and an 31 Integrated Program.  Targeted and integrated social development programs to the urban poor.  Establishment of the UPP Social Program, on a pilot basis, in the State slums. Transport  Creation of the  Adopted of the Single Ticket Urban Transport (Bilhete Unico) for Master Plan public (PDTU) and a transportation in the stakeholder RJMR. mechanism to  accompany implementation.  Implementation of a policy to physically integrate municipal and State- managed mass transit systems.  Adoption of a new policy to promote non-motorized transport.  Adoption of a policy for competitively awarding licenses for bus routes, instituting a results- based management system for 32 concessions, and increase transparency on bus service performance. DRM  Radar system.  Establishment of a  DRM. State disaster risk policy. Natural  Strengthening of Resource the Institute of Management Environment to capacity to protect environmental assets. Improvement of effective watershed management and decentralization. PDO Rating Satisfactory Satisfactory Moderately Satisfactory Moderately Satisfactory Satisfactory (ISR) Not yet initiated 33 Annex 7. List of Interviewees The following individuals were interviewed during the preparation of the report. Paulo Tafner Deputy Secretary for Finance, Secretariat of Finance, GORJ João Marcos Meneses Simas de Souza Finance Analyst, Secretariat of Finance, GORJ Luisa Mazer Finance Analyst, Secretariat of Finance, GORJ Eugenio Manoel Chief Auditor, Secretariat of Finance, GORJ Robson Ramos Audit Manager, Secretariat of Finance, GORJ José Correa Superintendent of Finance Information Records, Secretariat of Finance, GORJ Sergio Abramovich IT Director, Secretariat of Finance, GORJ Julio Montovani Finance Planning Director, Secretariat of Planning, GORJ Eduardo Brandão Finance Analyst, Secretariat of Finance, GORJ Pablo Villarim Superintendent of Fundraising, Subsecretary of Finance, GORJ Fátima Leite Planning Manager, Secretariat of Planning, GORJ Claudia Bezerra Health Director, Secretariat of Health, GORJ Deborah Linhares Assessor Subsecretariat of Finance, Secretariat of Finance Roland Clarke Task Team Leader, World Bank Rafael Chelles Barroso Economist, World Bank Barbara Bruns Lead Education Economist, World Bank 34 Annex 8: Map State of Rio de Janeiro 35