IEG Report Number: ICRR14892 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 02/17/2016 Country: Brazil Is this Review for a Programmatic Series ? Yes No How many operations were planned for the 1 series? How many were approved ? 1 Series ID: S126465 First Project ID : P126465 Appraisal Actual Project Name: Rio State Project Costs (US$M): 300.0 300.0 Development Policy Loan Iii L/C Number: Loan/Credit (US$M): 300.0 300.0 Sector Board: Public Sector Cofinancing (US$M): Governance Cofinanciers: Board Approval Date : 08/30/2012 Closing Date: 01/31/2014 01/31/2014 Sector(s): Sub-national government administration (60%); Health (20%); General education sector (20%) Theme(s): Public expenditure; financial management and procurement (30%); Health system performance (20%); Education for all (20%); Tax policy and administration (20%); Managing for development results (10%) Evaluator: Panel Reviewer: ICR Review Group: Coordinator: Mauricio Carrizosa Jorge Garcia-Garcia Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The objective of the operation (Program Document, PD, p.v and 35) was “to assist the Government of Rio de Janeiro in strengthening its tax administration, improving the efficiency of public financial management, increasing the quality of public service provision in education and health and ensuring that policies adopted are both consistent with priorities of the State Government and with resources likely to be available in the medium term. b. If this is a single DPL operation (not part of a series), were the project objectives/ key associated outcome targets revised during implementation? No c. Policy Areas: The program supported three policy areas (PD, para. 93): The Development Policy Operation (DPO) supported three policy areas (PD, para. 93): 1. Tax administration . This area covered implementation of the synchronized taxpayer registration system to simplify the registration procedures and to increase compliance; procedures to allow for the introduction of a unified Taxpayer Ledger; and implementation of a performance-based management system for the Secretariat of Finance (SEFAZ) in Tax Administration. 2. Public financial management . This area covered adoption of formal procedures for evaluation, selection and approval of public investment projects; and carrying out a detailed analysis of all ongoing major investment projects to document costs and future operating and maintenance costs. 3. Public education and health . On public education, this area covered adoption and implementation of a transparent, merit-based selection process for regional directors and school directors and adoption and implementation of annual school-level targets for improving school results and bonus pay for school that achieve their targets. On public health, this area covered adoption of a legal and administrative framework to assess and contract social organizations to manage public hospitals and health units and definition of evaluation mechanisms to transfer bonuses and monetary incentives to municipalities, municipal hospitals, and regional state hospitals. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The program consisted of one Development Policy Loan (DPL) for $300 million. It was approved on August 30, 2012 on the basis of prior actions implemented during the course of 2011 and 2012, declared effective on November 16, 2012 and closed on January 31, 2014, as expected. The DPL was fully disbursed. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High. The DPO’s objective is consistent with the State’s development priorities. These include: a) raising more taxes to fund structural increases in expenditures as well as investments (including those for the 2014 World Cup and the 2016 Olympic Games), a possible fall in tax revenues from the sharing of royalties on oil production, and revenue shortfalls derived from adverse fluctuations in country macro variables that might materialize; (b) using scarce resources more effectively and preserving fiscal sustainability; and c) promoting social inclusion, income equity, labor productivity, and growth. On fiscal sustainability, the PD noted a declining trend in the state’s primary surplus. On social development, the PD noted weaker progress in reducing inequality in the State, compared to the average progress in Brazil. On education, school performance had reached the second lowest level in Brazil in 2009, with high repetition and low graduation rates. On health services, challenges include the lack of coverage and low quality of urgent and emergency services in the poorest neighborhoods, inadequate access by the poor to hospital care, particularly in small municipalities; and poor quality, efficiency and coordination of medium to high complexity care by regional hospitals in the interior of the state. Furthermore, prevention of chronic diseases is undermined by a low coverage of the Family Health Program. It is the lowest among the states in the south-east region and considered essential for health promotion and prevention, especially for the increasing incidence of chronic diseases that result at the advanced stage of the demographic transition. The DPO’s objective is also consistent with the World Bank Group’s 2012-2015 Country Partnership Strategy (CPS), notably with its objectives to increase the efficiency of public and private investments; and improve the quality of public services for low income households, and expand their provision through public and private channels. The July 2014 CPS Progress Report confirmed those objectives. The DPO’s objective is also consistent with Rio State’s 2012-2015 Multiannual Plan, which included fiscal management, health, and human capital among its five priorities. b. Relevance of Design: Modest. The planned reforms could reasonably help achieve expected results, albeit possibly not within the seventeen-month timeframe of the operation. On taxes, better taxpayer registries, consolidated taxpayer accounts, and results-based tax administration can be expected to increase tax collection. On Public Financial Management, sound investment project evaluation, adequate costing of investments and of their recurrent costs, and a restructuring of the state audit institution (covering a focus on quality assurance of internal control systems, internal career paths, and reporting) can be expected to improve public investment quality, the consistency of investments with resource envelopes, and stewardship of the State’s resources. On education, merit-based selection of regional and school directors, and establishment of school results targets with bonus pays for achieving targets, can raise education quality. On health, contracting of services from non-profit providers and bonuses and monetary incentives to municipalities, municipal hospitals, and regional state hospitals can help increase coverage and quality. However, prior actions consisted of initial steps (e.g., on taxes, establish a committee for the tax registration system, adopt standards to improve collection, establish a process to implement results-based management), so that it would take time to have the reforms up and running. This means that actually achieving results (e.g., higher revenues), not just outputs (e.g., a functioning taxpayer ledger), would take even more time. This suggests that either prior actions should have reflected a more advanced stage of reforms in the single tranche operation, or that a multi-tranche operation, with e.g., reforms completed in the first stage and sustained in the second, would have been better results-oriented. Although the fourth DPL also covers financial management and fiscal sustainability issues, the outcomes it seeks do not follow up on those of the third DPL (e.g., on the screening process for public investment). The Bank selected a single-tranche (PD, para. 75) because of the long partnership with the State, the State’s high capacity and commitment, and difficulties with the approval process of single loans in a programmatic series under Brazil’s Fiscal Responsibility Law (PD paras. 75-76). 4. Achievement of Objectives (Efficacy): Strengthen tax administration . Modest. Reforms supported by the operation were expected to raise state revenues from the State’s taxes on goods and services and on estates and gifts, and make further progress on tax administration reforms, as reflected in the targets discussed below. However, the Government of Rio de Janeiro [GORJ] only partially completed the tax reforms that the operation supported. The taxpayer registration system had been in place before the operation was approved (a prior action). The tax ledger system is at an experimental and testing stage (the target of establishing it with taxpayer access to accounts was not met). The Tax Administration Office adopted the 20 performance indicators it expected to adopt (target met). The target with respect to revenues from the tax on goods and services (R$28.67 billion) was surpassed (R30.75 billion). Similarly, the target with respect to revenues from the estate and gift tax (R$470 million) was surpassed (R$644.6 million). But it is difficult to ascertain the significance of these targets for three reasons. First, the tax revenue outcome indicators were defined in monetary terms, not normalized to a measure of the economic activity generating those taxes (e.g., the State’s GDP), so that revenue increases linked to increases in activity could be isolated. Second, the early stage of the reforms suggests that their impact would take time to materialize beyond the target year (2013); accordingly, the observed revenue increases may only be partially, if at all, attributed to the reforms. Finally, the indicators do not provide a measure of tax administration effectiveness (e.g., such as revenue per dollar of expenditure). Improve the efficiency of public financial management and ensure that policies adopted are consistent with priorities of the State Government and with resources likely to be available in the medium term . Modest. The reforms supported by the operation were expected to improve the screening of public investments, ensure consistency of investment decisions and policy commitments with fiscal sustainability, and improve internal audit controls, as reflected in the targets discussed below. Little progress was achieved. There was no progress on investment screening, where no projects above R$50 million were subjected to the new technical screening process for investments, whilst the target was to screen five projects. There was very limited progress on ensuring consistency of public investment and policy commitments with fiscal sustainability. The target to publish information on future annual cost commitments from investments was not met and only modest progress was achieved in costing major policy programs and publishing their results as a means of achieving consistency with resource envelopes. The target was to cost four major policy programs and publish the results. Three programs were costed (but ex-post, not ex-ante as intended) and the results were not published. The best of the planned outcomes was the publication of an action plan to improve internal controls and of an annual report on human resource reforms and performance of internal control systems corresponding to 2013. But this outcome reflects only an action plan, not actual implementation. Other outputs (covering audit regulations, human resource management, decentralization to executing agencies, and procedures for focusing on results and ex-ante reviews) were only first steps in a lengthy process of incorporating modern audit and accounting systems. Increase the quality of public service provision in education . High. The operation was expected to implement actions towards improving the quality of education services, as reflected in the targets discussed below. To improve the quality of education services, the GORJ introduced a new merit-based selection process for regional and school directors, as well as a system of school results targets with staff bonus pays for achieving 80 percent of targets. The number of selected regional directors (35) exceeded the target (28) and the number of selected school directors (164) also exceeded the target (82). The target of paying bonuses for 2012 by June 2013 was met. The ICR notes that the staff in about 30 percent of the schools received bonuses (amounting to approximately 25 percent of the annual salaries) for meeting their targets and that the number of schools achieving 80 percent of their annual targets increased by 23 percent in comparison with the first year. It further notes an impressive rise in the State’s ranking in the national index of development of basic education (IDEB) for secondary education quality, from 26th in 2009 to 4th in the latest (2013) national IDEB scale. It also notes a consistent improvement in the annual Index of Educational Development of the State of Rio de Janeiro (IDERJ), an index that the State designed to measure its math and language proficiency results as well as its completion rates. All this suggests that the reforms may have helped increase quality. Increase the quality of public service provision in health . Substantial. The operation was expected to improve health care management, as reflected in targets on contractual arrangements, hospital coordination for complex treatments, and the quality of patient treatment discussed below. The State carried out the reforms to contract social organizations to manage health emergency services, covering 30 emergency care units, well over the target of 22. The ICR notes contracting has helped refocus the Secretariat of Health’s activity towards contract management rather than oversight of direct provision of services, although reimbursements need to focus more on performance targets, as opposed to service provision. Contracts appear to have been extended to some local hospitals, which was not originally planned. The supply of complex treatments to patients from other municipalities (25 percent of total, year not indicated in the ICR), an indicator of coordination, exceeded the 2011 and 2012 baselines (respectively 16.78 and 20.00 percent) and the DPL target (24 percent). On hospital patient care, the number of hospitals engaging ombudsman services (63), which provide a redress mechanism for citizens that are not satisfied with the services provided, surpassed the target (53) and the baseline (44). However, the number of those hospitals (36, as informed by the TTL) that had infection evaluation committees did not meet its target (55). The ICR reports those committees (a surveillance mechanism for intra-hospital infections) were not found to be very effective in achieving their goals or to operate as expected. In lieu of this indicator, the Government now monitors other indicators, including hygiene standards and hospital waste disposal. 5. Efficiency (not applicable to DPLs): 6. Outcome: The DPL sought objectives that were relevant. They were consistent with the Government’s and Bank’s strategies and addressed important development issues of fiscal management and public sector service delivery. Design focused on relevant reform areas but underestimated the time needed to implement some of the reforms initiated by the prior actions, which were insufficient to achieve objectives, particularly those on tax revenues and public financial management. Progress achieved in education was the highlight of the program. In the health sector, the evidence suggests improvements in the administration of services. Results on tax revenues met targets, but attribution to the reforms supported by the DPL is in question. There was some advance in the costing of programs to seek their consistency with resource envelopes. Little progress was achieved on investment management. Audit reforms are at a very early stage. On balance, the outcome is rated Moderately Satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: With the end of the administration supporting the reforms in 2014, the ICR identifies reversal risks in tax policies and of costing programs, where changes enacted through directives are not fully consolidated. This means that there is a risk that reforms in those areas may not be sustained. In contrast, the ICR reports that changes in the education and health sectors are taking root, with constituencies built behind them, and are unlikely to be reversed. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Quality at entry benefited from strong relevance of objectives and from a selection of reform areas that were appropriate to those objectives. Nevertheless, the time horizon needed for full implementation of reforms was probably underestimated, resulting in ambitious targets. The reforms were expected to have positive poverty and social impacts (PD, para. 138), with no positive or negative environmental effects. The financing provided by the DPL was consistent with the state’s declining debt-to-revenue ratios, considering the State’s fiscal policies and projected growth. Implementation arrangements were straightforward. The Finance Secretariat, in collaboration with the Planning Secretariat, was to lead the program, implement the fiscal and PFM reforms, and collect results data; and the education and health secretariats were responsible for the reforms associated to their sectors. A Project Implementation Committee made up of officials from these agencies was to monitor implementation. Funds were to disburse to a commercial bank that was deemed acceptable to the World Bank, as it met financial, audit, and capacity criteria (PD para 159). M&E design suffered from pitfalls (Section 10). The PD identified relevant fiscal, external, political, and sector (education, health) risks and relevant mitigation measures. Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: Supervision had a good focus on development impact in that it collected and monitored progress in results indicators. However, supervision inputs may have been inadequate. The ICR reports that during the 18 months of implementation there were (two) full formal supervision missions, mainly geared at obtaining information on progress. In addition, the Bank followed implementation progress closely and assisted the authorities on a continuous basis only in a few areas. Not enough resources were allotted to engaging closely with the authorities and helping to develop remedial measures or otherwise assist with implementation beyond the prior actions. For example, on tax administration, where results called for actions following the DPL’s prior actions (e.g., establishing a unified taxpayer ledger), the ICR reports that the Bank did not have a close presence. Nevertheless, the ICR also notes (Para. 79) that most of the Bank’s work in this operation took place as part of the dialogue with GORJ as part of other loans. The Bank filed three ISRs that indeed reflect good collection of information on the progress made in results indicators. Their ratings may have been less than fully candid, as some of the results indicators suggested that achievement of the PDO and implementation progress may have been less than satisfactory. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Strong Government engagement in dialogue with the Bank and substantive prior actions in the different policy areas suggest substantive ownership and commitment towards the objectives, although follow-up after the prior actions was not as strong in all areas (e.g., on public investment management). In addition to providing the enabling sectoral environment through prior actions, the Government also ensured a good macro/fiscal environment to allow for the financing. The Government held consultations covering the reforms with local and sector stakeholders, including teachers, parent-teacher associations, health workers and health service providers (PD, paras.65-66.) Readiness for implementation appears to have been stronger in some areas (e.g., education) than others (PFM). As with the Bank, attention to resolving implementation issues may not have been sufficiently strong. M&E arrangements worked well in terms of collection of information and not so well in terms of utilization (Section 10). The ICR did not report on coordination or the role of other donors. The Government and the Bank continued some of the reform efforts of the DPL through a follow-up operation that includes medium-term planning and monitoring of public expenditures as one of its policy areas. Government Performance Rating : Moderately Satisfactory b. Implementing Agency Performance: Implementing agency performance was uneven. The ICR notes the Finance Secretariat demonstrated good leadership as coordinator of the program. Its role in implementation (taxes) was weaker, as the tax reforms initiated by the prior actions fell short of completion. The Planning Secretariat achieved little progress on investment management reform, where coordination with Finance Secretariat appeared poor (ICR, para 57), and some progress in costing of programs. The Audit Agency’s performance was strong on its limited target of pursuing a modernization plan and beginning to change the focus of its oversight. The Education and Health Secretariats did well in implementing respectively the reforms to select and reward well-performing education staff and implement performance-based management in health services. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Objectives were reasonably clear. However, most indicators measured outputs (e.g., selection of school directors), not outcomes (e.g., education quality indices) reflecting the objectives. The one numeric outcome indicator (tax revenues) was not well designed, as it was not normalized by economic activity. An indicator linked to economic activity (ratio of ICMS to state GDP) would have been more helpful. Several indicators reflected actions (e.g., establishment of a taxpayer ledger), not numeric measures indicative of progress (as, e.g., number of taxpayer queries to their ledgers). Since indicators largely reflected actions, data collection was largely limited to recording and documenting those actions. Given the focus of “outcomes” on actions, timely availability of baselines was straightforward (e.g., the baseline for establishing a ledger is “no unified general taxpayer ledger”). It cannot be said that the M&E design was well-embedded institutionally, as the ICR reports (para. 43). b. M&E Implementation: Since collection of information on actions was fairly straightforward, M&E arrangements worked well in terms of collecting information. Baseline data was collected and indicators measured. Government ownership of M&E appears limited. The ICR notes that the committee formed to monitor agreed actions worked mostly to satisfy the needs of the periodic supervision missions of the Bank and dedicated groups that closely monitored the advances of reforms in the education and health were essentially set up for implementation purposes. The ICR did not discuss the reliability or quality of the data. Except in the education sector (where there are ongoing impact evaluations), the evaluation dimension of M&E does not appear strong. c. M&E Utilization: With data collection alone and no evaluation, utilization of M&E remained weak as a tool to address implementation issues. For example, the data in the March 2013 ISR was showing good progress in ICMS and ITD tax collection. An analysis of that data could have sought to uncover the sources of that progress and its link, if any, to the reforms, in order to help the authorities make mid-course corrections as needed in those reforms. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: No safeguards policies were triggered by the series. b. Fiduciary Compliance: The ICR did not report on fiduciary compliance. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate The ratings in the ICR are inconsistent Outcome: (“low or negligible” in the ratings summary section; “moderate” in the section on Assessment of Risk to Development Outcome). Reversal risks in tax policies and costing programs suggest a “Moderate” rating. Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR suggests the following three lessons: 1. A reform where the interests of implementing agencies are not well-aligned has limited chances of being achieved. This seems to have been the case of the public investment reform, where the Finance Secretariat coordinated poorly with the Planning Secretariat, the agency in charge of that reform. 2. A combination of better incentives and institutions is a sound underpinning for successful reforms. The education reforms supported by the DPL combined an institutional change (merit-based appointments) with a good incentive (bonuses on good results) and seems to have been successful. 3. A consistent and persistent engagement with a client will help improve results by helping build knowledge and expertise. In particular, the ICR underscores that programs are more likely to succeed when there are high level professional teams involved in the design stage. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: This is a concise, candid and thoughtful ICR. It reports the evidence on results indicators, albeit with gaps in the indicators relating to health service outcomes. The ICR provides a good account of the reforms and a thoughtful analysis of results. Discussion of relevance of objectives and design could have been more specific with regard to Rio State challenges and programs, to Bank strategy, and to the results chains underpinning the four dimensions of the PDO. a.Quality of ICR Rating : Satisfactory