Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) Report Number : ICRR0021355 1. Project Data Project ID Project Name P097026 NG-State Gov & CB TAL 2 (FY10) Country Practice Area(Lead) Additional Financing Nigeria Governance P146147,P146147 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46670 30-Nov-2015 71,374,993.83 Bank Approval Date Closing Date (Actual) 29-Jun-2010 31-Oct-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 120,000,000.00 0.00 Revised Commitment 120,000,000.00 0.00 Actual 71,374,993.83 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Objectives (PDO) as stated in the Financing Agreement (Schedule 1, page 5) are: "To improve transparency, accountability and quality in public finance and human resource management systems with a view to strengthen governance in the Participating States." The PDO as stated in the Project Appraisal Document (PAD, page 10) was: "To improve transparency, accountability and quality in public financial management in the participating states." Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components This project, with six components, was implemented in eleven participating states, split into three categories. The Two Tier I states (Ondo and Kogi) which met the full selection criteria at appraisal, were to implement the first three project components. Component four activities were to be implemented in seven Tier II states (Abia, Adamawa, Ekiti, Imo, Kebbi, Niger and Plateau), based on their ability to meet the selection criteria. The activities for the Tier II states were limited to putting in place the necessary legal framework for Public Financial Management, strengthening tax authorities and automating state treasuries. Component five, consisting of scaling up activities, was to be implemented in two states (Bauchi and Kadua) which were carrying out activities under a prior Bank financed project (First State Governance and Capacity Building Project). According to the Financing Agreement, (pages 5 and 6) and the PAD (pages 11-12), the components were: One. Development and Modernization of Public Financial Management (PFM) Systems. Appraisal estimate US$39.10 million. Revised estimate after restructuring US$26.70 million. Actual cost US$22.80 million. This component aimed at supporting fiscal planning and standardization of PFM procedures, processes and reporting among participating states. Activities included technical assistance for: (i) improving accounting and financial reporting; (ii) strengthening capacity for preparing budgets and public investment financial reporting; (iii) strengthening internal and external audit and oversight role of the State Houses of Assembly; (iv) improving tax administration through reform of the State Tax Authority; (v) modernizing and implementing the State Integrated Financial Management Information System;(vi) supporting the development of a social accountability mechanism in key areas such as planning, budgeting and public procurement through the involvement of the public and the civil society organizations and dissemination of budget information and analysis using the media; (vii) strengthening the capacity of the State governments to make and execute budgets and enabling the governments to give due attention to sectors like water, health and education that are critical for service delivery and poverty alleviation; (viii) enacting a modern legal and regulatory framework for public procurement and technical assistance to strengthen capacity for public procurement; and, (ix) pilot capacity building for selected local governments. Two. Human Resource Management Strengthening. Appraisal estimate US$9.30 million. Revised estimate after restructuring US$6.20 million. Actual cost US$5.29 million. This component aimed at developing a structured training program, targeted capacity-building to PFM-related Ministries, Departments and Agencies (MDAs) and training personnel and payroll systems staff. Activities included an Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) assessment of existing skills and training needs of MDAs, providing logistics support, developing service standards for selected MDAs and modernizing personnel records in the state public service. Three. Monitoring and Evaluation System Development. Appraisal estimate US$7.50 million. Revised estimate US$5.00 million. Actual cost US$4.27 million. Activities in this component included: (i) a diagnosis of the existing Monitoring and Evaluation (M&E) systems; (ii) designing and implementing a new M&E system; (iii) formulation of policies and drafting enabling regulations; (iv) training for the staff of State Planning Commissions and MDAs;(v) establishment of an M&E Secretariat to coordinate M&E activities across the state alongside capacity building; and, (vi) pilot evaluations and service delivery satisfaction surveys for key public services in the participating states. Four. Improvement of Public Financial Management in Selected States. Appraisal estimate US$24.00 million. Revised estimate US$16.80 million. Actual cost US$17.61 million. This component aimed to support the Tier II states in achieving Tier I status. Activities included technical assistance aimed at improving the public finance and the procurement legal and regulatory frameworks and accounting and financial reporting and if deemed to be necessary, analytic work to review public expenditure and financial accountability. Five. Scaling up of Selected Activities Under Original Project. Appraisal estimate US$22 million. Revised estimate US$14.70 million. Actual cost US$14.51 million. This component supported scaling-up activities in the states which had benefited from the previous First State Governance and Capacity Building Project through financing for a Budget and Treasury Management Information System, support for budget preparation, procurement, human resource management and capacity building activities to consolidate the outcomes of project-supported reforms and strengthen M&E. Six. Project Coordination. Appraisal estimate US$8.50 million. Revised estimate US$4.50 million. Actual cost US$3.84 million. This component was to fund project coordination at federal and state levels (including centrally provided technical, quality assurance and other support services to states on PFM, procurement, financial management and audit. It also financed the operating costs for the National Project Coordination Unit and the State Project Coordination Units. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate US$110.40 million. Revised estimate US$73.90 million. Actual cost US$68.32 million (see sub-Section on dates below for an explanation of these differences). Project financing. The project was financed by an IDA Credit of US$120.00 million. US$40.00 million was cancelled after restructuring (discussed below). The revised estimate of the IDA Credit at restructuring was US$80.00 million, of which US$71.37 million had been disbursed by closure. There was parallel financing for complementary activities associated with Public Financial Management in selected states (Anambaca and Yobe) from the European Union (EU). Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) Borrower contribution. None was planned or materialized. Dates. The project was approved on June 29, 2010. It was originally to be implemented in 16 states. After Board approval, and following a change of government in Nigeria, scope and design were changed through a Level 2 restructuring on December 28, 2011, before the signing of the Financing Agreement: (i) the closing date was extended from November 30, 2015 to December 31, 2016; (ii) the geographic scope of the project was reduced and four states were dropped from the IDA Credit Financed Trust Fund. Reforms in these four states (one Tier I state, two Tier II states and one state which was scaling up activities under a prior Bank-financed project) were to be financed by an EU Trust Fund; (iii) US$40.00 million of the IDA grant related to activities in the five dropped states was cancelled; (iv) the results framework and the monitoring arrangements were revised; (v) the project became subject to the 2011 Bank Procurement Guidelines; and, (vi) the project name was changed to the Public Sector Governance Reform and Development Project. A further Level 2 restructuring came into effect on 20 December 2016 with the following changes: (i) the closing date was extended by ten months to complete ongoing activities; (ii) two activities were added to the Tier II states: first, reform of the State Tax Authority to enhance domestic revenue mobilization in the face of declining international oil prices; and, second, adoption of the Financial Management Information System (FMIS); (iii) project costs were realigned to US dollar availability following dollar depreciation relative to Special Drawing Rights; and, (iv) project disbursement bottlenecks were addressed. According to information subsequently provided by the task team, these bottlenecks related to the delays in replenishing credit funds to the seven Tier II states, which were financed out of a Joint Designated Account, managed centrally by the National Project Coordination Unit (NPCU), at the Federal Ministry of Finance. These arrangements were revised to enable the Bank to finance the states directly rather than through the NPCU. The project closed October 31, 2017, about two years behind the original schedule. 3. Relevance of Objectives Rationale The PDOs were highly relevant to country conditions. Nigeria has a federal political framework, and the states have significant autonomy in matters pertaining to the delivery of public services. Under the revenue- sharing formula, about 55%, 25% and 20% of federal revenue were allocated respectively to the federal, state and local tiers of government. Given that the sub-national tiers controlled close to half the public purse, the quality of expenditures at the sub-national levels affected fiscal management, service delivery and poverty reduction. In the years before appraisal, although the government had instituted reforms at the federal level (such as, medium-term policy-based expenditure framework, procurement and budget reforms), Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) these reforms did not automatically apply to states and, with some exceptions, the latter lagged behind on public sector reforms. At appraisal, the PDOs were consistent with the federal government's Vision 2020 and the medium-term strategy for realizing that vision - The Transformation Agenda 2011-2015. Both documents emphasized structural reforms, governance, institutional strengthening, and transparency, particularly in the management of public funds (PAD, page 9). The PDOs were also consistent with the government's Seven Point Agenda and the State Economic Empowerment and Development Strategy. The PDOs were well-aligned with World Bank Group strategy. One of the strategic clusters of the Country Partnership Strategy for the 2014-2017 period (the latest available strategy document) highlighted the need for strengthening governance and public-sector management. The CPS also supported the government's Transformation Agenda goals of achieving socially and regionally inclusive economic growth. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve transparency in public finance and human resource management systems. Rationale Theory of Change. The modernization of the public financial management systems (PFMs) was to be achieved through project- supported capacity building activities focused on accounting and financial reporting, preparing budgets, conducting internal and external audits, improving tax administration, and enhancing monitoring and evaluation (M&E) systems, as well as through the implementation of a modern legal and regulatory framework for public procurement. Capacity building activities for the personnel and payroll systems staff of the respective Ministries, Departments and Agencies (MDAs) and the modernization of personnel records in the state public service were expected to strengthen human resource management. It was also anticipated that, taken together, these activities would enhance transparency in the use of public resources in the Tier I states. Installing the Budget and Treasury Management System (BATMIS) and technical assistance support for budget preparation, procurement, human resource management and strengthening M&E, was expected to improve transparency in public finance and human resource management in the states which had implemented the previous First State Governance and Capacity Building Project. Greater transparency in Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) public financial and human resource management in the Tier II states was expected to be achieved by targeted technical assistance activities. Outputs (ICR, pages 33-47). In all the participating states. • With the support of the project, fiscal responsibility laws were enacted in all the eleven participating states. Training was provided to the staff on implementing the new laws. The new laws were disseminated to the State Houses of Assembly, business community, representatives of civil society organizations and local government officials. The office of the State Auditor General and internal audit was restructured. • Revised legal frameworks for commitment control and payment authorization/approval limits were adopted in all the eleven participating states. This exceeded the target of nine states. • By the end of 2014, the budget classification system of the participating states adhered to the IMF’s Government Financial Statistic (GFS) and the OECD’s Classification of the Functions of Government (COFOG) standards and the participating states used these standards for preparing the 2015 budget. • Eight participating states adopted an Integrated Financial Management Information System by project closure. This exceeded the target of five. • Public Procurement laws were adopted by the participating 11 states. Procurement regulatory systems were established in nine participating states. This exceeded the target of two. • The Human Resource Management System (HRM) of the Tier I states and the states which scaled up activities under a prior closed Bank states were modernized for reducing the discrepancies between the personnel records and the payroll data and thereby minimizing abuse fraud. Where personnel records and the payroll system are managed and operated by the same governmental institution, there is greater vulnerability to payroll fraud, including the insertion of fictitious names (ghost workers) into the payroll. Separating the two functions, with personnel records managed by Human Resources and payroll by, say, the Accountant General’s office, could create checks and balances, which, in turn, could be expected to reduce, if not eliminate, this kind of fraud. The activities supported by the project in this context included, migration of the personnel data to the State Integrated Financial Management Information System (SIFMIS) module, monthly reconciliation of pay and nominal rolls, introducing use of biometric data features for identification and enumeration of civil servants and training the officials of the respective ministries, departments and agencies on payroll management, Human Resource Management and computer skills. In the Tier I States. •A multi-year budget framework and three year rolling sector strategies and fiscal planning were developed in 20 Ministries, Departments and Agencies (MDAs) of the two Tier I states. Two MDAs in Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) these states produced financial statements using the new State Financial Management Information Systems. This was short of the target of five MDAs. • 21 key MDAs submitted annual reports on their activities with updated data on agreed results indicators in the two states. This exceeded the target of ten states. • 34.20% of the project beneficiaries (staff of MDA) participated in project-supported training courses, just short of the target of 35%. • A state-wide survey for identification of new taxpayers was completed in the Tier I states. The basic Information and Communication Technology (ICT) for taxpayer database management and tax administration was developed. A review of the organizational structure for the State Boards of Internal Revenue (SBIRs) was completed and the staff of the SBIRs were trained on key aspects of taxation and ICT maintenance. The regulations and tools for the implementation of presumptive tax regimes for the informal sector were developed. • 72% of the public contracts above threshold were awarded through open competition in the two states as compared to 20% at the baseline. This exceeded the target of 70%. In the states which scaled up activities under the prior closed Bank-financed project. • The Budget and Treasury Management System (BATMIS) was implemented in 20 line ministries of the two states which scaled up activities under the prior Bank-financed project, as compared to the target of 15 line ministries. The Medium-Term Expenditure Framework and the Medium-Term Sector Strategy was implemented in 24 sectors of the states. This exceeded the target of 15 sectors. • The two states submitted annual reports of their activities, using the results-based M&E systems as targeted. In the Tier II States. • The draft of the fiscal responsibility legislation was submitted to the state legislature in Seven Tier II States as compared to two at the baseline. This exceeded the target of six Tier II states. • 46% of public contracts were published in the Tier II states at project closure. This exceeded the target of 35%. Outcomes. • The number of taxpayers in the two Tier 1 States (Kogi and Ondo) increased by 750% and 200% respectively at project closure. This exceeded the target of 30% relative to the baseline. Projected Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) collection of Internally Generated Revenue from income tax increased from 60% at the baseline to 79.50% at project closure and as compared to the target of 90%. With the support provided to the Internally- Generated Revenue component since the end of 2016, three Tier 11 states added taxpayers (Kebbi added 160,000 business enterprises and Imo and Adamawa added 200,000 and 31,000 taxpayers respectively). • The deviation of actual expenditure from budgeted capital expenditure decreased from 55% at the baseline to 29.40% at project closure. This represented better performance than the revised target of 35%. • Verification of personnel data indicates that the discrepancies between personnel records and payroll database fell to 2.5 percent at closure (and subsequently below one percent), compared to a baseline of 25 percent and a target of 5 percent. There is no evidence, however, about reductions in the number of identified ghost workers. Indicators for the states which scaled up activities under a prior Bank-finance project (such as implementation of the, of the Budget and Treasury Management System in the line ministries, Medium Term Expenditure Framework and the Medium- Term Sector Strategy), were output-oriented. However, given that outcomes for the two Tier I states were for the most part realized, as were those for some Tier II states, this Review concludes that the project made a substantial contribution to realizing the objective of improving transparency in public finance and human resource management systems. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve accountability in public finance and human resource management. Rationale Theory of Change. Technical assistance activities in the participating states were designed to facilitate engagement of the public and civil society organizations in budget formulation, implementation and monitoring in conjunction with the dissemination of budget information and analysis through the media. These actions were expected to increase the public's awareness of budget planning and implementation and thereby contribute to enhanced accountability in public finance and human resource management. Outputs. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) • In addition to those described above, an additional output was relevant to this objective: Kaduna State implemented a M&E system which provided a platform for citizen feedback.. Outcomes. • 97.50% of the accounts of the State Government entities, representing at least 75% of total public expenditure, were audited annually in the two Tier I states as compared to 50% at the baseline. This exceeded both the original and revised targets of 75% and 85% respectively. • The Annual State Auditor General's Report were submitted to the state legislature of the Tier I states within seven months from the end of the financial year as compared to 12 months at the baseline. This was short of the target of six months. • The time taken by state legislature to scrutinize and issue report on external audit report in the Tier I states reduced to 20 months at project closure as compared to 48 months at the baseline and as compared to the revised target of 12 months. Given that outcomes for the two Tier I states were for the most part realized, this Review concludes that the project made a substantial contribution to improve accountability in public finance and human resource management. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective To improve quality in public finance and human resource management systems. Rationale Theory of Change. Technical assistance activities aimed at strengthening the capacity of the state governments to formulate and execute budgets in sectors, like water, health and education, that are critical to service delivery and poverty alleviation. Outputs. • An additional output was relevant to this objective: Kaduna State which was implementing activities under a prior Bank-financed operation, included a pension module in the Integrated Financial Management Information System (IFMIS). Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) Outcomes. The indicators used, such as development of a multi-year budget framework and three year rolling sector strategies and annual audits of the State government entities were output-oriented. While tax revenues improved, the theory of change linking the improvements to project interventions was less than robust and there is no evidence to back the assertion that the adoption of the Medium Term Sector Strategy and changes in budget classifications improved the quality of budgeting and financial reporting. In the absence of evidence regarding the quality of service delivery, the efficacy of the objective of improving quality in public finance and human resource management is rated modest. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale Efficacy of the two objectives, to improve the transparency and accountability in public finance and human resource management system, are rated as substantial, as the outcomes, were for the most part realized in Tier 1 States. Efficacy of the third objective, to improve the quality of public finance and human resource management system - is rated modest, in view of lack of relevant evidence. With two of the three objectives rated as Substantial, overall efficacy is Substantial. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Economic Analysis. No economic analysis was conducted at appraisal, given that the project activities focused on capacity building and technical assistance, and the expected economic benefits of public service institutional reforms were expected to be realized in the medium or long term. The PAD (page 20) notes that the potential benefits were expected to come from increased revenue collection by the state and better delivery of services due to improvements in the quality of public expenditure and budget performance. There was no ex post economic analysis. The ICR (page 21) notes that the overall administrative cost at project closure was 4.5% and that this is comparable to similar state-focused projects in Nigeria. The ICR (page 39) reports that the project opted for cost-effective options, such as, contracting local and foreign institutions to organize joint training sessions in country, and that this helped to save funds that would have otherwise been spent on traveling and logistics. . Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) Further, the states established their local implementation units staffed with trained personnel and this generated savings on travel and salaries. Administrative and Operational Issues. The uncertainty created by the initial delays (see Section 2e above) contributed to a loss of implementation momentum among the participating states in the initial years. Implementation was also slowed by the fact that the National Project Coordination Unit (NPCU) did not hire the necessary experts (such as an internal auditor and procurement officer). This issue was addressed by giving the participating states greater autonomy in project management. The project team clarified that one Bank directly disbursed to one of the seven Tier II (Abia), while the other states continued to be funded through the old disbursement arrangement. The anticipated quality assurance consultant for the FMIS and the M&E expert to support the states were not recruited until shortly before closure, Management capacity of some states remained weak throughout, as did the commitment of another, which did not sign the Subsidiary Agreement for most of the implementation period (ICR, paragraphs 54 and 56). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the PDOs to the Government and Bank Strategy is rated High. Efficacy is rated modest. The two objectives - to improve transparency and to improve accountability - was substantially achieved, since most of the intended outcomes were realized in the Tier 1 states. However, efficacy of the other third objective - to improve accountability and to improve quality in public finance and human resource management systems - is rated Modest in view of insufficient evidence. With two of the three PDOs rated as Substantial, overall efficacy is Substantial. Efficiency is also rated Modest in view of notable administrative and operational inefficiencies during implementation. Overall outcome is assessed as moderately satisfactory. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome There is a substantial risk to outcomes particularly in some Tier II states, which either lack management capacity or, in at least one case, commitment to the development objectives. Although the Tier 1 states and the states that had implemented activities under a prior Bank-finance project performed well, it is not clear if the state entities have either sufficient autonomy or appropriate incentives to sustain the project gains. 8. Assessment of Bank Performance a. Quality-at-Entry Preparation drew on the lessons from prior Bank-financed projects (State Governance and Capacity Building Project and the Economic Reform and Governance Project) and ongoing projects (Public Sector Governance Reform project and the State Employment and Expenditure for Results project). These included: focusing limited resources at the sub-national level, involving key stakeholders from government and non-government in the participating states, integrating the Project Management Unit within the government (as opposed to having independent units) and avoiding a uniform approach to the participating states, given their different stages of reform (PAD, pages 13-14). Risks identified at appraisal included capacity limitations at the sub-national level, and governance and security risks (both rated substantial). Mitigation measures included arrangements for reallocating funds if implementation was halted in any state due to political factors and involving third party monitors for supervising implementation. Appropriate arrangements were made at appraisal for compliance with safeguards (see Section 10 below). Despite the intention to avoid a homogenous approach, provision made for addressing capacity and commitment variations across states and entities was inadequate. Variations in ownership and commitment across different states affected project performance. The capacity of the National Project Implementation Unit's ability for overall coordination was over-estimated. This contributed to delays during implementation. There were weaknesses in M&E design, with no appropriate indicators for monitoring outcomes with respect to one PDO (see Section 9 below). Overall, shortcomings in Quality-at-Entry are considered as Substantial, and the rating is moderately unsatisfactory. Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Twelve Implementation Status Results (ISR) Reports were filed over a seven-year period. The project was mainly supervised from the country office, and this helped to maintain close contact with project decision makers. The supervision team had the appropriate skill mix and was supplemented by consultants with specialist skills in areas such as Integrated Financial Management Information Systems and Public Financial Management. During the initial delays following the change in government, the team supported implementation activities that could still be undertaken ahead of the signing of the Financing Agreement through making provisions for retroactive financing, counterpart funding and a project preparation advance. This enhanced readiness for implementation. In the wake of the major fiscal crisis in 2015 due to the sharp drop in international oil prices, the project scaled up support to Tier II states in tax administration and Financial Management Information Systems .Following the Level 2 restructuring on December 20, 2016, the Bank amended the disbursement arrangements with creation of a separate Designated Account (DA) for every participating state, thereby giving the participating states greater autonomy in project management. There was no attempt to address the M&E design weakness during implementation. Overall, shortcomings in Quality of Supervision are considered moderate, and the rating is moderately satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Of the key outcome indicators, the indicators pertaining to the growth in the percentage of projected collection of internally generated revenue, decrease in deviation of actual expenditure from budgeted expected in each participating state and reduced discrepancy between personnel database and payroll database in each participating state and the percentage of public contracts above threshold awarded through competitive open process, were appropriate for monitoring performance, with respect to the Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) first objective of improving transparency in public finance and Human Resource Management (HRM) Systems. The indicators pertaining to the improved coverage of audit, improvements in the timelines of access to audit reports by elected representatives and improved timelines of the review/audit scrutiny by elected representatives in Tier I states, were appropriate for monitoring performance, with respect to the objective of improving accountability in public finance and HRM systems. However, there were no appropriate outcome indicators aimed at the objective of improving quality of public services in key sectors, such as water, health and education). Further, since there was no provision for a beneficiary survey, there was no way of measuring improvements in service delivery. Responsibility for managing and operating the M&E framework was assigned to the National Project Coordination Unit. b. M&E Implementation During implementation, there was some attempt at refining the M&E design. For instance, the number of project beneficiaries for monitoring purposes was based on the number of persons trained. However, no key outcome indicators were added for monitoring performance to measure improved quality of public services. According to the ICR (page 27), the states performed “reasonably well” with regard to M&E. They submitted progress reports, which the Bank was able to use for reviewing progress, providing implementation support where needed and responding to issues as they arose. However, the National Project Coordination Unit did not perform the functions for which it was responsible - consolidating states’ reports, sharing them with the Bank, analyzing M&E issues as they arose in states, and responding to them. These weaknesses persisted throughout the implementation period. c. M&E Utilization Indicators were used for monitoring implementation progress. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) The project was classified as Category B for purposes of environmental assessment. Only the Environmental Assessment safeguard (OP/BP 4.01) was triggered (PAD, page 24). According to the PAD (page 23), the project envisioned minor civil works associated with the rehabilitation of existing structures. These were not expected to cause any significant adverse effect on the environment, and issues regarding land acquisition, involuntary settlement or restricted access to sources of livelihood were not anticipated. An Environmental and Social Management Framework (ESMF) was prepared to provide mechanisms for addressing potential environmental issues at appraisal (PAD, page 23). The ICR contains little discussion of compliance with OP/BP 4.01, but reports (page 28) that there were no environmental or social issues during implementation. b. Fiduciary Compliance Financial Management. A Financial Management Assessment conducted at appraisal concluded that arrangements both at federal and state levels, were satisfactory (PAD, page 44). The ICR (paragraph 72 and 73) notes that the financial management performance of the Tier I states was satisfactory during implementation. However, the capacity of the National Project Management Unit (NPMU) to address financial management issues remained weak. The NPMU managed a joint Designated Account (DA) for the Tier II states. There were delays on the part of the NPMU in replenishing disbursement requests from the states and disbursement arrangements had to be changed (see Sections 5 and 8 above). The ICR does not discuss external auditing of project accounts. Procurement. An assessment at appraisal concluded that some ministries did not have adequate procurement capacity due to lack of experience with Bank-financed projects (PAD, page 56) Corrective measures included, recruiting relevant and experienced procurement officers by each State Project Coordination Unit (SPCU), engaging experienced procurement consultants in the states where capacity was lacking, and training of relevant staff on Bank procurement procedures on a continuous basis during implementation. A Procurement Action Plan was developed at appraisal (PAD, page 56). The ICR notes that there were initial procurement delays after approval and delays during implementation due to the turnover of procurement staff. These were reportedly resolved, and procurement management was deemed to be satisfactory (ICR, paragraph 71). There were no reported instances of misprocurement. c. Unintended impacts (Positive or Negative) --- d. Other --- Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR pages 79) draws the following four lessons from the experience of implementing this project. (1) A realistic assessment of federal and state level ownership of ambitious institutional reforms projects is essential at design. In the case of this project, the commitment of some participating states remained weak. This in conjunction with the weak capacity of the National Project Implementation Unit contributed to delays during implementation. (2) Project implementation timelines need to match implementation capacity. This is particularly so where reforms are difficult or involve design and installation of Information Communication Technology (ICT) such as the Integrated Financial Management Information Systems. (3) In federal countries, when supporting implementation of reforms at the state level, it would be desirable to give appropriate autonomy to state implementation entities. In this project, there were delays in state-level implementation, due to the failure of coordination arrangements at the federal level. (4) In designing operations intended to be implemented by states in federal countries, it is desirable to have clear and predictable criteria for selecting participating states. This would help in addressing issues relating to capacity and commitment variations across states and entities. IEG draws the following lesson. (1) A robust theory of change requires a clear causal chain between project activities and expected outcomes, as well as indicators which enable progress in meeting objectives to be measured. In this case, there were no outcome indicators suitable for monitoring performance with respect to the objective of improving the quality of public finance and Human Resource Management systems. 13. Assessment Recommended? Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-State Gov & CB TAL 2 (FY10) (P097026) No 14. Comments on Quality of ICR The ICR is reasonably clear and candidly discusses the delays that arose in the initial years of the project. It is also candid inidentifying the issues which the Bank could have addressed at design. It provides a clear description of how the Bank helped in addressing problems following the fiscal crisis and the issues associated with the Joint Designated Account. The ICR is largely consistent with the guidelines. It could, however, have been more concise (there is frequent repetition). There could have been less reliance on acronyms and care taken to ensure that acronyms are explained at the beginning of the ICR. There was no consideration of external auditing of project accounts. a. Quality of ICR Rating Substantial Page 17 of 17