Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) Report Number: ICRR0021713 1. Project Data Project ID Project Name P107722 TZ- NEAS STATCAP Country Practice Area(Lead) Tanzania Poverty and Equity L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-48950 31-Oct-2016 27,880,146.91 Bank Approval Date Closing Date (Actual) 24-Mar-2011 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 0.00 Revised Commitment 29,871,248.97 0.00 Actual 27,880,146.91 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as set out in the Financing Agreement (page 5) was to develop a national statistical system (“NSS”) that effectively and efficiently delivers reliable and timely statistics in accordance with international standards and best practices. Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) This Review will carry out a split evaluation. While the PDO formulation was maintained in its original form, key outcome indicators, and supporting indicators, were revised in May 2015, changing the focus of the project from institutional development to data production. At the time, 55 percent of the IDA Credit had been disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components The project had six components: Component 1: Institutional development and legal reform (estimated cost at appraisal US$1.7 million; actual cost US$0.2 million). This component was to support a comprehensive organizational restructuring of the National Bureau of Statistics (NBS) and the Office of the Chief Government Statistician (OCGS). This was to include improvements in financial and human resource management and communications, and the development of related management information systems; and strengthening coordination across ministries, departments and agencies (MDAs) and local government authorities (LGAs) within the NSS. Existing legislation was to be revised to give NBS more professional and technical independence and strengthen the accountability of official statistics producers, as NBS was to become an autonomous public office governed by an Executive Board of Directors. The primary role of the Board was to safeguard the professional and technical independence and integrity of the NSS and to hold the Director General of NBS accountable for his/her performance. The Board was also to ensure that statistical activities across the NSS are harmonized and coordinated. These improvements were expected to enhance coordination within NSS and result in better and improved dissemination of statistical outputs to decision-makers, researchers and the general public. The new legal status was also to allow NBS to develop a new staff scheme of service with compensation based on a competitive, performance-based core salary rather than arbitrary allowances. Component 2: Human resource capacity development (estimated cost at appraisal US$5.0 million; actual cost US$1.0 million). A human resource strategy was to be developed for NBS, OCGS and most MDAs. It was to include new job profiles and salary scales for NBS and OCGS, a new career structure and a performance management system with rewards and incentives for good performance, skills improvement for statisticians and others with managerial responsibility, and improvement of skills of staff responsible for budgeting and procurement. A professional statistical training program was to be developed for statistical staff. Component 3: Development of statistical infrastructure (estimated cost at appraisal US$8.6 million; actual cost US$5.4 million). This component was to improve the underlying systems and databases needed to maintain a NSS. It was to invest in a Central Register of Establishments and enterprises (CRE); a National Master Sampling frame for household-based censuses and sample surveys; a compendium of concepts, definitions, and methodologies for application in statistical data collections and analysis; and a Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) spatial Geographical Information System (GIS) for providing maps for survey field work and for providing the basis for dissemination of geographically-disaggregated and referenced statistical data; and national classifications of economic and social statistics. These investments were to be used to ensure that standardized concepts, definitions, and methods are applied by NBS, OCGS and MDAs. The IS activities were to build on existing work, and establish a GIS unit in OCGS, as well as improve the GIS database in use in NBS. Component 4: Data development and dissemination (estimated cost at appraisal US$11.7 million; actual cost US$19.0 million). This component was to improve the quality, use and timeliness of socio-economic data. Sources for such data (either surveys or routine data systems) were to be improved. Better user access to available socio-economic statistics was to be established strengthening routine data collection, storage, compilation, statistical analysis, reporting and communication within the NSS; and improving social and economic statistics within the NSS. Component 5: Physical infrastructure and equipment (estimated cost at appraisal US$14.7 million; actual cost US$11.1 million). New office buildings were to be constructed for NBS and OCGS. Both buildings were to contain a computer centre, conference and training rooms and library facilities. Investments were also to be made in upgraded information technology and communications infrastructure needed to collect, process and disseminate of statistical data in NBS, OCGS, MDAs and LGAs Component 6: Project management (estimated cost at appraisal US$3.4 million; actual cost US$1.9 million). This component was to finance project planning and budgeting, management, monitoring and evaluation activities. NBS had formed a Tanzania Statistical Master Plan (TSMP) Coordination Team (TCT) under overall responsibility of the Director General. The TCT was to be strengthened with short-term and extended-term consultants to ensure compliance with the procurement, disbursement, and financial management (FM) policies and procedures, as well as train staff across NSS on fiduciary practices and good use of IT facilities. Within the first two years of the project implementation, core NBS staff were to take over these responsibilities. Significant Changes During Implementation The project went through four restructurings: the first, approved on July 9, 2013, added missing milestones and target data to some indicators; the second, approved on May 29,2015, introduced major revisions in indicators, refocusing the project on data collection, and extending the closing date; the third, approved on December 10, 2015, reallocated funds among expenditure categories, essentially increasing the infrastructure component; and the fourth, approved on December February 24, 2017, relocated construction sites of new buildings. The most important restructuring was the second. It reflected changes that had been introduced into the TSMP results framework, which also served as the de facto results framework for the project. Implementation of the TSMP was progressing slowly, and both the government and external partners participating in its financing, recognized that the timetable for implementation was too ambitious for extensive reforms proposed. The scope of the project therefore shifted, away from the original focus on legal and organizational changes and emphasized instead the production and dissemination of statistical products and services. At the same time, performance indicators were adjusted, including a revision of the three PDO-level indicators (see Section 9 below). Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs. Total project costs were estimated at US$63.18 million at approval; actual cost at closing was US$49.20 million. While the costs of the Data Development and Dissemination component increased, all others were reduced. Financing. Of the originally approved IDA Credit of SDR19.3 million (US$30 million equivalent), US$29.87 million equivalent was disbursed, representing 56 percent of total project costs. The equivalent numbers for the other participating external partners were: Canada US$4.89 million, US$4.46 million, 9 percent; United Kingdom US$16.17 million, US$8.07 million, 16 percent; In addition, the European Commission originally committed US$7.11 million, but it was never disbursed (the ICR does not say why). Borrower Contribution. The Borrower contribution was originally estimated at US$5.0 million, then increased to US$8.8 million. Dates. The project was approved on March 24, 2011, and became effective on August 9, 2011. Its original closing date, October 31, 2016, was extended once, on May 29, 2015, to June 30, 2018 in line with a revision to the schedule for the TSMP. 3. Relevance of Objectives Rationale The project objective was highly relevant to the country context, government policy, and World Bank Group strategy. Strong economic growth had a limited impact on poverty, raising the need for better information to monitor changes in well-being and evaluate the impact of policy interventions on the population. At the same time, popular demand for more transparency in government and accountability for policy was rising. Recognizing this, the government launched a statistical masterplan (TSMP) in June, 2010, financially supported by development partners, including the Bank. The project was consistent with the World Bank Group's Country Assistance Strategy (CAS) at the time of appraisal and remains consistent with the Country Partnership Framework (CPF) for FY18-FY22. In accordance with the CAS and the CPF, it emphasizes support for results-based policies, and good governance and accountability. Due to the cross-sectoral nature of statistical systems, the project objective is broadly in line with all three focus areas in the CPF. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) OBJECTIVE 1 Objective To develop an NSS that effectively and efficiently delivers reliable and timely statistics in accordance with international standards and best practices Rationale The objective was to be achieved by developing better systems for data production and data sharing, involving: (i) improvements in the legal and institutional environment based on the introduction of new statistical legislation; (ii) organizational changes and human resource development; and (iii) upgrading statistical processes by improving data collection by MDAs and LAs, and supporting data development by NBS and OCGS. Success was to be measured by user satisfaction, publicly available information, and timely implementation of scheduled censuses and surveys (key outcome indicators). This results framework could reasonably be expected to deliver the sought-for outcome - a credible system for data production and data sharing. The indicators could, however, have been stronger - they did not include any that explicitly addressed features raised in the PDO - effectiveness, efficiency, reliability, timeliness, international standards and best practice. The ICR suggested means of determining these indicators, which are discussed below in the Summary section. Outputs under the original design  The Statistics Law was passed in 2015, corresponding regulations in 2017, and amendments to the Law in 2018. While the target - passage of the law - was met, it took considerably longer than anticipated and this affected adversely the reorganization of the system and human resource management. Moreover, the Bank in 2018 considered that the relevant agencies were unlikely to be able to comply fully with the law: a Bank mission judged that the authorities were unable to produce many of the key economic statistics at the recommended frequency and in accordance with international standards, and were likely to have a serious impact on the generation and use of official and non-official data.  The Statistics Law stipulates the setting up of a governing board for the NBS; although a chairman has been appointed, the full board is not yet in place – target not met  Human resources capacity training (training in modern methods to collect and compile economic statistics) has only been partly completed. During the 2012-2017 period, 45 percent of OCGS staff received planned training, as only 54 percent of planned training courses was implemented. NBS completed 63 percent of its planned staff training – targets not met  Regarding physical infrastructure, office space became available, as the OCGS Zanzibar building and the NBS building were completed in 2018 – target met  Two upgrades of the computer system were undertaken – one in 2012, and another one was underway during the ICR mission – target met Outcomes under the original design Drawing on the original outcome indicators:  In 2013, 88 percent of planned surveys for 2012-2013 were implemented, exceeding the target of 75 percent. All were uploaded to the National Data Archive. Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722)  A survey of user satisfaction was undertaken in 2014 to establish a baseline for measuring future progress. It was followed by a focus group discussion with the main stakeholders to define strengths and weaknesses in the statistical system and develop a roadmap for addressing weaknesses.  57 percent of all data releases and planned publications were released on time, against a target of 75 percent. Summary. Outputs under the original design were partly met, as were outcomes. Slow processing of the Statistics Act affected implementation of other components, while subsequent amendments to the Act departed from international standards. Outcome indicators did not adequately address the full set of outcomes specified in the objective, notably effectiveness, efficiency, reliability, and timeliness. The ICR attempted to compensate for this deficiency, though the results of these efforts have little impact on the efficacy assessment. It suggested that “effectiveness” and “efficiency” could be measured by “statistical capacity,” using a Bank-developed Statistical Capacity Index. When applied, this measure indicates a one percentage point decrease in capacity between project effectiveness and closing. “Reliability” might be assessed through a Data Quality Assessment Framework prepared by the National Bureau of Statistics, which describes various standards and guidelines drawn from international best practice to be applied to ascertain reliability. However, this framework has not yet been adopted, and faces challenges of poor data quality at sub-national levels. “Timeliness” could be determined by the production and dissemination of data according to the 2012-2022 Survey Calendar; almost all surveys were conducted as scheduled. The ICR does not explain how the amendments to the Act affected outcomes. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Same as the original objective Revised Rationale During implementation, project design was recognized by external partners and government alike as being too ambitious to be pursued over the timetable set for the TSMP. While the objectives were maintained, the way their attainment was to be measured was redefined: the revised project focused on production and dissemination of statistical products and services, and specifically on improving the geographical information system in anticipation of the 2022 census, strengthening key economic statistics for key price indexes, and improving access to data bases. Outputs in the revised design Outputs under the revised design include the original outputs plus the following ones:  23 surveys implemented under the TSMP compared to a target of 18 – target met Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722)  Delivery of annual statistical products and services against the annual release calendar was 75 percent in January 2018, against a target of 72 percent - target met  To guide implementation of a quality assessment framework, three sectoral assessment reports were issued, equaling the target - target met Outcomes  The National Geographic Information System was strengthened as the preparatory geographic work for the 2022 census got underway (recruitment of core team, geographic demarcation of survey areas, development of a census plan) – outcome achieved  Economic statistics were strengthened with a revised base year for national accounts, and a revised quarterly production price index set against a 2012 basket – outcome achieved  24 data sets uploaded to the National Data Archive (NADA), equaling the target – target met Targets were met, since they had been set to a more realistic timetable. However, the revised results framework only addressed the PDO very partially. According to the analysis in the ICR, improved data collection does not reflect improved effectiveness, efficiency, or reliability of national statistics, while statistical capacity was enhanced. There were some modest improvements in timeliness in that most of the 2012-2022 Intercensal Surveys were reported to have been completed. Revised Rating Modest OVERALL EFF TBL OLD Rationale A weighted overall efficacy rating is provided in accordance with the percentage disbursed before the revision, i.e. 49.4 percent. A scale of 4 to 1 is used for the efficacy ratings (high, substantial, modest, negligible). Consequently, the weighted value is as follows: 2 for modest x 49.4 percent before the introduction of the 2015 restructuring = 0.99. The weighted value under the 2015 restructuring is 2 for modest x 50.6 percent = 1.01. The weighted average score is 0.99 + 1.01 = 2. 0. The rating for efficacy is therefore modest. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency While conventional quantitative measures of efficiency may not easily lend themselves to assessing project efficiency in this case, the ICR undertook a comparative cost analysis of the data development/dissemination component (essentially consisting of surveys and routine data collection) that represented 40 percent of project costs. Comparisons with the cost of conducting similar surveys in some other African countries showed Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) Tanzania falling significantly below the average. However, design deficiencies led to implementation delays that, with the added impact of cumbersome procurement processes stretched over almost three years. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high - they responded to country needs, as well as government and Bank strategy. Under the original design, efficacy is rated modest, since intended outputs and outcomes were only partially achieved. Efficiency is also rated modest. The outcome is assessed as moderately unsatisfactory. For the revised design, indicator targets for the modified results framework were met, but these did not adequately measure the intended outcomes of effectiveness, efficiency, reliability, timeliness, or attainment of international standards - all that continued to feature in the unchanged statement of objectives. Consequently, efficacy is rated modest and the outcome is assessed as moderately unsatisfactory. With outcomes for both the original and revised design rated moderately unsatisfactory, the overall outcome rating is also moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) 7. Risk to Development Outcome Although the project did not achieve most of its intended results, the TSMP has, according to the ICR, been formally endorsed by the Cabinet and supported by the development partners, through a basket fund managed by the National Bureau of Statistics. The slow pace of change presents a risk, as do shortfalls in some key areas that will determine how sustainable reform will be – for example, concerns about a sufficiently broad political acceptance of the legal framework and its institutional implications (reflected in the uncertainties surrounding the application of the Statistics Act). A recent country diagnostic focusing on the NSS points to the need for additional support in order to arrive at a quality system. So far, there does not appear to be any follow-up on the Bank side in the areas addressed under this project. While other external partners also support the TSMP, they are focused on other areas, such as macroeconomic statistics (IMF), and national panel surveys (EC). 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant, focusing on improving the information base for policy making in Tanzania, and especially in trying to lay the basis for pro-poor policy, which so far had not been altogether successful. The Bank assumed a leadership role among external partners. However, the project was over-ambitious, and more consideration should have been given during preparation to the scope of the challenges that were involved in raising information standards. As a result, key elements of the project had to be redesigned in mid-stream and some targets revised. This is despite the fact that project complexity was flagged as a considerable risk during preparation. The level of generality of the PDO provided space for flexibility, but also created uncertainty as to how outcomes should be interpreted. There was an imbalance in design which overemphasized the supply side of statistical services as opposed to the demand side. There were weaknesses in M&E design. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The project was regularly supervised – based on archived ISRs roughly every seven to ten months. Still, it was only in late 2014, some three years after project effectiveness that action was taken to revise the project timetable and its emphasis, belatedly recognizing the difficult challenges of legal and organizational reform, and giving more focus to data and data dissemination. While this shift in emphasis attempted to address key informational gaps needed for economic policy and the upcoming census, while giving more time to prepare the ground for difficult institutional change, it was not accompanied by any change in the Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) project's objectives. A disconnect was therefore engendered between post-restructuring achievements and the unmodified project goals. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Achievement of the overall development objective was to be assessed through three key indicators measuring user satisfaction, and the availability, quality, and timeliness of statistical products and services. At the output/intermediate level, progress was to be measured by the frequency of data download, skills development, and progress in the area of legal, institutional, and human resource policy reform. Neither the original nor the revised indicators selected for monitoring progress towards the project objective included all outcomes implied by the PDO statement. The original set of three indicators did indirectly address the effectiveness and efficiency of producing and disseminating data, drawing on user satisfaction, and the timeliness of data production and dissemination; it is not clear to what extent reliability or accuracy was being monitored. Likewise, the revised indicators did not include all the implied outcomes, focusing only on strengthening the national geographic information system and economic statistics, and the uploading of data onto the national data archive. The Ministry of Finance through NBS was the main executing agency for the project. For TSMP implementation, a coordinating team was established at NBS. The coordinating team was responsible for regular monitoring of indicators. b. M&E Implementation The revised results framework simplified outcome indicators to focus on the pre-census data system, on economic data, and access to data, backed up by seven activity-level indicators covering the five project components. The new set of indicators, as noted above, were inadequate gauges of attainment of the unmodified project objectives. c. M&E Utilization Indicators were used to monitor project progress. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category ‘B’ for Environmental Assessment purposes and OP/BP 4.01 was triggered. All environmental impacts were linked to the construction of new office buildings for NBS and OCGS. According to the ICR, an Environmental Impact Assessment was conducted for each site by the national authorities in accordance with national law. It was found that the anticipated impacts would be short term, site specific, confined, and reversible and could be adequately managed by NBS and OCGS through the proposed Environmental Management Plans. b. Fiduciary Compliance Fiduciary compliance was not discussed in the ICR. c. Unintended impacts (Positive or Negative) None indicated. d. Other None indicated. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Efficacy is assessed as modest both prior to and after the 2015 restructuring. Efficiency is Moderately Outcome Satisfactory assessed as modest. Those Unsatisfactory ratings result in a moderately unsatisfactory rating for Outcome. Moderately Moderately There are significant Bank Performance Satisfactory Unsatisfactory shortcomings in Quality at Entry Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) and lesser ones in Quality of Supervision. Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The following lessons are drawn from the ICR: Statistical capacity building is a long-term process. Statistical capacity building encompasses institutional and human capacity building as well as data development and dissemination. The project sought to address this by creating an enabling environment encompassing all these elements over a period of five years. This proved to be too ambitious: effective development policy making requires careful monitoring of processes to correctly adapt policy to circumstances in the environment. In Tanzania, an ambitious plan has been put into place to develop a solid data base for policy making, but this requires time and patience. The project had to be restructured as a result of a too ambitious agenda for implementation of the country’s statistical plan. Absence of a legislative framework can be an important factor in advancing or retarding reform. The readiness of a statistics law can crucially influence the pace and scope of implementation of statistics reform. The law is central to any organizational, institutional and human resource changes that are being sought. In Tanzania, the passage of the law, and its ultimate formulation, was a long process that would involve many stakeholders, making passage even slower. IEG adds the following lesson: Significant changes in outcome indicators and targets during implementation can reduce (or increase) the relevance of those indicators to the originally formulated project development objectives. When this occurs, as in this case, greater coherence would be obtained through appropriate modification of the statement of objectives. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR contains most of the elements needed for assessing the project. This was a complicated project, and the ICR authors made a commendable effort in assessing achievements under the project. The results framework offered structured guidance in analyzing outputs and outcomes, and the discussion surrounding M&E design was more thorough than often is the case in ICRs. More rigor could have been introduced Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ- NEAS STATCAP (P107722) by juxtaposing the two results frameworks with each other and noting potential distinctions in meeting the PDO, would have provided a more solid basis for assessment. Observations on fiduciary compliance was absent. a. Quality of ICR Rating Substantial Page 13 of 13