Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) Report Number: ICRR0021795 1. Project Data Project ID Project Name P126875 TZ-Energy Sector Capacity Building Prj Country Practice Area(Lead) Tanzania Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-52170,TF-15902 31-Dec-2018 13,904,149.54 Bank Approval Date Closing Date (Actual) 26-Mar-2013 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 21,460,000.00 4,852,955.00 Revised Commitment 26,312,955.00 4,852,955.00 Actual 14,658,211.65 3,902,982.81 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan Fernando Manibog Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (p.5) dated April 18, 2013, the Trust Fund Grant Agreement (p.7) dated November 4, 2013 and the project appraisal document (PAD, p.7), the objective of the project is "to strengthen the capacity of the Recipient [United Republic of Tanzania] to develop its natural gas sub-sector and public private partnerships for the power generation sector." Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project had five components with numerous sub-components (for a detailed explanation of the components, please see Schedule 1 of the Financing Agreement dated April 18, 2013): A. Petroleum Policy and Legal Framework. (Appraisal cost: US$6.55 million; Actual cost: US$0.45 million) 1. Petroleum Policy and Strategy. Carrying out feasibility studies, developing and updating sector policies, strategies and plans, and promoting local industry participation. 2. Legal and Regulatory Framework. Strengthening the gas sub-sector legal and regulatory framework, supporting Occupational Safety and Health Authority (OSHA) in health and safety in the gas industry, providing legal advisory service to Energy and Water Utilities Regulatory Authority (EWURA) in regulatory reform process. B. Strengthening Institutional Sector Management, Coordination and Governance. (Appraisal cost: US$12.46 million; Actual cost: US$6.27 million) 1. Sector Coordination and Governance. Training on revenue streams, financial modeling and physical audit processes in the gas sub-sector, undertaking overseas study tours, supporting oil and gas sector policy integration and implementation, providing technical assistance to enhance sector coordination, knowledge sharing and government’s capacity to negotiate natural gas projects. 2. Enhancing Organizational Capacity. Support to EWURA’s gas division in gas standards and transmission and distribution codes, in carrying out a technical skills capacity assessment and implementing a training program, and in developing technical designs and purchasing software and hardware for tariff models; support to EWURA’s energy division in developing feed-in tariffs, in developing regulations for power purchase feed-in tariffs, and in purchasing software and hardware; providing technical assistance to Ministry of Energy and Minerals (lately Ministry of Energy – MoE) in policy development for natural gas projects, in carrying out of a capacity and capability assessment and implementing a training program, in transactional advice, and in implementing a technology based data solution; and support to Tanzania Petroleum Development Corporation (TPDC) in acquiring reservoir simulation and geological modeling hardware and software, in acquiring geo- chemical laboratory equipment, in developing structured reservoir modeling, and in training on gas value chain and liquefied natural gas technology. 3. Environmental and Social Management. Conducting a Strategic Environmental and Social Impact Assessment, and support to National Environmental Management Council (NEMC) in environmental aspects of the petroleum sub-sector, development of a resourcing and training plan, hazardous waste management, gas venting and flaring, development of oil spill contingency plans, community- based early warning and response system, oil spill contingency planning, and in acquisition of equipment for monitoring compliance with environmental regulations. 4. Health and Safety Management. Support to OSHA in developing a health and safety training program and acquiring equipment for monitoring compliance with health and safety regulations. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) C. Education and Skills Development. (Appraisal cost: US$2.85 million; Actual cost: US$0.20 million) Support to Vocational Training and Education Authority (VETA) through the provision of: (i) technical assistance for a vocational skills assessment of the oil and gas sub-sector; (ii) technical assistance for the development, implementation and supervision of an education development plan; (iii) training of instructors and curriculum developers; (iv) administrative and logistics support for VETA training centers; and (v) acquisition of learning materials, tools and equipment. D. Power Generation and Natural Gas Public-Private-Partnership (PPP) Projects Capacity Building. (Appraisal cost: US$6.39 million; Actual cost: US$2.46 million) 1. Support to MEM PPP Node for the development of a sector PPP strategy, preparation of a PPP project pipeline, establishment of coordination mechanisms, and development of a PPP training program. 2. Support to Tanzania Electric Supply Company (TANESCO) PPP Node for the acquisition of software for power system master plan models; engagement of a technical advisor to mentor staff, review and update the action plan, and develop a plan and operating guidelines for PPP projects; implementation of a training program including communication aspects of PPP projects; and feasibility studies. E. Project Coordination. (Appraisal cost: US$1.50 million; Actual cost: US$0.48 million) Support to the Project Steering Committee and Project Management Teams of Project Implementing Entities and financing of costs associated with the implementation of the Project through the provision of technical advisory services, training, acquisition of goods and financing of operating costs.   Project components were not revised during implementation, but almost half of the project activities could not be completed (ICR, p.12) due to issues related to project implementation and procurement. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was originally estimated at US$35.00 million including US$5.25 million for physical and price contingencies. In December 2018, the project closed with a total actual cost of US$14.66 million*, or 49.3 percent of the appraisal estimate excluding physical and price contingencies. * Table 4 (p.20) and Annex 3 (p.40) of the ICR provide the breakdown of actual costs per component, which are reported in the above section 2.d Components. Actual component costs add up to US$9.86 million, which is lower than US$14.66 reported in the Data Sheet of the ICR. The project team did not provide clarification about this discrepancy. Therefore, the actual cost and the actual disbursement amounts— reported under Financing below—are taken from the Data Sheet of the ICR (p.2). Financing: At appraisal, the International Development Agency (IDA) credit was estimated at US$21.46 million (including US$3.22 million for contingencies), and the Canadian International Development Agency (CIDA) grant at US$13.54 million (including US$2.03 million for contingencies). At project closing, the IDA financing stood at US$10.76 million and CIDA financing at US$3.90 million. The World Bank project team Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) and the Government of Tanzania agreed to cancel the IDA funds at project closing. CIDA funds were extended until November 2020 to finance the remaining technical assistance activities. On September 24, 2019, the project team received a letter from the MoE with a list of proposed activities to be financed by CIDA funds amounting to US$9.00 million. Borrower contribution: At appraisal, the borrower’s in-kind contribution in the form of office space and staff salaries was estimated at US$1.5 million. The ICR does not provide information about the actual value of the borrower’s in-kind contribution at project closing. Restructurings and Dates: There was only one project restructuring in March 2012 immediately after project approval, but before project effectiveness. In that restructuring, upon the request of the Government of Tanzania (GoT), the Ministry of Energy and Minerals (from now on MoE—Ministry of Energy) and EWURA became the two main project implementing agencies, with TANESCO and TPDC being placed under MoE’s supervision (ICR, p.13). The original implementation arrangements included four project implementing agencies and four additional beneficiaries. There was no project closing date extension. 3. Relevance of Objectives Rationale The discovery of substantial onshore and offshore natural gas reserves during early 2010s was timely to address the economic growth and development issues in Tanzania through power generation by natural gas and exportation of liquified natural gas. However, Tanzania lacked the legal framework and institutional capacity to develop the natural gas sub-sector and to attract private investments in the natural gas and power sectors. The project intended to address these two issues by supporting the Tanzanian government to implement its National Natural Gas Policy (NNGP). The main objective of the NNGP was "to maximize the benefits from the utilization of natural gas to transform Tanzania’s economy" (ICR, p.6), which would contribute to the reduction of poverty as envisaged in the National Strategy for Growth and Reduction of Poverty. Therefore, the project objectives to strengthen the capacity in Tanzania to develop the national gas sub-sector and public private partnerships for power generation are still highly relevant to the country context in Tanzania. However, it should be noted that the focus has been shifting towards the development of hydropower and renewable energy sources, such as wind and solar (ICR, p.25). Furthermore, the objective, although relevant to the country context, was overly ambitious because its achievement required project implementation by eight separate institutions despite the general lack of capacity in the country, uncertain cooperation among these institutions and unclear government ownership of the project (PAD, p.18). The project objective was substantially aligned with the World Bank’s country strategy at the time of project closing. The project objective corresponds to the 2018-2022 Country Partnership Framework’s (CPF) Focus Area 3, Modernize and Improve the Efficiency of Public Institutions, Objective 3.2, Improve efficiency and competitiveness in public investments. The country strategy aims to "enhance the synergy between public and private investments in the economy" initially focusing on the energy sector by supporting the power utility, TANESCO and private sector power generation investments (CPF, p.31). The World Bank country Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) strategy does not include any interventions to strengthen the institutional capacity to further develop the natural gas sector in Tanzania. The World Bank’s involvement in the development of natural gas sector and related power generation infrastructure dates to 1990s when the International Development Agency (IDA) funded the Songo Songo Gas Development and Power Generation Project. In early 2010s, the World Bank responded to the financial crisis in the energy sector through a series of development policy operations, i.e., First and Second Power and Gas Sector Development Policy Operations. Given the World Bank’s experience in this sector, the project objectives were adequately challenging. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To develop the natural gas sub-sector in Tanzania. Rationale The first objective was broad and overly ambitious for a technical assistance project providing only support to strengthen the institutional capacity in the country. The project design was very complex; components had sub-components and some sub-components had more sub-components. A very broad causal link can be established between the achievement of the objective, and the technical assistance activities listed in the first three components of, i.e., Petroleum Policy and Legal Framework; Strengthening Institutional Sector Management, Coordination and Governance; and Education and Skills Development. The expected outputs were submission of gas midstream regulations to the authorities for approval, training of staff of key sectoral institutions on the management, coordination and governance of natural gas resources and revenues, purchase of equipment to model, monitor and simulate natural gas developments, and availability of vocational training specific for oil and gas sub-sectors. The expected outcomes were the establishment of a policy, legal and regulatory framework and enhanced institutional capacity, but the expectation that these would lead to the broad objective of developing the natural gas sub-sector The achievement of this objective was not measurable; hence, the project development objective indicators in the results framework were at the output level, rather than outcome level. Furthermore, establishing attribution among the project activities and the achievement of this objective is not straightforward, since there were separate projects funded by other donors with similar objectives (ICR, p.6-7). Overall, the theory of change for the first objective was not robust. The project activities related to the petroleum policy and legal framework under the first component were not implemented, i.e., development of feasibility studies, support for cross-sectoral strategies and plans to facilitate integration of natural gas projects into wider economy, and programs to enable local industry participation in gas industry developments. Training activities were also not implemented, such as secondment training in other countries, support for a planned Center for Excellence for the energy sector, Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) modeling infrastructure for Energy and Water Utilities Regulatory Authority’s (EWURA) electricity division, and advisory services and training for National Environment Management Council (NEMC) in environmental management under the second component. On the other hand, the project fully or partially achieved the following outputs related to this objective:  Gas Policy was approved by the Cabinet of Ministers; EWURA developed ten regulatory tools, service standards and guidelines related to the natural gas sub-sector; NEMC developed the Strategic Environment and Social Impact Assessment to support the development, operation and monitoring of the natural gas sub-sector; and Operational Safety and Health Authority (OSHA) developed Occupational Safety and Health Regulations in line with international best practices.  A total of 128 members of staff from Ministry of Energy (MoE), Tanzania Petroleum Development Corporation (TPDC), NEMC, OSHA and EWURA were trained in managing, regulating and monitoring gas sub-sector. The target was 123 members of staff.  TPDC upgraded and installed software for geological modeling, engineering and reservoir simulation to enable forecasting and data analysis for inhouse exploration and modeling of natural gas reserves. Training and hardware were also provided.  NEMC developed control and monitoring procedures, manuals and processes with respect to hazardous waste management.  Vocational training program for gas sub-sector was developed, as planned. Vocational Education and Training Authority (VETA) developed an operational manual for occupational health, safety and environment standards, and a curriculum for welding, scaffolding and mechatronics.  Two feasibility studies were to be delivered for making policy and investment decisions and attracting financing for concrete investment opportunities. None could be delivered because MoE had not contracted a consultant six months before project closing date.  Ninety-six visits to gas exploration and production sites were planned to supervise compliance, auditing and monitoring of safety, but only four visits were achieved. The visits were to take place after the enactment of health, safety and environmental regulations, which were completed only four months before the project closing date; therefore, there was not enough time to arrange the site visits.  Ten VETA members of staff were to benefit from the vocational training program, but only three could complete the program, because the program could only be finalized two months before the project closing date. The project was partially successful in delivering the project outputs as listed above. However, there is insufficient evidence whether these outputs led to the development of Tanzania’s natural gas sub-sector. Annex 6 of the ICR, which summarizes the status of the energy sector after project closing, there was no new gas field development planned beyond the existing two fields of Songo Songo and Mnazi Bay. Supported by other development partners, TPDC completed some studies for local distribution of natural gas, but there is no attribution between these studies and the project activities. The main contribution of the project in the development of the natural gas sub-sector in Tanzania was its support in the development of policies and regulations, such as the Gas Policy, gas sub-sector specific health and safety regulations by OSHA, and draft regulations, rules and guidelines for the mid-stream natural gas sector by EWURA (ICR, p.48-49). The project’s impact on the development of the natural gas sub-sector in Tanzania through other project activities, such as trainings, vocational training and installation of software for monitoring, simulating and planning natural gas reserves, is not measurable. Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) Despite its impact on the development of overall policy, and legal and regulatory frameworks related to the natural gas sub-sector in Tanzania, the achievement of this objective is rated modest due to insufficient evidence whether the project’s limited outputs contributed to the development of the sector. Rating Modest OBJECTIVE 2 Objective To develop public private partnerships for the power generation sector in Tanzania. Rationale Compared to the first objective, this objective was outcome-oriented and measurable. A causal link could be established between the project activities and the expected outcome. The project was intended to support the MoE in developing a public private partnership (PPP) strategy and in preparing a PPP project pipeline in the natural gas and power sector, while supporting Tanzania Electric Supply Company Limited (TANESCO), the power utility, in strengthening its capacity in PPP through software update, technical advisor support, review and update of PPP action plan, training and feasibility studies. These project activities were expected to result in the financial closure of one PPP power generation project, which was an ambitious outcome target for a technical assistance project to be implemented in five years. However, it is also noted that a considerable time is needed before a PPP project could be brought to financial closure (ICR, p.19). Overall, the theory of change for the second objective was adequately sound and the results framework included measurable indicators to monitor the achievement of this objective. However, the project failed to implement most of the project activities to achieve the objective of developing power generation PPPs. Although PPP units were established at both the MoE and TANESCO, these units could not become functional to develop a PPP strategy because of the government’s decision to amend the PPP law, which could only be passed in 2018, two months before project closing. Therefore, only communication training was delivered to TANESCO staff, and an advisor was hired to structure the 320 MW Mkuranga-Somanga Fungu Combined Cycle Power Plant. The latter led to the development of the legal and technical tender documents for an international competitive bidding tender for the first PPP gas-based power project in Tanzania (ICR, p.18). However, this process, too, had to stop due to the amendment of the PPP law. The ICR (p.19) claims that much progress had been made to achieve the financial closure of the first PPP power generation project in Tanzania, which awaited the formal approval of the regulations based on the amended PPP law. At the time of project closing, a PPP power generation project was not brought to financial closure, nor was a PPP strategy developed; therefore, despite substantial progress in preparing bidding documents, the development of PPPs for power generation sector did not materialize; hence, the achievement of this objective is rated negligible. Rating Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) Negligible OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project was not successful in achieving the second objective to develop PPPs for the power generation sector in Tanzania. However, the project was partially successful in supporting the development of a natural gas sub-sector by supporting the development of legal and regulatory framework, the passing of the national gas policy, and the enactment of sector specific health and safety regulations. Therefore, the achievement of the project development objective is rated modest, but barely. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency Traditional methods of economic analysis, such as economic rate of return or net present value, could not be applied in this technical assistance project. The main assumption at appraisal for the economic justification of the project was that "[s]uccessful implementation of similar technical assistance projects has resulted in substantially increased investment, improved government oversight over the sector, and greater accountability of revenue streams generated by the sector" (PAD, p.19). Furthermore, the ICR (p.19) states that "[technical assistance projects] must, instead, be evaluated on the basis of the cost-effectiveness of achieving of their objectives." Both approaches implicitly assume the successful achievement of the project objectives. However, in this project, the achievement of the project development objective was barely modest. Furthermore, although the ICR (p.19) claims that the costs of the completed project activities were comparable to the cost of similar training and technical assistance projects, it does not provide any evidence to support this claim. Due to lack of borrower’s ownership of the project and very slow procurement process, only 49.3 percent of US$29.75 million of funds—excluding US$5.25 million for contingencies—provided by the IDA and the CIDA could be disbursed. Signing of some consultancy contracts were delayed by couple of years. The complexity of the project design and the existence of numerous project implementing agencies also adversely affected project efficiency. Until the completion of the Training Needs Assessment (TNA) in early 2018, the training programs were implemented on an ad hoc approval of requests, which delayed the implementation of project activities. After the TNA, implementation of project activities gained pace, but there was not enough time to complete all the project activities before project closing date in December 2018. The Project Steering Committee, which was established to oversee project implementation, did not function. The issues during project implementation required a closer Bank supervision, which resulted in a very high supervision cost of US$2.05 million compared to US$0.55 million estimated at appraisal. Overall, the project efficiency is rated negligible. Efficiency Rating Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Although there seems to be a shift in the government’s preference towards the development of hydro and renewable energy sources for power generation, the project objective is still substantially relevant to the country context in Tanzania given that huge proven natural gas reserves are not fully tapped yet. The project objective is also substantially relevant to the Bank strategy at the time of project closing. The project was partially successful in developing some legal and regulatory framework for the development of natural gas sub-sector and not successful in developing public private partnerships in power generation, despite the delivery of some consultancy work, training, and purchase of software and hardware. Therefore, the project’s efficacy in achieving its development objectives is rated modest. The efficiency of the project is rated negligible because of serious implementation delays resulting from slow procurement, an overly complex project design, and numerous project implementing agencies, all leading to very high Bank supervision costs. The overall project outcome is rated unsatisfactory. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome There is no substantial risk to the limited development outcome achieved by the project. The global demand for natural gas and liquefied natural gas (LNG) has been increasing despite an increase in the use of renewable energy sources; therefore, it is expected that the Government of Tanzania will be willing to continue to further develop its natural gas sub-sector. The development of downstream industries, such as gas related chemical industries, and distribution of natural gas for domestic use are additional reasons why the Government of Tanzania should continue with the development of the sector. However, domestic political instability, lack of government ownership, failure to maintain trained personnel in the sector may slow down or stop the development of the natural gas sector. Furthermore, a less favorable investment environment may prevent private sector to invest in industries, such as power generation and chemicals. Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) 8. Assessment of Bank Performance a. Quality-at-Entry The objectives are strategically relevant to the country conditions, but the first objective to develop the natural gas sub-sector through capacity building was not measurable and theory of change was inadequate for the achievement of this objective. The second objective to develop PPPs for power generation was better supported by the theory of change, but it was overly ambitious for a technical assistance project with only a five-year implementation period. The project design was very complex both in terms of the variety of project activities and the number of project beneficiary institutions—four project implementing and four beneficiary institutions. The project implementing agencies lacked capacity to implement the project activities. These shortcomings, i.e., large number of project implementing and beneficiary institutions, complex project design and lack of project implementation capacity, were identified as key risks during appraisal (PAD, p.18). These risks were to be mitigated by establishing an independent project management unit (PMU), separate project management teams (PMTs) in each institution and a Steering Committee that would closely coordinate and supervise the implementation of project. However, the Ministry of Energy (MoE), the primary project implementing agency, was not adequately consulted during project preparation, which was primarily negotiated with the Ministry of Finance (ICR, p.29). Therefore, after the approval of the project, the project implementing arrangements had to be revised upon the request of the MoE. An independent PMU was not established and there were various issues in procurement activities due to lack of capacity at the MoE, which adversely affected project implementation. M&E arrangements were in place, but with a focus on the outputs of the project activities, rather than the achievement of the project objectives. Addressing training requests on an ad- hoc basis did not ensure the implementation of appropriate programs to meet the needs of diverse project beneficiary institutions (ICR, p.24-25). This was corrected late in project implementation after the completion of a Training Needs Assessment. Although the need for strong government ownership was defined as one of the lessons learned from previous similar projects (PAD, p.14), the project lacked full government ownership (ICR, p.23). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision There were ten supervision missions during project implementation—two in each calendar year. The Bank supervision was focused mostly on improving the procurement process by making the project implementation arrangements more efficient. Following the mid-term review in late 2016 and follow-up meetings in early 2017, the project team was successful in revising the implementation arrangements and each project beneficiary institution established its own project management unit, rather than MoE being in charge of the procurement activities of other six institutions, except EWURA, which is under the Ministry of Water. After this change and the completion of a Training Needs Assessment, the project implementation gained paced in the last year of the project. Upon the insistence of the project team, a dedicated accountant was also assigned to the project in December 2017, after which procurement activities and Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) fiduciary aspects of the project improved (ICR, p.24). During project implementation, lack of full government ownership of the project was an obstacle for effective bank supervision. Undisbursed IDA funds were cancelled at project closing in December 2018, while the CIDA funds were extended until November 2020 to finance the remaining technical assistance activities, ensuring continued support to the development of natural gas sub-sector in Tanzania. The project team’s supervision of fiduciary and safeguard aspects of the project was adequate. The Implementation Status and Results Reports (ISRs) were candid in performance reporting and in explaining the shortcomings in project implementation. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Project objectives were overly ambitious for a technical assistance project of this size and short implementation duration. The theory of change was not adequate in showing how project activities and outputs would lead to the outcomes. M&E framework did not fully capture the achievement of the project outcome, especially in developing the natural gas sub-sector through capacity building. The project indicators were at the output level, rather than the outcome level. The indicators were easy to measure since the measurement units were either Yes/No or numbers, such as number of members of staff trained or number of visits made. b. M&E Implementation There was no change in the M&E framework during project implementation. Although the measurement of the indicators was rather straightforward, because they were directly related to the completion of project activities or counting the number of members of staff trained or visits conducted, there were delays in receiving the current status of the project activities from the beneficiary institutions (ISR, no:8). c. M&E Utilization The M&E framework did not provide sufficient evidence for the achievement of the project outcomes. Due to the slow progress in procurement, the project team’s focus was to accelerate the delivery of the project outputs. To this end, the Mid-Term Review included detailed suggestions on how to improve the implementation efficiency of the project. The findings of the Mid-Term Review led to the changes in the project implementation arrangements and project activities gained pace only during the last year of the project. As a result of the M&E findings, the project team suggested restructuring the project including an extension of the project closing date, one year before project closing, but the MoE did not support this strategic change claiming that this might cause further delays in project implementation (ICR, p.25). On Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) the other hand, the parties agreed to extend the availability of the CIDA funds to finance remaining project activities. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Category B under Environmental Assessment (OP/BP 4.01) and did not trigger any other safeguard policies. Environmental Assessment (OP/BP 4.01): The project did not finance any construction activities, and on that count, no significant adverse environmental or social impacts were expected. However, since feasibility studies to be prepared under the project could lead to future investment activities in the gas sub-sector, the project supported NEMC in the preparation of a Strategic Environmental and Social Impact Assessment. The final report was prepared in May 2018 incorporating the World Bank Group’s Environment, Health, and Safety Guidelines, World Bank Safeguards Policies and IFC Performance Standards. It is expected to provide guidance to NEMC and MoE "on systematically integrating environmental and socio-economic concerns in development, operations, and maintenance activities in the oil and gas sector" (ICR, p.26). b. Fiduciary Compliance Financial Management After the approval of the project, the Government of Tanzania requested to restructure the implementation arrangements, and MoE and EWURA were designated as the project implementing agencies. Both project implementing agencies had adequate financial management arrangements to support the implementation of the project (Mid-Term Review Report, p.10), but slow disbursement, weak implementation activity in the designated accounts and the absence of a dedicated accounting position were issues throughout project implementation. MoE was late in submitting interim financial reports and there were delays in EWURA’s submission of annual audit reports (ICR, p.27). The last audit of the project by the Controller and Auditor General covered the fiscal year ending on June 30, 2018, and it was unqualified (Audit Report AR/ESCBP/2017/2018, dated December 2018). A dedicated accountant started working for the project in December 2017, but the position was not filled when the accountant left in September 2018. This prevented an efficient closure of the project in December 2018 (ICR, p.24). Lastly, at project closing, only the IDA portion of the project was closed, but the CIDA funds were extended until 2020 to finance the remaining technical assistance activities. On September 24, 2019, the World Bank received a letter from the MoE listing the proposed activities to be financed by CIDA funds amounting to US$9.00 million. Procurement Weak procurement arrangements were the main reason for long delays in project implementation and low Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) disbursement amount (ICR, p.24). Until after the Mid-Term Review, MoE oversaw the procurement for six project beneficiary institutions, but rather than establishing an independent project management unit (PMU), members of staff were assigned to work part time for the project, while carrying out their existing line duties. Procurement was further adversely affected by the frequent change of personnel. Introduction of a new computerized system to manage procurement activities, without adequate training, resulted in cancellation of procurement processes (ICR, p.27). Furthermore, there was lack of coordination between the MoE and project beneficiary institutions in procurement activities (ICR, p.27-28). Procurement accelerated after the assignment of a dedicated accountant to the project and the establishment of project management teams (PMTs) at every institution in December 2017, but there was not enough time left to complete all project activities before project closing in December 2018 (ICR, p.28). The World Bank project team provided support to the PMTs in procurement activities according to the World Bank guidelines. c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Relevance of objectives is substantial and the achievement of the project objectives is modest, but due to major shortcomings in the efficiency of Moderately Outcome Unsatisfactory project implementation, Unsatisfactory complexity of the project design and costly Bank supervision, the efficiency of the project is rated negligible. Hence, the overall outcome rating is unsatisfactory. Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Modest 12. Lessons Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) Three lessons are taken from the ICR with some adaptation of language. The absence of an independent project management unit (PMU), especially in projects with multiple beneficiaries and complex designs, will highly likely result in poor project implementation and failure to achieve the project outcomes. The Ministry of Energy (MoE), the main project implementing agency, did not establish an independent PMU, but assigned members of staff on a part-time basis, who also continued with their daily line duties. This resulted in coordination problems with other project beneficiary institutions—six under MoE—and adversely affected the procurement activities. Project implementation efficiency and procurement pace improved only after the assignment of dedicated staff for project implementation and establishment of project management teams in beneficiary institutions. The project objective to develop public private partnerships (PPPs) in power generation was overly ambitious and unattainable for a technical assistance project with only a five-year implementation period. Structuring a PPP is a very complex process. In addition to the establishment of the legal and regulatory framework, it requires lengthy and very detailed negotiations with potential investors. Even with a well-established legal and regulatory framework, achieving financial closure for a PPP project in five years is highly optimistic. A more realistic objective would be the formation of a sound legal framework and preparation of the bidding documents. The preparation of a Training Needs Assessment (TNA) in advance can improve project implementation and project efficiency. The Government of Tanzania was supposed to complete a TNA before the start of project implementation to be used as guidance for training demands. The TNA was not prepared, and the project had to address training requests on an ad-hoc basis without adequate prioritization. This resulted in inefficient use of project funds and slowing down procurement activities. A TNA was prepared after the Mid-Term Review and, with other changes in project implementation arrangements, the project implementation gained pace in the last year of the project. Nevertheless, these changes were not sufficient to complete most of the project activities and almost half of the project funds could not be disbursed at project closing. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is concise and candid about the shortcomings of project design and implementation. It provides a comprehensive overview of the project. The discussion on Lessons and Recommendations is adequate and supported by evidence from the ICR. The narrative is internally consistent, but only loosely supports the ratings. The focus is mostly on the outputs, rather than the outcomes and the evidence whether they were achieved. The discussion of efficacy is centered on the achievement of the indicators’ targets, which do not fully encompass the achievement of the project objectives. There is insufficient evidence to support the outcome achievements of the project. The Bank Performance and M&E sections could have benefited from a more Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TZ-Energy Sector Capacity Building Prj (P126875) detailed discussion. The actual project cost and disbursement amounts given in various parts of the ICR do not match. The ratings for Bank Performance on pages 2 and 29 do not match. The text includes numerous spelling and grammatical errors, which make the reading of the document difficult. a. Quality of ICR Rating Modest Page 15 of 15