Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Report Number: ICRR0021668 1. Project Data Project ID Project Name P120825 MW: Mining Technical Assistance Project Country Practice Area(Lead) Malawi Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-49010,TF-12935 30-Sep-2016 28,002,618.63 Bank Approval Date Closing Date (Actual) 31-Mar-2011 31-Jul-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 25,000,000.00 2,514,028.34 Revised Commitment 29,724,426.54 2,410,129.68 Actual 28,002,618.63 4,683,958.08 Prepared by Reviewed by ICR Review Coordinator Group Dileep M. Wagle Fernando Manibog Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective, as mentioned in the Project Appraisal Document (PAD), as well as the Financing Agreement (FA), was to improve the efficiency, transparency and sustainability of mining sector management b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) No c. Will a split evaluation be undertaken? No d. Components A. Managing Mineral Rights and Operations (estimated cost at appraisal: US$6.5 million; actual costs: US$6.5 million) The object of this component was to build an efficient, transparent and environmentally and socially sustainable framework for managing mineral rights and operations. Towards this end, the following sub- components were supported: Sub-component A.1: Communications, Outreach and Stakeholder Dialogue - by setting up the structure and support arrangements for continuous and formalized dialogue among stakeholders on mining policy issues, oversight of policy implementation, and design of MNREE's communications and outreach functions. Sub-component A.2: Reforming Minerals Legislation - by drafting instructions for the new mining law and by developing the requisite regulations, covering mineral licensing, monitoring and inspection, health, safety and environmental standards, and a model Mining Development Agreement. Sub-component A.3: Modernizing Minerals Licensing - by establishing a modern computer-based cadastral system and administrative arrangements for the efficient and transparent allocation and management of mineral exploration and mining rights by the Ministry of Natural Resources, Energy and Environment (MNREE). Sub-component A.4: Strengthening Supervision of Mineral Operations - through capacity-building of MNREE for undertaking regular monitoring and inspections of exploration, mining and mineral processing operations. Sub-component A.5: Strengthening Environmental and Social Management of Mining - by establishing a comprehensive environmental and social management framework and robust monitoring systems at central, district and community levels. B. Generating and Managing Mineral Revenues (estimated cost at appraisal: US$1.55 million; actual costs: US$1.55 million) This component aimed to support the authorities in developing transparent arrangements for optimal generation and use of mineral revenues. Towards this end, the following sub-components were supported: Sub-component B.1: Reforming the Mineral Royalty and Tax Regime - by completing the establishment of a coherent and standardized fiscal regime for mining by designing and implementing appropriate royalty and tax regulations. Sub-component B.2: Maximizing Mineral Revenue Collection - by building capacity in royalty and tax administration. Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Sub-component B.3: Building Mineral Revenue Transparency - allowing revenue collection arrangements to be subjected to scrutiny and assurances given to the public that revenues are being properly accounted for. Sub-component B.4: Developing Mineral Revenue Forecasting as part of the Governments annual budget preparation cycle and medium-term macro-economic planning. Sub-component B.5: Developing Mineral Revenue Management - to deal with potentially large and volatile mineral revenue flows and determine allocations made from funds collected. C. Promoting the Mining Sector (estimated cost at appraisal: US$17.0 million, actual costs US$17.0 million) This sub-component supported the Government in improving the enabling environment for mining sector development, by acquiring and disseminating geo-data, fostering more sustainable artisanal and small scale mining, augmenting the supply of Malawians trained at tertiary level in mining-related disciplines, and improving the policy environment for mining-related infrastructure development. Towards this end, the following sub-components were supported: Sub-component C.1: Program of Geo-data Acquisition, Interpretation and Promotion - including a comprehensive country-wide air-borne geo-physical survey to acquire new data, interpretation studies and resource assessments to identify promising mineral opportunities and activities to promote investor interest. Sub-component C.2: Establishing a Geo-Data Management Center - to digitize, store and increase on- line access to geo-data. Sub-component C.3: Program of Training and Awareness-Raising for Artisanal Mining - to gradually improve artisanal and small-scale (ASM) practices and institutional support to the ASM sector. Sub-component C.4: Building Capacity for Tertiary Education in Mining - to increase the supply of Malawians qualified at tertiary level in mining-related disciplines, so as to help build up the regulatory capacity of government institutions and help alleviate the chronic skills shortage faced by a fast-growing mineral sector. D. Project Management (estimated cost at appraisal: US$2.70 million; actual costs: US$2.72 million) This component was to enable MNREE to undertake project management in accordance with the World Bank's fiduciary and other guidelines, through provision of incremental costs, equipment, training on fiduciary and project management issues, project audits and engagement of technical advisers to provide technical expertise on project performance monitoring and planning. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: There was minimal divergence between project costs estimated at appraisal and actual costs at project closing. The only component where costs diverged slightly at closing was the project management component, where actuals ended up higher by US$0.02 million. Though the project was restructured four times, this did not impact on costs in any way. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Financing: The sources of funding for this project at appraisal consisted of IDA resources of US$25.0 million, augmented by cofinancing of US$5.66 million (Euro 4.1 million) from the EU, made available through a grant. Borrower Contribution: No borrower contribution was originally envisaged for this project, though a contribution of US$4.5 million from the borrower was subsequently added at restructuring, of which US$3.7 million was actually disbursed. Dates: The project was originally envisaged to close on September 30, 2016. However, the closing date was extended by nearly two years through restructurings in May 2016, May 2017, October 2017 and March 2018, to a final date of July 31, 2018. From project approval on March 31, 2011 to project closing on July 31, 2018, the project took 7 years, 4 months to complete. 3. Relevance of Objectives Rationale Alignment with Strategy: The objectives of the project were consistent with the broad objectives of the World Bank's Country Assistance Strategy for Malawi (FY13-16), under the theme of promoting sustainable, diversified and inclusive growth (Results area: strengthening social accountability for service provision). Citing the project, the CAS indicated (Page 41) that while sustainable growth of the mining sector from its then-presently low base was not assured, it was nevertheless possible to achieve - if geology campaigns and exploration revealed more resources, and if projects then in the pipeline qualified for investment decision in the medium term. On account of a delay in finalizing the subsequent CAS, the FY13-16 CAS remains the most current at project closing. The project was also consistent with the larger objectives of the Malawi Government's Growth & Development Strategy (2017-22). Though not a priority area in its own right, the enhanced production and management of mineral resources was mentioned under the theme of Energy, Industry and Tourism Development (Page 43), with the aim of improving transparency in the management of resources through the Extractive Industries Transparency Initiative (EITI), reviewing the legal and institutional framework, and increasing exploration and mining of mineral resources. Country Context: The PDO of improving the efficiency, transparency and sustainability of mining sector management was relatively clear and measureable, and likely to help Government improve sector governance and create an enabling environment for mining. For many resource-rich countries, the EITI approach was an important entry point to the sound management of their resource sectors, by incentivizing transparent revenue reporting by government and industry, accompanied by actions along the entire management chain to improve governance and provide for institutional capacity-strengthening. Previous Sector Experience: The World Bank's engagement in the sector in Malawi started in 2007, through preparation of a Mineral Sector Review, which underpinned a National Mines and Minerals Policy, in 2011, setting overarching objectives for the management of the sector. The current project, initiated in that year, and co-funded by the EU, was developed in recognition of the fact that the wider benefits of mining could not be achieved without good governance, with clear regulatory mandates and strong institutions operating in a transparent manner. Recent mineral exploration, facilitated by the geological data made available Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) through the project, appears to have revealed some promising mineral deposits, from which extraction is expected to commence soon. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective “To improve the efficiency of mining sector management” Rationale Theory of Change: The PAD does not discuss the project's theory of change. Its Annex 1 presents the project's Results Framework, which contains year-by-year targets for the intermediate and final outcome indicators. Actions and specific percentages are also delineated; however, the Results Framework does not analyze causality and attribution, which a credible theory of change would require. The ICR does however present the project's theory of change, with a diagrammatic description of the results chain, indicating the linkages between activities and outputs, and between outputs and outcomes. A broad causal link can be drawn between the projects activities, which included the introduction of modern mining legislation and a computerized cadaster, development of a standardized fiscal regime, together with mineral tax administration and revenue management, and completion of health, safety environmental and social management policies, among others, and the expected outcomes in terms of efficiency, sustainability and transparency of mining operations and its revenue utilization. These in turn would be linked to long-term outcomes via the increased contribution of mining to sustainable growth and development in Malawi; a key underlying assumption being that once appropriate exploration licenses were allocated to mining companies by leveraging the geo-data made available though the project, exploration would yield favorable results in terms of technical and economic viability for extracting the mineral resources. A second implicit assumption was that international commodity prices would remain high and stable enough to encourage mineral extraction and spur private investment in the sector. It should be mentioned that the theory of change discussion does not analyze whether these activities were the most appropriate ones to achieve the desired outcomes, as well as whether they were of adequate scale and timing. For instance, while the introduction of mining legislation and a computerized cadaster would no doubt facilitate greater efficiency, in the absence of other, specific improvements to approval procedures, it could not directly ensure this result. The discussion also does not encompass the extent to which the results framework was adequate to capture the intended results. The issue of weaknesses in the results framework is discussed in more length in Section 8, under Quality at Entry. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Outputs: - A geo-data management center was established, with a functioning geo-data management information system (GDMIS). However at project closing, the system was not accessible on-line, as planned, which limited remote accessibility of detailed geo-data by investors. - Computer-based cadaster system went live in April 2017. All license data from 20009 to project closing was captured in the database, and all license applications were being processed through the system by a trained team of officers. Target of a computerized cadaster being in use was thereby achieved) - A number of public service personnel (179, including 32 women), were trained in sector-relevant fields (against a target of 115). This output supported the achievement of all three pillars of the PDO (efficiency, sustainability and transparency). - A number of students (149, including 61 women, against a target of 150) were newly enrolled in mining- related tertiary education programs (this output similarly supported the achievement of all three pillars of the PDO (efficiency, sustainability and transparency). Some project activities associated with these outputs were only partially completed by project closing on account of resource constraints: these included hosting and maintaining the established cadaster system through cloud-based services, developing a Local Area Network for the mining cadaster offices, and making the established GDMIS available to the public by displaying non-confidential data online. However, it could be argued that to some extent these were optional enhancements to the system, which limited their impact on the PDO-level indicators. Outcomes: Despite achievement of the above outputs, the project was only modestly successful in improving the efficiency of mining sector management. Though the project did facilitate the issuance of mineral rights by increasing the knowledge of Malawi's geological potential and helping to streamline some administrative processes, this did not translate into a significant improvement in approval processing times. Despite some improvement, by close of project it still took as long as 64 days to issue the various licenses associated with mineral rights, against a target of 45 days. The original baseline of 90 days was revised at a late stage (March 2018) to 102 days, but in performance terms the overall outcome was nevertheless disappointing. Based on the above, efficacy of this objective is rated Modest. Rating Modest OBJECTIVE 2 Objective “To improve the sustainability of mining sector operations” Rationale Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Outputs: The following intermediate outcomes were achieved: - Policy framework for health, environment and social management was completed and monitoring systems were put in place, along with development of new implementation tools. - A large number (179, against a target of 115) of public service personnel were trained in sector-related fields. As mentioned above, this output supported the achievement of all three pillars of the PDO (efficiency, sustainability and transparency). - A number of students (149, including 61 women, against a target of 150) were newly enrolled in mining- related tertiary education programs (this output similarly supported the achievement of all three pillars of the PDO (efficiency, sustainability and transparency) Outcomes: As indicated by the PAD, this objective was framed in terms of the establishment and better enforcement of environmental, social, health and safety standards in the mining industry, which would have domestic as well regional benefits. As such, the key results indicator was expressed (somewhat narrowly) in terms of the percentage of mining operations that would be subjected to environmental certification. On this basis, the objective was substantially achieved. The project facilitated enhanced sustainability of mining sector governance, in part by increasing knowledge of existing social and environmental baselines and advising on approaches to environmentally/socially sustainable management of the sector through a Strategic Environmental and Social Assessment (SESA) and a SESA Action Plan; in part by facilitating the development and adoption of health, safety environmental, and social management policies and monitoring systems by the Government, and also by helping to increase the capacity of trained public service personnel. This results indicator was also achieved. On the basis of environmental compliance audits conducted by the Environmental Affairs Department, the percentage of mining operations found to have been compliant, increased from 50 percent in 2012 to its 90 percent target value by project closing. Rating Substantial OBJECTIVE 3 Objective “To improve the transparency of mining sector management” Rationale Increasing the transparency of the mining sector - especially in terms of revenue reporting - was a key focus of the Extractive Industries Transparency Initiative (EITI). Outputs: Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) - New Mining Bill was drafted and approved by Cabinet Committee, and by Parliament in December 2018. In addition, a new artisanal policy was drafted and adopted. - Standardized royalty and tax laws and regulations were drafted, in support of the establishment of a globally competitive fiscal regime for mining operations in Malawi. This was accompanied by capacity building of personnel on issues related to mineral tax administration and revenue sharing. - A study, providing options on improving mineral tax administration and revenue management, was completed. One of the study's conclusions established that the low levels of revenue collections, coupled with macro-economic considerations, did not warrant the setting up of a Mineral Revenue Fund. - EITI Multi-stakeholder Group, consisting of industry, government and civil society representatives, was established and began meeting on regular basis. The Group released the first EITI report in April 2017, disclosing Malawi's mineral sector revenues, and the second report in June 2018. - The geo-data management information system (GDMIS) was up and running, as mentioned under PDO1. However, on-line access to the system had not been made available by project closing, thereby diminishing some of the utility of the system. Outcomes: Considerable progress was achieved towards the objective of increasing transparency and governance in Malawi's mining sector. Under the project, modern transparent legislation and regulatory frameworks such as the Mining and Minerals Bill, a new artisanal mining policy, and a standardized fiscal regime were introduced. In addition, tools for transparent sector governance such the GDMIS and a multi-stakeholder forum for implementing EITI in Malawi, had been brought into play. The broad objective of ensuring annual public reporting of mineral sector revenues, which was the PDO indicator, could be considered to have been substantially achieved. Based on the above, efficacy of this objective is rated Substantial. Rating Substantial OVERALL EFF TBL OLD Rationale The project helped enhance knowledge of Malawi's geological potential, establish modern tools for sector governance, and build capacity in personnel for this purpose. In keeping with the objectives of the EITI initiative, these outcomes helped foster new private sector investments, by helping alleviate country risk perceptions of potential investors. At project closing, total investments from new exploration activities that had been stimulated amounted to some US$659 million. The project broadly met its objective of ensuring increased sustainability and transparency in the mining sector, but was only partially successful in ensuring improved mining sector efficiency. Based on this, and taking into account implementation delays and the fact that some (albeit non-critical) project activities were left unfinished at closing, overall efficacy is rated Substantial. Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Overall Efficacy Rating Substantial 5. Efficiency Administrative and Operational Efficiency: The project faced delays in implementation, which led to a nearly two-year extension in project closing date, from September 30, 2016, to July 31, 2018. These delays were ostensibly to allow for better coordination with the co-financing provided by the EU; however, it is noted (ISR, p.22) that despite this extension, on account of apparent resource constraints, not all project activities were completed by closing. Project costs remained much the same at closing as at inception, though some activities did experience overruns, so that some other tasks (e.g. hosting and maintaining the established cadaster through cloud-based services, or developing a local area network for the mining cadaster offices) could not be completed by project closing. Operating costs, as reflected in amounts actually utilized towards project management, were capped at 9 percent of total costs, which does not appear to be particularly low. The ICR does argue that this is a low percentage, given the complexity of the project, but does not offer any benchmark comparators, by way of evidence. Project Management costs did exceed their budget by US$200,000, which was in keeping with the extended duration of the operation. [It should be mentioned that the team subsequently provided examples of three recent mining projects, whose ICRs indicated ratios of actual project management costs to total disbursements in excess of 10 percent. It is however not clear to what extent these can be adopted as benchmarks, since two of the projects were rated modest for efficiency by IEG in reviews of their ICRs, and the third review is awaited]. Economic and Financial Efficiency: As this was a technical assistance operation, no detailed economic analysis was conducted either in the PAD or the ICR. The PAD mentioned that the value of minerals output, projected to reach around US$250 million in 2012 (World Bank's Mineral Sector Review, 2009), could potentially double by the end of the decade, if binding constraints could be lifted. The project, which was intended to enhance Government's ability to manage the country's resources wisely, was expected to contribute to that, helping to expand mineral investment flows and optimize their contribution to sustainable development. To some extent it succeeded in doing that. The increased geological data made available as a result of its activities resulted in more than eight new mineral exploration activities, of which four have already yielded positive results in terms of viable mining options. As indicated earlier, expected investment flows were of the order of US$659 million. That said, this does not mean that the project design necessarily provided the least-cost approach to these economic benefits, or alternatively that higher benefits could not have been obtained through better design or speedier implementation. The ICR does not provide any information to be able to consider any such counterfactual. Taking into account the implementation delays that did occur, and the fact that some activities remained incomplete by closing, in the absence of other information, the project's efficiency is rated Modest. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were substantially relevant to the Government's strategy, as well as to the World Bank's Country Assistance Strategy, and continues to be so. The achievement of PDO1 was found to be modest, on account of shortcomings in meeting results targets, but the project substantially achieved its objective of improving the sustainability of mining sector operations (PDO2) as well as of improving the transparency of operations (PDO3). Overall efficacy was Substantial. Efficiency was rated Modest, on account of shortfalls in administrative and operational efficiency. Taking all of this into account, the overall project outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Exogenous factors, especially the possibility of volatility in global commodity prices, constituted an important area of risk to the country's extractives sector. While the improvements in governance and in the enabling environment for the mining sector operations under the project would be expected to have a positive impact in terms of private sector investments, the risk of a global decline in commodity prices could have serious adverse effects on investor perceptions of risk. Interestingly, the Operational Risk Assessment Framework (ORAF) presented in the PAD made no reference to this potential risk, focusing instead on stakeholder, implementing agency and project-related risks, which - it concluded were medium to low. In practice, by project closing commodity prices appeared to have bottomed out and faced a relatively strong outlook for such minerals as niobium, graphite, heavy sands and rare earth metals, for which exploration in Malawi had yielded favorable results to date. That said, mineral commodity prices are expected to be cyclical in nature over the medium term, and the improvements in transparency, efficiency and sustainability brought about by the project will no doubt stand the country in good stead during any future downturn. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) One implementation risk that should be flagged would be the incomplete nature of some of the project's activities, such as the hosting and maintenance of the established cadaster system through cloud-based services, the development of a local area network for the mining cadaster offices, and providing public access to GDMIS (for non-confidential data), which if not corrected in the foreseeable future - could prevent the full development impact of the project from being realized in the medium term. 8. Assessment of Bank Performance a. Quality-at-Entry The ICR does not provide any details on the preparation of the project; however, the design of the project was relatively straightforward and, in terms of strategic relevance, appropriate to the country's needs. As outlined in the PAD, the project was designed consistent with the key elements of the Extractive Industries Transparency Initiative (EITI), incorporating the latter's focus on effective management of oil, gas and mining revenues, all along the entire value chain. Project design was based on the simple logic that effective management of Malawi's mining sector would require increased information on the country's geological potential, modern computerized systems, streamlined governance frameworks and increased sector transparency to better regulate mineral sector operations. The project put in place an implementing agency (Project Management Team - PMT) under the oversight of a Project Steering Committee (PSC), which included key senior members of the Ministry of Natural Resources, Energy and Environment (MNREE). Recognizing that the implementing agency did not have any experience in implementing a World Bank-funded project, mitigation measures in the form of training to accounting staff from MNREE were to be provided, along with a Financial Management Specialist, recruited for up to 18 months to coach the staff in the Ministry in the use of accounting software provided by the project. Other risk mitigation measures were however less well thought out. The project's design was largely based on the then upward trends in mineral and commodity process. As mentioned earlier, the risk of a potential decline in prices was not given any consideration in the formulating the project's Operational Risk Assessment Framework (ORAF). Mineral commodity prices did however begin to decline in 2013-14 and continued to decline steeply during the project's implementation period, which, while not directly affecting the project's outputs, did impact negatively on the performance of the sector, and hence on potential investor interest. Another issue was the design of the results framework. This incorporated indicators relevant to all elements of the PDO, but was relatively narrow in scope and oriented towards outputs rather than higher- level outcomes. The outcome indicator on efficiency faced some attribution weaknesses, since it was not clear to what extent the targeted improvement in processing of mineral rights was due specifically to project-related activities. At the same time, the fact that the baseline for this indicator had to be revised late in the implementation process suggested that the targets were based on insufficient prior analysis of actual processing speeds; nor was it clear that the targets were set on the basis of any best-practice benchmarks. Similarly, the indicator on transparency, which focused on the simple act of public reporting of mineral sector revenues, was non-specific on the quality of data that would actually be reported. Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project does not appear to have been intensively supervised. Between effectiveness in August 2012 and closure in July 2018, the project received a total of only six supervision missions (ICR, p.27), or one a year. Though there were three task team leaders (TTLs) in charge of supervising the project at different times during its implementation, it does not appear that any of them was based in the country office. Some additional field-based resources were however provided in the form of a short-term consultant located in Malawi, to follow up on activities and provide interface with the implementing agency and beneficiary institutions. A Mid-Term Review was held in December 2014. Supervision of M&E and Safeguards aspects of the project appears to have been adequate. M&E results were made available to the project team on a timely basis, and any discrepancies in measuring indicators were addressed in time through collaboration between the team and the implementing agency. The ICR indicated (p.25) that fiduciary mechanisms were adequate, with regard to financial management and procurement, but provides no details on whether any implementation support was needed to be provided by the team in this regard.. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The design of the M&E system reflected the results chain and incorporated indicators measuring all elements of the PDO. The results framework was fairly detailed and measurable. However, many of the indicators took the form of policy triggers (e.g. "achieved" vs."not achieved"), or provided no basis for judging quality and adequacy (e.g. whether the training courses administered to public service personnel was appropriate, of adequate quality, and whether the threshold for achieving the training targets was simply attendance at courses offered or whether there were performance benchmarks incorporated). That said, the intermediate indicators themselves adequately captured progress towards completion of the project's main activities. Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) b. M&E Implementation Data for M&E were collected and analyzed in a systematic manner. Baseline data were first collected by the implementing agency and subsequently examined by the project team for quality and accuracy. This process allowed for corrections to be made in timely manner - such as the baseline on the outcome indicator for Efficiency, and output data for number of students and personnel trained. c. M&E Utilization M&E data shared with World Bank supervision teams and with project management proved useful in monitoring implementation progress. Such information was utilized effectively in decision-making, which for instance helped bring about an increase in project disbursement in the latter stages of the operation. M&E data found its way into World Bank Implementation Status & Results (ISR) reports, highlighting important issues for action by the Bank's country management. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Environmental and Social Compliance: The project provided technical assistance and did not directly support any specific mining activity, so its component activities were not expected to have any direct environmental or social impacts. According to the PAD, it would instead have significant positive impact, through improved environmental management of the mining sector and by enhancing environmental capacity in the Ministry of Natural Resources, Energy and Environment. The project was however classified as Category B for social and environmental purposes, triggering OP/BP 4.01 (Environmental Assessment), due to the potential impact of any follow-on investment projects in the minerals sector. One consequence was the need for preparation of a Strategic Environmental and Social Assessment (SESA). This was deemed to be appropriate to help the Government (a) diagnose key environmental and social problems associated with the anticipated growth of Malawi's mining sector, (b) identify the policy, regulatory and other actions needed to minimize the adverse environmental and social impacts of future mining operations, and (c) propose specific measures that Government could implement in the foreseeable future to improve the environmental and social sustainability of mining in the country. At appraisal, the project disclosed terms of reference for consultants to assist with the preparation of a mining sector SESA. Preparation of the SESA within 18 months of Effectiveness was one of the conditions of the Financing Agreement. This condition was duly complied with. In addition, the project consistently complied with all required environmental and social safeguards in a satisfactory manner through its lifetime, and was rated Satisfactory on both the overall safeguards ratings and OP/BP 4.01 (Environmental Assessment) ratings in the final ISR. Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) b. Fiduciary Compliance The PAD noted (p.13) that on account of a lack of capacity in dealing with World Bank projects, the overall financial management risk for the project was assessed as Medium-Impact during preparation and Medium-Likelihood during implementation. Mitigation measures were planned accordingly, including ensuring that accounting staff of MNREE be sufficiently trained before and after implementation, recruiting a Financial Management Specialist to coach current Ministry staff, and providing for a pool of accounting staff, including the Chief Accountant, for the project management team. It was agreed that the Ministry would provide to the World Bank quarterly unaudited interim financial reports (IFRs) on the source and use of funds within 45 days after the end of the quarter, and annual audited financial statements within six months after the end of the Government fiscal year. The ICR does not report on whether these covenants were complied with, other than a general statement that fiduciary mechanisms were found to be adequate with regard to financial management. Subsequent information provided by the team however indicates that the project generally complied with FM requirements, including financial covenants. Most interim financial reports and audited statements were submitted on a timely basis, and all audited reports had unqualified opinions, except for the one for 2016-17. The ICR provides no information on procurement-related issues. Information provided subsequently by the team indicates that the lack of procurement capacity identified during appraisal was mitigated during implementation by recruiting a dedicated, experienced Procurement Specialist. There were no cases of mis-procurement declared during implementation. c. Unintended impacts (Positive or Negative) There were no other unintended impacts.. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Relevance of objectives is rated Substantial. Efficacy is rated Substantial. Efficiency is rated Outcome Satisfactory Moderately Satisfactory Modest on the basis of delays in implementation and information provided by the ICR. Quality at Entry rated Bank Performance Satisfactory Moderately Satisfactory Moderately Satisfactory on account of design weaknesses. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) Quality of Supervision rated Moderately Satisfactory on the basis of the details on supervision activities provided by the ICR. Quality at Entry rated Moderately Satisfactory on account of design weaknesses. Quality of Supervision rated Quality of M&E Substantial Substantial Moderately Satisfactory on the basis of the details on supervision activities provided by the ICR. Quality of ICR --- Modest 12. Lessons  The design of extractive sector projects needs to take commodity price projections into consideration. Commodity prices being typically cyclical in nature, the viability of an operation cannot be based only on the prevailing level of prices at appraisal, should they be at a peak, but needs to take into account the real possibility of a decline over the short to medium term. To the extent that the operation can focus on the basics, such as enhancing geological knowledge, streamlining administrative processes, improving sector governance, etc., exposure and hence vulnerability to exogenous factors such as commodity price fluctuations can be reduced.  Project preparation needs to include detailed analysis to ensure the accuracy of baselines upon which results indicators are based. Equally, target values need to strike a balance between ambition and reality. In the project, for instance, the Efficiency indicator of number of days for issuance of mineral rights would have been more convincing if the target number of days was shown to have been based upon global or regional benchmarks for similar approval processes and the baseline to have been correctly estimated at appraisal. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clearly written, concise and generally consistent with guidelines. It provides a good description of the projects components and a detailed theory of change (which was missing in the PAD). The analysis is broadly evidence-based, though not especially critical of the quality of the evidence. The ICR does not provide any Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW: Mining Technical Assistance Project (P120825) detail whatsoever on fiduciary compliance in respect of financial management and procurement. Secondly, while discussing Bank performance, it does not provide adequate information on the manner in which the project was prepared and appraised, nor on the adequacy of supervision resources and inputs. a. Quality of ICR Rating Modest Page 16 of 16