Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004768 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF A1838 ON A SMALL GRANT IN THE AMOUNT OF USD 0.405 MILLION TO MONGOLIA FOR Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) June 17, 2019 Governance Global Practice East Asia And Pacific Region Regional Vice President: Victoria Kwakwa Country Director: Martin Raiser Senior Global Practice Director: James A. Brumby Practice Manager: Alma Kanani Task Team Leader(s): Badamchimeg Dondog ICR Main Contributor: Erdene Ayush ABBREVIATIONS AND ACRONYMS AG Auditor-General CPD Continuing Professional Development MNAO Mongolian National Audit Office INTOSAI The International Organization of Supreme Audit Institutions INTOSAI IDI International Development Initiative of the International Organization of Supreme Audit Institutions IPSAS International Public Sector Accounting Standards ISSAI International Standards of Supreme Audit Institutions SAI Supreme Audit Institution SAI CDF Supreme Audit Institution Capacity Development Fund UK NAO National Audit Office (The United Kingdom) QA Quality Assurance TABLE OF CONTENTS DATA SHEET ............................................................................ ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES............................................................. 5 II. OUTCOME ................................................................................................................................ 9 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME..................................... 13 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ..... 15 V. LESSONS LEARNED AND RECOMMENDATIONS..................................................................... 16 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ................................................................. 18 ANNEX 2. PROJECT COST BY COMPONENT................................................................................... 25 A The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Capacity Building for the Mongolian National Audit P155305 Office Country Financing Instrument Mongolia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Ministry of Finance Mongolia National Audit Office Page 1 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Project Development Objective (PDO) Original PDO The main objective of the proposed project is to strengthen MNAO’s capacity, primarily in financial audit and audit firm contracting and oversight. To achieve this objective a comprehensive and systematic continuing professional development strategy together with a time-bound action plan that is needs-driven, tailored and sustainable will be developed and implemented. For this purpose, the current capacity strengths and gaps in financial audit and related areas will be independently assessed by a peer SAI. In parallel to this assessment, MNAO personnel will be surveyed and participate in focus groups to identify their requests for and perceptions of their professional development needs and alternative means to achieve these objectives. The results of the financial audit peer review and the knowledge and learning needs assessment to be carried out under this proposed project, as well as the 2010 performance audit peer review, will feed into development of a CPD strategy for the institution that will then be developed into a 3-year knowledgeand learning program. Furthermore, as part of the project, tailored training materials will be developed and actual training will be delivered to MNAO staff as per the CPD strategy and the knowledge and learning program. While the specific content and depth of topics to be covered under the CPD program are largely to be identified and tailored as per the assessments to be carried out,an early indication of capacity gaps reveals such areas as MNAO staff’s knowledge and understanding of the relevant standards, i.e. IPSAS and the newly translated ISSAIs, quality control, contracting and quality assurance of private audit firm engagements, and audit of the consolidated financial statements of the Government. PDO as stated in Legal Agreement (if different from Project Paper) The objective of the Project is to strengthen the Mongolian National Audit Office (“MNAOâ€?)’s capacity, primarily in financial audit and audit firm contracting and oversight. FINANCING FINANCE_T BL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A1838 405,000 405,000 403,380 Total 405,000 405,000 403,380 Total Project Cost 405,000 405,000 403,380 KEY DATES Approval Effectiveness Original Closing Actual Closing 22-Jan-2016 04-Feb-2016 30-Aug-2018 31-Dec-2018 Page 2 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 26-Jun-2018 0.27 Additional Financing Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 29-Jun-2016 Satisfactory Satisfactory 0.08 02 16-Dec-2016 Satisfactory Satisfactory 0.08 03 29-Jun-2017 Satisfactory Satisfactory 0.10 04 04-Jan-2018 Satisfactory Satisfactory 0.21 05 13-Jun-2018 Satisfactory Satisfactory 0.27 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Victoria Kwakwa Country Director: Bert Hofman Martin Raiser Senior Global Practice Director: James A. Brumby James A. Brumby Practice Manager: Roberto Tarallo Alma Kanani Regis Thomas Cunningham, Badamchimeg Dondog, Regis Task Team Leader(s): Badamchimeg Dondog Thomas Cunningham ICR Contributing Author: Erdene Ayush Page 3 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Page 4 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Its abundance of natural resources presents Mongolia with a unique opportunity and challenge to ensure that the wealth generated from its resource utilization is transformed into sustainable and equitable growth. In recent years, the country has adopted a series of reforms to manage public resources in a more sustainable, transparent and decentralized way. As commonly observed in many economies around the world that are reliant on mineral exports, the Mongolian economy has experienced a high degree of volatility in public revenues and has been characterized by boom-and-bust cycles. In reaction to the 2008-2009 crisis, the Government of Mongolia (GoM) adopted a Fiscal Stability Framework in 2012, followed by passage of an Integrated Budget Law that sets forth rules for budget planning and decentralization to achieve strengthened accountability and improved distribution of public spending. Following the decentralization of budget accountability to the local municipal authorities, as introduced by the Integrated Budget Law, the Mongolian Law on State Audit was amended in 2013 to enhance independence of the country’s Supreme Audit Institution (SAI) – the Mongolian National Audit Office (MNAO) – both in terms of its financing and human resource management. The new Law provides MNAO an organizational structure and personnel to provide oversight, enhance accountability and support implementation of the decentralized budget nationwide at all levels of government. 2. As the country’s supreme audit institution, MNAO is mandated to conduct financial, performance and compliance audits. The Integrated Budget Law of Mongolia also stipulates MNAO’s responsibilities for auditing the annual government accounts and the relevant timeline for doing so. 3. In the early stages of its financial audit capacity development, MNAO received support from the UK National Audit Office (NAO) and generally adopted that SAI’s financial audit methodology and manual. In the past few years, in consonance with the Strategic Plan for 2013-2018, MNAO has updated its financial audit manual to make it more adapted to local conditions, developed audit software and translated the ISSAI standards. 4. Given MNAO’s wide audit mandate and its current staffing level, MNAO engages over 40 private audit firms to conduct the annual financial audits of a large number of budget entities, including IFI-funded projects, and state-owned enterprises. In regard to IFI-funded audit engagements, IFI procurement procedures have been followed to contract audit firms due to perceived weaknesses with MNAO’s own audit firm contracting procedures. The need to strengthen and streamline MNAO’s private audit firm selection and contracting processes has therefore been an important area needing improvement and could lead to the expanded use of country systems in this area. 5. An international team from Norway and Sweden conducted a peer review of MNAO’s performance audit capacity in 2010 and peer review results and recommendations were well received by MNAO management. However, MNAO’s financial audit capacity and practices had not been externally reviewed in recent years. 6. The MNAO 2013-18 Strategic Plan was developed with support from the INTOSAI International Development Initiative (IDI). It provided the institution with a strong strategic direction and had been driving the actions taken by MNAO in past years towards improving its systems and processes (e.g. legal framework, audit methodology Page 5 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) and standards, and collaboration with stakeholders including the Parliament). MNAO’s leadership considered human resource development at the heart of its strategy and a critical goal in the 2013-18 Strategic Plan was to provide MNAO personnel with the requisite professional skills to conduct their duties effectively and efficiently. 7. MNAO is mandated to provide methodological support and training to its auditors both at the central and provincial levels. However, an appropriate continuing professional development policy, program and training curricula had not been in place and MNAO lacked qualified trainers with up-to-date professional knowledge, experience and training materials. MNAO had therefore not been able to organize effective, targeted training for its auditors on a regular basis to implement one of its core strategies. The revised organizational structure, particularly the incorporation of provincial auditors, new MNAO legal framework, and move to ISSAIs and accrual-based accounting, had created a vastly more complex set of professional development requirements and need for a carefully and professionally developed CPD program. 8. The need to establish a well-structured, needs-driven and sustainable continuing professional development strategy together with an action plan and a training program was consequently a priority area for MNAO. The practice of exposing senior and junior auditors to the same training content on an irregular basis, irrespective of their level of experience and areas of expertise, called for a more systematic approach to addressing capacity gaps. Trainings and continuing professional development had become even more crucial yet challenging for MNAO as under the revised legal framework the Auditor-General became directly responsible for keeping the professional knowledge and practices up to date for its over 380 auditors, inclusive of those at the provincial level, as compared to the cohort of about 60 auditors at the central level previously. 9. The Capacity Building for MNAO Project funded from the Supreme Audit Institution Capacity Development Fund (SAI CDF) financed by the Swiss State Secretariat for Economic Affairs (SECO) in the amount of USD 0.405 million provided assistance to the MNAO for three years to help address some of the above-referenced challenges through developing a continuing professional education program, delivering the relevant priority training and developing a procedure for selection of private audit firms contracted to conduct audits of government entities. Project Development Objectives (PDOs) 10. The PDO of the project was to strengthen the MNAO’s capacity, primarily in financial audit and audit firm contracting and oversight, through development and implementation of a comprehensive and systematic continuing professional development strategy together with a time-bound action plan that is needs driven, tailored and sustainable. Key Expected Outcomes and Outcome Indicators 11. Under the main objective of strengthening the MNAO’s capacity, the project expected to carry out the following activities: (a) assessing the current MNAO capacity strengths and gaps through a SAI peer review, (b) conducting an internal organizational needs assessment among MNAO staff on professional development and possible solutions, (c) developing and implementing a CPD strategy and action plan, (d) providing targeted Page 6 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) trainings and developing training materials, and (e) developing and adopting proper procedures on audit firm contracting and oversight. 12. The key expected performance indicators, against which results were measured included the following: i. MNAO has made demonstrable progress in implementing the targeted improvements which emerge from the financial audit peer review; ii. MNAO auditors have been trained in the priority areas identified by the CPD strategy and action plan; iii. MNAO has utilized a strengthened procedure for selection of private audit firms contracted to conduct audits of government entities. Components 13. The project consisted of 5 main activities: (a) Peer review of MNAO’s financial audit activities, (b) Comprehensive training needs assessment of the new MNAO structure, (c) Design a comprehensive and sustainable continuing professional development strategy together with a detailed action plan, (d) Development of customized training materials and delivery of the training as per the training program tailored to the capacity building needs identified by the peer review and training needs assessment and reflected on the continuing professional development strategy and program, and (e) Design and apply an updated private audit firm contracting process. Activity 1: Peer review of MNAO’s financial audit activities (USD 0.070 million at appraisal, USD 0.065 million actual) 14. The objective of this activity was to conduct a peer review of the MNAO’s financial audit activities by an experienced SAI using the INTOSAI’s SAI PMF methodology as deemed appropriate. A peer review had never been performed on MNAO’s financial audit capabilities; hence the MNAO critically needed, and its management supported, a peer review concentrating on its financial audit capacity and related domains. The peer review report was quality reviewed by the International Donor Initiative (IDI) of the International Organization of Supreme Audit Institutions (INTOSAI). The peer review provided an objective, third-party assessment of its strengths, weaknesses and priority improvement needs. These priority areas for improvement were then incorporated in the organization’s CPD strategy and the relevant knowledge and learning program. Activity 2: Comprehensive training needs assessment of the new MNAO structure (USD 0.040 million at appraisal, USD 0.040 million actual) 15. In parallel to the SAI peer review activity mentioned above, this activity aimed to carry out a comprehensive MNAO knowledge and learning needs assessment covering the entire organization, including its local branch office staff. The knowledge and learning needs assessment was carried out to identify capacity gaps from within the organization. This assessment covered all MNAO staff, both at the central and local branch offices and utilized various surveying methods such as enquiry, questionnaire, focus group discussions, individual interviews, etc. Page 7 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Activity 3: Design a comprehensive and sustainable continuing professional development strategy together with a detailed action plan (USD 0.040 million at appraisal, USD 0.042 million actual) 16. Under this activity, a comprehensive, needs-driven and sustainable continuing professional development strategy (also known later as the Learning and Development Strategy) together with a detailed action plan for three years was developed based on the peer review conducted at MNAO and the knowledge and learning needs assessment. Activity 4: Development of customized training materials and delivery of the training as per the training program tailored to the capacity building needs identified by the peer review and training needs assessment and reflected on the continuing professional development strategy and program (USD 0.235 million at appraisal, USD 0.236 million actual) 17. The purpose of this activity was to develop customized training materials tailored to MNAO’s capacity building needs described in the CPD strategy and the knowledge and learning program and to deliver actual training to MNAO personnel during most of the three-year period of the CPD program. While the specific content and depth of topics to be covered under the training program were largely to be identified and tailored as per the assessments to be carried out, an early indication of capacity gaps at appraial included areas such as MNAO staff’s knowledge and understanding of the relevant standards, i.e. IPSAS and ISSAIs, quality control, contracting and management of private audit firm engagements, and auditing of the consolidated financial statements of the Government. Under the project, training delivery was also expected to include innovative modalities such as online/virtual training using a newly established VPN connection available throughout MNAO offices. To minimize project costs, MNAO was expected to seek to utilize, translate and customize for the local context as necessary, existing ISSAI and other training materials available from the IDI or other SAIs. Activity 5: Design and apply an updated private audit firm contracting process (USD 0.020 million at appraisal, USD 0.020 million actual) 18. The project provided technical assistance to establish an efficient and fully transparent and documented procurement process for contracting private audit firms by the MNAO. The MNAO’s practice at the time of engaging private audit firms to conduct the annual financial audits of budget entities and state-owned enterprises on behalf of MNAO was analyzed to improve documentation and transparency. Such efforts were timely and beneficial as, in addition to budget entities, IFI-funded projects had started to engage in a similar practice where MNAO carries out auditor selection for the projects but by following the IFIs’ own procurement procedures. Hence, the project proposed strengthening and streamlining MNAO’s private audit firm selection processes carried out for budget entities and IFI-funded projects. For these audits conducted by private audit firms, MNAO also applies its quality control and quality assurance framework. Page 8 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) II. OUTCOME Assessment of Achievement of Each Objective/Outcome 19. The assessment of the PDO has been mainly organized around the three PDO-level results indicators. The main data source for measuring the results indicators has been the project progress reports, which detailed out the achievement and progress status for each of the results indicators. As the set of results indicators for measuring the PDO achievement was quite straightforward, the data and information in the progress reports were sufficient to reach a conclusion that the project had achieved all the targeted results in its Results Framework. Results indicator 1. MNAO has made demonstrable progress in implementing the targeted improvements which emerge from the financial audit peer review 20. Upon receipt of the final peer review report prepared by UK NAO and quality reviewed by the INTOSAI IDI, the MNAO has developed a timebound plan of actions, with the responsible units/personnel identified, for implementing the key peer review recommendations (actions ranging from revisions to the legal framework to training staff in certain skills development). At completion of the project, implementation of this plan has been satisfactory with all of the recommended actions already taken by the relevant departments of the MNAO. Results indicator 2. MNAO auditors have been trained in the priority areas identified by the CPD strategy and action plans 21. Priority trainings under the newly approved Learning and Development Strategy and the Action Plan have been successfully delivered in the form of two regional trainings in 2017 and 2018, respectively. The first round of regional trainings covered 334 staff out of total 365 staff members (coverage rate of 91.5%) of the MNAO in 5 different geographical regions. The second set of trainings were delivered to 347 staff out of 365 total staff (coverage rate of 95.06%) through 10 regional trainings in 2018. The content of the regional training was duly informed by the findings of the peer review and, where appropriate tackled, tehcnical areas where weaknesses were identified e.g. the need to better understand and implement the concepts of materiality, sampling, considerations to fraud, etc. Attempts to deliver targeted and more tailored training were also undertaken with the support of the project e.g. orientation training for newly recruited staff, training of trainers, managers’ training, etc. Results indicator 3. MNAO has utilized a strengthened procedure for selection of private audit firms contracted to conduct audits of government entities 22. The MNAO has successfully carried out selection of audit firms for the FY2017 and FY2018 outsourced audits following the newly developed “Regulation on selection, contracting, and evalutaion of independent audit entities to conduct audit of financial statements, and monitoring the quality of performed auditsâ€? which was approved by the order No. A/191 of the Auditor-General (AG) of Mongolia on October 26, 2017. The MNAO deems that the new regulation allows for a more transparent selection process of private audit firms for contracting audit work and for a clearer evaluation and monitoring of performance quality of contracted audit work. When the MNAO Page 9 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) carries out the auditor selection for the FY19 audits of the Bank-financed project, the Bank may assess more closely the possiblity of using the above-mentioned regualtion for the IFI project auditor selection. Outcome assessment by Activity 23. The project activitites were aligned closely with the PDO. The project in its entirety and its individual activities that formed the core of positive advances in strengthening MNAO’s capacity were fully achieved, if not overachieved. Capacity Building for MNAO project has not only successfully achieved its development objective in strengthening MNAO’s capacity primarily in financial audit and audit firm contracting and oversight, it has also contributed to strengthening MNAO’s capacity in other types of audits through development and implementation of a comprehensive and systematic Learning and Development Strategy together with a time-bound Action Plan that cover learning and development objectives in all three audit areas within MNAO’s mandate: financial, performance and compliance. Initially when the project was being prepared, a Continuing Professional Development (CPD) Strategy and Program were envisioned to be developed to mainly provide specific guidance on developing necessary technical and soft skills and learning opportunities for primarily financial audit staff. However, the end result of the project in this regard has surpassed the original targeted intent and produced an even more comprehensive Learning and Development Strategy and Plan documents that covered not only the necessary knowledge and learning needs for finincial auditors but also for perofmance and compliance auditors, and addressed the MNAO’s skills and expertise, as an organization, needed to deliver on its strategic plan and to fulfill its responsibilities to improve the audit standards across the public and private sectors in Mongolia. Assessment of contribution by each project activity to the achievement of specific outcomes is highlighted in the following paragraphs: Activity 1. Peer review of MNAO’s financial audit activities 24. This activity has been satisfactorily completed as the MNAO peer review report by the UK NAO was finalized and quality reviewed by INTOSAI IDI that developed the SAI PMF methodology for reviewing performance of Supreme Audit Institutions. The final quality-assured peer review report delivered to the MNAO was quite comprehensive in nature and content. Upon receipt of the final report, the MNAO has developed a plan of actions for implementing the key and priority peer review recommendations and, at the project closing, the implementation of this plan has been found satisfactory with all of the recommended actions already taken by the relevant departments of the MNAO. 25. Based on the peer review findings, the financial audit procedures of the MNAO were revised and updated in accordance with INTOSAI’s ISSAI and were approved by Order No. A/63 of the AG. These financial audit procedures are being followed starting from 2019. In addition, according to the peer review recommendations, activities such as developing the toolkits for materiality, sampling and risk assessment were reflected the MNAO’s 2019 work plan and implementation has started. As intended, the peer review findings were duly considered for the Learning and Development Strategy developed by international consultants and subsequently approved by the AG, as well as for the contents of the regional trainings held in 2017 and 2018. Furthermore, the peer review identified six major issues for resolution at the institutional level. Those included development of the State auditing policy document, draft proposals for amending the Constitution of Mongolia and the State audit law of Mongolia (for enhanced independence), Quality assurance procedure, Quality control procedure, and a Stakeholder engagement strategy. Three out these six documents, that could be resolved within the mandate of the AG, have Page 10 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) been finalized and approved while the remaining three documents have been submitted to the Parliament for review and approval. Activity 2. Comprehensive training needs assessment of the new MNAO structure 26. The MNAO had successfully carried out the training needs assessments at various levels of the organization including central and local branches. Relevant surveys and focus group discussions took place in January 2017 and a final report on results of the training needs assessment was submitted to the MNOA management and shared with the Bank in March 2017. The findings of the training needs assessment, coupled with the peer review recommendations, have helped shape the Learning and Development Strategy and plans for the MNAO as well as the associated training efforts under the project that were delivered prior to the project closing. Activity 3. Design a comprehensive and sustainable continuing professional development strategy together with a detailed action plan 27. An international consulting firm from UK drafted a comprehensive set of Learning and Development documents including a strategy, a program and a detailed plan which were reviewed and commented by the management of MNAO and were subsequently finalized and approved on September 19, 2017 by the order No. A/164 of the AG. MNAO has found the exercise to be very useful as it helped MNAO better understand how learning and development process as a whole should be looked at from an institutional perspective. 28. Based on the newly adopted Learning and Development Strategy, 36 trainers/instructors were prepared from within the organization through a program for training trainers. Eight trained instructors led the regional trainings conducted in 2018. This way, trainings conducted in 2018 were proven to be more cost effective and higher in quality. Currently, the trained instructors continue to work in their local branch offices providing relevant training and advice. In addition, this strategy created the base to improve the SAI’s human resource training system. A five-year training plan (2018-2022) was approved by AG. The plan was closely aligned with the Learning and Development Strategy and sought an establishment of a Learning and Development Unit staffed with a newly appointed Learning and Development Manager who will be supported by the MNAO’s Training Expert in the Human Resources team as well as the Training Specialist in the Strategic Management Team. 29. Another key aspect of the Learning and Development Strategy was development and adoption of a Competency Framework, which establishes the skills the MNAO needs to meet its strategic objectives and to clarify the standards of performance expected by all staff at all levels in central and local branches. A draft of the framework has been developed with the support from international consultants. However, its finalization and adoption by MNAO is being delayed due to the newly adopted Law on Civil Service in Mongolia, effective as of January 2019. MNAO is currently awaiting completion of the corresponding regulations to the law pertaining to state organizations to ensure that the newly developed Competency Framework for MNAO is consistent with the new law. Activity 4. Development of customized training materials and delivery of the training as per the training program tailored to the capacity building needs identified by the peer review and training needs assessment and reflected on the continuing professional development strategy and program Page 11 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) 30. Trainings delivered under the project as part of the Learning and Development Strategy were critical in further strengthening the MNAO’s capacity. In accordance with the results of the needs assessment, numerous online and regional trainings were organized with the project financing. Necessary equipment was also purchased under the project for the MNAO as well as the 22 branch offices to enable online training facilities. Online meetings and on-the-job trainings among MNAO’s central and local audit offices became cost effective and time efficient through the use of the online training facilities. 31. Regional trainings, which covered significant thematic areas identified by the peer review, were successfully held with 91.5 percent participation from staff members in 2017 and 95.06 percent in 2018. Training materials were developed for the regional trainings covering specific topics such as planning financial audits, setting the materiality level during the audit fieldwork, enhancing auditors' responsibilities related to fraud in audit, conducting audit risk assessment, doing sampling, and evaluating results. 32. Attempts at providing tailored and fit-for-purpose training to the staff were initiated under the project as the first-ever orientation training for new staff were organized in 2018 with support of the project with 38 new staff enrolled. Traditionally, MNAO has been more accustomed to delivering the same training content despite the experiences and specific learning needs of the staff. 33. Furthermore, as recommended by the peer review, on-the job trainings by international consultants for financial and performance auditors were conducted in November/December 2019. International consultants carried out planning for 6 pilot audits together with the MNAO pilot audit teams. MNAO pilot audit teams were coached on the key principles and concepts in audit planning and its relationship to efficient and effective field work and quality audit reports. A training workshop was held for the MNAO auditors at the end of the assignment focused on areas of technical weaknesses identified during the pilot audits e.g. while the ISSAI 3000 identifies 3 different approaches to performance audits, the MNAO uses only risk-oriented approach. Therefore, the performance audit consultant, who led the on-the-job training, coached the MNAO auditors and together they planned a pilot audit using the problem-oriented approach. Subsequently, the MNAO continues to work on developing in more detail this new approach. Pilot audit team members and the trained auditors are implementing the knowledge acquired from on-the-job trainings in the audits they are conducting and sharing their knowledge with other peer auditors. Activity 5. Design and apply an updated private audit firm contracting process 34. With support from a local consultant, the MNAO developed a “Regulation on selection, contracting, and evaluation of independent audit entities to conduct audit of financial statements, and monitoring the quality of performed auditsâ€? and obtained approval of the AG by the order No. A/191 on October 26, 2017. The MNAO has since successfully carried out selection of audit firms for the FY2017 and FY2018 outsourced audits following this new regulation. To help monitor performance of the outsourced audits, under the project the MNAO also had contracted ICT Group, a local consulting company, to develop a simple software for evaluating the work carried out by the private audit firms in accordance with the approved regulation. The software development has been completed and the MNAO now plans to incorporate its use in the evaluation process in practice. Page 12 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Overall Outcome Rating 35. Based on the detailed assessment by activity, the project’s relevance has been rated High given the project’s full alignment with the priorities in the World Bank Country Partnership Strategy for Mongolia for FY2013-2017 both at the appraisal and completion stages. Its efficacy has been rated Substantial considering its satisfactory achievement of the PDO and efficiency has been rated Substantial for its profitability and long-term significant benefits. As such, the overall outcome rating has been assessed as Satisfactory for the project. Other Outcomes and Impacts Mobilizing MNAO’s Resources 36. The project implementation was managed by full-time MNAO staff, mainly from its Strategic Planning Department. By leveraging its own staff for the project implementation, the MNAO was able to spend the grant proceeds entirely for planned project activities, without having to hire external consultants for implementation/operational support. This arrangement not only ensured ownership for the project leading to timely achievement of targets set by project activities but also helped the relevant MNAO staff gain experience and capacity in project management and implementation of Bank-financed project. Adopted Best Practices 37. Preparation of the Learning and Development Strategy for the MNAO drew heavily from lessons and best practices of other countries’ SAIs. The Learning and Development Strategy, adopted by the MNAO, was based on feedback from the SAIs of UK, Netherlands and New Zealand with useful insights from feedback provided by SAIs of China, Japan, Denmark and Uganda. A key insight drawn from these countries was that their SAIs are developing their continuing professional development strategy within the context of their organizational strategy that determines the future skills, knowledge and behaviors required to meet its objectives. Another was that SAIs are increasingly linking the CPD with their broader human resource management strategies so not to have it as a stand-alone CPD strategy but rather produce learning and development strategies that are linked with all aspects of support provided to employees from entrance until final departure from that organization and include matters relating to team development, management and leadership development and organizational change. Therefore, what was envisioned to be a CPD strategy development for the MNAO during project preparation, ultimately, turned into a Learning and Development Strategy due to the lessons and best practices drawn from others. Other Unintended Outcomes and Impacts Not applicable. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 38. Setting priorities right. Equitable and transparent management of public revenues and expenditures has been a priority area of the Bank’s engagement in Mongolia, a resource-rich country with rapidly rising government revenues from its mineral exports. A strong public sector audit can increase the effective, Page 13 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) efficient and economic use of public resources. Under Pillar 1 of the 2013-2017 CPS, the Bank reiterates its commitment to improved implementation of planning, budgeting, financial reporting and auditing policies of the government services to the public, especially the poor. The Capacity Building for MNAO project’s aim of further strengthening public sector auditing in the country was in full alignment with the CPS and was also fully consistent with MNAO’s own Strategic Plan that was developed in coordination with the INTOSAI. 39. Results Framework and M&E. Overall, the Results Framework was sound with PDO indicators aligned with the operational objectives. All the intermediate results indicators were aptly set up to capture the real achievement of each of the planned activities, except for the indicator “Materials developed for 50% of the courses in the CPD action planâ€?, which was proven in practice a bit difficult to measure because training materials for courses to be delivered over a three-year training program could not all be developed in advance but rather be prepared as part of planning for the actual training event/course to ensure up-to-date information is incorporated into the training materials. This approach was found suitable in practice and was used appropriately. The project’s M&E framework was largely designed adequately with clear and measurable baselines and targets. Methodologies, frequencies, and responsibilities for data collection to evaluate the achievement of the PDO indicators were clearly defined and followed through during implementation. 40. Adequacy of risk and mitigation measures identification. Critical risks as to project implementation capacity and sustainability were adequately identified and the corresponding mitigation measures were integrated into the project design. The overall project risk was rated Moderate after mitigation measures were identified for all risks. B. KEY FACTORS DURING IMPLEMENTATION 41. Dedication and professionalism of the MNAO staff contributed to successful project implementation. Although Bank-financed project management was new to many of the staff involved in the project implementation, the responsible MNAO staff were sufficiently qualified and knew well the intrinsic issues involved in the project activities and, therefore, were able to successfully carry out the project activities. In addition, the Director of the Strategic Planning Department of the MNAO who oversaw the overall project implementation provided strong leadership and guidance throughout the project implementation period which positively contributed to successful completion of the project. 42. Adequate financial management and M&E systems were in place during the project implementation. The MNAO staff responsible for the project made efforts to ensure that all project interventions were consistent with the fulfillment of the PDO. Throughout the project implementation, the M&E system proved to be comprehensive and thorough, thus the project progress reports and other necessary supporting information and documents were provided timely to the Bank. 43. Significant attention given to the project by the MNAO management has contributed to successful implementation of the project. MNAO management was quick to resolve issues to keep the project moving and the project plans and implementation progress were reviewed by the Project Steering Committee semi-annually. Page 14 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) 44. The overall responsibility for project monitoring and evaluation, as well as reporting rested with the MNAO. The day-to-day duties were carried out by an MNAO staff from the Strategic Planning Department who was responsible for the project M&E with the necessary guidance from the Bank’s task team. Implementation progress and performance were assessed in line with the indicators identified in the Results Framework and were supplemented by detailed project progress reports and financial management reports in a format agreed with the Bank. Responsible MNAO staff coordinated all project evaluation activities based on the performance indicators included in the Results Framework and carried out own assessments at various stages of project implementation and consolidated the results in the semi- annual progress report. A final completion report covering the latest progress of project activities and results achieved was provided to the Bank at project completion. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 45. Environmental and social safeguards. There were no environmental or social safeguard policies triggered for the project. 46. Financial management. In compliance with the Bank’s fiduciary requirements, the required interim financial reporting and annual audit reports were submitted to the Bank largely on time and the project audits were unqualified. Supervision missions regularly reviewed the project accounts and procedures to ensure compliance with the fiduciary requirements. Financial management assessment was carried out during preparation and concluded that project’s financial management arrangements were adequate. As part of the project implementation support missions, FM supervisions were conducted to ensure fiduciary requirements were being complied with at all levels. 47. Procurement. Procurement activities were mainly carried out by the MNAO staff with assistance from the Bank when needed. Individual contracts established under the project were relatively small in value. The key risk associated with the project procurement was inexperience of the MNAO staff handling the project procurement work and their unfamiliarity with the Bank’s procurement procedures. Despite the initial concern, given the project was light on its procurement needs mostly with small value consulting contracts, there no significant procurement issued encountered during the project implementation. C. BANK PERFORMANCE Quality at Entry 48. The project design was informed by the MNAO’s own Strategic Plan, which was developed in coordination with the INTOSAI and focused on the needs that were deemed to be of paramount importance to the MNAO in building its capacity. MNAO’s leadership considered human resource Page 15 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) development to be at the core of its strategy; thus a critical goal of its Strategic Plan was to provide MNAO personnel with the requisite professional skills to conduct their duties effectively and efficiently. During preparation, the Bank team worked closely with the MNAO team to delve into identifying core activities in support of building its capacity further and mutually agreed to collaborate on developing a comprehensive CPD strategy and plan. In that respect, the project design appropriately captured MNAO’s needs and demonstrated good collaboration between the Bank and the MNAO teams. Quality of Supervision 49. The Bank supervision missions had been fielded regularly with a stable team. The task team was quick in identifying and addressing key implementation issues such as possible delays in certain areas and provided necessary support to keep the project moving forward. Having a country office-based co-TTL for the project made it easier for the MNAO to have quick and efficient access to the necessary guidance and advice timely. As most of the project activities were straightforward and were implemented rather smoothly, the progress towards achievement of PDO and overall implementation progress ramained Satisfactory throughout the project implementation period. 50. Overall, the Bank performance is rated as Satisfactory based on (a) satisfactory quality entry for the appropriate project design and implementation arrangement; (b) overall good quality supervision to address implementation and sustainability issues; and (c) fair and consistent ratings of implementation status. D. RISK TO DEVELOPMENT OUTCOME 51. The risk to development outcome is rated Moderate. Adoption of the Learning and Development Strategy by the MNAO and continuous implementation of the Learning and Development Plan to further materialize the direction identified in the strategy will require the MNAO to have sizeable budget allocations beyond the lifetime of the project. There is a risk that the allocated budget will not be sufficient to carry on the work after the project completion. Currently, this risk will be addressed, to a certain extent, owing to the support the MNAO is expecting from another Technical Assistance Project recently approved by the Asian Development Bank (ADB). Through implementation of the ADB project, there is an opportunity for the MNAO to sustain the momentum in implementing the Learning and Development Strategy and Plan developed under the Bank financed project. Furthermore, the EU-funded Strengthening Governance in Mongolia Project of the Bank is also providing support to the MNAO in advancing its public outreach agenda to help strengthen the institution’s engagement with the public. V. LESSONS LEARNED AND RECOMMENDATIONS 52. The following lessons have been drawn from the implementation of the MNAO capacity building project: (a) The impact on the MNAO of a well-developed Learning and Development Strategy document prepared by experienced international consultants with previous experiences of working for a SAI was Page 16 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) significant. It allowed the MNAO to better understand the breadth and depth of what is needed to achieve the skills and expertise required to deliver on its strategic plan and to fulfill its responsibilities to improve the audit standards across the public and private sectors in Mongolia. The Learning and Development Strategy was informed by international best practices used by SAIs in other developed countries. (b) The fact that MNAO staff were in charge of project management and implementation created opportunities for sustaining and building on the project achievements. The strategic leadership and ownership of the project within the MNAO has been a key aspect for ensuring stability in the operating environment of the project. (c) Language barrier was a challenge that had to be overcome during the project implementation. Many of the MNAO staff lacked English language skills and, therefore, had to rely on the few staff who have sufficient English language capabilities needed for managing and engaging in the project implementation, in particular for working with international consultants. Also, the MNAO accountant did not speak English and had to rely on English speaking colleagues when preparing the Interim Financial Reports, handling disbursement-related correspondence with the Bank, and communicating officially with the Bank task team. This placed undue burden on the few English speaking MNAO staff who were working with full work programs of their own on the side of the project. Hence, other interventions in the future could either consider support to enhancing the MNAO staff’s English language proficiency and/or providing the necessary translation support as needed. . Page 17 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The Project Development Objective (PDO) is to strengthen the Mongolian National Audit Office ("MNAO")'s capacity, primarily in financial audit and audit firm contracting and oversight. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion MNAO has made demonstrable Percentage 0.00 50.00 50.00 100.00 progress in implementing the targeted improvements which 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 emerge from the financial audit peer review Comments (achievements against targets): MNAO developed a dedicated plan of actions for implementing the key priority peer review recommendations. As per the latest progress report received from the MNAO as at December 18, 2018, 100% of the activities on this action plan had been implemented. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion MNAO auditors have been Percentage 0.00 50.00 50.00 95.06 trained in the priority areas identified by the CPD strategy 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 Page 18 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) and action plans Comments (achievements against targets): By project closing, two rounds of regional trainings had been carried out based on the MNAO's Learning and Development Strategy and Plan. The second round of regional trainings were organized in 10 locations and covered 95.06% of all state auditors in Mongolia. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion MNAO has utilized a Yes/No N Y Y Y strengthened procedure for selection of private audit firms 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 contracted to conduct audits of government entities Comments (achievements against targets): "Regulation on selection, contracting, and evaluation of independent audit entities to conduct audit of financial statements and monitoring the quality of performed auditsâ€? was developed and approved on October 26, 2017 by ord er no. A/191 of Auditor-General of Mongolia. The procedures stated in the said regulation have been followed by the MNAO for two consecutive years' outsourced auditor selections since its approval. A.2 Intermediate Results Indicators Component: Peer review of MNAO’s financial audit activities Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion A peer review report including Yes/No N Y Y Y Page 19 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) recommendations for 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 improvement of MNAO’s financial audit capabilities submitted Comments (achievements against targets): The United Kingdom (UK) National Audit Office has conducted a peer review of the MNAO in 2016. The final peer review report which was quality reviewed by INTOSAI International Development Initiative (IDI) was delivered to MNAO on July 15, 2017. Component: Comprehensive training needs assessment of the new MNAO structure Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Report on MNAO training Yes/No N Y Y Y needs assessment 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): A comprehensive MNAO knowledge and training needs assessment was conducted by local consultants and the report was finalized in March 2017. Component: Design a comprehensive and sustainable continuing professional development strategy together with a detailed action plan Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion CPD strategy and an action Yes/No N Y Y Y plan have been developed and adopted 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 Page 20 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Comments (achievements against targets): The Learning and Development Strategy, Program and Plan were developed by international consultants (with prior work experience at the UK NAO) and approved on September 19, 2017 by order no. A/164 of Auditor-General of Mongolia. Component: Development of customized training materials and delivery of the training as per the training program tailored to the capacity building needs identified by the peer review and training needs assessmen Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Materials developed for 50% of Percentage 0.00 50.00 50.00 50.00 the courses included in the CPD action plan 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Training materials for courses to be delivered over a three-year training program could not all be developed in advance but rather be prepared as part of planning for the actual training event/course to ensure up-to-date information is incorporated into the training materials. This approach was found suitable in practice and was used appropriately; hence the result indicator is deemed to have met the target. Component: Design and apply an updated private audit firm contracting process Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Selection procedure for private Yes/No N Y Y Y audit firms that would perform contracted audits developed 11-Jan-2016 31-Aug-2018 31-Dec-2018 31-Dec-2018 Page 21 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) Comments (achievements against targets): Selection procedure for contracting private audit firms was developed and formalized through the "Regulation on selection, contracting and evaluation of independent audit entities to conduct audit of financial statements and monitoring the quality of performed audits" which was approved on October 26, 2017 by the Auditor-General of Mongolia. Page 22 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 – Strengthened MNAO capacity 1. MNAO has made demonstrable progress in implementing the targeted improvements which emerge from the financial audit peer review 2. MNAO auditors have been trained in the priority areas identified by Outcome Indicators the CPD strategy and action plans 3. MNAO has utilized a strengthened procedure for selection of private audit firms contracted to conduct audits of government entities 1. A peer review report including recommendation for improving of MNAO’s financial audit capabilities submitted 2. Report on MNAO training needs assessment 3. CPD strategy and an action plan have been developed and adopted Intermediate Results Indicators 4. Materials developed for 50% of the courses included in the CPD action plan 5. Selection procedure for private audit firms that would perform contracted audits developed 1. Peer review of MNAO’s financial audit activities was conducted by UK National audit office 2. Comprehensive MNAO knowledge and learning needs assessment was carried out covering the entire organization, including its local Key Outputs by Component branch office staff. (linked to the achievement of the Objective/Outcome 1) 3. Comprehensive and sustainable continuing professional development strategy together with a detailed action plan was developed. 4. Customized training materials were developed. Training as per the Page 23 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) training program tailored to the capacity building needs identified by the peer review and training needs assessment and reflected on the continuing professional development strategy and program was delivered. 5. New regulation for private audit firms contracting process was developed. Software for evaluation of work process of private audit firms was developed. Page 24 of 25 The World Bank Capacity Building for MNAO through Establishing a Countinuing Professional Development Strategy and Program (P155305) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Conduct a peer review of MNAO's financial audit 0.070 0.065 92.8% activities Carry out a comprehensive knowledge and learning 0.040 0.040 100% needs assessment of MNAO Develop a comprehensive, needs-driven and sustainable continuing professional 0.040 0.042 105% development (CPD) strategy together with a detailed action plan for three years Develop customized training materials tailored to MNAO's capacity building needs 0.235 0.236 100.4% described in the CPD strategy and the knowledge and learning needs assessment Technical assistance to establish an efficient and fully transparent and 0.020 0.020 100% documented process for contracting private audit firms Total 0.405 0.403 99.5% Page 25 of 25