Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) Report Number : ICRR0020224 1. Project Data Project ID Project Name P087347 MZ Tech & Voc Edu & Training (FY06) Country Practice Area(Lead) Additional Financing Mozambique Education P125283,P125283 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-41560,IDA-49630,TF-56866 31-Oct-2011 74,500,000.00 Bank Approval Date Closing Date (Actual) 21-Mar-2006 30-Sep-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 67,000,000.00 0.00 Revised Commitment 66,980,624.07 0.00 Actual 67,106,711.34 0.00 Sector(s) Vocational training(53%):Central Government(47%) Theme(s) Education for the knowledge economy(29%):Improving labor markets(29%):Decentralization(14%):Gender(14%):HIV/AIDS(14%) Prepared by Reviewed by ICR Review Coordinator Group Susan Ann Caceres Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The Lending Agreement (p. 20) stated the project objective as "to support the Borrower's technical and vocational education system to respond more effectively and equitably to labor market needs, using a framework based on occupational standards and being sensitive to gender and HIV/AIDS issues." The appraisal document stated the objective differently, as "to facilitate the transition of the existing technical and vocational education and training (TVET) system to a demand-led training system and provide beneficiaries with more market-relevant skills and improved economic opportunities." This review utilizes the version noted within the lending agreement, as per IEG/OPCS guidelines for differences between documents in the statement of objectives. During a the second project restructuring in 2011, the project development objectives were revised: "to improve the quality and relevance of technical and vocational training in Mozambique, with a focus on selected key TVET institutions and programs." This statement was noted in Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) both the additional financing legal agreement and project paper. Consistent with harmonized criteria, this review will assess outcomes under the original and revised objectives over the course of the operation and weigh based on disbursement before and after project restructuring. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 06-Jun-2011 c. Components There were four components (within each, gender and HIV/AIDS prevention were integrated into activities): • Development of an Institutional Framework (appraisal US$6.0 million; actual, US$17.5 million) was to reform the governance and management structure for the TVET system. It was to support the development of an institutional framework, capacity building to establish labor market monitoring, and decentralized arrangements for management of TVET institutions. Studies were undertaken to facilitate the transition to a demand-led TVET system. The aim was to garner active participation of industry and firms in the management of TVET institutions in relation to the development of programs. It also supported an Executive Secretariat that coordinated the implementation of the training reform program, as well as sensitized relevant stakeholders regularly to the program. • Standard-based Qualifications and Training System (appraisal US$2 million; actual, US$9.1 million ) was to develop occupational competency standards in at least 20 occupational areas in four sectors (i.e. tourism and hospitality, management, industrial maintenance, and agro-industry). The standards were then to be used to develop training program, assessment, and certification systems. • Quality Improvements in Training Institutions (appraisal US$19.1 million; actual, US$38.1 million) was to strengthen the National Directorate for Vocational and Technical Education and National Institute for Employment and Training via inputs such as: office equipment; staff training; rehabilitation of offices; new learning materials in four pilot sectors; and internships in industry. Sixteen institutions were to pilot the new competencies. • Skills Development Fund (FUNDEC) (appraisal US$ 6.4 million; actual, US$5.2 million ) was to fund grants to be awarded competitively, not limited to pilot sectors. Grants were to promote quality of training and increase access for out-of-school youth and adult population in rural areas. Grants were to be earmarked for capacity building and training activities of public providers (not part of the pilot) and private providers. The grants were also meant to respond to underserved groups located in rural areas to improve their livelihood opportunities. The additional financing provided resources to each of the components, but also eliminated support for the National Training Authority, National Directorate for Vocational and Technical Education, and National Institute for Employment and Training, as the Government’s own resources were later designated to support these entities. Two additional competency standards (information and communication technology (ICT) and mining) were added, as well as additional rounds of competitive grants. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs: Total project costs were US$ 73.5 million. The IDA grant fully disbursed. Actual costs were higher than revised project allocations for the Institutional Framework and Standards-based Qualification Training System components. Quality Improvement in Training Institutions was consistent with the revised allocation, while Skills Development Fund (FUNDEC) disbursed approximately half (46%) of the revised amount. The ICR indicated that costs for rehabilitating TVET institutions were higher than anticipated due to price inflation experienced in the region due to the World Cup games (ICR, p. 23). The ICR did not explain why the actual costs were higher for Institutional Framework and Standards-based Qualification Training System even with the revised allocation and some planned activities not actually financed by the project. The ICR also did not explain why FUNDEC did not fully utilize the reallocated amount, despite more phases of grants. Financing: This operation was financed by an IDA grant of US$30 million and additional financing of US$ 37 million, as well as a Dutch Trust Fund of US$6.57 million. The Trust Fund was lowered from its planned amount of US$7.35 million because of budget cuts due to the economic crisis. A grant was provided from the Government of Japan (Population and Human Resources Development PHRD) in the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) amount of US$695,000 for preparation. A Norwegian Education Trust Fund grant in the amount of US$505,000 provided technical advisory services and consultants. Borrower Contributions: The Borrower made no contributions, nor were any planned. Dates: In the June 6, 2011 level one restructuring the objective and key performance indicators were revised, and some activities were scaled back. The objective was simplified to more accurately reflect the intended outcome of the project. Additional financing was given to fill a resource gap and the project was extended to complete project activities from September 30, 2011 to September 30, 2014. The project was subsequently extended on June 27, 2014 from September 30, 2014 to September 30, 2015 (ICR p. 9). The project operated nearly four years longer than initially planned (i.e. 47 months). 3. Relevance of Objectives & Design a. Relevance of Objectives Original Objective: While primary schools had reached near universal access at appraisal, many students were not reaching seventh grade and even fewer entered and completed high school. Thus, the country’s workforce was poorly educated and lacked technical skills. There was a small formal employment sector, but it was growing and demanding more skilled labor. In recognition of this need, the Government launched the Integrated Program for Training Reform meant to transform the quality and relevance of technical and vocational education and training in close connection with industry to more effectively respond to the needs of the labor market. Key challenges within the technical and vocational education and training included: lack of involvement from employers and industry in defining occupational standards and learning content; low quality; underqualified and inadequately skilled instructors; shortage of teaching aids and workshops; shortage of places due to restricted opportunities in technical schools; fragmented governance of TVET; and resource constraints. As a result, students entered TVET as a second-best option. The system was inefficient, as an estimated 45% of students (in some levels) repeated, and drop-out rates ranged between 15-20%. The objectives of this operation substantially aligned with the Government’s priorities and policy to reform the TVET system to make it labor market-oriented, but they were awkwardly worded and difficult to interpret. They are consistent with the Government’s Action Plan for Reduction of Poverty that emphasized skill development of citizens, as well as the Bank’s Country Assistance Strategy (FY04-FY07; FY08- FY11) at preparation and closure (FY12-FY15), which prioritized support for TVET and post-primary education to produce a better educated and skilled workforce. The objectives were also consistent with the aims noted in the Employment and Vocational Training Strategy (2004- 2010) and the Education Sector Strategic Program (2005-2009). Thus, the objectives were substantially relevant throughout the life of the operation. Revised Objective: The revised objectives (i.e. improve the quality and relevance of technical and vocational training) are clearer and narrower in scope than the original objectives, but still address the previously noted imperatives to transform TVET to more effectively respond to the needs of the labor market. The revised objectives are clearer from an evaluation perspective in comparison to "respond more effectively and equitably to labor market needs" and thus easier to evaluate. The ICR reported that the revised objectives "more accurately reflected the intended outcome of the project". The revised objectives substantially align with Government and Bank’s strategies over the course of the operation. Rating Revised Rating Substantial Substantial b. Relevance of Design Original Design: The theory of change behind the reform of the TVET system was to use occupational standards to make the training system more labor market-oriented and demand-driven. Design comprehensively addressed the TVET system. The three-level implementation structure was used to advance and support the Government reform. An institutional framework and financing mechanism were created to set up the governance of the system. A competitive fund was expected to encourage other demand-led changes and address access and equity. The operation predominantly focused on public institutions, but private institutions were also included. The demands from formal sector employers predominated, despite the fact that it is a small segment in comparison to the informal sector. The needs of the informal sector were expected to be satisfied via a Commission (an Executive Body composed of 12 members from government and private sector-including trade unions and other organizations) and grants (window supporting professional training for associations). Gender and HIV/AIDS prevention were included in each qualification and curriculum, but no outputs or outcomes were tracked in relation to these two aspects, except for female Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) project beneficiaries. These activities adequately addressed the objective to more effectively respond to labor market needs, as the system was trying to be more demand-oriented, but there was inadequate attention to activities to equitably respond to labor market needs. A subset of grants was meant to respond to underserved groups located in rural areas to improve their livelihood opportunities, but this was not sufficient to attain the objective. The reform was intended to be phased with this operation piloting reforms that would be scaled up in other occupations and institutions in the second phase, and in the final phase extend to all TVET institutions. However, the scope and time to accomplish the first phase were ambitious. Revised Design: The activities were largely unchanged in the restructuring, thus, what was previously noted remained applicable as the activities to support the TVET system and making the system respond more effectively to labor market needs aligned with improving both relevance and quality. The focus on occupational standards and curriculum and assessment related to these standards addressed both objectives. Rating Revised Rating Modest Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To respond more effectively to labor market needs using a framework based on occupational standards and being sensitive to gender and HIV/AIDS issues. Rationale During the course of the operation, several donors (i.e. Denmark, Spain, and Norway) also financed support for TVET (estimated US44.5 million ICR p. 4). Germany International Cooperation provided substantial technical assistance for project implementation to the implementing agency, training of teachers, and provision of school-based advisers. Thus, the attainment noted below cannot be exclusively attributed to the financing of the World Bank. Outputs: An institutional and regulatory framework was established in collaboration with industry representatives to ensure responsiveness to labor market needs. The framework created the financing and governance for the TVET system. • A national regulatory body (National Training Authority) was established (ANEP). The creation of the body was informed by a study and stakeholder consultation in relation to governance, management, and financing of the TVET system. • A national TVET Law was created, submitted to National Assembly, and enacted in March 2016. The law establishes a National Training Fund, with resources contributed from both public and private sector. Private sector contributions come from a payroll levy system. The law also created ANEP, which is the regulatory body for TVET. • Commission for Professional Education and Training Reform Board (COREP) became functional. Private sector representatives are included in the Board. The Board conducted a marketing campaign for secondary students to raise awareness of TVET programs. • Standard-based assessment and certification system was developed with representatives from the private sector, civil society, and government based on a qualification framework. Guides and manuals supporting the qualification system were developed. Curricula were developed for the areas of industrial maintenance, hospitality and tourism, agriculture and agro-business, and administration and management, and were piloted in five schools. Approximately 25 students per sector completed the course. • Strategic plans were developed in all pilot training institutions. Revenue generated by all targeted institutions was retained within all the institutions. All targeted institutions participated in tracer studies. • 67 competency standards were developed and approved by COREP, which is higher than planned (28). All of the programs were designed with private sector participation. In addition to the targeted sectors, occupational standards were also completed for the gas hydrocarbons sector. 25 partnerships were formed between training providers and private institutions. Representatives from the private sector had an advisory role in relation to the development of occupational standards and internships (ICR p. 44). 5500 students (or half of them) were provided with internships, which exceeded the target (3300 students or 30% of students). • Labor Market Observatory was established. One labor market report and two bulletins were produced (the target was annually per sector). 9 technicians from the Ministry of Labor, Employment and Social Security were trained in the Labor Market Observatory. 58 staff have been trained in the labor market information system. Labor Market Observatory was expected to be expanded to one or two provinces Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) to identify specific labor market needs for the province. Five sector profiles were conducted (i.e. Agriculture, Fisheries, Construction, Energy and Mining). Priority sectors such as tourism and agriculture received project focus as they are important for economic growth and development in Mozambique (ICR p. 32). Outcomes: • 80% of employers were satisfied with the quality of the new competency based training (CBT) program, which exceeded the target (60%). However, this does not tell us whether employers were satisfied with the skills of employees participating in the competency program. • 57% of graduates (in targeted programs) found a job (or created a related job) within 6 months of graduation, which is slightly lower than the target of 60%. • Graduates of CBT courses had higher earnings than non-CBT graduates (ICR p. 28). Around 31% of CBT graduates earn over 10,000 MT per month while only 23% of non-CBT graduates earn this level of income. • TVET graduates (beyond CBT programs) earned 10% more than general secondary education students (ICR p. 28). Rating Substantial Revised Objective To improve the relevance of technical and vocational training in selected key TVET institutions and programs. Revised Rationale Note: During restructuring, the previous statement of objective was revised to improve the relevance of TVET. The revised objective statement in this review is interpreted as equivalent to the original objective statement. The logic is that responding effectively to labor market needs is equivalent to improving relevance of education because relevance is examined in relation to labor market needs. Thus, the same evidence and rating apply to both the original and revised objective. Outputs and outcomes are the same as noted under the original objective 1 above. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective To respond more equitably to labor market needs using a framework based on occupational standards and being sensitive to gender and HIV/AIDS issues. Rationale Outputs: • 189 grants were executed and 126 institutions benefitted from the grants (ICR p. PDO indicator chart) with approximately 30,000 beneficiaries (ICR p. 26). An evaluation of the Fund found improved competencies and improved livelihoods, but the ICR (p. 26) did not Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) report the data. Outcome: There has been an increase in the percentage of females graduating from TVET institutions from 22% in 2009 to 38% in 2015. The ICR (p. 47) reported efforts were made to make a more gender equitable TVET system through interventions such as: free participation in entry examination for women, marketing materials for women, and uniform subsidies for women, but it did not state how widespread they were applied across TVET institutions. The female pass rate of CBT course was 85% and the male pass rate was 84%, which exceeded the target (65%). Pass rates in non-CBT courses were not provided. The ICR also reported that 42% of project beneficiaries were women, which exceeded the target of 38%. The aspect of gender equality in TVET enrollment is important consider that the rate of female labor market participation is higher than male (WB Country Partnership Strategy FY12-15). Rating Substantial Revised Objective Improve quality in selected institutions and programs Revised Rationale Outputs: • 9 classrooms were rehabilitated and equipped, which is lower than what was planned (10). However, the ICR noted that subsequent to project closing the final classroom was completed with government resources. • 2,027 teachers received training related to the new CBT system. Since the total number of teachers trained exceeded the target (450), the total number of females trained also is higher than the target, but the proportion is lower than expected (11% in comparison to the target of 15%). 162 school managers (in institutions implementing the new CBT) received training, which is higher than the target (60). Prior to the training, a capacity building study was conducted, which informed the content in the training program for teachers and managers. Topics included developing a management strategy and utilizing the information management system. • 25 TVET institutions utilized the competency-based testing (CBT) system, which met the revised target. Non-pilot institutions in Chokwe, Nacuxa, and Boane also utilized the agriculture curriculum (ICR p. 48). The CBT system now recognizes students' prior learning wherever it took place. • A task force was established to learn from the pilot experience and make further improvements in the CBT process (ICR p. 48). • 11,413 TVET students were assessed using the new competency-based training, which exceeded the target (1300). • A strategy for teacher training was implemented with a new certification system containing three levels: (1) managers/coordinators/lead teachers; (2) TVET teachers at the pre-secondary and secondary level; and (3) TVET instructors in professional training centers. • A management information system was developed. All the targeted institutions utilized the MIS to submit enrollment data and grades. Outcomes: The ICR provided information about attainment of outputs, but did not contain data that demonstrated improved quality in targeted TVET institutions and programs. Revised Rating Modest 5. Efficiency The project's economic rationale in the appraisal document and the ICR (p. 28) was the high returns to education in Mozambique. Every year of Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) education is associated with 12.4% higher earnings among salaried employees. There were implementation efficiencies in the project. For example, FUNDEC grants had a high execution rate, as 73% of allocated grants were fully executed. Also, institutions that were awarded grants (in approximately 68% of cases) also provided their own financial resources to augment the projects grants, scale up the activities, and make the activities more sustainable. However, there were also several inefficiencies in the project. Initial delays (in enacting institutional reforms, recruiting a technical firm to develop the operational standards, and building civil works) negatively impacted implementation and created the need for multiple extensions. The project operated nearly four years longer than initially planned (i.e. 47 months). Additional financing nearly doubled the amount of money devoted to the civil works component (US$40 million instead of US$23 million), yet the number of institutions rehabilitated decreased from 12 to 10, due to inflation in the price of construction materials (and underestimation of the initial amount needed). The ICR did not explain why the actual costs were higher for Institutional Framework and Standards-based Qualification Training System even with the revised allocation and some planned activities not financed by the project. The ICR also did not explain why FUNDEC did not fully utilize the reallocated amount, despite more phases of grants. The ICR did not provide evidence of what was actually attained with the grants or whether the grants were used efficiently, reporting only that that the grants adhered to the planned expenditure categories. Considering all these factors, the efficiency of the project is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under original objectives Under revised objectives Relevance Relevance of objectives: Substantial Relevance of objectives: Substantial Relevance of design: Modest Relevance of design: Substantial Efficacy Original Objective 1: Substantial Revised Objective 1: Substantial Original Objective 2: Substantial Objective 2: Modest Efficiency Modest Modest Outcome rating Moderately Satisfactory (4) Moderately Satisfactory (4) Outcome under the original objectives is rated Moderately Satisfactory, based on substantial relevance of objectives and modest relevance of design, substantial achievement of original objectives 1 and 2, and modest efficiency. Outcome under the revised objectives is rated Moderately Satisfactory, based on substantial relevance (of objectives and design), substantial achievement of original objectives 1 and 2, and modest efficiency. The overall project outcome is based on the average of these ratings, weighted by the amount of the total loan disbursed before (42%) and after (58%) restructuring, as follows: Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) Outcome, original objectives = Moderately Satisfactory (4) x .42 = 1.68 Outcome, revised objectives = Moderately Satisfactory (4) x .58 = 2.32 Total: 4 or Moderately Satisfactory a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Despite the gains made in the project, there are still some risks to the development outcome. First, while the supply of TVET instructors increased during the project, there is still need for more of them to meet the demand within the sub-sector. The CBT have been piloted, but more resources are needed to scale up. While institutions are able to retain their generated income, more resources are needed to maintain workshop training. Responsibility for TVET has changed from the Ministry of Education to the Ministry of Science, Technology, Higher Education and Technical Training. The impact that this will have in moving the reforms forward is unclear. However, the advances made during the project helped transform the TVET system. Capacity has been built in key TVET organizations and a sustainable financing mechanism has been developed. Private sector participation is part of the system. With the new law, TVET institutions have greater autonomy; they are required to have Boards. Further efforts are planned on the framework. With the Government’s consent, US$2 million in funds will be allocated to sustain the work of ANEP (i.e. partnerships with private sector, decentralized institutional management, accreditation, and further work on the framework) from the additional financing to the Bank-financed Higher Education Science and Technology project. The Government has reiterated its commitment and intent to provide adequate resources to the TVET subsector. With the recently enacted law, the Government and private sector have agreed to both put forth resources for training. Likewise, the government budget allocation has risen between 2009 and 2015 (178%). Several donors such as German International Cooperation, Italy, Norway, Canada, UK Department for International Development, and Korea continue supporting the TVET subsector and are expected to support the Government’s Reform Program. All of these factors help to minimize the risk to the development outcome, thus the risk to the development outcome is rated modest. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry Preparation was thorough and not rushed, taking 18 months for the whole process. It benefitted from a grant from the Government of Japan Population and Human Resources Development of US$695,000. The Bank worked closely with other partners who were involved in supporting TVET in the country. Preparation was informed by analytical work (i.e. Skills Development in Mozambique, Expenditure Review, employer surveys, etc.) and lessons from TVET reforms in other country (such as Chile, Mexico, Zambia, and Ethiopia). The Bank had funded previous education sector operations in Mozambique and so was able to learn from those experiences. Several reviews were held during preparation (i.e. Concept and Appraisal Review meetings and Quality Enhancement Review). The suggestions reinforced the need to ensure employer involvement in planning, design, and governance of the TVET sector and ensure autonomy to TVET institutions in decision-making. Despite a solid design of the reform, there was an inaccurate assessment of the time to complete curriculum, develop the organizing body, develop baseline indicators, enact legislation, and implement TVET governance. Project costs were also underestimated. While the team identified the risks and attempted to mitigate them, preparation did not fully understand the implementation realities. There were also weaknesses in the results chain. Quality-at-Entry Rating Moderately Unsatisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) b. Quality of supervision A strong working relationship was developed between the Bank team and the Government, which was facilitated by the fact that some were located in Maputo. A multidisciplinary team (e.g. financial and procurement, construction, monitoring/evaluation, and safeguards) supervised the operation, including specialists in technical and vocational education. The Bank harmonized its efforts with the support of other partners. The operation also benefitted from a Norwegian Education Trust Fund grant of US$505,000 that provided technical advisory services and consultants. The Bank was proactive and monitored all aspects diligently. It requested additional financing to see the activities to completion. In 2011 the project was formally restructured. To ensure sustainability of the reforms, the Bank reminded the Government of the need to provide its budgetary resources. A mid-term review was conducted to improve implementation effectiveness. This review focused on cost of rehabilitating TVET institutions (cost overruns), private sector involvement, and financial support from government. Another MTR was held post-additional financing in 2013 to review progress and concluded that progress was on track to meet the revised development objectives. The Bank reviewed M&E data regularly to assess progress in attaining the development objective. However, there were weaknesses in the results chain and selection of indicators. Safeguard compliance was monitored by the Bank team, including cases involving resettlement and compensation of families. Indicators and targets were revised numerous times during implementation, both formally and informally (when baseline data was established in 2008 and during MTR, as well as during the restructuring). Supervision reports included a new indicator that was deemed important to track, yet was not included in the formal restructuring. Thus, the implementing agency did not track and monitor the indicator. The Bank did not adequately utilize supervision ratings to signal to the Implementing Agency and Government early performance shortcomings. For example, despite noted problems during the initial years, rating were only lowered at the MTR from Satisfactory to Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Commitment was evident and noted throughout the operation. At the planning stage the Government expressed its commitment to define and implement necessary policies, support institutional and financing strategy and design a framework for the system, and assign funds from the budget to support the reform process. The project became effective three months after Board approval. The Government developed: (1) an inter-ministerial Commission consisting of representatives from the Ministries of Education, Labor, Higher Education, and Economic Associations; (2) an Executive-level Commission; and (3) an Executive Secretariat to implement the new TVET policy. When additional financing could not cover all the planned activities, the Government still proceeded with developing the oversight body and finished the remaining civil works not completed by project closing. It also enacted the law after the project closed. The government shared and disseminated the findings from the evaluation studies conducted. There were delays in recruiting staff. The government fell behind in its budget contribution to the subsector and reform program, but this was subsequently addressed, and substantial resources were provided to ensure the reforms were developed. Project leadership was changed during implementation as a result of a Government reshuffle between leadership from the Ministry of Education and Culture to the Ministry of Science and Technology, Higher Technical and Professional Education. This slowed progress in the development of the institutional framework. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance During preparation the Ministry of Education and Culture worked closely in designing the elements of the project and determining where Bank support would be beneficial. Initial delays developed in relation to the occupational standards, as it was difficult to find capable firms in the target sectors. The initial bidder went bankrupt and another had to be found. This created delays in relation to curriculum development and training. Disbursements during the initial years lagged by two years. Baseline data were also delayed as it took more time than expected to Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) survey employers and graduates. Implementation began to improve once most of the staff members were appointed. After the additional financing, progress was reported within all the components with the exception of slow recruitment of TVET instructors and slow development of the institutional framework. M&E staff regularly shared data with the Bank. While data was initially weak, the data improved over the course of the project. Procurement delays experienced during the initial years were addressed, and financial management performance improved. Prior to effectiveness a procurement manual was developed and utilized during the course of the operation. Financial management complied with Bank procedures. When minor issues were identified, they were addressed. During the final years of the project, a person was assigned to act as a focal point to coordinate safeguard compliance. The Implementing Agency performance significantly improved over the course of the operation and provided detailed project progress reports to the Bank. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The original measures were adequate to measure attainment of the first objective (respond effectively to labor market needs), but were inadequate in relation to the second one (respond more equitably). Originally there were indicators to measure employers’ perceptions of skills of recent graduates, earnings of graduates, and number of days to find a job or become self-employed, but all of these were dropped and replaced with something else: employer satisfaction with competency training, and number of students assessed with CBT. The rationale for these changes was time. This concern seems unsubstantiated as the project was extended twice. Thus, there was time to systematically assess the first cohort benefiting from training who graduated in 2011 (and second cohort) before the project closed in 2015. Several of the key performance indicators were revised when the objectives were restructured. Some were clarified, but others were dropped or replaced with something entirely different. Some of the changes lowered the bar. There was inadequate attention to measures that would demonstrate improved quality in selected TVET institutions. Assessment of the skills of the trainees, earnings, or changes in the drop-out rate would have documented improved quality. Gender and HIV/AIDS prevention were included in each qualification and curriculum, but no outputs or outcomes were tracked in relation to these two aspects, except for female project beneficiaries. The monitoring and evaluation plan provided a schedule for data collection including responsibilities; it also linked the activities with key annual events. Four evaluation studies (i.e. tracer studies, effectiveness of FUNDEC, progress on gender, and progress in relation to mainstreaming the content of HIV AIDS prevention policy) were included. b. M&E Implementation During the initial years data were gathered through reports from TVET institutions, but this arrangement did not provide accurate and detailed data and so new baseline data were collected. Baseline data were completed in 2007 and 2008, and subsequently revised. Significant technical assistance was provided to strengthen the data coming from TVET institutions. These efforts improved data quality and reliability. The assessment of the FUNDEC competitive fund was undertaken, as well as two tracer studies. The gender study was funded by the Spanish Cooperation Agency was carried out in all TVET institutions. The HIV study was completed. c. M&E Utilization As a result of the initial monitoring and evaluation experiences, data collection and reporting were strengthened. The gender and HIV studies informed the issues to be addressed in designing the qualifications. The tracer studies have been used by institutions to evaluate their own programs and to understand how the private sector perceived their programs. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) M&E Quality Rating Modest 11. Other Issues a. Safeguards The project triggered two safeguard policies: Environmental Assessment (O.P. 4.01) and Involuntary Resettlement (O.P. 4.12). The project was classified category B for Environmental Assessment due to the planned rehabilitation of TVET institutions. An Environmental and Social Management Framework was developed. At the MTR, safeguard compliance was reviewed. School officials were trained in environmental management for waste management of hazardous materials. In cases where land was purchased, families were compensated and the Bank team reviewed how these cases were handled. Both Environment Assessment and Involuntary Resettlement were rated Satisfactory during the course of the operation. b. Fiduciary Compliance Procurement and financial monitoring activities for the Japanese grant were carried out by the Department of Administration and Finance in the Ministry of Education and Culture. This department was familiar with Bank procedures, as it handled them in two previously funded education projects. Procurement was rated Moderately Satisfactory or Satisfactory during the operation. Assessment of compliance with Bank procurement procedures revealed minor deficiencies such as lack of delivery schedules in contracts and arithmetic errors. Procurement improved over the course of the operation. Audits were unqualified (clean) and were provided on time. Financial management was rated Satisfactory over the course of the project. An internal auditor performed an audit including a verification of the use of project funds. The project monitored the execution of the FUNDEC grants. There were a few instances where standards were not adhered, and so these sub-projects were interrupted or cancelled (ICR p. 29). Financial reports were provided on time and with acceptable quality. When issues were detected by the Bank team, they were minor and subsequently addressed. The final audit is expected to be finalized by June 30, 2016. c. Unintended impacts (Positive or Negative) None reported. d. Other None reported. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Modest Modest --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons IEG has summarized the lessons presented in the ICR (p. 37-38) as the following: • Realism is needed when undertaking a far-reaching reform within a given sector. First, there has to be consistency in how the Bank and Government view the reform, which was established in this operation as there was strong alignment between the project and the Government’s own reform. As a result, the Government is maintaining the program. Second, there has to be an understanding of what is required on the part of the Government. A realistic plan and timeline then need to be established. In this operation, it took many more years than planned to achieve some of the initial activities such as developing standards, framework, and curriculum, as well as enacting the law, because the implementation realities were not adequately understood. In the end it took additional financing and time to transform the TVET subsector from a supply-driven system into one that is demand-driven and relevant to the country’s development needs. • Active participation of the private sector is instrumental in TVET reforms. Through a collaborative and consultative progress with TVET training providers and institutions, private sector firms and representatives helped to define and develop the occupational standards and associated training programs created in the project, thus ensuring their relevance and link to the needs of the labor market. 14. Assessment Recommended? Yes Please explain TVET is an under-evaluated sub-sector within IEG’s PPARs and thus this operation could yield valuable lessons related to improving quality and reforming systems to be more demand-oriented, as well as yield lessons about the Bank's support of TVET. 15. Comments on Quality of ICR The quality of the ICR is substantial. It provides a concise description of the project, its activities, and the factors affecting implementation. More discussion regarding achievements of grants would have been beneficial. Important information was presented in the annex, rather than the text. The ICR correctly calculated the outcome rating, based on OPCS guidance on rating outcomes of projects with formally revised objectives (OPCS Guidelines Appendix B). This review was based on the objectives in the Lending Agreement, while the ICR utilized those contained in the appraisal document, thus contributing to minor differences in sub-ratings. a. Quality of ICR Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Tech & Voc Edu & Training (FY06)(P087347) Substantial