GOVERNANCE GLOBAL PRACTICE FEBRUARY 2018 THE MALAYSIA DEVELOPMENT EXPERIENCE SERIES Improving Education Sector Performance in Malaysia Lessons from a Delivery Unit Approach CONNECT WITH US wbg.org/Malaysia @WorldBankMalaysia @WB_AsiaPacific blogs.worldbank.org/category/ countries/malaysia FEBRUARY 2018 THE MALAYSIA DEVELOPMENT EXPERIENCE SERIES Improving Education Sector Performance in Malaysia Lessons from a Delivery Unit Approach GOVERNANCE GLOBAL PRACTICE About KNOWLEDGE & RESEARCH The World Bank Group’s current partnership with Malaysia is focused on knowledge-sharing. It is centered on support for Malaysia’s vision to join the ranks of high-income economies by 2020 through inclusive and sustainable growth, and to share its lessons with developing countries. In March 2016, the World Bank Group officially launched its Global Knowledge and Research Hub in Malaysia. The Hub is the first of its kind, serving both as a field presence in Malaysia and as a global knowledge and research hub. It focuses on sharing Malaysia’s people-centered development expertise and creating new innovative policy research on local, regional and global issues. Knowledge & Research reports are flagship work emanating from the teams based in the Malaysia Hub. The Malaysia Development Experience Series captures key lessons from Malaysia relevant for emerging economies in Asia, Africa and elsewhere that are transitioning out of poverty and into shared prosperity. Cover Photo: The Education Performance and Delivery Unit (PADU) The findings, interpretations, and conclusions expressed in this report do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Please contact the lead authors of this report, Jana Kunicova (jkunicova@worldbank.org), Jeevakumar Govindasamy (jgovindasamy@worldbank.org) and Lars M. Sondergaard (lsondergaard@worldbank.org), if you have questions or comments with respect to content. Table of Contents Acknowledgements 4 List of Acronyms 5 List of Figures 6 List of Tables 6 List of Boxes 6 Foreword 7 Executive Summary 9 Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? 14 Improving education outcomes is one of the top government priorities 15 High-level priorities were underpinned by action-oriented programs to deliver change 18 The design and implementation arrangements of one such program - LINUS - were thoughtful and recognized previous implementation challenges 22 A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? 24 Enhanced tools to screen students 25 New learning material and pedagogical approaches 28 FasiLINUS: support and guidance to schools and teachers 29 What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? 30 Strong program design 31 Delivery units improved coordination and performance incentives 33 Substantial resources were allocated to finance the program 37 What were the Challenges in Applying the DU Approach in Education in Malaysia? 38 Assessments of student learning outcomes is difficult 39 Attribution of results to LINUS is difficult 41 Lack of targeting and expenditure tracking leads to cost efficiency questions 42 What can Other Countries Learn from Malaysia’s Experience with Transforming Education Sector Performance? 44 Learning from the experiences that supported LINUS’ implementation 45 Learning from the challenges 46 References 47 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 3 Acknowledgements This report was prepared under the Public Sector Performance theme of the World Bank’s Global Knowledge and Research Hub in Malaysia outbound knowledge program. The World Bank team consisted of Jana Kunicova, Jeevakumar Govindasamy, and Lars M. Sondergaard. Other colleagues, including Tan Mei Ling, Ricardo Alfredo Habalian, Angela Chen, Chancey Lee Pacheco and Alan Sie Ping Lau provided useful inputs. The team worked under the overall guidance of Robert Taliercio and James Brumby, and closely coordinated with Ulrich Zachau and Faris H. Hadad-Zervos. The team is grateful to peer reviewers, Shomikho Raha, Marie-Helene Cloutier and Myrna Machuca-Sierra for their valuable comments. This report benefited from presentations, fruitful discussions, comments, and information from various sections of the Education Performance and Delivery Unit (PADU), Ministry of Education (MOE), and Performance Management and Delivery Unit (PEMANDU) in the Prime Minister’s Department. We are indebted to the PADU and the MOE for their ongoing collaboration with the World Bank and in particular their strong leadership in the launch of this report. Puan Khadijah Abdullah (CEO of PADU), Dr. Azwan bin Abdul Aziz, Dr. Habibah binti Abdul Rahim, Dr. Noorliza binti Hj. Zakuan, En. MD Yaisa bin Yahya, Puan Masreen, and also Tengku Nurul Azian Tengku Shariman from PEMANDU and her team, Ungku Fazuin and Nigel Fong, were exceptionally forthcoming and proactive in providing information and coordinating with other line ministries, departments, and agencies. A special not of thanks to Dato’ Noor Rezan Bapoo Hashim and Dato’ Lee Boon Hua for their valuable input and insights. 4 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach List of Acronyms BPK Curriculum Development Division, Ministry of Education CEO Chief Executive Officer DMO Delivery Management Offices DU Delivery Units ETP Economic Transformation Program FasiLINUS Facilitators for LINUS GTP Government Transformation Program LINUS Literacy and Numeracy Screening MDAs Ministries, Departments and Agencies MDU Ministerial Delivery Unit MOE Ministry of Education NKRA National Key Result Area NTP National Transformation Program PADU Education Performance and Delivery Unit PEMANDU Performance Management and Delivery Unit PM Prime Minister PMD Prime Minister's Department PMDU Prime Minister's Delivery Unit PROTIM 3R Remedial Program KIA2M Early Intervention for Reading and Writing KPI Key Performance Indicator MDG Millennium Development Goals NTP National Transformation Program PIRLS Progress in International Reading Literacy Study TIMMS Trends in International Math and Science Study Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 5 List of Figures Figure 1. Malaysia’s National Transformation Program (NTP) 16 Figure 2. Public spending on education as a share of GDP 17 Figure 3. Public spending on education as a share of total public spending 17 Figure 4. Illiteracy among Malaysia’s 5th graders by state (2008 estimates) 18 Figure 5. LINUS Screening Instrument for Literacy 26 Figure 6. LINUS Screening Instrument for Numeracy 26 List of Tables Table 1. Categories of LINUS Students 27 Table 2. Estimating the LINUS Cost (MYR) 37 List of Boxes Box 1. Eight Steps of Transformation 19 Box 2. PADU: A small, high-powered delivery unit to drive the Education Blueprint delivery 21 Box 3. How do PEMANDU Labs work? 32 Box 4. De-Bottlenecking by Delivery Units: The Escalation Process 35 Box 5. Process Evaluation of LINUS: Is the screening tool producing accurate results? 40 Box 6. Randomized Evaluation 43 6 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Foreword There is an increasing demand for accountability establishing its credibility and earning the trust for delivering performance to obtain the expected among its stakeholders. Thus, a lot of effort was results. Lessons learned have led to more efficient spent on building strong operational governance, approaches in translating aspirations into tangible getting the right talent in the team, adding value outcomes. It usually starts with a very good plan, to the Ministry by providing solutions to problems, but often falters at the point of execution. Yet strengthening effective communication and implementation is the most crucial aspect of arriving change management, as well as building strong at the desired outcomes. Given the increase in engagement with all the stakeholders. public demand for government reforms to improve performance, the establishment of an independent One of the key challenges was to change the mindset delivery unit is an option to consider in helping the of the stakeholders to embrace transformation, government to successfully deliver on its priorities which involved the buy-in to do things differently. It in a timely manner. required high perseverance and tenacity to stay on course, focused, and unwavering. As a result, even Realizing the importance of efficient delivery of during PADU’s early years, it was very encouraging strategic goals and the country’s aspirations in to see some of the initiatives demonstrate success. education, the Government of Malaysia established Among the most notable initiatives that showed the Education Performance and Delivery Unit great achievement is the Literacy and Numeracy (PADU) in 2013. PADU’s role is unique and Screening (LINUS) program. LINUS was well- distinctive. It partners with Ministry of Education received and gained tremendous momentum since to facilitate, support, and deliver the vision of the it was first rolled-out in 2009. In ensuring that the Ministry in transforming the national education programs achieve their goals and targets, PADU system through the successful implementation of employs tools and performance mechanisms to the Malaysia Education Blueprint (MEB) 2013-2025. plan, track, and monitor activities carried out under It should be emphasized that the measure of success the LINUS initiative. Also, PADU was always involved is not only about meeting the tangible targets but in providing solutions when there were issues to also by focusing on the intangibles such as values, resolve. As the programs moves into its 8th year ethics and spirituality so as to produce students now, PADU is conducting a comprehensive ‘health who are balanced and well rounded – excellent in check’ of this initiative and intends to elevate it to academic and at the same time well entrenched in the next level by pegging the outcome to be at par the noble values as aspired in the Blueprint. to international standards. In ensuring the successful implementation of It is indeed an honor that the World Bank is the MEB, it is important that PADU works in total documenting lessons from delivery units like PADU alignment with the Ministry. Working closely with in delivering performance improvement initiatives. strong mutual support is critical in order to develop This report is timely as it offers insights on the setup action plans, monitor and track implementation, of a delivery unit, description of our journey, the as well as provide solutions and interventions to success story of a program, as well pointers on the eliminate stumbling blocks. lessons learned. We hope that it will be a useful reference to countries that may have similar needs. Admittedly, managing change is never easy, especially when it involves a fundamental transformation of the system. Understandably, Khadijah Abdullah in the early years of the MEB implementation, CEO, The Education Performance and Delivery Unit (PADU), PADU as a newly set up unit faced a challenge in Ministry of Education Malaysia Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 7 Executive Summary Executive Summary This report is part of the World Bank’s Malaysia Education’s inclusion in this list reflected two things: Development Experience Series that strives to Malaysia’s education system was not delivering, and document the country’s approach to improving the previous attempts at reforming the sector were public sector performance. The report builds on not bringing about the desired transformation. A the previous installment in the same series that more drastic overhaul was needed. focused on Malaysia’s experience with driving performance from the center of government The more drastic overhaul of the sector was through the Prime Minister’s Management Delivery envisioned to come from the application of Unit (PEMANDU).1 While the previous assessment the DU approach to transforming sector focused on the delivery unit approach in general, performance. PEMANDU introduced its signature this report attempts to drill down into the education “8 steps of transformation” methodology. This sector to illustrate how the delivery unit (DU) approach is designed to take the sector from method worked to implement a particular education strategic direction, through operationalizing high- program, Literacy and Numeracy Screening (LINUS). level goals into projects or programs during the It can thus be read as a companion piece to the Lab phase and setting key performance indicators, PEMANDU assessment. Education is a particularly to monitoring, troubleshooting and communicating interesting sector in Malaysia to examine the DU the results to the public. Crucially, the first step along approach, in part because the Ministry of Education this path– providing strategic direction – is more (MOE) set up its own ministry-level DU, known than just providing direction: PEMANDU supported as the Education Performance and Delivery Unit the Ministry of Education (MOE) in identifying the (PADU), to help MOE deliver on its objectives and game changing initiatives or programs that could to interface with PEMANDU. Therefore, the role of help drive the ministry in the desired direction of DUs in the sector has been especially prominent. improving student learning outcomes. In 2009, Malaysia’s education system embarked Literacy and Numeracy Screening (LINUS) was on a transformation. Although Malaysia reached one of the five game changers that is worth an adult literacy rate above 90 percent, OECD’s examining for a number of reasons. First, despite PISA scores provided hard evidence of what many being an upper middle income country with a Malaysians already knew: more and more students well-developed education system, Malaysia had were graduating, with diplomas certifying they a sizeable share of its early graders struggling to had completed more years of schooling, but the master basic numeracy and literacy, although the education they received – as reflected in the overall magnitude of the problem was unclear due to lack poor mastery of English language skills, critical of systematic measurement of those skills. Malaysia thinking, and analytical reasoning skills – was not introduced Literacy and Numeracy Screening of the quality needed for a country with aspirations (LINUS) in 2009 as a systematic and comprehensive of reaching a high income status. In 2009, the effort to screen and improve literacy and numeracy government launched the National Transformation standards of primary school students. Second, how Program (NTP), an ambitious effort to transform the LINUS was conceptualized is a good example of country. One of its original six National Key Results “evidence-based decision making” at work, with Areas (NKRAs) was “assuring quality of education.” wide-spread applications for other countries and 1 World Bank. 2017. Driving performance from the center: Malaysia’s experience with PEMANDU. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/318041492513503891/Driving-performance-from-the-center-Malaysia-s-experience-with-PEMANDU Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 9 Executive Summary programs. Third, LINUS is also worth examining This report describes how the DU method worked as it provides good insight on how delivery in both the design and implementation stages units like PEMANDU and PADU influenced the of LINUS. This includes the interface between program’s design, institutional arrangements and MOE and PEMANDU, as well as PEMANDU and coordination, as well as monitoring. Finally, the PADU. The core of the document focuses on challenges that Malaysia encountered during the implementation details: the institutions, leadership, implementation provide valuable lessons for other and incentives that contributed to the program’s countries. This includes challenges in designing a success. It also describes how the implementation program to enable impact evaluations. agencies worked together to implement LINUS. How did the DU approach improve implementation of education programs? Including education into the NTP as one of the Elevating education to such high priority level top national priorities provided the mandate to provided opportunities and incentives for re-strategize efforts to improve literacy and coordination and collaboration. The NKRA numeracy. These centered on better collaboration Education Lab provided a viable platform for the top among MOE Divisions, as well as the central, state, management of MOE to bring together different district, and school levels. Unlike previous literacy MOE Divisions to work more collaboratively. Labs and numeracy interventions, a cross-division were 6-9 week stakeholder workshops designed to task force (LINUS Taskforce) was established “unpack” the strategic goal of improving education to coordinate the different divisions in MOE, quality into concrete programs with timelines, KPIs, including the State Education Departments and and responsible agencies. The way the labs have District Education Offices to implement LINUS in been designed and implemented by PEMANDU schools. A more collaborative effort, the LINUS also enabled the top management of MOE to bring initiative benefited in terms of intellectual capital together different MOE Divisions to work towards that shaped its overall program design; screening developing a tangible implementation plan by the instruments, learning and teaching modules, and end of the education lab duration. The education teaching pedagogy. The institutional setup for NKRA Lab also provided a good platform for the implementation of LINUS was complex, but collaboration, as well as to seek out fresh new ideas provided the right combination of leadership, and validation points. The NKRA Education Lab accountability, and incentives for the program to was instrumental in shaping the program details succeed. The Curriculum Development Division of LINUS. The lab resulted in the establishment of (BPK) has led the LINUS Task Force and worked the LINUS Taskforce to oversee the continuation of closely with other MOE divisions, as well as with the development, implementation and monitoring PADU and PEMANDU.` efforts of LINUS. The lab also defined the key 10 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Executive Summary milestones of LINUS and set the stage on how the culture into the MOE, with their private sector ethos, performance of the MOE Divisions involved in the hard deadlines, reporting routines, and breaking LINUS delivery chain is to be measured. down hierarchical thinking. This also created performance incentives for schools and teachers Linking the center of government with the on the ground. The involvement of DUs in the education sector entities through the NTP design and implementation of education programs improved monitoring and problem-solving. The like LINUS also created several adjustment issues LINUS Task Force, PEMANDU, and PADU have been within the ministry. These adjustments include successful in providing an effective coordination, how performance is to be accurately measured, tracking, monitoring and reporting framework interpretation of data, and the validation process. for LINUS. The per formance management framework is anchored on using the best and LINUS program was well-resourced. The NKRA leading transformation practices. This involves status of LINUS as well as the launch of the Malaysia the introduction of performance tools as follows: Education Blueprint helped in getting the required (i) Development of granular implementation plans, human, financial and technical resources required (ii) Dashboard to track and monitor progress of to implement the program. Ample resources were initiatives, (iii) Mid-Year Review, (iv) End-year Review, allocated to the LINUS program to finance a strong (v) Problem-Solving Meeting for the issues to be design, pay for start-up costs, and for its continued escalated and decided, and (vi) Annual Report that implementation. communicated the achievements for the year. In addition, the delivery units infused performance What were the challenges for the DU approach in education? No reliable estimate of early grade literacy process that resulted in inaccuracies of the and numeracy skills existed prior to LINUS, assessment of student abilities. Commissioning so a dependable baseline was not available. In an external process evaluation of LINUS was an addition, assessing student learning outcomes is important step toward refining the program and its inherently difficult. Are the standards sufficiently accuracy, but LINUS program design did not enable hard? Does the assessment accurately measure rigorous impact evaluations. students’ level of mastery of those standards? How are consistent assessment-taking conditions and In the absence of impact evaluations, it is unclear consistent grading ensured? Although students whether LINUS was the sole intervention that took repeated screenings to improve the accuracy resulted in improved literacy and/or numeracy of results, there were still questions whether LINUS skills of early graders. The program was rolled-out was accurately measuring what students could nationwide, which did not allow the implementers or could not do. The Deloitte performance audit to specify “treatment” and “control” groups for of LINUS, commissioned by PEMANDU in 2011, LINUS. As such, under the current implementation revealed several weaknesses in the screening arrangements, it is not possible to ascertain whether Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 11 Executive Summary LINUS or something else helped improve literacy numeracy might not be a nationwide problem, but rates. Also, it is difficult to tell what part of the rather concentrated in several districts. It is possible LINUS program (e.g. FasiLINUS; remedial teachers; that a narrower and much cheaper program or the focus on measuring learning outcomes) was focusing on the struggling states and districts more impactful in improving learning outcomes. could have achieved the same or better results for While the in-depth process evaluations helped less. However, it is difficult to say with certainty identify strengths and weaknesses of the LINUS how much the LINUS program cost. The way the implementation and measurement of what students costing information has been pieced together and knew at each grade, an impact evaluation would subsequently tracked, without taking into account be beneficial to determine which interventions the salaries of teachers, makes estimating the actual have worked. This would include steps to establish cost to run a program like LINUS challenging. This appropriate baselines enabling future efforts (e.g., is problematic for both establishing the value for next 5 years) to undertake a more robust program money of the program, but also for other countries evaluation. considering to replicate a similar initiative. The lack of targeting and expenditure tracking raises questions about the cost efficiency. The trends in the data seem to suggest that literacy and What are the key lessons for other countries? Malaysia’s DU approach to improving education Third, fostering per formance culture in the outcomes offers important lessons for other implementing ministry will improve incentives on countries, both in its successes and its the ground. Finally, conducting process evaluations limitations. On the positive side, this approach and learning from them will help improve programs improved top-to-bottom incentives to implement by iteratively adapting their design. LINUS, fostered performance culture, and enabled adaptive learning. However, it is also important Malaysia’s challenges in implementing LINUS to learn from the pitfalls in measurement, impact are equally instructive. Before embarking on a evaluation, and cost tracking. national program of this magnitude, it is important to ensure that there are appropriate baselines to Malaysia’s experience with LINUS point to track literacy and numeracy rates. Also, it is helpful four key lessons that have supported the to build in impact evaluations into the program implementation of LINUS. First, making improving design to address the issue of attribution and cost education outcomes a national priority will effectiveness. Furthermore, before embarking on a elevate the profile, the stakes, and the resourcing nationwide program, countries may wish to consider for the initiative. Second, creating institutional whether a targeted approach will achieve the same mechanisms to ensure collaboration, coordination, results at a lower cost. Finally, it is critical to track and commitment among all stakeholders, from the full program costs to establish value for money and highest levels of the ministry to the districts, schools, flag cost overruns. and teachers is key for better implementation. 12 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach CHAPTER 1 Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? 14 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? Many developing countries adopt sensible strategies to improve education outcomes, but few are able to successfully implement them. Implementation challenges take on many forms, including lack of prioritization or appropriate diagnostics of the underlying challenge, poor coordination, or insufficient monitoring and evaluation that allow for course correction. To overcome these challenges, Malaysia used a DU method to tackle one of its most persistent service delivery challenges – improving the quality of its primary education. Malaysia has declared assuring education quality one of its key national priorities, or National Key Results Areas (NKRAs) in 2009, signaling a high-level leadership commitment to improving student outcomes and assuring that this area would receive resources, attention, and support from the center of government. Moreover, this high- level commitment was translated into granular programs with assigned responsibilities, concrete milestones, and key performance indicators (KPIs). Finally, the design and implementation arrangement of one such program, Literacy and Numeracy Screening (LINUS), took into account the lessons from previous unsuccessful initiatives and provides a good example of evidence-based policy making. Improving education outcomes is one of the top government priorities Shortly after the current Prime Minister Najib came to power in 2009, his government promulgated its top priorities in the National Transformation Program (NTP). The NTP was grounded in the existing national development planning framework such as 10th and 11th Malaysia Plan, but prioritized a limited number of the Prime Minister’s policy objectives that reflected his vision and the motto “People First, Performance Now.” NTP was divided into two parts: the public service delivery portion – the Government Transformation Program (GTP) – and the industrial policy portion, the Economic Transformation Program (ETP). Under GTP, there were eight NKRAs, including reducing crime, fighting corruption, and improving quality education. These NKRAs would receive additional funding, scrutiny, monitoring, and trouble-shooting, and would be supported from the center of government by the Prime Minister’s Performance Management and Delivery Unit (PEMANDU).2 2 For details on Malaysia’s experience with the NTP and PEMANDU, see World Bank. 2017. Driving performance from the center: Malaysia’s experience with PEMANDU. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/318041492513503891/Driving-performance-from-the-center-Malaysia-s-experience-with-PEMANDU Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 15 Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? FIGURE 1 . Malaysia’s National Transformation Programme (NTP) The 7 NKRAs under the Government Trasformation Programme (GTP) and the 12 NKEAs & 6 SRIs under the Economic Transformation Programme (ETP), all form part of the National Transformation Programme (NTP) FOCUS COMPETITIVENESS “DRIVERS” to ensure focus on high-impact areas “ENABLERS” to ensure competitiveness 7 NATIONAL KEY RESULT AREAS 12 NATIONAL KEY ECONOMIC AREAS 6 STRATEGIC REFORM INITIATIVES (NKRAs) (NKEAs) (SRIs) Addressing Reducing Oil, Gas & Palm Oil & Rubber Financial Services Competition Public Finance Public Service Cost of Living Crime Energy Standards & Reform Delivery Liberalisation Fighting Improving Student Tourism Business Services Wholesale & Retail Narrowing Reducing Human Capital Corruption Outcomes Disparities Government’s Development Role in Business Raising Living Improving Rural Education Healthcare Communications Standards of Low- Development Content & Income Households Infrastructure Improving Urban Electronics & Agriculture Greater Kuala Lumpur Public Transport Electrical / Klang Valley Source: PEMANDU 16 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? As one of the NKRAs, Assuring Quality Education became one of the top service delivery priorities of Malaysia’s government. Although Malaysia is an upper middle income country with a well-developed education system, a sizeable share of its early graders struggle to master basic numeracy and literacy. Malaysia’s education system made tremendous progress since the country gained independence in 1957, when over half of its population had no formal schooling. By 2011, the enrollment at the primary level was near universal at 94 percent. The percentage of students dropping out of primary school had been significantly reduced to about 0.2 percent. The Government of Malaysia sees education as one of its key priorities: public spending on education hovers around 5 percent of GDP (Figure 2) and the education budget is consistently the largest component of the total Federal Government Budget, rising from 14 percent of total spending in 2008 to around 20 percent in 2015 (Figure 3). FIGURE 2. Public spending on FIGURE 3. Public spending on education education as a share of GDP as a share of total public spending 6 6 25 25 5 5 20 20 % of total public spending % of tot l public spendin 4 4 % of GDP 15 15 % of GDP 3 3 10 10 2 2 5 5 1 1 0 0 0 0 MalaysiaM l si Sin Singapore pore Th Thailand il nd Vietn m Vietnam M l si Malaysia Sin pore Th il ndThailand Singapore Vietn mVietnam 2008 20082010 20102015 2015 2008 2008 2010 20102015 2015 Source: UNESCO Institute of Statistics Although the education system is well-funded, a sizeable share of students in Malaysia has struggled to master basic literacy and numeracy skills. Despite large sums of resources invested in education, the 2008 Government of Malaysia estimates3 show that 13 percent of Year 1 students did not have basic literacy skills, while 24 percent among Year 4 students did not have basic numeracy skills. Similarly, test results from 2008 indicate that three states, in particular Sarawak, Selangor and Johor, had alarming rates of 5th graders who did not have basic literacy skills (see Figure 4). For a country with aspirations to achieve high income status within a decade, and for a larger proportion of its workforce to have a tertiary education, these results were worrisome. Moreover, research by MOE revealed that one factor contributed to drop-out rates was the inability of students to cope with the syllabus taught. In 2008, almost 32,000 students dropped out of school at various stages.4 3 LINUS Lab Report 4 Improving Students Outcome, GTP roadmap Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 17 Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? FIGURE 4 . Illiteracy among Malaysia’s 5th graders by state (2008 estimates) Number of Year 5 students without basic literacy skills in 2009 (Percentage of total students without literacy skills nationwide) 437 (0.7%) 5,767 (9.6%) Perlis 5,022 (8.4%) Kedah 3,402 (5.8%) 2,151 (3.6%) Pulau Pinang Kelantan Terengganu Perak 3,999 (6.7%) Pahang 3,414 (5.7%) 115 (0.2%) Selangor 9,252 (15.4%) W. P. Kuala Lumpur 682 (1.1%) W. P. Putrajaya Negeri Sembilan W.P. Labuan Melaka Sabah Johor 6,829 (11.4%) 2,761 (4.6%) 1,036 (1.7%) 64 (0.1%) 5,508 (9.2%) 9,511 (15.9%) Sarawak Low illiteracy Total No. of Students Medium illiteracy in PROTIM (2M) ~59,950 High illiteracy Source: PROTIM Scores for 2M, 2009 Source: PEMANDU Lab Report (2009) High-level priorities were underpinned by action- oriented programs to deliver change Because the goal of improving the quality of education became one of the NKRAs, it was supported by PEMANDU, the delivery unit in the Prime Minister’s Department. PEMANDU means “driver” in Malay, which is a fitting metaphor for its role in achieving the NKRAs: the actual implementation of these goals rests with the ministries, departments and agencies (MDAs) such as the Ministry of Education (MOE). To support and literally “drive” these efforts, PEMANDU developed its signature eight-step methodology, known as Eight Steps of Transformation. This approach is designed to help determine the strategic direction for the sector; operationalize the high-level goals into projects or programs during the Lab phase; set the key performance indicators (KPIs) with targets; monitor them during the implementation; troubleshoot implementation problems; and eventually evaluate and communicate the results to the public. Details of this methodology are further described in Box 1. 18 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? BOX 1 Eight Steps of Transformation PEMANDU acts as a convener, 2 coordinator, and technical support to MDAs on performance management, LABS Establish in detail what monitoring and reporting. Although needs to be done these are distinct roles, they are well- defined and structured in PEMANDU’s “Eight steps of transformation.” 3 1 PEMANDU acts as a convener at Step 1, OPEN DAYS STRATEGIC Share Lab output DIRECTION when multiple cabinet retreats provided with people To ascertain the and seek their strategic direction strategic direction and alignment at the feedback required highest leadership level. PEMANDU also convenes and coordinates 6-9 week 4 stakeholder workshops known as “labs” (Step 2) that break down the strategic ROADMAP Tell the people what we are going priorities into concrete interventions and to do create ownership among stakeholders. 8 5 PEMANDU continues in the same ANNUAL KPI TARGETS role by holding Open Days (Step 3) REPORTS Setting KPIs for Tell people what we monitoring and and publishing Roadmaps (Step 4), have delivered tracking ensuring that the public is aware of the government plans and holds it accountable for delivery. In Step 5, 7 6 PEMANDU switches gears and acts as technical support to MDAs in setting IPR & AUDIT IMPLEMENTATION External Problem-solving validation on on the ground and monitoring KPI targets. If problems results achieved implementation and bottlenecks occur in during implementation (Step 6), PEMANDU becomes a convener and coordinator again, escalating the process through the Steering Committee and Problem Solving Meetings (see Box 6 for details). In Steps 7 and 8, PEMANDU is again a technocratic agent overseeing that the reported KPI targets are validated by the third party and compiled into the Annual Report. Source: World Bank 2017. Driving Performance From the Center: Malaysia’s Experience with PEMANDU. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 19 Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? After improving the quality of education was chosen as an NKRA in Step 1, PEMANDU ensured through the subsequent steps that this high-level goal was broken down into specific programs with assigned responsibilities, deadlines, and key performance indicators (KPIs). PEMANDU supported the Ministry of Education (MOE) in identifying the game-changing initiatives or programs that could help drive the ministry in the desired direction – raising student learning outcomes. Five such potentially game-changing education initiatives were identified in 2009. These included: the Pre- School program; the Literacy and Numeracy Screening (LINUS); the High Performing School program; the New Deal program, introducing a new performance-based assessment for Head Teachers to innovate, improve upon and deliver high performance at their schools; and the School Improvement Program. These five priority initiatives would later be confirmed and expanded upon as part of the preparation of a comprehensive Malaysia Education Blueprint5 for the education sector that took place during 2011-2013. MOE, the main implementer of the NKRA education program and also the new Malaysia Education Blueprint (MEB), saw the need for a ministry level DU, the Education Performance and Delivery Unit (PADU). All MDAs responsible for the implementation of NKRAs had some kind of an institutional interface with PEMANDU, most often in the form of Delivery Management Offices (DMOs). MOE, like other ministries also had an institutional interface with PEMANDU. MOE saw the need and advantages of establishing a DU at the ministry level (i.e. PADU) to help implement the Malaysia Education Blueprint (MEB). MOE’s institutional interface with PEMANDU on the Education NKRA was then streamlined under PADU. This included reporting the KPIs to PEMANDU on a monthly, quarterly, and annual basis. In turn, one of PEMANDU’s tasks was to regularly update the NKRA dashboard that is periodically reviewed by the Prime Minister, elevating the importance of achieving the targets and of trouble-shooting any shortfalls early in the process. 5 Malaysia Education Blueprint 2013 – 2025 (Preschool to Post-Secondary Education), https://www.moe.gov.my/images/dasar-kpm/articlefile_file_003108.pdf 20 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? BOX 2 PADU: A small, high-powered delivery unit to drive the Education Blueprint delivery Drawing from international experiences in successfully managing the transformation with the necessary rigor and discipline, the MOE established the Education Performance and Delivery Unit (PADU). This small and high-powered program office drives the delivery of all initiatives across the Ministry and schools. In practice, PADU monitors progress, problem solves, delivers, and manages the communication about the transformation. Specifically, PADU drives the execution of the Malaysia Education Blueprint (MEB) initiatives and interacts with the public to keep them informed of the Blueprint progress and to gather ongoing feedback. PADU integrates the functions of the MOE’s Delivery Management Office (DMO) and the Project Management Office (PMO). Driving the execution of education initiatives extends beyond tracking progress. PADU provides on-the-ground problem solving and acts as the first point to escalate issues for the Ministry on a day-to-day basis. However, it is important to note that PADU plays a supporting role; the accountability for delivering remains with the Ministry. The composition of PADU is critical to its success. At the helm of PADU is a Chief Executive Officer (CEO). The rest of the PADU comprises highly skilled employees, from both the public and private sector. The Ministry established PADU - a dedicated transformation office to ensure delivery of outcomes WHAT IT DOES... Education Performance and Delivery Unit (PADU) Debottlenecks problematic situations and take corrective action to address lagging performance Creates transparency by tracking and monitoring implementation and performance Provides stakeholder engagement support Mission: To be a partner to the Ministry of Education in facilitating transformation of the Malaysian education system both internal and external Source: Adapted from the Malaysia Education Blueprint and PADU Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 21 Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? This sophisticated and systematic design and implementation arrangement is particularly noteworthy because public sectors in developing countries – even upper middle income ones such as Malaysia – often struggle with implementation. They often know what needs to be done and can produce well written strategies and policies, but the implementation often falls short. Malaysia MOE’s efforts to combat illiteracy and innumeracy during the early 2000s is a case in point: the problem was well-diagnosed through numerous assessments, and programs were in place to address the problem. But despite sustained efforts over several years, the problem did not go away. Introducing the DU approach – both at the Prime Minister and ministry level – were seen as a way to improve implementation. This can provide valuable lessons for other countries what works and what does not. The design and implementation arrangements of one such program – LINUS – were thoughtful and recognized previous implementation challenges LINUS, a systematic and comprehensive effort to screen and improve literacy and numeracy standards of primary school students, was one of the five game-changing initiatives introduced in 2009 to achieve the Education NKRA targets. The aim of the program was to ensure that all students acquire basic literacy and numeracy skills at the end of three years of primary schooling. The rationale for launching LINUS was rooted in the country-wide assessments from the 2006-2009 showing that a sizeable share of students in Malaysia did not have basic literacy and numeracy skills (see Figure 4). The design and implementation arrangements for the LINUS program explicitly took into account past implementation weaknesses. First, substantial outside help was mobilized to help the MOE design the program. Choosing education as one of the NKRA signaled the high-level commitment at the Prime Minister’s level to improving student outcomes. “Labs” were conducted to help MOE design the program with wider participation from within the ministry, including the state, district, and school levels, as well as to obtain more structure and quality feedback from stakeholders. PEMANDU and the international consulting firm McKinsey played an important part in running the Labs alongside MOE to ensure it met its objectives. Second, to ensure effective implementation, a sectoral delivery unit (PADU) was created by MOE to function as an effective project management office at the ministry level. PADU does not operate like a conventional ministry department, given its private-sector-inspired structure, performance management and staffing. Aside from interfacing with PEMANDU at the central level, PADU was put in charge of overseeing implementation coordination, monitoring and trouble-shooting when problems arose. Third, process evaluations, or “performance audits,” were commissioned from the outside by the independent evaluator (Deloitte), to highlight strengths and weaknesses in the implementation of the program. 22 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 1: Why is Malaysia’s Delivery Unit Approach for Improving Education Sector Performance Worth Studying? LINUS is worth focusing on because its conceptualization and implementation has demonstrated some elements of evidence-based decision making at work, with wide-spread applications for other countries and programs. Evidence-based decision making involves six steps, some of which were employed during the design and implementation of LINUS: i. Acknowledging that there is a problem to be solved. This includes ensuring high- level leadership commitment to solving it. ii. Quantifying the problem in order to establish a baseline and track progress. In LINUS’ case, efforts were made to look into the data from previous assessments to establish a baseline. While there are still areas for improvement in developing a reliable baseline, efforts to quantify the problem during the Labs can be seen as commendable. iii. Identifying interventions that could help address problem. In LINUS’ case, this brainstorming was done in “Labs” which resulted in identifying the key interventions making up the LINUS program (see next section). iv. Designing implementation in such a way that the interventions can be both monitored and evaluated, to ascertain whether they have an impact on desired outcomes. For LINUS, this meant conducting in-depth “process evaluations” to help identify strengths and weaknesses of implementation. v. Evaluating which interventions have worked. vi. Building on the interventions that worked. The remainder of this assessment focuses on lessons learned from applying the DU approach to the design and implementation of the LINUS program under the Education NKRA. Chapter 2 provides a primer on LINUS, including its components and how it was meant to drive improvements in literacy and numeracy. Chapter 3 unpacks how the DU method, or the 8 Steps of Transformation, worked in LINUS’s case. Chapter 4 examines what can be learned from the limitations in the design and implementation of LINUS. Finally, Chapter 5 summarizes the lessons learned for other countries from both LINUS’s successes and challenges. An important caveat: As will discussed in more details below, this report does not contain evidence that LINUS has achieved its intended objective, i.e. improved reading and numeracy skills of early graders. Such an assessment is not possible due to the lack of key data. Rather, the report focuses on assessing the design of the program; and the implementation arrangements that supported its ongoing implementation. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 23 CHAPTER 2 A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? 24 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 2: A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? LINUS built on earlier attempts in Malaysia to screen and improve literacy and numeracy standards, taking on board lessons from earlier programs. Before introducing LINUS in 2009, there had been no screenings in Malaysia that would capture both literacy and numeracy standards for primary school students in Years 1 through 3. Prior to LINUS, the Early Intervention for Reading and Writing (KIA2M) was used to assess students’ reading and writing skills in Year 1, while the 3R Remedial Program (PROTIM) was designed to assess reading, writing and arithmetic for Year 4 to Year 6 primary students. LINUS drew lessons from these programs and sought to strengthen them in three key ways: i. Enhancing tools to screen students. LINUS ramped up efforts to identify struggling students earlier to allow for an earlier intervention for both literacy and numeracy. For example, the KIA2M’s Year 1 screening was expanded to several screenings in Year 1 – and additional screenings in Year 2 and 3. ii. Introducing new learning material and pedagogical approaches. iii. Introducing FasiLINUS – a cadre of experienced teachers offering support and guidance for teachers and schools in implementing LINUS: FasiLINUS teachers mentored and guided other teachers and clarified roles in providing remedial education. Enhanced tools to screen students With LINUS, students were screened earlier, more comprehensively, and more systematically than before. LINUS consists of two screening instruments: one for literacy and one for numeracy. The instruments were developed by MOE based on the national curriculum for primary school. The instruments are made up of 12 “constructs,” designed to measure students’ mastery of different aspects of literacy and numeracy. For literacy, the easier constructs focused on the ability to read, write, and understand simple words. The harder constructs measured students’ ability to understand sentences, and the most advanced constructs gauged their ability to apply the knowledge in learning and everyday communication. Similarly, for numeracy, easier constructs measured whether a student could recognize numbers, while the harder constructs eventually measured students’ ability to apply mathematical skills in everyday life. Figures 5 and 6 list these constructs. Unlike the earlier screening efforts, LINUS screening involved all primary schools in every state and districts, using a standard screening instrument developed by MOE. In addition, the previously used PROTIM’s Year 4 screening was also improved to continuously help students who have undergone LINUS in 2010 to 2013 after they have completed Year 3 of primary schooling. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 25 Chapter 2: A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? FIGURE 5. LINUS Screening Instrument for Literacy Screening instrument - Literacy The screening instrument is developed based on the definition of literacy, i.e. Ability to read, write and understand words, simple and complex sentences (using conjunctions) in Bahansa Malaysia and apply such knowledge in learning and everyday communication Instrument Constructs 1. Ability to pronounce and write vocal and consonant alphabets 2. Ability to pronounce and write open and close ‘suku kata’ 3. Ability to pronounce and write words of ‘suku kata terbuka dan tertutup’ 4. Ability to pronounce and write words to distinguish between vowel of ‘e taling’, ‘e pepet’ and ‘o’ 5. Ability to pronounce and write words containing “suku kata tertutup ‘ng” 6. Ability to pronounce and write words containing dipthong and ‘vokal berganding’ 7. Ability to pronounce and write words containing “digraf’ng’, ‘ny’, ‘sy” 8. Ability to pronounce and write word with prefixes and suffixes 9. Ability to pronounce and write simple sentences 10. Ability to pronounce and write complex sentences 11. Ability to read and understand paragraphs 12. Ability to relay information in stimulus materials in oral and written form Source: PEMANDU Lab Report on LINUS (2009) FIGURE 6. LINUS Screening Instrument for Numeracy Screening instrument - Numeracy The screening instrument is developed based on the definition of numeracy, i.e. Ability to solve basic mathematical operations and understand the idea of simple mathematics and able to apply mathematical skills to everyday life Instrument Constructs 1. Ability to recognise numbers in oral and written form 2. Ability to count 3. Ability to understand the value of numbers 4. Ability to arrange numbers in sequence 5. Ability to perform basic mathematical operations 6. Ability to recognise Malaysian currency 7. Ability to tell time 8. Ability to perform basic mathematical operations using Malaysian currency 9. Ability to measure length, mass and volume of objects 10. Ability to translate normal sentences into mathematical equation and vice versa 11. Ability to apply knowledge and skill of round number in everyday life 12. Ability to apply knowledge and skill which involve currency, time and measurement in everyday life Source: PEMANDU Lab Report on LINUS (2009) 26 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 2: A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? The aim of the screenings was to detect by teachers from the mainstream classes. Students struggling students, and help teachers pinpoint who are able to master constructs 1 to 2 only are where students were struggling so they could called the LINUS group and are taught by LINUS offer the appropriate support. Classroom teachers Remedial Teachers. Those who cannot master administered the screening for each student constructs 1 to 2 are categorized as the LINUS individually at the beginning of the academic Tegar (hardcore) group. This group are not only year to determine if the student would stay in the taught by remedial teachers, but also undergoes mainstream class or placed in the remedial class health screenings to detect any learning disabilities. (see Table 1). Students who mastered all but the Students with identified learning disabilities are most difficult constructs were categorized as the channeled to special education classes. As part Arus Perdana (mainstream) group. Having mastered of the LINUS program, MOE deployed 1,098 LINUS up to construct 8, this group of students additional remedial teachers to primary schools requires the least form of intervention and is taught across Malaysia.6 TABLE 1. Categories of LINUS Students GROUPING OF STUDENTS BASED DETAILS ON LINUS RESULTS Students who could not master LINUS Constructs 9 – 12 Arus Perdana (Mainstream) Students are taught by mainstream teachers Students who could not master LINUS Constructs 3 – 8 LINUS Students are taught by LINUS Remedial Teachers Students who could not master LINUS Constructs 1 – 2 LINUS Tegar (Hardcore) Students are taught by LINUS Remedial Teachers Health screening to detect learning disabilities Source: Authors 6 LINUS operational documents, i.e. information on operations and governance of LINUS, are available (in Malay) at: https://www.slideshare.net/MuslimahMuhammadNoh/buku-pengoperasian-linus Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 27 Chapter 2: A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? New learning material and pedagogical approaches LINUS teaching and learning materials have been developed, piloted, and validated by the Curriculum Development Division, Ministry of Education (MOE). The teaching and learning module is based on several concepts to enable students to master literacy and numeracy. These concepts involve an integrated approach to literacy and numeracy, with elements of progression (ansur maju), edutainment (didik hibur), integration (penggabunglingan), and repetition (pengulangan). The literacy and numeracy standards are anchored in the national curriculum for primary education. The module encompasses information pertaining to the list of skills to be taught, as well as suggested activities and training. The materials, consisting of learning modules and teaching guides, are prepared in a CD-ROM format and distributed to the State Education office for printing and distribution to all schools. Enhancement of teachers’ pedagogical skills is an integral part of the program. Recognizing the lack of expert teachers (guru pakar) for teaching basic literacy and numeracy, intensive teachers’ training was conducted to strengthen the knowledge and pedagogy, as well as to identify students with multiple abilities and to conduct teaching and learning effectively. Development of training modules, as well as the pre-service and in-service training are the responsibility of the Teachers’ Development Division (BPG) in the MOE. During this process, BPG in MOE recorded the best practices and developed support programs for teachers. In-service training for teachers included the training of trainers (ToT) of about 60 teachers at the national level and 800 teachers at the state level during the inception phase of LINUS. About 17,000 teachers were trained from 2009 to 2010.7 Pre-service training includes the revised curriculum for Teaching and Learning (T&L) of literacy and numeracy. Manual and teaching materials were also prepared to guide teachers.8 The focus on training and upskilling the teachers to enable them to become effective remedial teachers has paid off, as demonstrated by the results of the different screenings throughout the year for the same cohort. 7 LINUS for early learning, The Star Online (2012) http://www.thestar.com.my/news/nation/2012/09/05/linus-programme-for-early-learning/ 8 Teaching modules for LINUS teachers are available (in Malay) at: https://www.academia.edu/9229034/Modul_Guru_Linus_Numerasi_Tahun_3?auto=download 28 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 2: A Primer on LINUS: How was it Meant to Improve Literacy and Numeracy? FasiLINUS: support and guidance to schools and teachers New roles for teachers and as well as specialized teachers were introduced to aid the implementation of LINUS in schools. Facilitators known as FasiLINUS were introduced and roles such the LINUS Teachers and LINUS Remedial Teachers were created. FasiLINUS are experienced teachers, sometimes also retirees, brought in to ensure implementation success of LINUS at the district level by providing support and guidance to schools and teachers. FasiLINUS provide training, coaching and mentoring to LINUS teachers as well as remedial teachers. They also assist schools to develop a suitable strategy, based on the screening results, to improve literacy and numeracy. FasiLINUS also advise the State Education Office on the deployment of remedial teachers. The PEMANDU lab estimated that 884 FasiLINUS were required, with a minimum of four and the maximum of twelve per District Education Office (i.e., about one FasiLINUS per 30 schools). FasiLINUS are placed in each of the 154 District Education Offices, serving as an intermediary between the MOE, State Education Office, District Education Office, and schools under their supervision. The establishment of FasiLINUS includes capacity building for FasiLINUS, teacher deployment to replace FasiLINUS, and Professional Support Plan for Teachers’ Capacity Building. LINUS teachers are the designated teachers involved with the administrative aspects of LINUS in schools. The roles of LINUS teachers include: (i) conduct screenings for literacy and numeracy; (ii) collect and maintain data to identify LINUS Tegar, LINUS, and mainstream (arus perdana) students; (iii) identify ways to improve the implementation of LINUS; and (iv) input LINUS results for each screening into the NKRA portal. LINUS Remedial Teachers teach students who did not pass the screening targets for literacy and numeracy. Under KIA2M, the program that preceded LINUS, there were already about 7,000 remedial teachers deployed, resulting in a 1:1 ratio between remedial teachers to schools. Under LINUS, the targeted ratio of remedial teachers to students was 1:15. At the inception of LINUS, this meant that additional 3,246 remedial teachers needed to be deployed. This number was eventually scaled down to about 1,048 additional teachers deployed as Remedial Teachers. The roles of Remedial Teachers include: (i) conduct screenings for literacy and numeracy within the stipulated time alongside LINUS teachers; (ii) assist LINUS teachers in the preparation of teaching materials; and (iii) plan the timetable and activities for LINUS Tegar students and other LINUS group of students. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 29 CHAPTER 3 What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? 30 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? The DU approach encompasses a holistic literacy and numeracy target for all primary school methodology that includes a number of well- Year 3 students in Malaysia. defined and sequenced roles for the DU. This methodology is illustrated by PEMANDU’s “Eight Another strong feature of the design was the Steps of Transformation.” The highlights of this openness to the outside guidance on design and approach that made LINUS successful include: (i) delivery of LINUS. The NTP and the DUs made MOE the strength of the program design that involved to revisit the status quo on how education programs stakeholder involvement and ownership through the were developed in the past. Through working with PEMANDU-facilitated Education Lab; (ii) institutional PEMANDU and others, MOE top management arrangements for implementation; (iii) allocation found ways to better leverage platforms such as of significant resources; and (iv) conducting and the labs. This resulted in a more inclusive approach learning from the external process evaluation of the to stakeholder discussions, and also involvement screening tool. of external consulting firms in the design process. For instance, McKinsey was hired to support the This section builds closely on the World Bank preparations for and the running of the Education assessment of the PEMANDU experience. 9 At the Lab. The design documents emerging from the Lab same time, it illustrates how a DU approach works in were exceptionally carefully crafted, containing: a specific sector implementing a specific program. As such, it can be read as a companion piece to i. clear descriptions of the program, underpinned the PEMANDU report that delves deeper into the by whatever data were available to paint a program design and implementation arrangements “baseline situation;” on the ground. ii. a solid discussion of how existing efforts could be amended to have more impact; iii. arguments for why new approaches were Strong program design needed, including references to international experience; iv. multi-year costing of the interventions; and Unlike the earlier literacy and numeracy v. detailed implementation arrangements. remedial programs, the development of LINUS was treated as a key priority program by the The NKRA Education Lab provided a viable MOE and went through a more rigorous process platform for the top management of MOE to involving more stakeholders. LINUS was among bring together different MOE divisions to work several key programs introduced to improve access together more collaboratively. The way the labs to and quality of education, led by MOE. Because were designed and implemented by PEMANDU it was a GTP initiative, the design of the program enabled the top management of MOE to bring was developed at the national-level problem- together different MOE divisions as well as the solving workshops known as “Labs,” facilitated State Education Department and District Education by PEMANDU. The Education Lab ran for eight Offices to work towards developing a tangible weeks and involved the entire MOE, including implementation plan by the end of the education lab State Education and District Education Offices, duration. The lab also defined the key milestones in and also multiple stakeholders from the academia, the implementation of LINUS and set the stage for government, and NGOs. During the lab, the LINUS how performance of the MOE divisions involved in program design was discussed and agreed upon. the LINUS delivery chain would be measured. This included its action plan, institutional setting and resourcing required to achieve the 100 percent 9 World Bank (2017). Op. cit. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 31 Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? BOX 3 How do PEMANDU Labs work? PEMANDU initiated labs to analyze how to The lab is a consultative process, where people implement Malaysia’s goal of becoming a high- work together iteratively to design solutions to income country by 2020. At the beginning of identified implementation challenges within a initiating the NTP, PEMANDU invited officials from strict timespan. Senior leaders act as “lab sponsors” line ministries and representatives of the private to support the conduct of the lab and endorse the sectors for 6-9 weeks and discussed sectoral and final lab output. The lab consists of three phases; the industrial priorities and implementation plans. pre-lab, the main lab and the post-lab. The pre-lab PEMANDU worked with line ministries and private session includes planning among key stakeholders, sector to set the objectives, actionable work plans, initial analysis, fine tuning the scope and boundaries specific deliverables, and timeline, as well as resource of the lab against the agreed true north and selection needs to implement policy objectives. In Dato’ Sri of relevant lab participants. The main lab session Idris Jala’s (CEO of PEMANDU) words, lab allowed is held at a physical location and the participants for “the descent from 30,000 feet to 3 feet” and work together to create a solution in a specific facilitated the prioritization of projects, initiatives and timespan. The participants select a lab leader among policies, KPIs and action plans of NKRAs and NKEAs themselves. PEMANDU deploys lab facilitators who that would help deliver Malaysia’s development moderate the discussions and ensure that lab leaders goals. undertake their assigned role. The session focuses on LEARNING BRAINSTORMING & • Fact-finding PROBLEM SOLVING • Research (best • Intervention design practices, success • Follow-up on stories, etc.) outstanding issues OUTCOMES INPUTS • Lab Environment • Documentation and • Skilled and • Working teams motivated teams • Lab Objectives data entry • Key stakeholders • Delivery plan for (internal and each national external) priority area • Data and • Resourcing background ANALYSIS DISCUSSIONS & DEFRIEFS strategy (people materials • Data analysis and and financing) number-crunching • Outcomes from external meetings • Stakeholder analysis and discussions • Root cause analysis • Syndication Source: World Bank (2017) 32 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? Delivery units improved coordination and performance incentives issue identification, learning, analysis, brainstorming and problem solving, stakeholder syndication, The NKRAs platform provided the mandate to discussions and debriefs. The lab produces a report re-strategize efforts to improve literacy and containing sectoral or thematic initiatives, detailed numeracy, centered on better collaboration implementation plan, lead agencies and identified KPIs. The report is then presented to the “lab among MOE divisions at the central, state, sponsor.” Once lab sponsor endorses the report, the district, and school level. A more collaborative post lab session of public consultation starts. If no effort, the LINUS initiative benefited in terms of significant public objections are received, the initiative the intellectual capital that shaped its overall will be implemented. The post lab also includes the program design; screening instruments, learning process of further syndication with stakeholders and and teaching modules, and teaching pedagogy. implementers who were not participants of the lab and as well as to design the delivery system that will The education NKRA lab also provided a good govern the implementation process. platform for collaboration as well as to seek out fresh new ideas and validation points. PEMANDU’s Big Fast Results Institute (BFRI) has been conducting labs in countries outside The NTP had an impact on the institutional of Malaysia. BFRI staff facilitated the running set up and human resource management of of the labs in the education sector and the retail MOE. Several new entities have been created in sector at the state level in Andhra Pradesh, India. MOE as a response to the national transformation The Education Lab focused on how to improve the agenda. Like all ministries that implement the quality of education across the pre-school, primary and secondary levels. The lab detailed the policy NTP, the MOE initially established a Delivery outcomes and KPIs to monitor the achievement of Management Office (DMO) to monitor and report the objectives. This included improving student on the Education NKRA to PEMANDU. Through learning outcomes, improving quality of teachers, the DMO, the ministry staff was exposed to new and enhancing the school system. The lab involved approaches and the ways in which issues were 92 participants from 26 organizations. Similarly, Retail escalated, solved, and the methods used for Lab focused on how to generate investment, jobs and income, involving 13 government agencies, 2 NGOs, progress tracking and reporting. and 18 private sectors. As a result of the Education Lab facilitated by PEMANDU, MOE created a taskforce for each Education NKRA initiative to oversee The Lab is a the operational and delivery aspects. The main consultative process, goal was to avoid the coordination problems of the previous literacy and numeracy interventions. where people work For LINUS, there was the LINUS Taskforce headed together iteratively by the MOE Curriculum Division (BPK). The LINUS to design solutions Taskforce is made out of eight different divisions within the MOE, including divisions that oversee to identified policy the state and district level education offices. The challenges within a cross-division approach is seen as an achievement strict timespan. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 33 Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? by the ministry, as it was not easy to break the silos Since 2012, PADU has worked alongside the that existed prior to introducing LINUS. The NTP LINUS implementing units in the MOE to ensure wave was seen by the MOE as a catalytic event the delivery of student outcomes. With the launch which gave the ministry a much-needed boost to of the Malaysia Education Blueprint, the performance restructure from within and work towards a common of the Education NKRA initiatives including LINUS goal. were tracked and monitored by PADU and reported further to PEMANDU. As a transformation office, The LINUS Taskforce ensured that the PADU was given the mandate to de-bottleneck program and delivery specifications which problematic situations and take corrective actions to was formulated during the lab is sustained as address lagging performance, create transparency well as implemented. The task force provided a through tracking and monitoring, and provide suitable platform for the implementing offices to stakeholder engagement support. work out the details of LINUS. This included the development, resourcing, and implementation plan The institutional setup for the implementation for the screening instruments, learning and teaching of LINUS was complex, but provided the right materials, teachers’ training modules, capacity combination of leadership, accountability, and building for FasiLINUS and Remedial Teachers, incentives for the program to succeed. The stakeholder awareness and engagement strategy, Curriculum Development Division (BPK) who led monitoring and evaluation, and equipment. the LINUS Task Force worked closely with PADU and PEMANDU. The LINUS Task Force, together A further evolution of institutions took place with PADU and PEMANDU, have been successful with the launch of the Malaysia Education in providing an effective coordination, tracking, Blueprint in 2012, when the MOE created its own monitoring, and reporting framework for LINUS. The DU, PADU, to replace the DMO. The institutional performance management framework was anchored changes in MOE also impacted its human resource in the Eight Steps of Transformation developed by management (HRM). Specifically, PADU does not PEMANDU and adopted by PADU. This involved function like a standard department or unit under the introduction of performance tools as follows: (i) the MOE, but rather like a corporation. It is headed Development of detailed implementation plans; (ii) by a Chief Executive Officer (CEO) and draws talent Tracking and monitoring the progress of initiatives from private sector, Government-Linked Companies through a dashboard; (iii) Mid-Year Review; (iv) End- (GLCs), academia, as well as MOE staff who are year Review; (v) Problem-Solving Meetings (issues seconded to PADU. From the HRM standpoint, to be escalated and decided – see Box 3); and (vi) this implies some form of autonomy granted from Annual Report, reporting of achievements for the the ministry to the DU for HR functions such as year. recruitment, compensation, and performance management. The autonomy in HR meant that PADU is also able to hire the right talent through a competitive process from the labor market as well as from the ministry. 34 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? BOX 4 De-Bottlenecking by Delivery Units: The Escalation Process PEMANDU monitors progress towards the NTP KPI targets on a weekly basis. MDAs submit information to PEMANDU every week. PEMANDU reviews the information and updates its dashboard. If the weekly monitoring uncovers implementation issues, PEMANDU refers them to the Technical Working Group, which consists of the staff from the relevant MDAs that are involved in implementing the relevant NKRA/NKEA/SRI. Implementation issues that cannot be resolved at the Technical Working Group are elevated to the Steering Committee. The Committee is typically chaired by one or two ministers, and also comprises secretaries-general, directors-general and CEOs from core ministries or agencies. The Committee serves as the principal decision-making forum (as opposed to the Technical Working Group that serves as the principal working session) and meets on a monthly or quarterly basis. The issues that remain unresolved at the Steering Committee level are then elevated to the Problem-Solving Meeting (PSM), which is held twice a year, chaired by the Prime Minister. The PSM is pivotal in resolving difficult, cross-ministerial issues. At mid-year and year-end, the Prime Minister also conducts reviews with the respective ministers, with the CEO of PEMANDU in attendance, where issues are highlighted for his direction and decision. It also provides an avenue for revision of targets, if and when endorsed by the PM. Source: World Bank (2017) Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 35 Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? LINUS created incentives for teachers and schools to improve implementation on the ground. Involvement of various MOE divisions in the implementation process of the program at the central, state, district, and school levels were limited prior to LINUS. With the introduction of LINUS, support mechanisms provided by MOE to schools and teachers included the introduction of professional help for teachers and also the emphasis on intensive training that exposed teachers to various teaching styles and techniques to enhance students’ learning. As for the buy-in, there were concerted efforts by the MOE to work alongside the District Education Office, headmasters, schools, and teachers to ensure the implementation and ownership of the initiative on the ground. For example, MOE did not hire new remedial teachers but placed a lot of emphasis on the development of training materials to support both remedial teachers and FasiLINUS in carrying out their roles effectively. MOE also played a more direct role in the implementation of LINUS by going to the field more often and ensuring that FasiLINUS, State and District Education Officers, and teachers received the support required to address the implementation issues of LINUS. This approach of MOE being more directly involved on the ground also created incentives for teachers and schools to ensure delivery. 36 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 3: What can be Learned from Using Malaysia’s Delivery Unit Approach to Implement LINUS? Substantial resources were allocated to finance the program The NKRA status of LINUS as well as the launch of the Malaysia Education Blueprint helped in getting the required human, financial and technical resources required to implement the program. Ample resources were allocated to the LINUS program to finance a strong design, pay for start-up costs, and for its continued implementation. However, estimating the total cost of the LINUS program is complicated because the LINUS budget allocation is only for capital expenditures (e.g., Labs, screening materials, teacher training). However, the LINUS cost also includes the hiring and salaries of 884 professional facilitators or FasiLINUS to support teachers in the implementation of LINUS, for which limited public data is available. In total, the LINUS program cost is estimated to fluctuate between MYR 53 million to MYR 78 million per year. In per student terms, this is equal to MYR 40-59. The non-salary cost of LINUS in the inception year (MYR 35 million in 2010) was higher than in the following year (MYR 28 million in 2011), reflecting the cost of the design stage, such as conducting Labs, designing the screening instruments and training materials. Table 2 provides estimates of the total cost of LINUS using the data from the NKRA Education Lab as well as the World Bank staff estimates. Salary data from the Malaysian Education Review conducted by UNESCO in 2015 were used to estimate the salary cost of FasiLINUS. TABLE 2. Estimating the LINUS Cost (MYR) World Bank Staff NKRA Education Lab estimates Items 2010 2011 2012 2016 FasiLINUS salaries (est.) 36,332,400 36,332,400 36,332,400 62,279,568 LINUS Budget (labs, screening and training materials, printing, 35,110,000 28,828,900 16,919,900 16,000,000 administration, etc.) Total LINUS Cost (MYR) 71,442,400 65,161,300 53,252,300 78,279,568 Number of Primary Year 1 - 3 1,389,210 1,366,000 1,338,654 1,328,090 Students LINUS Cost per Student (MYR) 51 48 40 59 Source: Authors Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 37 CHAPTER 4 What were the Challenges in Applying the DU Approach in Education in Malaysia? 38 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 4: What were the Challenges in Applying the DU Approach in Education in Malaysia? There are three key challenges. First, assessments of student learning outcomes are difficult to get right. Process evaluation revealed that LINUS was likely overestimating improvements made, which led to adjustments in the program. Second, even if literacy and numeracy have improved in Malaysia, it is hard to quantitatively attribute the improved literacy to LINUS in the absence of rigorous impact evaluations, although compelling qualitative arguments can be made. Finally, lack of targeting to the most struggling schools and districts calls into question the cost effectiveness of LINUS. Relatedly, the costing and expenditure tracking of the program could be improved to establish a full program cost. Assessments of student learning outcomes is difficult Because LINUS was part of an NKRA, the accuracy of its results and associated KPIs came under more scrutiny. In Step 8 of the eight-step methodology, PEMANDU communicates the results achieved by the NKRAs in its Annual Report, and the KPI targets reported there are independently audited by PricewaterhouseCoopers. This means that all KPIs, including those achieved by LINUS, require more diligent ex ante quality control. This includes the accuracy of the LINUS screening tool. Assessments of student learning outcomes are difficult to get right. Are the standards sufficiently hard? Does the assessment accurately measure students’ level of mastery of those standards? How are consistent assessment-taking conditions and grading ensured? Although students took repeated screenings to improve the accuracy of results, there were still questions whether LINUS was accurately measuring what students could or could not do. Therefore, in 2011, PEMANDU commissioned a process evaluation, or a performance audit, of LINUS by a third party, Deloitte Consulting Malaysia. The Deloitte performance audit revealed several weaknesses in the screening process that resulted in the inaccuracies of the assessment of student abilities. The audit undertook an independent assessment of 1st, 2nd and 3rd graders and found that their reading and math skills were significantly lower than what the results from LINUS suggested. Box 5 describes this evaluation in more detail. However, it is uncertain whether these inaccuracies are due to a faulty design of the screening tool or its inappropriate use. The current screening is based on students’ ability to master each of the 12 constructs for Literacy and Numeracy and the student performance is tracked based on this framework. It is unclear if the constructs are customized to the different grades of students (i.e., Year 1, Year 2 and Year 3) and if the difficulty level increases with the students’ grade level. Commissioning the external evaluation of LINUS was an important step toward refining the program and its accuracy. The audit report provided the following recommendations: (i) improve outcomes by maintaining clear standards for the LINUS assessment and administrative process; (ii) improve teaching capabilities and capacity to ensure that students receive the best possible outcomes from LINUS; (iii) align LINUS with the mainstream curriculum to provide a logical development outcome for students; (iv) enhance remedial efforts to ensure that targeted students receive the development needs required; and (v) address social environmental challenges to enable better student learning outcomes. A process evaluation such as the Deloitte CHECK LINUS report is a good practice where additional effort was taken to verify and address the reliability concerns of the literacy and numeracy screening. One key change implemented in the wake of this evaluation was to reduce the number of screenings from three times a year to two times a year. This was to allow schools and teachers more time to focus on the remedial intervention and also to reduce the administrative burden of preparing the screenings. Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 39 Chapter 4: What were the Challenges in Applying the DU Approach in Education in Malaysia? BOX 5 Process Evaluation of LINUS Is the screening tool producing accurate results? The process evaluation of LINUS delivery, and also determined good practices was conducted by an external audit and key challenges in the implementation the company, Deloitte Consulting, in 2011. program at the school level. An average of 20 The performance audit, also known as the students per school were selected from 199 CHECK LINUS project, sampled a total of schools in 13 states, including the Federal 3,982 students. This included 1,934 of Year Territories of Kuala Lumpur and Putrajaya. 1 students and 2,048 Year 2 students. Both A total of 639 teachers were also surveyed cohorts were the Arus Perdana (mainstream) nationwide. Four audit teams were deployed students who were assessed on their literacy covering on average 2 schools per day, and numeracy competencies. The focus spending about 4.5 hours per school over a areas of the audit were as follows: period of one and a half months. • Performance: Student achievement The findings documented the encouraging specifically in the Arus Perdana level activities and dedication from the District group Education Office, teachers, and school • Assessment: The approach to screening administrators in their effort to implement the students the LINUS Program. However, there were • Administration: The overall process also several challenges, including: focused on the integrity of NKRA portal data • LINUS results seem to over-estimate the • Environment: Factors that impact student students’ reading and math ability; outcomes (parents, medical, transfers, • An inconsistent understanding of the and similar). marking scheme among teachers; • The short time gap between screenings The audit process consisted of three that did not allow a sufficient time for main activities: (i) LINUS Documentation meaningful remedial actions; Review; (ii) Student Assessment Tool; and • Data entry discrepancies; (iii) Audit Interviews and Teachers’ Survey. • Teachers were burdened with LINUS The documentation review was conducted administrative functions, together with to determine LINUS data accuracy and multiple school roles and/or curriculum consistency. The student assessment tool administrative duties; was developed in consultation with the • LINUS constructs were too basic and not Curriculum Development Division and the aligned to the mainstream curriculum; and Examination Syndicate. The interviews and • Lack of standardized remedial teaching surveys gathered feedback from teachers on aids and instructional support for LINUS the LINUS program teaching and learning students. Source: CHECK LINUS Report (2011), Deloitte Consulting Malaysia 40 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 4: What were the Challenges in Applying the DU Approach in Education in Malaysia? Attribution of results to LINUS is difficult For a number of reasons, it is hard to quantify the impact LINUS has had on improving early grade reading and numeracy skills. First, a dependable baseline for the pre- LINUS intervention– i.e., a measurement of students’ reading and numeracy skills, against which progress could be measured - was not available. Specifically, the two programs (KIA2M and PROTIM) that existed before LINUS did not generate measurements which could easily be compared to LINUS’ results. KIA2M focused only on grade 1 students and used a methodology of scoring students which makes comparisons with LINUS difficult, and PROTIM focused on students that were older than those tested by LINUS (in grade 4 and 6). Second, LINUS is one among several key programs designed to improve access to and quality of education. LINUS was not launched in isolation: for example, a curriculum review conducted in 2011 resulted in the introduction of the Primary School Standard Curriculum (KSSR), which may have well contributed to the improvements in the literacy. Similarly, the district transformation program, school performance banding, and other initiatives could have plausibly influenced the improvement in literacy. MOE and PADU provide compelling qualitative arguments in support of LINUS playing an important role in this improvement. First, early and systematic identification of weaker students allow teachers to pinpoint where students are struggling. This, in turn, allows teachers to provide more targeted remedial activities. Second, the new cadre of professional supporters, the FasiLINUS and Remedial teachers, have been well-received and recognized for their dedication and commitment to address learning problems. They are seen as valued delivery partners and welcome assistance to teachers and school administrators. However, it is difficult to rigorously verify these conjectures. Had the program design built in impact evaluation features,10 it would be easier to attribute the results to LINUS or some other parallel intervention. When designing and implementing new sizeable programs, international best practices and experience point toward design features that would allow for a rigorous impact evaluation, such as randomized treatment and control groups. This means designing implementation in such a way that one group of students receives “treatment” (i.e., is exposed to LINUS interventions) while the other (“control”) group does not. This would allow the interventions to be both monitored and evaluated to ascertain whether they have an impact on desired outcomes. LINUS program design does not include elements that would enable a quantitative impact evaluation. The program was introduced nationwide in one go, making rigorous evaluations impossible. Specifically, the introduction of LINUS nationwide does not allow researchers to identify a “treatment” and “control” groups for LINUS. As such, under the current implementation arrangements, it is not possible to ascertain whether it was 10 For a comprehensive compendium on impact evaluations, see: Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch. 2011. Impact Evaluation in Practice. 1st ed. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/2550 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 41 Chapter 4: What were the Challenges in Applying the DU Approach in Education in Malaysia? LINUS or something else that helped improve literacy rates. Also, it is difficult to tell what part of the LINUS program (e.g. FasiLINUS; Remedial teachers; the focus on measuring learning outcomes) was more impactful in improving learning outcomes. While there were in-depth process evaluations conducted both internally and externally to help identify strengths and weaknesses of the LINUS implementation, an impact evaluation would be beneficial to determine which interventions have worked. Impact evaluation features would also include steps to establish appropriate baselines enabling future efforts (e.g. next 5 years) to undertake a more robust program evaluation. Box 6 provides further explanation of impact evaluations based on randomized treatment and control groups. Aside from problems of attribution, impact evaluations would allow to determine the program’s cost effectiveness. Currently it is difficult to tell which of the interventions embedded in LINUS contributed more than others – or whether all were required, or only some. This means that it is impossible to establish whether the interventions included in LINUS were “cost effective,” or whether some were better “value for money” than others. Lack of targeting and expenditure tracking leads to cost efficiency questions LINUS was designed as a nationwide program, but there seems to have been no consideration whether a more targeted effort could have resulted in the same outcome. LINUS was rolled out to all early primary students in 7,764 primary schools across Malaysia. All 154 District Education Offices in the country were involved in hiring FasiLINUS teachers. While there are some districts that truly struggled with illiteracy and innumeracy (see Figure 4), in other districts the illiteracy and innumeracy rates are very low. LINUS results show that there is a number of districts where a high percentage of students (90 percent or more) who are able to master the highest constructs for LINUS. The trends in the data seems to suggest that literacy and numeracy might not be a nationwide problem, but concentrated in several districts. It is possible that a narrower and much cheaper program focusing on the struggling states and districts would have achieved the same or better results for less. The way the costing information has been pieced together and subsequently tracked makes estimating the actual cost to run a program like LINUS a challenging task. This also poses a limitation for understanding the actual cost of the program. First, it is unclear how the original costing was constructed, which in turn affects estimating cost overruns, cost reviews and adjustments. Second, the main cost driver in the program, the salaries of the FasiLINUS teachers, is captured under the MOE’s regular budget, which risks masking the true costs of the program. Estimating the full cost of LINUS would not only help establish value-for-money for the Malaysian authorities, but also could help other countries that might be interested in potentially replicating the program. 42 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 4: What were the Challenges in Applying the DU Approach in Education in Malaysia? BOX 6 Randomized Evaluation It is not always obvious which policy will have the most desirable effects on educational outcomes. Should scarce funds be spent on school uniforms, treating ailments that keep students away from the classroom, textbooks, or something else? What is the best way to help students who are falling behind? Does performance-based pay for teachers improve learning, or does it promote “teaching to the test”? To design good policy in an environment in which programs compete for limited funding, we need to know whether and how well a program works, and whether it provides good value for the money relative to other options. Are there alternative ways of achieving the same (or better) outcomes at a lower cost? Are some components of a program ineffective and superfluous? Random assignment offers a simple way to answer these questions. In randomized evaluations, individuals or schools are selected to receive a program based on lottery. Those who do not receive the program form a “control” group. Because the selection process is random, the two groups are similar in every respect, except that one group receives the program, while the other does not. Therefore, if after the program is implemented, the group that received the program has different outcomes (e.g. improved or worsened teacher attendance, higher or lower test scores), we know that this difference was caused by the program. This clear attribution of which effects were caused by the program can provide insights about its effectiveness. Randomized evaluations are particularly appropriate when programs are oversubscribed, scheduled to be rolled out in a gradual fashion, or initially tested with pilot programs. In those cases where some potential participants would inevitably be denied access, randomization is one of the fairest and most transparent ways of determining participation. Randomized evaluations of development programs are a relatively recent innovation, largely pioneered by J-PAL and its affiliates, and the potential for introducing an element of randomization into the process of evaluation continues to gain recognition. When properly designed, randomized evaluations can provide insight not only into whether a program works, but also why it works, allowing for potential scale up of successful innovations to other areas. Source: Adapted from Evidence-Based programming in Early Grade Reading, USAID & Abdul Jameel Latif, Poverty Action Lab, 2012 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 43 CHAPTER 5 What can Other Countries Learn from Malaysia’s Experience with Transforming Education Sector Performance? 44 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach Chapter 5: What can Other Countries Learn from Malaysia’s Experience with Transforming Education Sector Performance? Malaysia’s DU approach to improving education outcomes offers important lessons for other countries, both in its successes and its challenges. On the positive side, this approach improved top-to-bottom incentives to implement LINUS, fostered performance culture, and enabled adaptive learning. However, it is equally important to learn from the pitfalls in measurement, impact evaluation, and cost tracking. Doing so will allow other countries to achieve better education outcomes faster. Learning from the experiences that supported LINUS’ implementation Make improving education outcomes a national priority. In Malaysia, elevating Assuring Education Quality to the national priority status (NKRA) raised the profile of the education programs. This meant that the stakes of all involved in delivering on the results substantially increased: the Prime Minister’s Department was watching the MOE and its ability to deliver. In addition, this helped to bring more resources to fund the key initiatives aimed at improving education outcomes. Ensure collaboration, coordination, and commitment of all stakeholders.11 PEMANDU- facilitated Labs involved a multitude of stakeholders across different levels of government (central, state, district, school). As a problem-solving platform, the labs created the initial buy-in and commitment to the high-level goal that was developed into more granular programs, such as LINUS, during the Lab phase. The voice of the rank-and-file MOE staff who took part in the Labs was equal to the voice of the MOE leadership, which broke down the hierarchy and made bottom-up voices heard. The Labs also planted the seeds for collaboration on the ground during the implementation phase. Coordination and collaboration were further enhanced by the institutional setup for LINUS implementation. Foster performance culture in the implementing ministry. PADU worked with PEMANDU and the Task Force to create performance culture within MOE. This included creating reporting routines on KPIs within ministry and operating its own KPI dashboard. In addition, PADU brought in a private sector ethos into MOE, including hard deadlines, project management, and a problem-solving approach. As a result, the old hierarchal approach within the MOE was challenged. Before, MOE saw itself a policy maker, while the states and districts were the implementers. With the advent of the NTP and DU approach, MOE became more involved in the implementation process on the ground, creating stronger performance incentives for schools and teachers. 11 Collaboration, coordination, and commitment are the “3 Cs” of the World Development Report (2017) on Governance and the Law. For more detail, see http://www. worldbank.org/en/publication/wdr2017 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 45 Chapter 5: What can Other Countries Learn from Malaysia’s Experience with Transforming Education Sector Performance? Learn from process evaluations to improve programs. PEMANDU’s audited annual reporting on NKRAs also created incentives for process evaluations of LINUS to ensure that the KPIs were accurate. PEMANDU commissioned an external audit of LINUS which found some measurement inaccuracies. These lessons were in turn used to improve the program and the design of the screening tool. Learning from the challenges Ensure that there are appropriate baselines to track literacy and numeracy rates. It is difficult to gauge how much LINUS improved literacy and numeracy. There were no previous screenings that had measured the literacy and numeracy skills of grade 1 to 3 primary students. Therefore, no comparable baseline exists to compare the impact of LINUS to literacy and numeracy rates before and after it was introduced in 2009-2010. Build in impact evaluations into the program design to address the issue of attribution and cost effectiveness. Once the baselines are established and improvements tracked, it is easy to determine if the outcomes are achieved. However, the attribution is uncertain: which program is driving these improvements? Impact evaluations can estimate quantitatively whether the improvements in literacy and numeracy are due to LINUS or a similar program, and which parts of these programs are the main drivers. This can improve cost effectiveness. However, impact evaluations are only possible if they are built into programs at the initial design stage that enables the comparison of treatment and control groups during program implementation. Consider whether a targeted approach will achieve the same results at a lower cost. While equal opportunity for all children is a well-accepted tenet in education programs, some districts and schools may be struggling more than others. In contrast, others may need little, if any, help. Consequently, before embarking on nationwide programs, it is worth considering an approach that would target the struggling schools, while providing lighter support to those schools and districts that already produce good outcomes. Track full program costs. Full program costs of LINUS or similar programs consist not only of training materials, but also of staffing costs, including hiring new teachers. Tracking these will improve cost efficiency, help identify program overruns, establish value for money, and also serve as a benchmark for other countries considering to replicate similar initiatives. 46 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach References Educational Planning and Research Division. 2016. “Quick Facts 2016: Malaysia Educational Statistics”. Ministry of Education Malaysia. http://www.moe.gov.my/images/Terbitan/Buku-informasi/QUICK-FACTS-2016/QUICK%20 FACTS%202016.pdf Educational Planning and Research Division. 2015. “Quick Facts 2015: Malaysia Educational Statistics”. Ministry of Education Malaysia. http://www.moe.gov.my/images/Terbitan/Buku-informasi/QUICK-FACTS-2015/QUICK%20 FACTS%202015.pdf Educational Planning and Research Division. 2014. “Quick Facts 2014: Malaysia Educational Statistics”. Ministry of Education Malaysia. http://www.moe.gov.my/images/Terbitan/Buku-informasi/QUICK-FACTS-2014/QUICK%20 FACTS%202014.pdf Educational Planning and Research Division. 2013. “Quick Facts 2013: Malaysia Educational Statistics”. Ministry of Education Malaysia. http://www.moe.gov.my/images/Terbitan/Buku-informasi/QUICK-FACTS-2013/QUICK%20 FACTS%202013.pdf Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch. 2011. Impact Evaluation in Practice. 1st ed. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/2550 J-PAL South Asia at the Institute of Financial Management and Research. 2012. “Evidence-Based programming in Early Grade Reading.” USAID & Abdul Latif Jameel Poverty Action Lab. https://www.povertyactionlab.org/sites/ default/files/documents/2012.07.22-USAID-web.pdf Malaysia, Performance Management and Delivery Unit (PEMANDU). 2009. National Key Results Area (NKRA) Education Lab Report, by PEMANDU. Lab report, Putrajaya. Malaysia, Ministry of Education. 2015. Annual Report 2015, by Performance and Delivery Unit (PADU). Annual report, Putrajaya. Ministry of Education Malaysia. 2013. Malaysia Education Blueprint 2013-2025 (Pre-School and Post-Secondary Education), Putrajaya: Ministry of Education Malaysia. World Bank. 2017. Driving performance from the center: Malaysia’s experience with PEMANDU. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/318041492513503891/Driving-performance-from-the-center-Malaysia-s- experience-with-PEMANDU World Bank. 2017. The World Development Report: Governance and the Law. Washington, DC: World Bank Group. http://www.worldbank.org/en/publication/wdr2017 Improving Education Sector Performance in Malaysia: Lessons from a Delivery Unit Approach 47 CONNECT WITH US wbg.org/Malaysia @WorldBankMalaysia @WB_AsiaPacific blogs.worldbank.org/category/ countries/malaysia