69742 Design of a new institutional arrangement for education quality assurance Lessons for Chile from international and local experiences The World Bank Human Development Department The Latin American and Caribbean Region Design of a new institutional arrangement for education quality assurance: Lessons for Chile from international and local experiences February 2010 The World Bank Human Development Department The Latin American and Caribbean Region Confidential Draft – For Official Use Only This report was written by Emiliana Vegas (Senior Education Economist and Task Team Leader) and Agustina S. Paglayan (Junior Professional Associate), from the World Bank’s Human Development Department. The authors want to thank Susan Sclafani and Keerti Hasija Kauffman, consultants from Madison Education Group, for their contribution to this report and to previous background reports. The team is especially grateful to the counterparts in the Ministry of Education and the Budget Office of the Ministry of Finance in Chile. Their close collaboration led to significant improvements in the report. Last but not least, the team is deeply grateful to the officials and education experts in Ireland, the Netherlands, New Zealand and Scotland who shared their time and knowledge about their system’s education quality assurance experience. All errors are the sole responsibility of the authors. Vice-President: Pamela Cox Country Director: Pedro Alba Sector Director: Evangeline Javier Sector Manager: Chingboon Lee Sector Leader: Hermann A. von Gersdorff Task Team Leader: Emiliana Vegas Contents Executive Summary ................................................................................. 7 1 Introduction ............................................................................................ 11 1.a Background ............................................................................................ 11 1.b Objetives ................................................................................................ 14 1.c Inputs for this report ............................................................................... 15 2 Methodological approach ....................................................................... 17 2.a Selection of OECD systems ..................................................................... 17 2.b Selection of local reforms ........................................................................ 17 2.c Conceptual framework ........................................................................... 18 2.d Data collection ....................................................................................... 20 2.e Summary ................................................................................................ 21 3 What dimensions of the selected OECD education quality assurance systems and recent history are most relevant to the proposed reforms in Chile? .................................................................................... 23 3.a Brief overview of the selected OECD education quality assurance systems ................................................................................... 25 3.b Summary ................................................................................................ 40 4 Implications of the analysis for the proposed reforms in Chile ................. 45 4.a Implications for the system as a whole .................................................... 45 4.b Implications for specific institutions ......................................................... 63 5 Conclusion ............................................................................................. 89 6 Bibliography ........................................................................................... 92 3 Abbreviations and Acronyms AP Action Plan CPA Certified Public Accountant CEO Chief Executive Officer CITO National Institute for Educational Measurement (the Netherlands) DES Department of Education and Science (Ireland) ERO Education Review Office (New Zealand) GTC General Teaching Council (Scotland) HMIE Her Majesty’s Inspectorate of Education (Scotland) ICT Information and Communications Technologies INCA International Review of Curriculum and Assessment Frameworks LOCE Ley Orgánica Constitucional de Enseñanza LTS Learning and Teaching Scotland MINEDUC Ministry of Education (Chile) NCCA National Council for Curriculum and Assessment (Ireland) NES National Education Service (Chile) NQA National Qualifications Authority (Ireland) NZQA New Zealand Qualifications Authority NZTC New Zealand Teachers Council OECD Organization for Economic Co-operation and Development PISA Program for International Student Assessment SEC State Examinations Commission (Ireland) SEED Scottish Executive Education Department SEP Subvención Escolar Preferencial (Preferential School Subsidy) SIMCE Sistema de Medición de Calidad de la Educación (Educational Quality Measuring System) SNED Sistema Nacional de Evaluación del Desempeño de los Establecimientos Educacionales (National School Performance Evaluation System) SQA Scottish Qualifications Authority SSA Scottish Survey of Achievement TC The Teaching Council (Ireland) UNESCO United Nations Educational, Scientific and Cultural Organization 5 Executive Summary This report seeks to inform the process of design and implementation of institutional re- forms related to the development of a new education quality assurance system in Chile. The report is a synthesis report that draws on previous work by The World Bank in the context of a two-year program of technical collaboration with the Government of Chile. As part of this collaboration, the World Bank produced two related reports, including: (i) a comparative report that analyzes the K-12 education quality assurance systems of Ireland, the Netherlands, New Zealand and Scotland to understand how quality assurance func- tions are distributed across central government institutions; the characteristics of these institutions; and the relationships between them; and (ii) a comparative report that ana- lyzes how these four OECD systems have evolved over time; what lessons emerge from their experience implementing institutional reforms in the past 15 years; and what les- sons emerge regarding the exercise of education quality assurance functions. In addition, the report draws on two research papers by local experts analyzing the implementation of institutional reforms in Chile’s health and criminal justice sectors, as well as on an analysis of the Chilean Ministry of Education’s current organizational and human resources capa- bilities conducted by Ministry of Education staff. At the time when the report was written, the Chilean parliament was discussing Law Proposal No. 609-356, which seeks to strengthen the education quality assurance system for pre-primary, primary and secondary education. The approval of this proposal would signify a substantial increase in the State’s role in the education sector, which historically remained very limited. The proposed change in paradigm is supported by the findings of two comparative reports by the OECD (2004) and The World Bank (2007). Both reports found that no high-performing education system has such limited State intervention in the education sector as does Chile. Also, when the report was written, Law Proposal No. 1151-356 had been submitted to parliament with the aim of strengthening public education. The approval of the first proposal (No. 609-356) would require the implementation of profound institutional reforms. Two new institutions would be created, the Agency for 7 Education Quality and the Superintendency of Education. The Agency would be respon- sible for evaluating the performance of individual students, teachers, school principals, schools and school owners; and reporting to the general public on the results of these evaluations. The Superintendency would monitor schools‟ financial practices and their compliance with laws and regulations; and determine what sanctions should be applied to schools and school owners failing to meet performance standards or statutory regula- tions. The reform would also require a substantial restructuring of the Ministry of Educa- tion, which would assume some new responsibilities and transfer others to the Agency and the Superintendency. The approval of the second proposal (No. 1151-356) would en- tail the creation of a National Education Service, which would be responsible for –among others functions- the provision of technical-pedagogical support to publicly-financed schools, including those managed by the public and the private sector. To inform the reforms ahead, the report summarizes the key lessons that emerge from the education quality assurance experience of high-performing OECD systems and the insti- tutional reform experience of local sectors. We will not attempt to summarize here all the findings discussed in the report, but instead will point out three critical lessons that stand out from the experience of the four selected OECD education quality assurance systems. First, consultations are an essential component of the education sector’s institu- tional culture. To set performance standards, extensive consultations take place early on with a broad range of stakeholders including representatives of students, parents, teach- ers, school principals, schools, school owners, education inspectors, education research- ers and experts, knowledge institutions, local educational authorities, and the range of government agencies that are part of the education quality assurance system. Consulta- tions will be decisive to establish performance standards that reflect what is generally understood as “quality education� by Chilean society. In turn, reaching a shared vision of what constitutes “quality� will contribute to the smoothness and legitimacy of the evalu- ation of performance against these standards. In the selected systems, the consultative nature of the education sector goes beyond the process of setting standards. For example, consultations are part of the process of inspecting and evaluating schools, as well as the process of designing an institutional reform. In the latter case, consultations with public-sector employees affected by the creation or restructuring of an institution have contributed to reduce anxiety about the implications of the reform, maintain employees’ motivation and build a sense of owner- ship of the reform’s objectives and outcomes. As Chile moves forward with the implementation of institutional reforms, it will be criti- cal for the Ministry, the Agency and the Superintendency to introduce early on a regular consultative process with the different actors affected by the reforms. Indeed, the imple- mentation of the proposed reforms presents an opportunity to introduce a cultural shift within the Chilean education sector. Second, in the selected systems there is an emphasis on building constructive relationships between the inspectors responsible for evaluating the quality of 8 schools, and the teachers and school principals affected by these evaluations. When inspectors arrive at a school, they do not just apply a “checklist� to evaluate the quality of education that is being provided, but also take the time to meet with teachers and administrators, provide them feedback, and listen to their views and reactions. In- deed, the external evaluation process presents an opportunity for inspectors to promote discussions between a school’s teaching and non-teaching staff about different dimen- sions of the school’s quality; build their capacity to use evidence to identify the school’s strengths and weaknesses; encourage them to think about ways in which they could address their weaknesses; and provide examples of how schools with a similar context have dealt with similar challenges. In addition, all systems have mechanisms in place to measure the effectiveness of the evaluation process –from “customer satisfaction� sur- veys that are filled in by schools to assess the extent to which the inspection process has contributed to their work, to more rigorous impact evaluations that are carried out by the Inspectorate or commissioned to independent researchers. Unquestionably, the experience of the selected OECD systems raises the bar for the kind of school inspections and evaluations commonly seen in middle-income countries. There is consensus in these systems that the extent to which external evaluations can affect schools’ practices and contribute to educational improvement is crucially contingent on the quality of the external evaluation process and, as part of that, the quality of the rela- tionship between inspectors, teachers and schools. As Chile moves forward with the implementation of the functions assigned to the Agency for Education Quality, it will be imperative to acknowledge the Agency’s role to “guide educational improvements�, which is recognized in Article 6 of Law Proposal No. 609-356. Fulfilling this role will require several steps, from recruiting the right leader- ship for the Agency to ensuring extensive training of inspectors and installing self-audit mechanisms. Third, the success of institutional reforms of the magnitude proposed for Chile’s education sector requires commitment at the highest political level and, with that, a multi-year commitment of the resources necessary to gradually imple- ment the reform. The education sector reform provides an opportunity to introduce modernization-of-the-public-sector features that would contribute to the effectiveness and efficiency of government institutions. Even if this opportunity is not fully seized, implementing the reform will require a substantial investment of public resources over several years. Making a commitment to the education sector is usually not politically at- tractive, because the short-term costs are likely to outweigh the short-term benefits, as the bulk of benefits would arise in the longer term. The successful implementation of the proposed reforms will require a political leader who has the vision and willingness to assume the costs that are necessary to overhaul the quality of education in Chile and put the country in a path where it will be able to compare itself with Ireland, the Neth- erlands, New Zealand, Scotland or any other high-performing OECD country. Given the magnitude of the reform, its implementation will likely take several years. Indeed, the international experience suggests the need for planning for a gradual, multi-year imple- mentation process. 9 Although the quality of education in Chile compares favorably to that of other Latin American countries, it lags well behind that of OECD countries. Some important reforms have already been introduced in recent years to contribute to educational improvement, including the General Law of Education and the Preferential School Subsidy. The law proposal to strengthen Chile’s education quality assurance system is also aligned with the aim of improving the quality of education for all children. Several challenges lie ahead as Chile moves forward with the implementation of the proposed changes. This report an- ticipates many of these challenges and provides lessons from the international and local experience that could contribute to the success of the institutional reforms. 10 1 Introduction a. Background The need for education reform in Chile Educational quality and equity have become central issues of concern in Chile. Chilean students’ learning outcomes have improved in the past years, both in absolute terms and compared to other countries (PISA 2000 and 2006). However, Chilean students’ performance in international tests of reading, math and science remains well below that of OECD students (PISA 2006). The comparison with the OECD is relevant because Chile aspires to become a developed country (Mineduc 2009, 16). In addition, there is a valid concern about the inequality of learning outcomes. Indeed, among the fifty-four coun- tries that participated in PISA 2006, Chile together with Argentina, Bulgaria and Indo- nesia, exhibited the highest level of disparity in student tests scores (Mineduc 2009). Empirical research has shown that test scores are relatively homogeneous within schools but differ considerably between schools (Murnane, Page and Vegas 2009), a finding that is consistent with research suggesting that the high level of inequality across Chilean students’ learning outcomes is mostly related to schools’ socio-economic segregation (Treviño, Donoso and Bonhomme 2009). In fact, two thirds of municipal schools serve a low- or middle-low-class population, whereas the vast majority of private subsidized schools serve middle- or middle-high-class students (The World Bank 2009). Social pressure to improve the quality and equity of education has mobilized important policy reforms. At the beginning of 2006, massive street protests by high-school students took place, in what became known as “the penguins’ revolution�. These protests cata- lyzed the political debate about how the State should address society’s demand for the availability of quality education services for all children. In this context, two major reforms were introduced and a third reform is being discussed by Parliament: • The Preferential School Subsidy (SEP, for its name in Spanish), a bill that had been developed and proposed to Parliament in 2005, before the student protests, was 11 approved at the beginning of 2008. SEP represents an unprecedented reform in education finance policy. It recognizes the need to award an extra per-student subsidy to children from socio-economically disadvantaged backgrounds and to schools with a high concentration of socio-economically disadvantaged children. At the same time, SEP-participating schools are subject to innovative self-evalua- tion and accountability rules that seek to ensure that the extra funds are focused on raising student learning. • The General Law of Education (LGE), approved in 2009, replaced the Organic Constitutional Law of Instruction (LOCE) which had been enacted during Au- gusto Pinochet’s authoritarian regime. The new law shifts the balance between teachers’ and schools’ right to decide how to impart education and students’ right to receive quality education, in favor of the latter. It sets more strict re- quirements to open a school, and establishes that the State must assure that all schools provide quality education regardless of whether they are managed by the public or private sector (Art. 6). • A bipartisan proposal for a new education quality assurance system was sub- mitted to Parliament in 2008. Several key aspects of this proposal are already acknowledged in the LGE, including the creation of a new Agency for Education Quality and a new Superintendency of Education. The Agency would be respon- sible for evaluating the quality of individual students, teachers and schools, and reporting to the general public on the results of these evaluations. The Superin- tendency would monitor schools’ financial practices and their compliance with laws and regulations. The reform would also require a substantial restructuring of the Ministry of Education, which would assume some new responsibilities and transfer others to the Agency and Superintendency. These reforms signify a change in paradigm in Chile’s education sector, as they entail more State intervention than in the past. In the 1980s, education reforms had aimed toward greater decentralization and privatization in the provision of education. The role of the central government was reduced to establishing rules of entry into the education sector; providing funds on a per-student basis to educational providers complying with these rules; assessing school performance; and making school performance information available to school and parents. The underlying logic was that, if schools are managed by the private sector and local governments, and if parents choose schools based on their quality, schools would compete for students, and this competition would drive improve- ments in educational quality. However, as empirical research has shown, the underlying assumptions of this paradigm are not so obvious. While parents value the quality of education, this is not the main criteria used to choose schools. Instead, schools’ geo- graphic location and the composition of their student population are more important when choosing for which schools to apply. Moreover, the extent to which quality is used to guide school choice decisions is inversely related to parents’ socioeconomic status and educational background (Elacqua, Schneider and Buckley 2006). Three other factors have also prevented pure competition between schools: (i) the fact that schools have 12 exercised the ability to select students, either directly (through achievement tests, parent interviews, background checks) or indirectly (by charging fees that only some families can pay); (ii) the different rules that apply to public and private school teachers; and (iii) in the case of rural or sparsely populated areas, an insufficient supply of private schools. In part because of this imperfect education market, predicted improvements in educational quality have been limited. International comparisons between Chile and high-performing education systems also suggest the need to strengthen the State’s role in education quality assurance. Australia, Canada, Finland, Hong Kong-China, Japan, South Korea, the Netherlands, New Zealand and The United Kingdom, all performed above the average of OECD countries in the PISA 2006. In addition, all of these countries have participated in international assessments of student achievement since 1965 and, since then, have had a constant performance at a high level or have improved over time (Barrera-Osorio 2009). The State’s role in educa- tion quality assurance varies across these countries, with countries like the Netherlands and New Zealand providing relatively more autonomy to schools, and others like South Korea exhibiting higher levels of central government regulation. However, none of these high-performing countries have such a limited State intervention in the education sector as Chile. This conclusion is consistent with the findings of a World Bank report commis- sioned by the government of Chile and published in 2007, and with an OECD report published in 2004. The World Bank’s report identified eight core functions that States in high-performing countries carry out to assure the quality of education, and observed that in Chile the State had historically been responsible for only a few of these functions. The report also highlighted the importance of clearly separating the policy-setting, policy oversight and service provision functions, a feature that was also not present in Chile’s education institutional arrangements. Current situation and proposed changes In this context, Law Proposal No. 609-356, which seeks to strengthen Chile’s education quality assurance system, puts forth that the State must carry out the following eight functions1: 1. Set performance standards for all education sector actors –students, teachers, school principals, schools and school owners. 2. Evaluate the performance of each of these actors. 3. Inform the general public on the performance of these actors. 4. Evaluate the impact of education policies and programs. 5. Set the requirements that actors must comply with to enter and remain in the education sector. 6. Provide technical-pedagogical assistance to front-line providers of education ser- vices. 1 These eight functions are listed in a slightly 7. Ensure that there is adequate funding for education and that it is distributed different order under equitably. Article 2 of Law Proposal 8. Apply performance-based accountability measures. No. 609-356. 13 The proposal also outlines how these functions ought to be distributed across public agencies so that policy-setting remains within the purview of the Ministry of Education, but oversight functions are delegated to the Agency and the Superintendency. Another law proposal, No. 1151-356, which seeks to strengthen public education, es- tablishes the creation of the National Education Service, a central government agency that would implement education policies and programs and provide technical-peda- gogical support to publicly financed schools (including municipal and private-subsidized schools). At the time when this report was written, the parliamentary discussion on Law Proposal No. 609-356 was more advanced than that of Law Proposal No. 1151-356. Two possible reform scenarios are considered in this report: one in which only Law Proposal No. 609- 356 is approved, and another one in which both proposals are approved. b. Objectives Important efforts would be required for the State to assume new functions for education quality assurance, and to restructure the education sector’s institutional setup. This report seeks to inform the process of designing and implementing a new institutional setup for education quality assurance in Chile. To this end, the report draws on the international experience of selected OECD education systems as well as on reform efforts in non- education sectors in Chile. The main questions that are addressed are the following: How are education quality as- surance functions distributed across central government institutions in high-performing OECD systems? How are these functions exercised in each of these systems? What is the relationship between the different institutions that compose an education quality assur- ance system? What lessons emerge from the international and local experience regarding the implementation of institutional reforms? What lessons emerge from the international experience regarding the exercise of education quality assurance functions? c. Inputs for this report The report draws on previous work by The World Bank, Chile’s Ministry of Education and local experts, which was conducted as part of the effort to inform the design and implementation of a new institutional setup for education quality assurance. This work includes: • Detailed written comments by The World Bank on Law Proposal No. 609-356. • Written comments by The World Bank on Law Proposal No. 1151-356. • A comparative report by The World Bank that analyzes the K-12 education qual- 14 ity assurance systems of Ireland, the Netherlands, New Zealand and Scotland to understand how quality assurance functions are distributed across central government institutions; the characteristics of these institutions; and the rela- tionships between them. • A comparative report by The World Bank that analyzes how these four OECD systems have evolved over time; what lessons emerge from their experience im- plementing institutional reforms in the past 15 years; and what lessons emerge regarding the exercise of education quality assurance functions. • Two research papers commissioned to local experts analyzing the implementa- tion of institutional reforms in Chile’s health and criminal justice sectors. • An analysis conducted by Chile’s Ministry of Education to understand its current organizational and human resources capabilities. 15 2 Methodological approach This section begins by describing the criteria used to select OECD education systems and local sectors for the analysis. It also outlines the conceptual frameworks developed and applied to analyze both the design of K-12 education quality assurance systems and the implementation of institutional reforms. The data collection processed is also described. At the end of this section, Table 1 summarizes the methodological approach. a. Selection of OECD systems The Republic of Ireland, the Netherlands, New Zealand and Scotland were selected for review of their education quality assurance systems. These four OECD systems meet the two selection criteria defined jointly by The World Bank, Chile’s Ministry of Education and the Ministry of Finance. First, the central government has formal authority over key edu- cation policy decisions, but the provision of education is decentralized and there is at least some private provision. Second, these countries have demonstrated high performance in international assessments of student learning and, in particular, they all performed better than the average of OECD countries in PISA 2006. b. Selection of local reforms The implementation of local reforms can provide lessons that have been validated in the Chilean social, political and economic context. Two specific reforms were studied: the health reform, which began to be implemented in 2005, and the criminal justice reform, whose implementation began in 1999. These reforms were selected in part because they are relatively recent, and therefore there is enough first-hand “institu- tional memory� to extract lessons on their implementation, but most fundamentally because both sought an improvement in the quality of services provided, and both brought about important institutional reforms in the public sector. In this sense, these 17 reforms have important commonalities with some of the key proposed reforms in the education sector. c. Conceptual framework Design of education quality assurance systems To analyze the design of education quality assurance systems in the selected OECD coun- tries, a conceptual framework was developed and applied. The framework, which builds on previous work by The World Bank (2007), looks into eight education quality assurance functions that may be carried by a national authority; seven education-sector actors who may be affected by the exercise of these functions; and three types of institutions who may carry out these functions. The eight functions, which are also presented in Chile’s Law Proposal No. 609-356, include to: (i) set performance standards; (ii) evaluate performance; (iii) report on per- formance to the general public; (iv) evaluate the impact of education policies and programs; (v) set requirements to enter and remain in the education sector; (vi) provide technical-pedagogical support; (vii) ensure adequate funding and its equitable dis- tribution; and (viii) apply performance-based accountability measures (sanctions and rewards). Each of these functions may affect one or more of the following participants: (i) students; (ii) teachers; (iii) school principals; (iv) schools; (v) school owners; (vi) local educational authorities; and (vii) the national education system. For example, performance standards may be set for students through a compulsory curriculum set at the national level, but there may be no national performance standards for individual schools. Or the perfor- mance of individual teachers and schools may be evaluated on a regular basis by a na- tional authority such as an inspectorate, but performance-based accountability measures may only be applied to schools and not to individual teachers. In addition, each of these functions may be carried out by three different types of institutions: (i) those primarily responsible for setting education policies; (ii) those primarily in charge of overseeing the implementation of those policies as well as compliance with regulations; and (iii) those primarily dedicated to the provision of education services. The selected OECD education quality assurance systems were mapped and analyzed by applying this conceptual framework. The detailed analysis is discussed in a separate re- port, Strengthening the quality assurance system for basic and secondary education in Chile – Comparison of education quality assurance systems and institutions in selected countries (The World Bank 2009a). The report also contains details about the organiza- tional and governance characteristics of national-level institutions involved in education quality assurance in the selected systems. 18 Implementation of institutional reforms To analyze the implementation of institutional reforms in the selected OECD systems, a separate conceptual framework was developed. The framework, which draws on knowl- edge from disciplines such as organizational behavior and public management, classifies national-level education quality assurance institutions into three categories, and looks into six specific dimensions of the implementation of institutional reforms. The three categories of national-level education quality assurance institutions are: (i) new, if an institution was created during the past fifteen years (between 1994 and 2009); (ii) restructured, if an institution has existed for more than fifteen years and was restructured during the past fifteen years; and (iii) long-existing, if an institution has existed, and was not restructured, during the past fifteen-year period. The fifteen-year cutoff was chosen to maximize the number of institutions that could be analyzed as new or restructured, while at the same time ensuring that there was enough institutional memory within or- ganizations so as to share knowledge of these implementation processes. The framework puts forth that implementation of institutional reforms can affect one or several of the following six dimensions: (i) the reform design process, including the rationale for reform, the actor(s) who had the initiative to introduce it, the actors in- volved in the reform design process and the extent of consultation during this process; (ii) the chronology of the implementation process, which refers to the duration of the implementation process, planning for the sequence of implementation, and the extent to which these plans, if available, were followed; (iii) general management practices and management of organizational change, which includes issues such as the dissemination of the reform, leadership, organizational culture, governance and division of responsibil- ity and authority within the organization, monitoring and evaluation of individual units, and coordination with other organizations; (iv) management of human resources, which involves policies to attract, recruit and retain personnel, initial training and continuous professional development, employment status, compensation and non-salary benefits, individual performance evaluation, the existence and content of a Code of Ethics, and any policies on personnel distribution; (v) management of financial and information resources, which includes the management of budgets and information systems; and (vi) institutional accountability and evaluation, which refers to the internal and external mechanisms that hold the organization accountable for its performance as well as any systematic efforts to evaluate the reform. Which of these dimensions may be most affected by a reform process is related to an institution’s category. In new organizations, implementation processes can be expected to affect all six dimensions. In restructured organizations, the process of designing the restructuring process is relevant, but the restructuring in itself may affect only one di- mension (e.g., general management practices), several, or all of them. In long-existing organizations, by definition, changes are not expected in any of these dimensions, but still it is important to characterize how these organizations operate (i.e., their current general, human, financial and information management practices, as well as account- ability mechanisms in place). 19 In addition to the analysis of the past fifteen years of institutional reforms, for each selected OECD system the historical evolution of the education quality assurance institu- tions was also reviewed. d. Data collection To analyze the design of education quality assurance systems in the selected OECD coun- tries, data were collected from individual institutions’ websites, official documents, Eury- dice (a database on education systems in Europe) and the International Review of Cur- riculum and Assessments Frameworks (INCA) online archive. To analyze the implementation of institutional reforms in these systems, three types of information sources were used. First, distinct questionnaires were developed for new, restructured and long-existing organizations. The World Bank team identified key infor- mants from each of these institutions, who generously responded to the questionnaires. Second, follow-up interviews were conducted to senior staff in some of the institutions. Third, relevant academic literature, official documents, press releases and transcripts of speeches delivered by key actors were reviewed. Throughout the analysis stage, extensive feedback was provided by staff at Ireland’s Edu- cational Research Center; the Netherlands’ Ministry of Education, Culture and Science, and Inspectorate of Education; New Zealand’s Ministry of Education, Education Review Office, and Qualifications Authority; and Scotland’s General Teaching Council, and Learn- ing and Teaching Scotland. The World Bank is very grateful to the individuals from these institutions who so generously devoted their time and contributed to ensure the veracity of the information and analyses contained in this report. 20 e. Summary Table 1. Methodological approach Objective Analysis Unit of analysis Conceptual framework Data collection Reference To inform the design Design of K-12 • Ireland 8 education quality assurance functions: • Institutions’ The World Bank and implementation education • The Netherlands (i) set performance standards; (ii) evaluate performance; websites (2009a) of institutional quality assurance • New Zealand (iii) report on performance; (iv) impact evaluation of policies and programs; • Official documents reforms in Chile’s systems • Scotland (v) set requirements to operate; (vi) provide technical-pedagogical support; • Eurydice education quality (vii) ensure adequate and equitable funding; (viii) apply accountability • INCA assurance system measures based on performance. • Other literature (EQAS) 7 actors: (i) students; (ii) teachers; (iii) school principals; (iv) schools; (v) school owners; (vi) local educational authorities; (vii) national system. 3 types of institutions: (i) policy-setting; (ii) oversight; (iii) provision. Implementation • Ireland, 6 aspects of institutional reform: • Questionnaires The World Bank of institutional • The Netherlands, (i) reform design process; (ii) chronology of implementation; (iii) general • Interviews (2009b) reforms • New Zealand, management practices; (iv) human resources management; (v) management • Written documents • Scotland of financial and information resources; (vi) institutional accountability and evaluation. • Chile’s health • Interviews Bitrán, Gómez 3 types of institutions: and Vergara and justice sector • Written documents (i) new; (ii) restructured; (iii) long-existing. (2009) and reforms Vargas (2009) • Chilean Ministry Mapping of 8 education quality assurance functions across the Mineduc of Education’s Ministry’s units and staff (unpublished) capabilities 21 What dimensions of the 3 selected OECD education quality assurance systems and recent history are most relevant to the proposed reforms in Chile? This section describes briefly what institutions are involved in K-12 education quality as- surance in the selected OECD systems, and how the eight quality assurance functions are distributed across institutions. The section also draws attention to specific dimensions of the design and recent history of these systems that might be particularly relevant to reforms proposed for Chile. An important issue to note is that, in all the systems, the exercise of quality assurance functions by the State affects public and private subsidized schools alike. The comparison between OECD systems and Chile is useful but must be made with cau- tion, as the context in which Chilean reforms would be introduced differs considerably from the context in the selected systems. First, the Chilean government has to provide services and ensure the quality of education within a larger territory: Chile is almost three times as large as New Zealand, ten times as large as Scotland and Ireland, and twenty-two times as large as the Netherlands. Second, reaching all the members of the population is particularly difficult in Chile because individuals tend to be more sparsely distributed than in the countries selected for comparison (with the exception of New Zealand). Third, although public investment in education has reached an unprecedented level in Chilean history, the country’s capacity to further invest in education and strengthen its quality assurance system faces a limit imposed by its income, which is much smaller than in se- lected countries. Fourth, evidence that public-sector governance in Chile is weaker than in the countries selected for comparison suggests that investments in education may be less efficient and/or less effective in the former. Fifth, the education sector in Chile serves a bigger number of students with relatively fewer teachers. 23 Demographic, economic, governance and education Table 2. differences between Chile and the selected OECD systems Republic of New Chile The Netherlands Scotland Ireland Zealand 1. Demography and economy Territory (‘000 sq.km)* 749 69 34 268 76 Population (millions)* 16.6 4.4 16.4 4.2 5.1 Density (inhab/sq.km)* 22 63 484 16 67 GDP per capita 13,108 40,168 36,580 25,306 32,766 (PPP, US dollars, 2005 prices)* 2. Public sector governance Participation and accountability 1.0 1.4 1.5 1.4 1.3 (-2.5 to 2.5; 0=world avg.)** Public sector employees’ capacity (1=below private sector; 7=above private 2.3 3.6 3.4 4.0 3.5 sector)*** Public sector efficacy 1.3 1.6 2.0 1.7 1.9 (-2.5 to 2.5; 0=world avg.)** Rule of law 1.2 1.6 1.8 1.7 2.0 (-2.5 to 2.5; 0=world avg.)** 3. Education system (primary and secondary)**** Students 3,574,419 822,373 2,598,000 759,906 712,298 Teachers 174,882 67,804 219,900 53,366 53,563 Schools 11,420 4,023 8,181 2,593 2,722 Sources: * World Bank, World Development Indicators. ** World Bank, Worldwide Governance Indicators. *** World Economic Forum, The Global Competitiveness Report. **** Mineduc, Anuario Estadistico 2007; Department of Education and Science, Education Statistics 2007/2008; Dutch Ministry of Education, Culture and Science, Key Figures 2004-2008; NZ Education Counts, Education Statistics of New Zealand 2007; Scottish Government, School Education Statistics 2008. 24 a. Brief overview of the selected OECD education quality assurance systems Post-reform Chile Given that two of the law proposals that have been submitted to Parliament could have implications for Chile’s education quality assurance system, two reform scenarios are considered. Under one scenario (which we refer to as “scenario 1�), Law Proposal No. 609-356 would be approved but Law Proposal No. 1151-356 would not, and therefore the education quality assurance system would be composed of four national govern- ment institutions: the Ministry of Education, which would be restructured; the Agency for Education Quality, which would be newly created; the Superintendency of Education, which would also be created; and the National Education Council, which would be a restructured version of the existing Superior Education Council. Under another scenario (“scenario 2�), both law proposals would be approved, and the education quality as- surance system would be composed of five national government institutions: the four already mentioned plus the National Education Service, which would be newly created. Figure 1 depicts the system’s institutional setup under each possible scenario. Chile’s education quality assurance institutions under two alternative reform Figure 1. scenarios Escenario 1 Escenario 2 Consejo Nacional de Educación Consejo Nacional de Educación Policy and Ministry of Education Ministry of Education Oversight Agencia de Superintendencia Servicio Agencia de Calidad Superintendencia Calidad de la de Educación Nacional de de la Educación de Educación Educación Educación “Sostenedores� “Sostenedores� (municipalidades, Corporaciones Locales de Provision (municipalidades o entidades privadas) Educación Pública o entidades privadas) Schools Schools 25 The distribution of quality assurance functions across institutions is depicted in Table 3. As can be observed, under both scenarios the Ministry of Education would be responsible for important policy decisions, including: to set performance goals for students, teachers, school principals, schools and school owners; to determine the requirements to operate as a school and to enter and remain in the teaching profession; and to provide funding Table 3.Distribution of functions across education quality assurance institutions in Chile National Agency for National Ministry of Superintendency of Educational Education Education Education Education Council Quality Service 1. Sets performance Yes No, but advises No No No standards No, but monitors schools/ 2. Evaluates performance No No, but advises Yes No owners 3. Reports on performance Yes No Yes Yes No 4. Evaluates the impact of Yes2 No No No No policies and programs 5. Sets requirements to Yes No No Yes No operate for municipal and private subsidized schools. Also under both scenarios, the Agency for Education Quality would be responsible for evaluating students, teachers, school princi- pals, and municipal and private subsidized schools, and for reporting to the general pub- 2 In addition to the Ministry of Education’s lic on these actors’ performance. These publicly available reports would serve as a repu- responsibility to evaluate tational reward or sanction for educational providers. The Superintendency of Education the impact of education would oversee how schools and school owners manage their resources and whether they policies and programs, the Ministry of Finance’s comply with statutory regulations, and it would intervene and/or apply other sanctions to Budget Office also under-performing schools. The National Education Council would have veto power over assesses the impact of some public-sector the performance goals developed by the Ministry of Education; the national curriculum; programs, including education programs and plans; the national evaluation plan; and national assessment some education sector policies. The Council would also advise the Ministry on policy matters; promote educa- programs. These impact evaluation studies are tional research and debate; and exercise quality assurance functions for higher education. usually commissioned to The main difference between scenarios 1 and 2 concerns the responsibility for providing external consultants. A similar role is played by technical-pedagogical support to municipal and private subsidized schools. This respon- the Netherlands’ Ministry sibility would fall under the purview of the Ministry of Education under scenario 1, and of Economic Affairs and under the National Education Service under scenario 2. New Zealand’s Treasury. 26 National Agency for National Ministry of Superintendency of Educational Education Education Education Education Council Quality Service No under 6. Provides/funds Yes under scenario 1; technical-pedagogical scenario 1; No No No Yes under scenario support No under scenario 2 2 7. Ensures adequate funding and its equitable Yes No No No No distribution 8. Applies performance- based accountability Yes No Yes Yes No measures Based on the three proposed categories for analyzing institutional reforms, the institu- tions that would be involved in education quality assurance in Chile could be classified as follows: Table 4. Evolution of education quality assurance institutions in Chile Ministry of Education Restructured National Education Council Restructured Agency for Education Quality New Superintendency of Education New National Education Service New 27 The proposed reforms entail two important features worth highlighting: • The responsibility by the State of four additional quality assurance functions, in- cluding to: (i) set standards for all teachers, school principals, schools and school owners; (ii) evaluate individual school principals, schools and school owners; (iii) provide technical-pedagogical support; and (iv) apply performance-based ac- countability measures. Some of these functions, such as the setting of standards or the evaluation of performance, are currently carried out by the Ministry of Education but affect only some participants (e.g., municipal ones, or those par- ticipating in SEP). The proposed reforms would require an extension of these functions to cover all participants. Other functions, such as the provision of technical-pedagogical support or the application of accountability measures, are new altogether. Although SEP was expected to serve as a pilot for the exercise of these functions, in practice, the systematic implementation of these functions has not yet taken place. • A profound reform of the institutional setup of the education sector. Accompa- nying the taking over of new quality assurance functions by the State, the insti- tutional setup of the education sector would also be altered profoundly. Under scenario 1, two institutions would be restructured (the Ministry and the Council) and two institutions would be newly created (the Agency and the Superinten- dency). Under scenario 2, in addition to these two institutions, a third institution would be created (the National Education Service). Republic of Ireland Figure 2 depicts Ireland’s current institutional setup for education quality assurance, and Table 5 shows how quality assurance functions are distributed across national-level institu- tions. Historically, all education quality assurance functions were concentrated under the purview of the Department of Education and Science (DES). The DES remains the central authority in Ireland’s quality assurance system, but over time it has delegated certain re- sponsibilities to subsidiary and affiliated organizations. In particular, since 2001, it has transferred: the administration of national examinations to the State Examinations Com- mission (SEC); supervision over the quality of teaching, the standards to enter and remain in the profession, and the provision of technical-pedagogical support to teachers to the Teaching Council; and the coordination and implementation of educational qualifications to the National Qualifications Authority (NQA). In addition, the National Council for Cur- riculum and Assessment (NCCA), which exists since 1987, was recognized as a statutory body in 2001, and provides advice to the DES and to educational providers on matters related to the curriculum, teaching and student assessment. The SEC, NCCA and Teaching Council are accountable to the DES, while the NQA is also accountable to the Depart- ment of Enterprise, Trade and Employment. The Inspectorate is a division of the DES that answers directly to the Secretary and Minister and is responsible for overseeing the quality of education provided by individual institutions. 28 Figure 2. Ireland’s education quality assurance institutions The Departament of Departament of Education and Science Enterprise, Trade Inspectorate Policy and and Empleoyment Oversight National Council National State Examinations Teaching for Curriculum & Qualifications Commission Council Assessment Authority Boards of Management Provision Schools Table 5. Distribution of functions across education quality assurance institutions in Ireland National Department of Council for State National Education Teaching Education and Curriculum Examinations Qualifications Inspectorate Council Science and Commission Authority Assessment 1. Sets performance No, but advises Yes No No No, but advises Yes standards DES 2. Evaluates No Yes No Yes No No performance 3. Reports on Yes Yes No Yes No No performance 4. Evaluates the impact of policies and Yes No No No No No programs 5. Sets requirements to Yes No No No No Yes operate 6. Provides/funds technical-pedagogical Yes No Yes No No Yes support 7. Ensures adequate funding and its Yes No No No No No equitable distribution 8. Applies performance- based accountability Yes Yes No Yes No Yes measures 29 Based on the system’s evolution over the past fifteen years, the institutions involved in education quality assurance can be classified as follows: Table 6.Evolution of education quality assurance institutions in Ireland (1994-2009) Department of Education and Science Restructured Education Inspectorate Long-existing National Council for Curriculum and Assessment Long-existing State Examinations Commission New National Qualifications Authority New Teaching Council New The design of Ireland’s education quality assurance system differs from the institutional setup proposed for Chile in several ways. In contrast with Chile, Ireland has specialized organizations for: the administration of national examinations (the SEC); the regulation of the teaching profession (the Teaching Council); and the establishment of qualifications for all levels and types of education (the NQA). In addition, Ireland’s Education Inspec- torate is not independent from the Department of Education and Science. Still, Ireland’s education quality assurance system and its recent evolution present three special features relevant to the reforms proposed in Chile: • The restructuring of the DES during the 1990s. In the mid-1990s the national government put forth a State modernization initiative that was to be implement- ed by all Departments, including the DES. The underlying aim of this initiative was to improve the efficiency and effectiveness of the public sector and, with that, the quality of service delivery and the contribution of the public sector to economic development. Profound changes took place as a result of that initia- tive: (i) the public sector culture of secrecy gradually evolved into one of greater openness and transparency; (ii) within the DES, each unit’s objectives were speci- fied in writing, targets were defined to measure their achievement, and regular performance evaluation of individual units was introduced; (iii) emphasis was placed on improving customer service, and mechanisms were introduced for more fluent communication with schools and to increase responsiveness to their needs and concerns; (iv) human resource policies underwent several changes aimed at making public sector employment an attractive option to qualified pro- fessionals; and (v) the importance of leadership within the DES was recognized. 30 While it is not possible to assess the extent to which these changes contributed to the improvements observed in Irish students’ educational outcomes, there is a generalized sense within the public sector that State modernization is neces- sary to bring about improvements in the quality of education services (Cromien Report 2000). • The 2001 reforms, which included the restructuring of the DES and creation of the SEC, NQA and Teaching Council. The 2001 reforms were triggered by a diagnosis that the DES was overburdened with daily operational work and had little time left to focus more broadly on education policy issues. In this context, many operational tasks were delegated to new institutions such as the SEC and the NQA. At the same time, the Teaching Council was created in response to teachers’ demand for greater authority over the regulations of their own profes- sion. These changes required important implementation efforts. However, they differ from the reform scenarios faced by Chile in a crucial way: in Ireland, no new quality assurance functions were captured by the State. Instead, functions that were already being carried out within the DES were transferred to new institutions. Moreover, the reforms were announced in 2001, but their imple- mentation has not yet been completed, and therefore extracting lessons from this implementation experience is relatively difficult. • Changes in the Inspectorate’s role from the 1990s on. Like in other parts of Eu- rope, the role of the Education Inspectorate has changed substantially since the 1990s. Two changes that are particularly relevant for the proposed reforms in Chile include: (i) during the 1990s, the taking over of the responsibility to evalu- ate the overall quality of individual schools; and (ii) since 2006, the dissemina- tion of public reports on the individual school performance. The Netherlands Figure 1 depicts the current institutional setup for education quality assurance in the Netherlands, and Table 7 shows how quality assurance functions are distributed across national-level institutions. The Dutch institutional setup is similar to that proposed for Chile, especially under scenario 1. The Ministry of Education, Culture and Science is in charge of setting policies and regulations, including setting standards for students, teach- ers, schools and school owners, and determining the entry requirements to operate in the education sector. It also administers the public budget for education; provides funding for schools to contract with independent providers of technical-pedagogical support; and is responsible for the impact evaluation of policies and programs, most of which are com- missioned to independent researchers. The Education Council is the main advisory body on education policy matters. It participates in the process of setting performance stan- dards and provides solicited and unsolicited advice to the Minister of Education, Parlia- ment and, sometimes, municipal governments. The Inspectorate of Education is a semi- independent agency which is funded by, and reports to, the Ministry but has autonomy 31 in determining how to carry out its functions. It is responsible for overseeing compliance with policies and regulations, and for evaluating, promoting and reporting on the quality of education provided by individual schools and by the system as a whole. The Central Funding of Institutions Agency, another semi-independent agency, is in charge of chan- neling public funds to schools’ competent authorities, providing relevant policy informa- tion to schools, and responding to providers’ questions or concerns regarding the impli- cations of policies and regulations. Although it does not have a responsibility to monitor schools’ financial practices, it is in an advantageous position to do so informally due to its daily and financial relationship with schools, and regularly informs the Inspectorate in cases where there is reason for concern. Figure 3. The Netherlands’ education quality assurance institutions Education Council Ministry of Education, Culture and Science Policy and Oversight Education Central Funding of Inspectorate Institutions Agency Competent Authorities Provision Schools 32 Distribution of functions across education quality assurance institutions in the Table 7. Netherlands Ministry of Central Funding Education Inspectorate of Education, Culture of Institutions council Education and Science Agency 1. Sets performance standards Yes No, but advises No No 2. Evaluates performance Yes No Yes No 3. Reports on performance Yes No Yes No 4. Evaluates the impact of policies and Yes No No No programs 5. Sets requirements to operate Yes No No No 6. Provides/funds technical- Yes No No No pedagogical support 7. Ensures adequate funding and its Yes No No Yes equitable distribution 8. Applies performance-based Yes No Yes No accountability measures Based on the system’s evolution over the past fifteen years, the institutions involved in education quality assurance can be classified as follows: Evolution of education quality assurance institutions in the Table 8. Netherlands (1994-2009) Ministry of Education, Culture and Science Restructured Education Council Long-existing Inspectorate of Education Restructured Central Funding of Institutions Agency Restructured 33 While the Netherlands’ institutional setup for education quality assurance is similar to that proposed for Chile, a key difference between them is that the current setup in the Netherlands is built on a two-century history of education quality assurance. Indeed, the Dutch Inspectorate of Education has been in place since 1801, even preceding the creation of the Ministry of Education. The most recent major institutional reform took place in 1994, when the Department of Education, which had grown and taken on many responsibilities during the 1970s, was divided into three agencies: the Ministry of Edu- cation, Culture and Science; the Inspectorate of Education; and the Central Funding of Institutions Agency. However, no new quality assurance functions were captured by the State during that reform. On the contrary, the reform brought with it a reduction of State intervention in favor of increased school autonomy. Despite these differences, the Dutch system and its recent evolution present several fea- tures which are particularly relevant to the reforms proposed in Chile: • Changes in the Inspectorate’s role during the 1990s. Four important changes took place within the Inspectorate during the 1990s: (i) an increase in the In- spectorate’s relative independence to carry out its functions; (ii) a new balance of functions in which relatively even attention was put to checking compliance with regulations, evaluating the quality of the education system and of individ- ual schools, and reporting on the state of education to Parliament, the Ministry and the general public; (iii) a new set of actions to carry out these functions, in- cluding the development and application of school evaluation frameworks, and the publication of school report cards; and (iv) a new relationship with schools, in which a school would remain largely autonomous as long as the Inspectorate judged that it was providing quality education. Many of these changes provide valuable learning opportunities for Chile, as they map well onto the proposed functions of the Agency for Education Quality. • Creation of the Knowledge Chamber within the Ministry of Education. The Knowledge Chamber was created in 2006 to increase the link between policy- making and educational research, by facilitating regular consultation between top-ranking officials and the managers of various knowledge institutions. Every spring, policymakers and researchers meet to plan the “knowledge agenda�, which is then reflected in the working plan of knowledge institutions. Every autumn, they meet again to discuss research findings and their policy implica- tions (Stegeman and Rouw 2007). The lessons from this experience may be particularly relevant for the Chilean Ministry of Education as it considers options to carry out its responsibility to evaluate the impact of education policies and programs, an important aspect of quality assurance. • Restructuring of the Inspectorate in 2008, including taking over the responsibil- ity to monitor schools’ financial practices from the Ministry. The practices of the Inspectorate’s financial oversight department may be relevant to Chile as they relate to some of the functions that would be carried out by the Superinten- dency of Education. 34 New Zealand Figure 4 depicts New Zealand’s current institutional setup for education quality as- surance, and Table 9 shows how quality assurance functions are distributed across national level institutions. The structure of New Zealand’s education system is highly decentralized with several agencies providing complementary services in the education quality assurance system. The Minister of Education and the Minister Responsible for the Education Review Office, who are appointed by the national Parliament, are the two main heads of education agencies. Agencies that fall under the purview of the Minister of Education include: the Ministry of Education, the New Zealand Qualifica- tions Authority (NZQA), and the New Zealand Teachers Council (NZTC). The Education Review Office (ERO) falls under the purview of its own Minister and, therefore, retains substantial independence. These agencies provide support to schools and relate to the individual school boards. The Ministry of Education sets education policies, evaluates their impact, develops national curriculum frameworks, and administers the education sector’s public budget. Part of this budget is used to fund the provision of technical- pedagogical support by independent, accredited institutions. The NZQA oversees the national examination system and sets standards for the National Qualifications Frame- work. The NZTC registers teachers and sets standards for teacher performance and qualifications. It also participates in research and other projects to support teachers and is in charge of accrediting initial teacher education programs. The ERO evaluates the performance and quality of schools and their teachers and informs the general public about the outcomes of these evaluations. Figure 4. New Zealand’s education quality assurance institutions Minister Minister of Education Responsible for Policy and the ERO Oversight Ministry of New Zealand New Zealand Education Education Qualifications Teachers Review Office Authority Council Boards of Trustees Provision Schools 35 Table 9.Distribution of functions across education quality assurance institutions in New Zealand New Zealand Ministry of New Zealand Education Qualifications Education Teachers Council Review Office Authority 1. Sets performance standards Yes Yes Yes No 2. Evaluates performance No Yes Yes Yes 3. Reports on performance Yes Yes No Yes 4. Evaluates the impact of policies and Yes No No No programs 5. Sets requirements to operate Yes No Yes No 6. Provides/funds technical-pedagogical Yes No Yes No support 7. Ensures adequate funding and its Yes No No No equitable distribution 8. Applies performance-based Yes Yes Yes Yes accountability measures Based on the system’s evolution over the past fifteen years, the institutions involved in education quality assurance can be classified as follows: Evolution of education quality assurance institutions in New Table 10. Zealand (1994-2009) Ministry of Education Long-existing New Zealand Qualifications Authority Long-existing New Zealand Teachers Council Long-existing Education Review Office Long-existing 36 A major institutional reform took place in 1989, when the Department of Education was replaced by the four agencies that are currently involved in education quality assurance: the Ministry of Education, the NZQA, the NZTC and the ERO. The reform sought to sepa- rate policy-setting functions from policy oversight and provision of support to educational providers. More importantly, the reform was accompanied by a change in the role of the State in education quality assurance. Since then, New Zealand has been characterized by a system in which schools remain largely autonomous as long as they comply with educa- tion regulations and performance standards. When they do not comply with established regulations and standards, the State ensures intense provision of support services and may resort to intervention (and ultimately closure) of under-performing schools. New Zealand’s system and its recent evolution present several features which are particu- larly relevant to the reforms proposed in Chile: • The 1989 reform. Of all the reforms analyzed, the 1989 reform in New Zealand is the one that most resembles the nature of the reform proposed in Chile, be- cause it entailed a major overhaul of the institutional setup for education quality assurance and, at the same time, signified a substantial change in the State’s role in the education sector. Unfortunately, because it took place twenty years ago, the institutional memory on the implementation of the reform is limited. • Evolution of the Education Review Office. The ERO, one of the products of the 1989 reform, deserves in-depth study as many of its functions map well with those that would be covered by the Agency for Education Quality. Particularly relevant aspects to consider include how the ERO relates to schools, how it built its legitimacy and credibility, how it relates to the Ministry and how it ensures its independence from it. 3 The Scottish Lifelong Scotland Learning Directorate does not play a role in As shown in Figure 5, five major national government institutions are responsible for educa- K-12 education quality tion quality assurance in Scotland. While the Scottish Executive Education Department retains assurance. It is involved in the development the policy-setting functions, over time it has delegated the responsibility for specific areas of policies related of education to specialized organizations. Her Majesty’s Inspectorate of Education (HMIE) is to adult education, community learning responsible for school inspections, the dissemination of information about individual schools’ and development, and quality, and the identification of promising practices that may help to improve the quality of education and training education at the school and local levels. Learning and Teaching Scotland (LTS) is responsible of individuals who are not in the formal for developing the curriculum and providing professional development options for teachers education system and and school principals. The General Teaching Council (GTC) is responsible for accrediting initial lack the skills to become employed. It has been teacher education programs, setting standards to enter and remain in the teaching profession, included in Figure 5 and registering qualified teachers. It also administers the tools used to evaluate teachers who to illustrate that the wish to be recognized as Chartered Teachers, a recognition which leads to a substantial salary Scottish Qualifications Authority reports to both increase not by promotion to a school management role but by staying in the classroom. The the Scottish Executive Scottish Qualifications Authority (SQA) is responsible for the development and assessment of Education Department and the Scottish Lifelong qualifications for students through upper secondary school and beyond. It reports to both the Learning Directorate. Scottish Executive Education Department and the Scottish Lifelong Learning Directorate3. 37 Figure 5. Scotland’s education quality assurance institutions Scottish Lifelong Scottish Executive Education Department Learning Schools Directorate Directorate Policy and Oversight HIM´s Learning and General Teaching Scottish Inspectorate Teaching Scotland Council Qualifications of Education Authority Local Educational Authorities Provision Schools Table 11. Distribution of functions across education quality assurance institutions in Scotland Scottish Learning and Scottish General Her Majesty’s Executive Teaching Qualifications Teaching Inspectorate Education Scotland Authority Council of Education Department 1. Sets performance standards Yes Yes Yes Yes No 2. Evaluates performance Yes Yes Yes Yes Yes 3. Reports on performance Yes No Yes No Yes 4. Evaluates the impact of policies Yes Yes No No No and programs 5. Sets requirements to operate Yes No No Yes No 6. Provides/funds technical- No Yes No Yes Yes pedagogical support 7. Ensures adequate funding and its Yes No No No No equitable distribution 8. Applies performance-based Yes No No Yes Yes accountability measures 38 Based on the system’s evolution over the past fifteen years, the institutions involved in education quality assurance can be classified as follows: Evolution of education quality assurance institutions in New Table 12. Zealand (1994-2009) Scottish Executive Education Department Long-existing Learning and Teaching Scotland Restructured Scottish Qualifications Authority Restructured General Teaching Council Restructured Her Majesty’s Inspectorate of Education Long-existing Although Scotland differs from Chile in size, it offers important insights for Chile: • The SEED has established an efficient system for providing quality education. Rather than expanding the national education department, SEED has chosen to provide key services through organizations that report to independent boards and take professional responsibility for supporting teaching and learning, the induction and professional competence of teachers, and the qualifications that students and adult take to demonstrate their competencies. To oversee the sys- tem and ensure quality in all schools, the inspectorate reviews all schools and lo- cal education authorities and makes recommendations for improvements. Thus, the government has established a strong focus on quality with the majority of the support provided by non-departmental entities and local education authori- ties. • Quasi-independent nature of the quality assurance entities ensures public trust. Each quality assurance agency has an independent board. HMIE enjoys a strong reputation for the quality of its inspections and its focus on improving outcomes for students. The publication of all inspection reports allows parents and com- munity members to gauge the quality of their institutions. The GTC is highly respected by teachers for the fairness of its processes for standards setting, ap- peals, and may soon become a completely independent entity, transferring au- thority for licensure and induction from the government to the profession, as has been the practice in other careers, including law and medicine. • Collaboration among quality assurance agencies leads to greater coherence of the instructional system. When the inspectorate reviews schools, it records in- 39 formation on promising practices and weaknesses in seven areas: curriculum, attainment, learning and teaching, support for pupil, ethos, resources and man- agement. Within each area, HMIE categorizes school performance into: major strengths, strengths outweigh weaknesses, significant weaknesses, or major weaknesses. Descriptions are provided for areas deemed very good and fair. This information is made available to LTS and the GTC to guide the develop- ment of their products and services. Similarly, the results of assessments carried out by the SQA are analyzed by the other three agencies to identify general areas in need of improvement in teaching and learning as well as specific needs of individual schools. In addition, cross-training of personnel from these agen- cies ensures that they understand the standards of quality in each and share a common vision of educational quality. This facilitates a continuous improvement process that contributes to strengthen teaching and learning and overall school performance. • The entire education system, both public and private, falls under the purview of the Ministry (SEED). If HMIE determines that a school or local educational au- thority has not made progress despite the recommendations and support it has received, the Scottish ministers have the statutory authority to step in to require improvement actions. Private independent schools must also demonstrate qual- ity to receive and maintain licensure from the government. b. Summary What dimensions of the selected OECD education quality Table 13. assurance systems and recent history are most relevant to the proposed reforms in Chile? Post-reform Chile Republic of The Netherlands New Zealand Scotland Ireland Scenario 1 Scenario 2 Distribution of education quality assurance functions across national-level institutions 1. Sets perfor- DES Ministry Ministry SEED Ministry mance stan- Advice of NCCA Advice of Education NZQA LTS Advice of National Education dards and NQA Council NZTC SQA Council Teaching Council GTC 2. Evaluates Inspectorate Ministry NZQA SEED Agency performance SEC Inspectorate NZTC LTS Superintendency ERO SQA Advice of National Education GTC Council HMIE 40 Post-reform Chile Republic of The Netherlands New Zealand Scotland Ireland Scenario 1 Scenario 2 3. Reports on DES Ministry Ministry SEED Ministry performance Inspectorate Inspectorate NZQA SQA Agency SEC ERO HMIE Superintendency 4. Evaluates DES Ministry Ministry SEED Ministry the impact of LTS policies and programs 5. Sets requi- DES Ministry Ministry SEED Ministry rements to Teaching Council NZTC GTC Superintendency operate 6. Provides/ DES Ministry Ministry LTS Ministry National funds technical- NCCA NZTC GTC Education pedagogical Teaching Council HMIE Service support 7. Ensures ade- DES Ministry Ministry SEED Ministry quate and equi- CFIA table funding 8. Applies per- DES Ministry Ministry SEED Ministry formance-based Inspectorate Inspectorate + NZQA GTC Agency accountability SEC NZTC + HMIE Superintendency measures Teaching Council ERO Classification of national-level education quality assurance institutions based on their evolution during 1994-2009 New SEC Agency for Education Quality NQA Superintendency of Education Teaching Council National Education Service Restructured DES Ministry of Education LTS Ministry of Education Inspectorate of Edu- SQA National Education Council cation GTC CFIA Long-existing Education Inspec- Education Council Ministry of SEED torate Education HMIE NCCA NZQA NZTC ERO 41 Post-reform Chile Republic of The Netherlands New Zealand Scotland Ireland Scenario 1 Scenario 2 Most relevant features of the education quality assurance system and its recent history • Restructuring • Changes in the • 1989 re- • Quasi- • The responsibility by the State of the DES Inspectorate’s role forms. independent of four additional quality assu- during the during the 1990s. • Evolution of nature of the rance functions. 1990s. • Creation of the the Educa- quality assu- • A profound reform of the insti- • 2001 reforms, Knowledge tion Review rance entities. tutional setup of the education which included Chamber within Office. • Collaboration sector. restructuring the Ministry of among quality of the DES and Education. assurance creation of • Restructuring agencies. the SEC, NQA of Inspectorate • The entire and Teaching in 2008, taking education Council. over the respon- system, both • Changes in the sibility to monitor public and Inspectorate’s schools’ financial private, falls role from the practices. under the 1990s on. purview of SEED. 42 43 44 Implications of the 4 analysis for the proposed reforms in Chile This section outlines the main lessons that emerge from the analyses conducted to inform the design and implementation of quality assurance reforms in Chile. The lessons are drawn from our analyses (detailed in previous reports) of: selected OECD education qual- ity assurance systems and reforms; the implementation of institutional reforms in Chile’s health and justice sectors; and the organizational and human resources capabilities that exist within the Ministry of Education. a. Implications for the system as a whole i. Design of a new institutional setup for education quality assurance Distribution of education quality assurance function across national-level institutions Education quality assurance functions are distributed across four national-level institu- tions in both the Netherlands and New Zealand, five institutions in Scotland, and six in Ireland. From a design point of view, this is in line with the institutional setup proposed for Chile: under scenario 1, Chile would have four national level institutions involved in education quality assurance, and under scenario 2 it would have five. However, Chile faces two key challenges that have not been faced by the selected OECD systems over the past fifteen years: a change in paradigm regarding the State’s role in the education sector, and the need to implement simultaneously profound institutional reforms in order to put in place the proposed arrangements for education quality assurance. 45 In addition, in the four OECD systems, there is a clear separation between policy-setting and policy-overseeing institutions. Typically, the Ministry or Department of Education is responsible for setting education policies and regulations, and the Inspectorate both oversees schools’ compliance with these regulations and provides an independent judg- ment of how the policies set by the Ministry affect schools’ day-to-day operations. This separation is not so clear-cut under the reform scenarios faced by Chile. In particular, the Superintendency of Education, would not only oversee schools’ and school owners’ com- pliance with educational laws and regulations, but would also set the binding regulations with which schools and school owners would have to comply in order to participate in the education sector and avoid sanctions.1 This is in contrast with the experience of the selected systems, where the Inspectorate oversees compliance with education regulations but does set those regulations. Table 14 lists the public-sector, national-level institutions involved in education quality as- surance in each system. The following design features are worth noting: • In the four OECD systems selected for review, a single institution is formally responsible for oversight of the education sector. In contrast, Chile would have two institutions with oversight responsibilities: the Superintendency, which would oversee schools’ and school owners’ financial practices; and the Agency, which would oversee the quality of education. These two functions are carried out within the same Inspectorate in the selected OECD systems. • None of the OECD systems reviewed carry out the performance evaluation function within a single agency. Typically, the evaluation of schools is per- formed by the Inspectorate, whereas student assessments are developed and applied by a different agency. In Ireland, there is a specialized agency in charge of determining the qualifications that must be met by students at the different levels of education (the NQA), and another agency administers the examinations that measure whether students have met the required qualifica- tions or not (the SEC). In New Zealand and Scotland, the determination of the qualifications and the administration of national examinations fall under the purview of the same national qualifications authority. In the Netherlands, qualifications are determined by the Ministry of Education, and examinations are administered by a private institution, CITO. These systems’ arrangements are different from those proposed for Chile, where the Agency would be in 1 “The object of the Superintendency will charge of evaluating both students and schools. While this design may make be to monitor the use sense for Chile today because it implies more manageable implementation of resources by school owners and schools that efforts, it is worth noting this difference to inform future directions in insti- are officially recognized tutional reform. by the State, and their compliance with the • Most selected OECD systems have a teacher council2 which is responsible for laws, regulations and instructions dictated by regulating the standards of competence and conduct necessary to enter and the Superintendency, remain in the teaching profession and for maintaining a registry of qualified also called ‘the educational norms’� teachers. Often, teacher councils may also be responsible for accrediting ini- (Art. 46, Law Proposal tial teacher education programs and may provide professional development 609-356). 46 and technical-pedagogical support to teachers and school principals. Teach- ing councils were introduced relatively recently and respond to years of de- mand by teachers to regulate their own profession. The reforms that have been proposed for Chile do not include a teacher council3. Again, this likely makes sense for Chile today, because it maintains the implementation efforts at a fairly reasonable level. But it is worth noting that the selected systems have evolved toward creating teaching councils, as this may inform future directions in institutional reform. 2 The functions, governance and • In two of the selected systems (Ireland and Scotland), there are specialized organizational structure agencies for the provision of technical-pedagogical support for teachers and of the teacher councils in Ireland, New schools4. This would also be the case in Chile under scenario 2, but not under Zealand and Scotland scenario 1. are described in The World Bank (2009a), Having agencies that specialize on a few quality assurance functions implies a trade- Strengthening the quality assurance system off. On one hand, specialized agencies are likely to carry out their assigned functions for basic and secondary more effectively than a single institution with responsibility for all functions. The 2001 education: Comparison of education quality reforms in Ireland illustrate how agencies can become overburdened and less effective assurance systems and when they concentrate on too many tasks. The downside of having specialized agen- institutions in selected cies includes greater implementation efforts to install these agencies in the first place countries. 3 In Chile, the National and, subsequently, to ensure coordination between them. When many agencies are Education Council involved in education quality assurance, a fair amount of institutional maturity is re- would be composed of ten members. At least quired for the system to function effectively as a whole. At this point, it seems wise for two of these members Chile to maintain the number of quality assurance institutions at four, as would be the must be teachers and, case under scenario 1. The implementation challenges under this scenario are already among these, at least one must be chosen by considerable, and a gradual implementation of the changes is advisable (see Table 16 the President of Chile for lessons on the pace of the implementation process). Creating a fifth institution, as in consultation with the most representative would be the case under scenario 2, may generate excessive simultaneous capacity teacher union in the demands to the Chilean public sector. country (Ley General de Educacion, No. 23.370, Art. 56). While the Council’s role and structure differs from that of teaching councils, the inclusion of teacher representatives in the Council may serve as a useful experience should Chile choose to create a teacher council in the future. 4 The functions, governance and organizational structure of specialized agencies that provide technical- pedagogical support to teachers and schools in Ireland and Scotland are described in The World Bank (2009a). 47 Table 14. National-level institutions involved in education quality assurance, by system and by type of institution Ireland The New Zealand Scotland Chile Netherlands Scenario 1 Scenario 2 Ministry or Department of Ministry of Ministry of Scottish Executive Ministry of Education Department of Education and Education, Education Education Education Science Culture and Department Science Education Education Inspectorate of ERO HMIE Agency for Education Quality; Inspectorate Inspectorate Education Advisory Council Superintendency Consejo de Consejo Nacional of Education Educación de Educación Qualifications/ NQA; SEC NZQA SQA Examinations Authority Technical- NCCA LTS National Pedagogical Education Support Service Authority Teacher Council Teacher Council NZTC GTC Funding Agency CFIA Relationship between the national level institutions involved in education quality assurance An important issue in the design of an education quality assurance system is the degree of independence between organizations that set education policies and regulations, and those that oversee how these policies translate into day-to-day operations in classrooms and schools. Law Proposal No. 609-356 formally recognizes the Agency for Education Quality and the Superintendency of Education as “independent� oversight entities. How- ever, the proposal also establishes that these entities will report to Parliament through the Ministry of Education. This means that the degree of independence that the Agency and Superintendency will actually enjoy will in large part be determined by practice. In particular, the leadership characteristics of the Minister of Education and of the heads of the Agency and Superintendency will crucially affect the latter institutions’ degree of independence. International experience may inform Chile as it shapes the relationship between the Ministry, the Agency and the Superintendency. In the selected OECD systems, a tenden- cy to award increasing independence to oversight institutions is observed. This increas- 48 ing independence arises from the recognition that in order to provide useful insight on the quality of education, Inspectorates should be allowed to question the policies and regulations underlying some of the problems they observe in schools and classrooms. For example, in the Netherlands, the Inspectorate of Education publishes an annual report assessing the quality of the education system. Through this report, or through the thematic studies that it also conducts and publishes, the Inspectorate can question the effectiveness of existing policies or regulations and identify potential regulatory loopholes. The Dutch Minister of Education must respond to the Inspectorate’s annual report before Parliament. The experience of the selected OECD education systems also emphasizes that, for over- sight institutions to retain their independence, their leaders should be able to form, stand for, communicate, and instill a shared vision for the organization. Similarly, the experience of the recent health sector reform in Chile, which led to the creation of a new Superinten- dency, highlights that the leadership traits of those in charge of setting up new organiza- tions are important determinants of the type of dynamics that are established between these and the Ministry. ii. Implementation of institutional reforms: Potential obstacles and lessons This section highlights the main lessons for Chile derived from our analysis of the imple- mentation of institutional reforms in the selected OECD systems as well as from recent in- stitutional reforms in Chile’s health and criminal justice systems. The section is organized to provide lessons about the six dimensions of the implementation process identified in the conceptual framework: the reform design process; the pace of implementation; general management practices and the management of change; human resources man- agement; management of information resources; and accountability mechanisms and evaluation. An important question for Chile as it moves forward with the implementation of educa- tion quality assurance reforms is the following: Will this reform be used as an opportunity to introduce a profound modernization of public sector governance in education-sector institutions? Or, will the reform need to be implemented based on the already existing general management practices, human resources policies, information management sys- tems, etc.? It is difficult to envision the success of the education quality assurance reforms without improvements in governance. In turn, the ability to strengthen education governance and administration in Chile will require an important commitment of public resources. A first step is identifying key priorities for improving education governance in Chile. This sec- tion provides lessons that may inform the implementation needs and the priority-setting process. 49 Reform design process The design of institutional reforms is an important stage of the change process not only because it sets the mission, functions and governance of new or restructured institutions, but also because it serves as an opportunity to establish the relationship dynamics be- tween those initiating the reform and those affected by it. The international experience provides valuable lessons for Chile as it moves on to design the details of the proposed institutional reforms. Table 15 summarizes these lessons. The reform design process: Lessons for Chile from the Table 15. international experience The reform design process in the Moving forward with reforms in Chile, it international experience would be important to: • Initiative to introduce institutional reforms comes • Tap into the latent social concern for education and from the Ministry of Education and/or the national use it strategically to bring legitimacy to the reform. government. • Acknowledge the importance of the education sector reform • Smoother reform design and implementation processes at the highest national level, communicate this priority are observed when the reform is triggered by social and support its implementation. pressure. Pressure from the national government also • Introduce consultation processes during the design contributes to make the reform process smooth. However, of institutional reforms. Consultations can contribute to when social pressure is not strong or the reform does raise the legitimacy of the reform, prepare education-sector not constitute a national government priority, the reform actors for change, and reduce the resistance that might moves more slowly and encounters more obstacles. be encountered when the actual implementation process • The Ministry of Education consults with internal and begins. external stakeholders, especially those directly affected • Consult, in particular, with those directly affected by the reform, to design the mission, functions and by the reforms. governance of the new or restructured institutions. Lessons about the inclusion of consultation processes during the reform design process are discussed in Box 1. These are based on the implementation of the 1989 reforms in New Zealand and the 2002 reform of Scotland’s General Teaching Council. 50 Extensive and careful consultation with civil servants during restructuring processes: box 1. Lessons from New Zealand and Scotland In New Zealand, the transition process for staff affected by the profound institutional reforms initiated in 1989 was defined through widespread and in-depth consultation with the various unions covering all of the employment agreements and staff. The reform dissolved the old Department of Education and replaced it with the Ministry of Education, Education Review Office, New Zealand Qualifications Authority and New Zealand Teaching Council. The new agencies’ CEOs were appointed through the State Services Commission. For all other positions within these agencies, an open recruitment process was initiated. In partnership with the unions, several aspects of the transition were defined: • The options available to staff were agreed upon. These included winning a similar position in one of the new agencies, retraining in readiness for a different position, and redundancy. The latter was a financially attractive option targeted to employees who did not want to start a new position, or funda- mentally against the philosophy of the reforms and unwilling be part of the changes and outcomes. • It was agreed that employees from the Department of Education would be given priority in the recruitment of new positions, and would be allowed to submit one single application for several positions. • Unions were involved in aspects of agreeing to salary provisions and employment conditions. This meant that the basic salaries and conditions in the new agencies for similar level positions were not any lower than what people had in their current positions. • As an outcome of consultation with unions, processes were set up to support staff during the transition. Staff had access to counseling, help with writing a CV, support to prepare for an interview, discussions around career paths, and financial planning. This later was particularly useful for those considering re- dundancy. The consultative nature of the restructuring process, and the wide range of options and positions available to staff during the transition, led to people feeling motivated and empowered –they were able to make choices for themselves, and most applied and were appointed to positions that they wanted and were interested in obtaining. However, the balance of involving staff in restructuring efforts must be carefully struck, as illustrated by the restructuring process that Scotland’s General Teaching Council (GTC) underwent in 2002. In this case, the non-promoted staff was given a major role in determining the new structures and organizational practices for the GTC. While this empowered the non-promoted staff, it made the senior executive team staff uncomfort- able because they felt their role and authority diminished. As a result, future changes in the GTC include not only the institutionalization of mechanisms that give greater voice to the employees within the organization, but also the restructuring of senior staff roles. Sources: Interview to staff member of New Zealand’s Ministry of Education (October 2009) and questionnaire responses submitted by Scotland’s General Teaching Council (March 2009). 51 Pace of the implementation process A smooth implementation process, in which most obstacles have been anticipated and other ones are effectively addressed as they arise, is desirable for many reasons. Through a smooth implementation process, (i) the elements of the reform, including the exercise of new quality assurance functions, can be materialized in less time; (ii) financial resources are spent more efficiently; and (iii) public-sector employees’ fatigue is minimized. Based on the analysis of institutional reforms in the selected systems, Table 16 summarizes the main observations regarding the pace of the implementation process, as well as valuable lessons for Chile. Table 16.Pace of the implementation process: Lessons for Chile from the international experience The chronology of the implementation Moving forward with reforms in Chile, it process in the international experience would be important to: • The length of the implementation process depends on • Develop an Action Plan. The AP should specify, for each the complexity of the reform. Reforms have taken 6 to institution and for the education quality assurance systems 36 months to be designed, and 12 to 48 months to be as a whole: (i) its objectives; (ii) targets and indicators implemented. to measure their achievement; (iii) activities; and (iv) the • The implementation of reforms is smoother when: timeline for completion of each activity and achievement of • There is an Action Plan (AP) each target. • The national government and Minister of Education • Adopt gradualism as an implementation strategy. are committed to the reform and communicate it What sources of gradualism are most adequate may vary clearly. across institutions. • Implementation takes place gradually. Sources of • Plan for the achievement of a few “quick wins�. gradualism differ (e.g., gradualism in the objectives, These are useful to maintain momentum and the reform’s functions that are assumed, regions that are affected, legitimacy. schools that are affected). Moving forward with reforms, it will be important for Chile to: • Develop an Action Plan (AP) for the entire education quality assurance system and for each institution. The importance of having an AP is illustrated by the recent reform experience of Learning and Teaching Scotland, described in Box 2. “Quick wins� should be integrated into the AP, as they help maintain mo- mentum for reform. An external institution (e.g., a special taskforce designated by the President, a taskforce within the Ministry of Finance, an inter-ministerial 52 committee) could monitor the accomplishment of the targets established in the AP. External oversight should not be done with the intention of sanctioning insti- tutions that are lagging behind their APs, but with the clear intention of helping those charged with the implementation of the reforms to continuously engage in self-monitoring, identify implementation challenges in a timely manner; and develop effective strategies to overcome these challenges. box 2. The value of having an Action Plan: Lessons from Learning and Teaching Scotland The restructuring of Learning and Teaching Scotland (LTS), initiated in 2000, sought to merge two previous insti- tutions, the Scottish Council for Educational Technology and the Scottish Consultative Council on the Curriculum. The logic behind this process included not only cost reductions and increased efficiency, but also the integration of educational technology into teaching and learning, which was difficult to achieve when work was divided between two organizations. However, by 2004 the two entities were still behaving as separate units. A new CEO was appointed. After a six month period of analysis by him, a team of educators and the LTS staff, an Action Plan was designed to restructure LTS. The AP had two phases. In the first phase, senior staff were removed; the work moved from a focus on individual projects to program areas, led by new senior staff; a performance management system was developed using a Balanced Scorecard based on data and evidence; staff were reeducated on work- ing in collaboration with each other, with staff of the other quasi-governmental organizations, and with practi- tioners on programs that made sense to practitioners. The new CEO focused his work with staff on developing a set of values for the organization and turning the view of staff outward to the needs of practitioners. Six staff members were added as area liaisons to work directly with local educators, sharing/translating to local needs what the LTS provides to schools, and taking back to LTS effective practices developed in the field. The second phase of restructuring, currently underway, seeks to align the senior staff domains to reflect their work with the GTC, HMIE, and other national organizations; to build capacity of the local educational authority leaders; and to focus on broad outcomes rather than individual projects. Source: Interview to staff member of Learning and Teaching Scotland (August 2009). • Set realistic expectations about the pace of the implementation. Some of the institutional reforms that were implemented in the selected systems took as much as 48 months to be implemented, and none of these reforms were as comprehensive as those proposed for Chile. In New Zealand, the implementa- tion of the institutional reforms introduced in 1989 took several years. Figure 6 illustrates the chronology of implementation of selected milestones in New Zealand’s education quality assurance reforms. 53 Chronology of selected events from the implementation of Figure 6. 1989 reforms in New Zealand October 1989: Education Act abolishes the Department of Education and establishes the Ministry of Education, Education Review Office and Teacher Registration Board. December 1989 (8 weeks after approval of Education Act): Lough Committee set up to review the process and outcomes of the institutional reform and recommendary improvements in the processes or the structures. July 1990: Education Act Amendment Act establishes the New Zealand Qualifications Authority. May 1991: Minister of Education´s announcement of strategy for curriculum review and development of national qualifications. August 1992: ERO commences assurance audits of schools (i.e., checkng compliance with laws and regulations) and signals evaluations of schools´ performance and quality to commence in 1993. September 1995: Standards for secondary school mathematics are the first to be registered on the National Qualifications Framework. January 1996: New English curriculum statement becomes compulsory for all schools. Source: Butterworth and Butterworth (1998). • Consider gradual strategies to implement the proposed reforms. When this report was finalized, the Education Commission of the House of Parliament had approved Law Proposal No. 609-356. The approval included an important modification to the original proposal: instead of requiring the Ministry of Educa- tion to define all performance standards within one year after the approval of the reform, the Education Commission extended this period to three years. This change sets a more favorable scenario for the gradual implementation of the reform, a recommendation which is consistent with the international experience 54 and, in particular, with the reforms initiated in New Zealand in 1989, the most similar in magnitude to the reforms now approved in Chile (see Figure 6). Gradualism is likely to take different forms across the different institutions. For example, for the Ministry of Education it might make sense to take on new functions gradually, beginning with the setting of performance standards. For the Agency and the Superin- tendency, school evaluation frameworks and financial monitoring tools may want to be tested in a representative sample of schools before they are applied to the whole system, in line with the international experience. In any case, the choice of gradual strategy and the subsequent development of an Action Plan should take into account the law’s requirements regarding the pace of the implementation process and the need to secure multi-year resources for the gradual implementation of the reform. General management practices and management of change General management practices include elements such as the dissemination of an insti- tutional reform; the leadership that guides this change process; the building of an orga- nizational culture; the setting of governance arrangements and division of responsibility within the organization; the monitoring and evaluation of individual units; and the estab- lishment of inter-agency coordination mechanisms. All of these are crucial to the success of an institutional reform. The international experience provides valuable lessons on the general management practices and the management of change, which are summarized in Table 17. A feature that stands out from the review of the international experience is the culture of consultations that is embedded in the education sector. Consultations form part of the process of setting performance standards, evaluating performance, evaluating the impact of policies and programs, restructuring an organization, and other core ac- tivities of the education quality assurance system. Implementing consultative processes requires, above and foremost, recruiting individuals who have the interpersonal skills necessary to engage in open and honest discussions with the different stakeholders of the education system. 55 General management practices and the management of change: Table 17. Lessons for Chile from the international experience General management practices and the Moving forward with reforms in Chile, it management of change in the international would be important to: experience • External communication of an organization’s activities and • Strengthen the consultative nature of education- consultation with external stakeholders occupy a central sector institutions, at least during the design and place in the daily functioning of public sector agencies in implementation of institutional reforms, but preferably, on a the selected education systems. more permanent basis that transcends the reform process. • Institutional reforms are disseminated through consulta- • Disseminate institutional reforms among the tion processes with all stakeholders directly affected, stakeholders who will be directly affected by it. during the reform design process. The Internet, Intranet, This requires: internal staff training activities and external newsletters • Identifying the actors who will be affected are also used to disseminate different aspects of a reform. by each reform. Not all actors would be directly In addition, managers are assigned the responsibility to affected by all the proposed reforms. For example, communicate the reform to the staff they oversee. The parents are likely to be directly affected by the creation media has sometimes been involved in dissemination of the Agency (through the publication of evaluative activities, but most agencies prefer to maintain a careful reports on individual schools), but are unlikely to be distance from it. directly affected by the restructuring of the Ministry • Leadership crucially affects the smoothness of the imple- of Education. Schools are likely to be affected by both mentation process; the functioning of quality assurance changes. agencies in general; an agency’s independence; and its • Analyzing how actors may be affected by the relationship with the education community. Leaders are reforms. Indeed, the creation of the Agency for judged to be most effective when they can articulate a Education Quality will likely affect schools in a very vision for the organization; inspire everyone in the orga- different way than parents. nization to work toward that vision; promote a culture of • Defining messages to communicate regarding constant adaptation, dynamism and readiness for change; the rationale, importance and implications of and recognize and build strategically on the organiza- the reform. These messages should take into account tions’ strengths and weaknesses. Leadership traits and how the reform will affect a specific actor. performance in previous relevant positions are the most • Tap into the latent social concern for education to important criteria used to recruit leaders. devise a dissemination strategy for the proposed reforms. • Multiple formal and informal mechanisms are used to • Disseminate the reform well in advance of its facilitate coordination between agencies. Informal mecha- implementation, during the design process. nisms are usually the most effective, but formal ones pre- • Assess the effectiveness of the different cede them and contribute to institutionalize inter-agency mechanisms already in place to communicate with coordination. teachers, schools and school owners. Some of these may effectively serve as mechanisms to disseminate the reform. 56 General management practices and the Moving forward with reforms in Chile, it management of change in the international would be important to: experience • Incentives for good performance are in place both for • Establish formal inter-agency coordination units within an agency and for individuals. In the case mechanisms and ensure flexibility for the creation of units, they must specify in writing their objectives, of informal mechanisms. Mechanisms may include define targets, and regularly monitor and report (usually regular meetings between senior-level officials of different annually) to senior management on the achievement of agencies, regular meetings between management staff those targets. Mid-level managers are held accountable of different agencies, the conformation of temporary for the performance of the unit that they monitor, with inter-agency task forces to work on a particular issue, results of the unit-level performance evaluation affecting the distribution of newsletters, or the appointment of their salary and chances of promotion, as well as their a spokesperson that communicates an organization’s unit’s budget. Individual employees are also evaluated changes and activities to other agencies. In addition, the regularly. The results of these evaluations rarely lead to agencies should provide cross-training opportunities to lead the firing of under-performing employees, but they do lead employees in each agency. to salary increases and better chances of promotion for • Clearly assign responsibility for the dissemination individuals who exhibit good performance. In the selected of the reform to managers. systems, evaluation of units is given more importance than • Recruit visionary, inspirational, dynamic and that of individuals. strategic individuals to lead the institutional changes within the Ministry, the Agency and the Superintendency. Search for individuals who have a record of successful leadership experience. • Monitor and evaluate the performance of the individual departments or units at least on an annual basis. These evaluations should include indicators of quality and achievements that enable senior management to assess a unit’s progress against pre-established targets set by the unit itself, as well as internal surveys of employee attitudes and suggestions for improved practices. 57 Human resources management Human resources policies, rules and incentives affect individuals’ motivation, knowledge, skills, adaptability and alignment with the organization’s mission. Thus, they are at the core of an organization’s performance. The analysis of human resources management across the selected education quality assurance systems reveals six generalized patterns: • Openness and competition in the recruitment of staff to new positions. • Allocation of approximately 3 percent of an institution’s budget to staff train- ing. • Increasing emphasis on specifying, and promoting respect for, an organization- specific Code of Ethics which outlines what is expected of employees within a particular agency. • Increasing need to recruit staff with the qualifications necessary to design and manage information systems, websites and large databases. • Focus on maintaining employees’ motivation during restructuring processes. • Regular performance evaluation of individual employees, usually on an annual basis. Evaluations usually integrate the views of the employee, the supervisor, peers and those affected by the employees’ work (e.g., teachers and principals in the case of inspectors). In addition to these patterns, distinct human resources management characteristics are observed across different types of institutions, such as Ministries or Departments of Edu- cation, or Education Inspectorates. These institution-specific findings are summarized in Section 4.b., which identifies specific lessons for the Ministry, the Agency and the Super- intendency. Management of information resources The international experience shows some recent trends in the management of informa- tion resources that are worth highlighting as they could inform the design and implemen- tation of institutional reforms in Chile. Table 18 summarizes these lessons. The need to install integrated information systems within agencies should be anticipated. In particular, it might be desirable for the Agency for Education Quality to create individ- ual school dossiers which contain student, classroom, and school test score information, information collected through inspection visits on the school’s educational outcomes and processes, claims submitted against or by the school, data on the school’s context. At the same time, it should be possible to integrate the information of several school dossiers so that, for example, average test scores can be obtained for students, classrooms, and schools in the same municipality or for schools that serve a similar student population. Be- ing able to integrate the information contained in all school dossiers should also facilitate the production of annual reports on the overall quality of education in the country. 58 It may also be desirable to have information sharing mechanisms between agencies and, in particular, between the Agency for Education Quality and the Superintendency of Education. In the Netherlands, certain features that may be observed during inspections to evaluate a school’s quality are used as triggers to indicate the risk of financial mis- management. A similar mechanism could be worked out between the Agency and the Superintendency so that the information produced by the Agency on individual schools automatically identifies potential financial management risks for the Superintendency to conduct more careful inspection of these schools. Management of information resources: Lessons for Chile from the international Table 18. experience Management of information resources in the Moving forward with reforms in Chile, it international experience would be important to: • There is an increasing need to have digitalized, easy-to- • Anticipate the need to install information sys- process and easy-to-analyze data about students, teachers, tems that facilitate analysis and rapid conversion schools and school owners. of collected and stored data into reports at the • Data storage mechanisms are being designed so that the classroom, school and system-wide levels. information that is collected and stored by institutions can • Anticipate the need to install integrated infor- be easily (and sometimes automatically) converted into mation systems within agencies (e.g., within the individual school reports, system-wide reports, thematic Agency, between the unit in charge of national examina- reports, etc. tions and the one in charge of evaluating the quality of • Data sharing mechanisms are also being designed, so that individual schools). information collected by one institution (e.g., the agency • Anticipate the need to install integrated infor- in charge of administering national examinations) can be mation systems across agencies (e.g., between the easily incorporated into the work done by another institu- Agency and the Superintendency). tion (e.g., the agency in charge of evaluating the quality of individual schools). Similarly, data sharing mechanisms within an organization are also important (e.g., within the Dutch Inspectorate, between the unit in charge of evaluating schools’ quality and the one in charge of overseeing schools’ financial practices). Accountability mechanisms and evaluation Table 19 summarizes the main findings on how institutions with responsibility for educa- tion quality assurance functions are held accountable in the selected systems, and how these findings may inform the design and implementation of institutional reforms in Chile. 59 Accountability mechanisms and evaluation: Lessons for Chile Table 19. from the international experience Accountability mechanisms and evaluation Moving forward with reforms in Chile, it in the international experience would be important to: • Internal mechanisms are in place for self-evaluation and • Design a comprehensive set of internal and exter- accountability purposes. They include regular performance nal accountability and evaluation mechanisms. This evaluation of units and individuals, and an audit unit in could include Annual Reports, performance evaluations of the case of Inspectorates. units and individuals, independent commissioned evalua- • To evaluate the overall work of individual institutions, tions, complaints procedures, customer satisfaction surveys, evaluations by independent organizations or individuals etc. Some mechanisms may be more adequate than others are also commissioned on a regular basis. for a particular institution. Institution-specific recommenda- • Other mechanisms regularly used to hold institutions ac- tions for accountability and evaluation mechanisms are countable include: Annual Reports, which contain indica- provided in Section 4.b. tors of the effect that the institution has had on the edu- cational community (students, teachers, parents, schools); procedures to file complaints against an organization or one of its employees; customer satisfaction surveys. 60 iii. Summary Table 20. Implications of the analysis for the proposed reforms in Chile Analysis Lessons for Chile from the international experience Design of a new Distribution of EQA • Number of institutions. The proposed number of institutions is in line with the international institutional functions across experience. arrangement for national-level education quality institutions • There is a trade-off in having agencies that specialize on a few quality assurance assurance functions. The implementation challenges faced by Chile under scenario 1 are already substantial. Those that would be faced under scenario 2 might become overwhelming given the current capacity of the public sector. • Separation between policy-setting and policy-overseeing institutions. This separation would not be so clear cut in Chile as it is in the international experience. The Superintendency would oversee schools’ financial practices and compliance with regulations, as well as set regulations that must be met by schools to avoid sanctions. • School inspections. Two institutions, the Agency and the Superintendency, would have oversight responsibilities over schools. This is in contrast with the international experience, where there is one institution in charge of overseeing schools. • Student assessments. The Agency would evaluate students as well as schools. This is in contrast with the international experience, where usually student assessments and school evaluations fall under different agencies. • Requirements to enter the teaching profession and performance standards for teachers. These would be set by the Ministry, in line with the experience in the Netherlands. In Ireland, Scotland and New Zealand, these are set by a Teaching Council. • Technical-pedagogical support. This service could be provided by the Ministry under scenario 1, and would be one of the functions of the National Education Service under scenario 2. The international experience is mixed. In the Netherlands and New Zealand, the Ministry provides funding for schools to hire private services. In Ireland and Scotland, there are specialized public agencies. Relationship between • Increasing independence for oversight institutions. There has been a tendency in the national-level selected systems to award increasing independence to the Inspectorate (analogous to the Agency institutions and Superintendency) with respect to the Ministry of Education. • Leadership as an important determinant of the independence that the Agency and Superintendency will have in practice. Leaders should be able to form, communicate and instill a shared vision in the organization. Implementation Reform design • Tapping into the latent social concern in education. of institutional process • Acknowledging the importance of the reform at the highest national level. reforms • Introducing consultation processes during the design of institutional reform. 61 Analysis Lessons for Chile from the international experience Implementation Pace of the • Developing an Action Plan. These should be developed for each institution and for the of institutional implementation system as a whole and should include (i) objectives, (ii) targets and indicators to measure their reforms process achievement, (iii) activities and (iv) a timeline. • Adopting gradualism as an implementation strategy. In some cases, for example in the Ministry, the new functions might be assumed gradually, beginning with the setting of standards. In other cases, for example the Agency and Superintendency, all functions might be assumed simultaneously, but beginning with a pilot experience in a representative sample of schools. • Planning for the achievement of “quick wins�. General management • Strengthen the consultative nature of education-sector institutions. practices and management of • Disseminate each institutional reform among the stakeholders directly affected change by the reform (during the reform design process). This requires (i) identifying the affected stakeholders; (ii) understanding how they may be affected; and (iii) defining messages to communicate regarding the rationale, importance and implications of the reform. • Tap into the latent social concern for education to devise a dissemination strategy. • Disseminate information about the reform early on, during the design process – well in advance of beginning its implementation. • Assess the effectiveness of the different communication mechanisms already in place. Some of these may effectively serve as a mechanism to disseminate the reform. • Establish formal inter-agency coordination mechanisms and ensure flexibility for the creation of informal mechanisms. • Assign responsibility for the internal dissemination of the reform to managers, and hold them accountable for that. • Recruit visionary, inspirational, dynamic and strategic individuals to lead the institutional changes. Search for individuals who have a record of successful leadership experience in the education sector. • Monitor and evaluate the performance of individual departments or units at least annually. Human resources • Defining all positions anew. Ensure openness and competition in the recruitment of new management positions, while at the same time giving priority to employees formerly employed by the Ministry when making assigning positions. Allow employees to choose where they would want to go, and allow them to apply to several positions, and then make decisions based on merit. Also, consider a “redundancy� package for staff not interested in being part of the reform • Allocate approximately 3 percent of the budget to staff training and professional development. 62 Analysis Lessons for Chile from the international experience Implementation Human resources • Create and enforce an institution-specific Code of Ethic. Codes of Ethics are particularly of institutional management important in oversight institutions, where the reliability of the evaluations and reports depend reforms on inspectors/auditors’ objectivity, fairness and integrity. Compliance with this code should be checked through regular individual performance evaluations, and through an established complaint process open to educators and the public. • Anticipate the need to recruit staff with skills and knowledge to design and manage information systems, website and large databases • Focus on maintaining employees’ motivation during restructuring processes. • Evaluate individual performance annually. This should include the views of the employee, the supervisor, peers, and those affected by the employee’s work (e.g., teachers and school principals in the case of inspectors). Management of • Anticipate the need to install information systems that allow for rapid conversion information resources of collected data into reports. • Anticipate the need to install integrated information systems within agencies (e.g. within the Agency, between the unit in charge of student assessments and the one in charge of schools evaluations). • Anticipate the need to install integrated information systems between agencies (e.g. between the Agency and the Superintendency, and each with the Ministry). Accountability • Design a comprehensive set of internal and external accountability and evaluation mechanisms and mechanisms. evaluation • Self-evaluation mechanisms may include: (i) a self-audit unit (especially relevant for the Agency); (ii) performance evaluation of units (these are usually awarded more importance than evaluations of individual employees); (iii) individual performance evaluations. • External evaluation mechanisms typically include the commissioning of reports to independent agencies to evaluate the overall work of an institution, or a part of its work. • External mechanisms to hold each institution accountable typically include an Annual Report (with indicators of how the institution has affected the education sector and contributed to improve education quality); and a procedure to file complaints against the institution or an employee. b. Implications for specific institutions i. Agency for Education Quality The international experience provides ample lessons for Chile’s Agency for Education Quality. The Agency, like the selected systems’ Inspectorates, would be responsible for evaluating and reporting on the quality of individual schools. In addition, the Agency 63 would evaluate individual students’ performance, a function often delegated to a sepa- rate organization in the selected systems. Institutional design Evaluating performance Table 21 compares the process of evaluating performance in the selected OECD systems to how it has been envisioned for Chile by Law Proposal No. 609-356. An issue that stands out from this comparison is the fact that several key decisions remain to be made in Chile regarding the evaluation of students and schools. The experience of the selected OECD systems may inform the following decisions: • Should there be a school-leaving examination at the end of secondary school? If so, should performance in this examination have direct conse- quences for students? In all the selected OECD systems, students must pass a school-leaving examination in order to obtain a certificate for the completion of the compulsory years of education. The exact timing of the examination varies across countries, as some mandate completion of ten years of education while others mandate twelve to thirteen years. The type of school-leaving examination also varies across and within countries. For example, in Ireland there are three types of Leaving Certificates (Established, Applied and Vocational Leaving Cer- tificates), all of which are administered by the SEC but provide information on different types of qualifications. In the Netherlands, all students must success- fully complete school-leaving examinations in the final year of secondary school, regardless of whether they attend pre-vocational, general or pre-university sec- 5 Each school prepares ondary education. These examinations are a combination of school-based tests1 its own tests in and national examinations2. The Ministry determines the examination syllabus, accordance with this syllabus, and submits and the Inspectorate’s approval is required for school-based tests. In some sub- them for approval jects, a passing grade in both the school and the national examination is re- by the Inspectorate. quired to obtain certification of completion of secondary education. In others, In general, school examinations consist a passing grade in the school tests may be sufficient. Students that do not pass of two or more tests the school-leaving tests do not receive a certificate of completion of secondary per subject, including oral, practical and/ education and therefore cannot go on to higher education studies. They can or written tests. They take the test again and, if they fail, they must repeat the final year or go to an must be completed and adult education institute (Béguin, Kremers and Alberts 2008). results submitted to the Inspectorate before the national examinations What criteria should be used to evaluate the quality of individual schools? In all are administered. the selected systems, the evaluation of schools’ quality takes into account: (i) student 6 National examinations are the responsibility outcomes (e.g., test scores, repetition and dropout rates, socio-emotional outcomes), of the Ministry, they (ii) educational processes (e.g., teaching methods, management of human and financial are developed and resources, self-evaluation processes); and (iii) the context in which these occur (e.g., infra- administered by the private-sector institution structure, pedagogical orientation, composition of the student population, composition CITO, and are marked of the teaching staff, position in the market, community characteristics). In recent years, by teachers within the school and checked by all countries have increased the relative importance given to learning outcomes. This a teacher in another shift in balance has occurred after years of emphasizing educational processes. However, school. 64 processes have not ceased to be considered in school evaluations because they inform inspectors’ judgment on a school’s strengths and weaknesses. A greater focus on school context indicators has also been observed in recent years, in recognition that outcomes need to be assessed taking into consideration the context in which they occur, and that recommendations for improvement must take contextual factors into account. What sources of information should be considered to evaluate the quality of indi- vidual schools? In all the selected systems, inspectors use the internal evaluation reports prepared by the school itself as the starting point for external evaluations. If a school’s self-evaluation report is judged to be reliable and indicates that the school is up to stan- dard, then the frequency of inspection decreases and the on-site inspection process is limited to gathering information about a reduced number of indicators. When schools are deemed not up to standard or their self-evaluation is not reliable, inspection occurs more frequently and information is gathered on a more comprehensive set of quality indi- cators. Typical sources of information used by inspectors include classroom observations, face-to-face consultations with teachers and management, and surveys to parents and/or students administered before or after inspectors’ visits. Limited information is requested from schools in advance. Cuadro 21. Características de las evaluaciones de desempeño In the selected OECD systems In Chile, based on the proposed reforms General • Two to five central government institutions evaluate the • One central government institution, the Agency for characteristics performance of students, teachers, school principals, schools Education Quality, would participate in the process and school owners. of evaluating the performance of students, teachers, schools and school owners. Another institution, the Superintendency, would also monitor –but not evaluate- schools and school owners. Evaluating • Students are usually evaluated by a specialized agency, • Individual students’ performance would be evaluated students separate from the Inspectorate that evaluates schools. by the Agency, who would also evaluate schools. • School exit examinations are administered at the end of • No decision has been made as to whether school compulsory education. Students must pass these examinations exit examinations would replace university entrance to obtain recognition for the completion of that level of examinations, or as to whether passing these education. examinations would be necessary to graduate from secondary education. Evaluating • In general, teacher performance evaluations take place • Individual teachers’ and school principals’ teachers and not on an individual basis but as part of the evaluation performance would be evaluated by the Agency. school principals of the overall quality of schools conducted by Education Inspectorates. Similarly, Inspectorates usually assess the quality of “management practices and leadership� but not a school principal’s individual performance. 65 In the selected OECD systems In Chile, based on the proposed reforms Evaluating • A single institution, the Education Inspectorate, is responsible • Two institutions would be responsible for overseeing schools and for evaluating schools and school owners (including the schools. The Agency would evaluate their quality. school owners assessment of their financial practices) and monitoring their The Superintendency would monitor their financial compliance with statutory regulations. practices as well as their compliance with statutory regulations. • External evaluations take into account: students’ cognitive • The specific aspects that would be taken into and non-cognitive skills; the overall quality of teaching and consideration to evaluate the overall quality of leadership; the responsiveness of the school environment to individual schools remain to be defined. students’ needs; the quality of internal evaluation processes; and the management of financial, physical and human resources. • External evaluations begin with a review of internal evaluation • External evaluations would take internal evaluations reports prepared by the school, and are complemented by as a starting point, but would not necessarily classroom observations, consultations with teachers and be complemented with classroom observations, management, surveys to parents, and on-site inspections. Little consultations with stakeholders or on-site inspections. information is requested from schools in advance. In the selected systems, every school is evaluated at least once every four years. Schools that are evaluated as having low levels of quality or are at risk of falling below the existing performance standards are inspected more often, usually every 12 to 24 months. In Chile, the frequency of external evaluations would be lower than in the se- lected systems. According to Article 18 of Law Proposal No. 609-356, schools falling well below the performance standards, classified as “unsatisfactory�, would be evaluated by the Agency at least once every 24 months, which is in line with the international experi- ence. However, schools falling just below the standards, classified as “regular�, would be evaluated at least once every four years, which is considerably less frequent than in the selected systems, where on average schools with regular performance are also evaluated every two years. Moreover, for the remaining schools in Chile, classified as “satisfactory� or “good�, the proposal does not establish any frequency of evaluation and leaves this decision to the Agency. It would be important for the Agency to establish a plan of regu- lar visits and evaluations for these schools, for example “at least once every five years�, as these are the schools that would enable the Agency to learn about best practices. Indeed, the ability to identify a diversity of best practices in different types of schools will likely affect the Agency’s ability to promote and support educational improvements in under-performing schools. The international experience provides several additional lessons about the evaluation of performance. These are summarized in the following paragraphs. The development of school evaluation frameworks should include extensive consultation with many different stakeholders. Evaluating the quality of individual schools will require a definition of what “quality education� means. This definition should 66 not be developed by the Agency alone –it should reflect consensus, or at least consider- able agreement, between students, teachers, schools, parents, educational experts and researchers, politicians, local authorities, etc. It should be a definition that captures what is generally understood as “quality education� within society. Schools will resist making changes that they do not trust will lead to improved results, and they will be less likely to carry out changes proposed by the Agency if they do not share the definition of “good quality� that underlies the framework used for inspection (van Bruggen 2001; de Wolfe and Janssens 2007). Distinct school evaluation frameworks may be developed for different levels of education and for different levels of educational quality. In the selected systems, there are distinct frameworks for the inspection/evaluation of early childhood, primary, secondary and special schools. Where applicable, there are also different frameworks for schools whose student population’s native language is not the mainstream language of instruction. In addition, there has been a more recent tendency to develop at least two types of frameworks for each level/type of education: a comprehensive framework for schools that exhibit low quality or are at risk of falling below the standard, and a narrower framework for schools that in the past were judged to be up to standard. Research can be used strategically to justify the inclusion of specific evaluation criteria. When discussing whether to include certain criteria in the definition of “quality education� and in school evaluation frameworks, the Agency could take advantage of validated research and empirical evidence to support its position. Assessment tools should build on already existing tools as well as past experi- ence and lessons from their application. This applies to tests for students, teacher performance evaluations, and inspection/evaluation frameworks for schools. Chile has previous experience with all of these tools. Student assessment instruments (e.g., SIMCE) have been applied to students, whereas teacher and school evaluations have historically been applied on a voluntary basis (e.g., to SEP-participating schools) or to the municipal sector only. Because the reform requires the application of assessments to all participants, the existing assessment instruments and systems will need to be adjusted. In addition, the reform provides an opportunity to improve these systems based on the lessons that have been learned over time through their application. Assessment tools should be tested before they are applied on a massive scale. In all the selected OECD systems, school inspection/evaluation frameworks were first tested on a representative sample of schools; then they were refined; and only afterward were they applied to all schools. Once an assessment tool is being applied on a massive scale, it should still be regularly revised and improved. In the case of school inspection/evalua- tion frameworks, these are usually revised based on (i) satisfaction surveys administered among teachers, parents, students and school principals after an inspection; and (ii) eval- uations of the effect that inspections/evaluations have had on educational improvement. This information is usually analyzed by a Self-Audit Unit within the Inspectorate, which is also in charge of coordinating consultations with stakeholders to revise and update the school inspection frameworks. 67 In addition to verifying the validity of assessment tools, the testing of these tools should be seen as an opportunity to identify undesirable side-effects. Negative side-effects typically include “putting on an act� on the day of the inspection, excluding low-performing students on the day of the assessment, increased stress among teachers, too much time dedicated to administrative tasks, and fear of introducing educational innovations. The testing of assessment tools should also be seen as an opportunity to eval- uate whether inspection leads to the desired outcomes. Desired outcomes may include the development and implementation of school improvement plans, improve- ments in teaching, and better educational outcomes (student achievement, progression in school, tertiary education and labor market opportunities, etc.). Finally, there is evidence from the international experience that school inspec- tions/evaluations are more likely to lead to educational improvement when the following conditions are met: • Changes proposed by inspectors or the weaknesses identified by them had already been thought of by the school leaders and teachers be- fore the inspection. School factors affect the extent to which inspections will result in educational improvements. A school’s culture, and in particular how open it is to receive feedback and criticism, is predominant among these factors. Also, research has found that schools are most likely to incorporate changes or address weaknesses that they had already identified. Similarly, teachers are more likely to implement changes if they were involved in devis- ing solutions. These findings have led to a recognition of the important value of ownership of educational improvement efforts among schools and teachers and of the need for inspectors to not only respect but promote it during their visits to schools (Standaert 2000; Ehren, Leeuw and Scheerens 2005; de Wolfe and Janssens 2007). • The inspection process contributes to strengthen schools’ institutional capacity. Even if schools decide to make changes in the way they provide edu- cation, they may not have the capacity to do so. Capacity that is conducive to the effective implementation of changes within schools requires resources and knowledge, but also organizational aspects such as cooperation between teachers and support (or at least non-resistance) from parents (Ehren, Leeuw and Scheerens 2005). The process of inspection can contribute to strengthen these organizational features and build capacity among schools, for example, by having inspectors engage in dialogue with groups of teachers, parents and students. • Inspectors have a good relationship with teachers and school principals, where there is mutual trust, respect and listening, and schools are allowed and encouraged to question inspectors. Schools need to experience the feedback provided by inspection as relevant, understandable, clear and useful. For this 68 to occur, inspectors must communicate clearly what they judge are a school’s strengths and weaknesses, and explain their judgments and analyses providing detailed supporting evidence. School principals and teachers are more likely to incorporate changes proposed by inspectors when they have received ample feedback during inspections, and when the relationship between inspectors and school employees is based on trust and stable over time (de Wolfe and Janssens 2007). Reporting on performance The international experience may inform Chile’s Agency and Superintendency, as they design the mechanisms to report on individual schools’ quality and compliance with regu- lation. In all the selected systems, the Inspectorate prepares detailed, individual reports about the quality of education of each reviewed school. • Schools and school owners are given the opportunity to comment on these reports before they become public. A school’s comments are included in the Inspectorate’s report or accompany it in a separate document. • The Inspectorate’s reports on the quality of individual schools are made available to the general public through its webpage. In most systems, the Ministry of Edu- cation’s website also has a link to these reports. The Inspectorate usually sends a hard copy of the final report to the school, and also provides printed copies to the public on an at-request basis. • Inspection reports usually include the following: (i) an overview of the school’s context and characteristics; (ii) the Inspectorate’s overall impression about the quality of education in that school, which summarizes and integrates the evalua- tion results under each of the aspects of quality considered as well as the school context; (iii) a detailed description of the school’s evaluation results under each of the aspects of quality considered (in some systems, a summary is provided for each quality aspect, and the full evaluation is included in an Appendix); (iv) an outline of the school’s strengths and weaknesses and recommendations to ad- dress these weaknesses; (v) an indication of how the Inspectorate will continue to supervise the school in the coming years. In some systems, the Inspectorate produces and publishes a second report directed to parents. These “school quality reports� or “quality cards� are briefer and simpler than the school inspection reports described above. Quality cards include information about: (i) a school’s achievement (e.g., the average level of achievement of its students, how it com- pares to the national and local average, how the actual level of achievement compares to the one that would have been predicted given the school context); (ii) the school climate; (iii) the teaching climate and teaching methods, and other features that are of particular interest to parents when choosing a school for their children. 69 In all systems, the Inspectorate also publishes: • A report on the overall quality of the education system. This report draws on the findings of individual school inspections. Over the years, information sys- tems have improved to facilitate the conversion of individual school reports into system-wide reports. • Several reports with the results of evaluations about particular aspects of the quality of education. These “thematic� evaluations are usually based on a sam- ple of schools. Examples of some of the quality dimensions that have been ad- dressed in these thematic evaluations are: teaching of specific subjects (e.g., language, mathematics, environmental issues), in-service teacher training, use of ICT in classrooms, discrimination at schools, school-leaving examinations in primary and secondary school, counseling services in secondary education, drop- out from secondary education, schools’ internal quality assurance mechanisms, and school admission policies. In Scotland, the Inspectorate also plays an important role in systematically identifying and disseminating best practices. This role is also played by the other systems’ Inspectorates but is exercised in a less systematic manner. The international experience provides several additional lessons specifically about the contents and use of performance reports: Reports should present information in clear, objective and simple terms. Any judgments or evaluations should be extensively supported. The underlying methodology of evaluative reports should be transparent and well disseminated. Public reports often affect the behavior of teachers and schools –who do not want to see their reputation harmed. Indeed, even if school evaluation frameworks do not represent statutory standards that need to be met in order to avoid sanctions, the reputational effect of school reports, which are based on these evaluations, acts as a powerful sanction or reward and therefore has an effect on teachers, schools and educational improvement. Public reports rarely affect parents’ choice of school. Consequently, public reports on performance may need to be complemented with other efforts, such as extensive dis- semination of the availability of these reports and how to access them, improvement in the way messages are communicated; and including other information valued by parents in order to encourage their reading. The publication of performance indicators may need to be complemented with other types of indicators. Information other than test scores tends to be highly valued particularly by parents and other stakeholders. For example, information about educa- tional processes, the pedagogical orientation of the school, and the school context is of interest to parents and communities. It is desirable to conduct consultations with parents and communities to identify the specific types of information they value. 70 Public reports on performance may have undesirable side-effects. These should try to be anticipated and mitigated. Examples of undesirable side-effects include fraud during the administration of standardized tests, increased school segregation, and misin- terpretation or misuse of reports by the media (e.g., ad-hoc creation of rankings). Implementation of institutional reforms: Potential obstacles and lessons This section addresses five implementation issues that are particularly relevant to the Agency for Education Quality: the pace of the implementation; leadership; human re- source management; management of information resources; and mechanisms to evalu- ate and hold the institution accountable. Most of these issues are related. For example, adequate leadership will be important to ensure that the Agency is, in practice, inde- pendent from the Ministry of Education. Independence, together with the mechanisms to evaluate and hold the institution accountable, will be important for the quality of the work done by the Agency. The quality of the Agency’s work will determine its legitimacy in the eyes of teachers, principals, schools, school owners and the general public. Pace of the implementation process Law Proposal No. 609-356 establishes that the Agency will have “a maximum period of one year, beginning when the performance standards for all actors have been approved, to determine and apply the methodology to classify schools� based on their performance (Transitory Article 2). The international experience suggests that this timeframe may be unrealistic. For instance, in Ireland it took two years (1998-1999) to develop the Whole School Evaluation framework and apply it in a small sample of 35 pilot schools; and an additional four years before the pilot experience became a general program applied at the national level. Similarly, in the Netherlands, it took four years (1991-1994) for a sample of 180 primary schools to be evaluated individually for the first time; and an ad- ditional five years for the evaluation framework to be refined and applied to 13 percent of all primary schools. Indeed, the international experience with the development of evaluation frameworks suggests the convenience of developing a pilot framework, testing it in a representative sample of schools, incorporating the lessons from the pilot experience to obtain a refined evaluation instrument, and only then moving on to scale up the application. This process requires not only a number of years but also extensive consultations with the various stakeholders of the education system, to ensure that the evaluation instruments are vali- dated by the educational community. Leadership The characteristics of the Agency’s leader will crucially affect the organization’s indepen- dence with respect to the Ministry of Education. Based on the international experience, a strong leader in the specific case of the Agency would be someone who can: • Develop a vision of where he/she wants to take the organization; 71 • Inspire everyone in the organization to get on board and work toward this vi- sion; • Be strategic, that is, acknowledge the weaknesses, strengths, threats and op- portunities of the organization, and get the most out of them; and • Be ethical, truthful, direct, and straightforward. Leadership characteristics that are not considered so critical to the success of Inspectorates include: being tactical (i.e., extraordinarily committed to results, giving great importance to facts, figures, numbers and data); being focused on the achievement of a limited and coherent set of results; being likeable and being open to feedback. In the selected systems, the heads of Inspectorates are individuals who, at the time of assuming this role, had at least a masters degree (but in several cases also had a doctoral degree) and had 20 years of professional experience in the education sector, during which they had held a leadership position for at least 5 years (and, on average, for 13 years). Nevertheless, neither the level of education nor the years of relevant professional experi- ence are seen as important factors when selecting who will lead these organizations. Similarly, the knowledge of the rules and procedures governing civil service and public administration has only moderate or little weight on the selection of leaders. The most important factor taken into consideration is an individual’s leadership capacity. Other very important factors that weigh into the decision are performance in previous relevant jobs, analytical skills, communication skills and managerial skills. Human resources management Inspectors play an important role in developing a strong relationship with teach- ers, school principals and the educational community. The international experience indicates the importance of having a strong relationship between inspectors and schools for the legitimacy of the Inspectorate’s work and, importantly, for evaluations to have an effect on educational improvement. A strong relationship is one where there is mutual trust, respect to each others’ views, and where schools are allowed and encouraged to question inspectors. In the selected systems, being an inspector has a certain social status that is associ- ated to their overall understanding of the education system and to the relatively demand- ing qualifications required to become an inspector. To become an inspector, experience in education is required, whether as a teacher, school principal or administrative staff. Skills that are highly valued among inspectors include knowledge of the education sector, ana- lytical skills, communication skills and interpersonal skills. Management skills have also acquired an increasing weight. The recruitment process for inspectors is highly competitive. All job vacancies are posted in several venues, including newspapers and online job-search websites. This is true even in the case of the 1989 reforms in New Zealand, when many of the inspectors that were assigned to the new Education Review Office came from the old Department 72 of Education. In that reform, all positions were advertised openly to everyone, but former employees from the Department of Education were given priority during the recruitment process, as well as training to be able to undertake the new job’s responsibilities. In most cases, a committee consisting of the chief inspector and other inspectors select a long list of candidates from the pool of applicants. Interviews are then conducted among these by the selection committee, and a short list is selected. Then, a second committee is put together that includes the senior chief inspector, the chief inspector, inspectors and other staff. They interview short-listed candidates and make a decision after, among other things, consulting the candidates’ former employers and colleagues. References are particularly important to make hiring decisions. All Inspectorates have their own Code of Ethics, which requires inspectors to be fair, impartial, responsible, and trustworthy. Most Inspectorates’ codes of ethics were reformed in recent years to explicitly include elements of personal integrity. In both the Netherlands and New Zealand, Inspectorates’ codes of ethics were also reformed to in- clude more specific standards of the behavior that is expected of inspectors. In New Zea- land, there are two codes of ethics: one, more extensive, for use by inspectors and inter- nally within the ERO; and another one, more concise, that is available to the general public so that they know what they should expect from inspectors. Compliance with the Code of Ethics is assessed through the individual performance evaluation processes in place. Initial training and supervision of new inspectors is important in all Inspectorates. During initial training, inspectors learn about the organizational mission of the Inspec- torate, how they are expected to contribute to that mission, and the contents of the Code of Ethics. Continuous professional development is less institutionalized than initial training but has increased steadily in the past two decades. Training activities for inspec- tors frequently include exchanges with inspectors from other education systems through participation in international workshops and through visits to other inspectorates. Around 1 percent of Inspectorates’ budget is allocated to initial training, and roughly 2 percent is allocated to continuous professional development. box 3. Training of inspectors in the Dutch Inspectorate of Education In the Netherlands, new inspectors are closely observed during the first months of service. A twenty-day program is organized under the leadership of a project manager who calls in inspectors, public officials and experts. After this period, each new inspector is assigned a tutor. Additional in-service training may be recommended based on performance appraisal and supervision. In terms of ongoing in-service training, the Netherlands attaches great importance to learning from the experience of Flanders (Belgium), North Rhine- Westphalia (Germany), Lower Saxony (Germany), England and Scotland. Sources: Standaert 2000. 73 Table 22 provides useful information to plan for the human resources needs of the Agen- cy for Education Quality. The number of inspectors ranges from 110 in Ireland to 250 in the Netherlands. The ratio of students to inspectors varies from around 5,000 to 10,000, while the ratio of schools to inspectors is in the mid-30s for Ireland and Netherlands, and around 17 for New Zealand and Scotland. Table 22. Number of inspectors in the selected OECD systems Republic of New Chile The Netherlands Scotland Ireland Zealand 1. Inspectorates Number of inspectors TBD 110 250 150 150 Students per Inspector TBD 7,476 10,392 5,066 4,749 Schools per Inspector TBD 37 33 17 18 2. Education system (primary and secondary)**** Students 3,574,419 822,373 2,598,000 759,906 712,298 Teachers 174,882 67,804 219,900 53,366 53,563 Schools 11,420 4,023 8,181 2,593 2,722 Note: TBD stands for “to be determined�. Sources: * World Bank, World Development Indicators. ** World Bank, Worldwide Governance Indicators. *** World Economic Forum, The Global Competitiveness Report. **** Mineduc, Anuario Estadistico 2007; Department of Education and Science, Education Statistics 2007/2008; Dutch Ministry of Education, Culture and Science, Key Figures 2004-2008; NZ Education Counts, Education Statistics of New Zealand 2007; Scottish Government, School Education Statistics 2008. Management of information resources The need to install integrated information systems in the Agency should be an- ticipated. In particular, the Agency for Education Quality may want to create individual school dossiers which contain a school’s test scores information, information collected through inspection visits on the school’s educational outcomes and processes, claims submitted against or by the school, data on the school’s context, etc. This should facilitate the generation of post-inspection school reports, as well as the provision of information about a school to be analyzed before an inspection. In addition, it would be desirable to have information systems that can integrate the data contained in several school dos- 74 siers so that, for example, average test scores can be obtained for schools in the same municipality or for schools that serve a similar student population. The ability to integrate the information contained in all school dossiers should also facilitate the production of annual reports on the overall quality of education in the country. The convenience of having information sharing mechanisms between agencies and, in particular, between the Agency for Education Quality and the Superin- tendency of Education, should also be considered. In the Netherlands, certain features that may be observed during inspections to evaluate a school’s quality are used as trig- gers to indicate the risk of financial mismanagement. A similar mechanism could be worked out between the Agency and the Superintendency so that the information pro- duced by the Agency automatically pinpoints to the Superintendency the risk of financial mismanagement in specific schools or school owners, leading to more careful inspection of these institutions. Accountability mechanisms and evaluation Measuring the extent to which the Agency’s work contributes to improve the quality of education will be crucial for the entity’s legitimacy to evaluate the quality of teachers, school principals, schools, and school owners. At the same time that the Agency will “demand� schools to be up to standard, so will schools demand the Agency’s work to be of high quality and relevant to improve their daily functioning. The Agency could benefit from having not only the accountability and evalua- tion mechanisms typically observed in quality assurance institutions (an Annual Report; internal performance evaluation mechanisms at the unit and individual employee levels; complaints procedures), but also an internal audit unit that is regularly in charge of evaluating the quality of inspections. The work of this unit may include: analyzing questionnaires distributed among parents, teachers, students and school prin- cipals at the end of each inspection, to gather information about their satisfaction with the inspection process; conducting ad-hoc parallel inspections to judge the consistency of inspectors’ judgment and their use of inspection frameworks; evaluating the impact of inspections on school improvement, and identifying obstacles to improvement as well as undesired side-effects from inspection. ii. Superintendency of Education Institutional design and implementation of institutional reforms: Potential obstacles and lessons Because the functions that in Chile would be carried out by the Agency and the Superin- tendency are carried out by one single institution in the selected OECD systems, most of the lessons outlined for the Agency also apply to the Superintendency. A few distinctions are outlined below. 75 Human resources management The nature of the functions that would be carried out by the Superintendency demand a different staff profile than that required for inspectors in the Agency. Important competencies among auditors in charge of monitoring schools’ financial prac- tices and their compliance with laws and regulations include analytical skills, communica- tion skills and toughness. In the selected systems, financial auditors are either Certified Public Accountants (CPAs), financial analysts, or have an educational or legal background and receive on-the-job training on specific financial skills relevant to the educational sys- tem. Auditors and school inspectors maintain a close relationship and collaborate to identify schools and school owners whose financial practices may represent a risk. Auditors train inspectors to recognize financial risks in the schools they visit, and to identify when underperformance by schools is more or less caused by bad financial man- agement by school owners. In the Netherlands, the financial oversight department within the Inspectorate has developed an early warning system by analyzing financial state- ments and other types of signals including informers and the press. In cases where risks are identified, teams of both auditors and inspectors monitor school owners’ changes to improve educational and financial processes. One option that might be considered for the Superintendency is the outsourcing of financial oversight activities. The Dutch experience may be particularly relevant to inform this option, and is summarized in Box 4. Outsourcing of financial oversight by the Dutch Inspectorate of box 4. Education In the Netherlands, independent CPAs perform audit services for roughly €30-40 million per year –about twenty times the total annual budget of the Inspectorate’s financial oversight department. They are the first-line over- seers, and the Inspectorate acts as the second-line overseer. To carry out its oversight functions, the Inspectorate ‘uses’ school owners’ own CPAs. It gives them extensive audit guidelines and monitors their performance. It uses their reports to monitor and sanction school owners (which happens in only about 50 cases each year, mostly for relatively small amounts). The Inspectorate has legal authority to sanction CPAs, but in general a very good working relation exists. In part, this good relationship is due to the fact that the department consults regularly with CPAs –for example, it organizes special briefings for CPAs to provide advice on the guidelines developed by the department. Source: Interview to staff member of the Financial Oversight Department within the Dutch Inspectorate of Education (April 2009). 76 iii. Ministry of Education Institutional design Setting performance standards A single most important finding from the analysis of the selected OECD systems is the extensive consultation processes involved in the exercise of several edu- cation quality assurance functions, including the setting of performance standards. Table 23 compares the process of setting performance standards in the selected OECD systems to how it has been envisioned so far in Chile. A key difference that emerges from this comparison has to do with the consultative nature of this process. In the selected OECD systems, formal provisions and institutional arrangements lead to the participa- tion of several institutions and actors in the process of setting performance standards. In Chile, based on formal provisions, only the Ministry of Education and the National Education Council would be part of this process. While these institutions could decide to involve teachers, schools, the Agency or the Superintendency during this process, this decision would be made on an ad-hoc basis. Institutionalizing the participation of the ac- tors who will be directly affected by performance standards would add to the legitimacy and enforceability of these standards. Indeed, the international experience suggests that: • It is important that participants (students, teachers, school principals, schools, school owners) understand the standards they are expected to meet and vali- date them. • Therefore, early consultations with those participants who will be directly af- fected by performance standards are important. It maximizes the probability that standards will be recognized as valid –instead of being perceived as unreal- istic, ideological, unrepresentative of what should be expected of these actors, or disrespectful of teachers’ and schools’ autonomy. • Consultations with public- and private-sector institutions that specialize in cur- riculum development, student assessment, teacher and school evaluations, teacher education and training, and other related areas, may also inform the process of determining performance standards for the different actors. 77 Characteristics of the standards-setting process in the Table 23. selected OECD systems In the selected OECD systems In Chile, based on the proposed reforms General Three to four central government institutions participate in Two central government institutions would participate in the characteristics the process of setting performance goals. process of setting performance goals. No single institution is solely responsible for setting The Ministry of Education would ultimately be responsible for performance goals for all education-sector participants. setting performance goals for all actors. Performance To set performance goals for students, regular consultations To set performance goals for students, the Ministry would goals for take place between the Ministry or Department of Education consult regularly with the National Education Council. students and other institutions (e.g., qualifications authority, advisory council). Performance The standards of competence and conduct for teachers are Performance standards for teachers would be set by the goals for set by a teacher council in three of the four systems. Ministry and would have to be approved by the National teachers Education Council to become effective. No formal provision requires the Ministry to consult with teachers in the process of setting the performance standards that apply to them. Performance The selected systems’ experience in this area is Performance standards for schools principals would be set by goals for school heterogeneous and does not provide common lessons7. the Ministry. The standards would have to be approved by the principals National Education Council. Performance The standards that must be set by schools and school Performance standards for schools and school owners would goals for schools owners are typically set by the Ministry or Department of be set by the Ministry of Education and approved by the and school Education in consultation with the Inspectorate. Indeed, the National Education Council. In contrast with the international owners Inspectorate plays a crucial role in this process. experience, no formal provision would require the Ministry to consult with the Agency in the process of setting performance standards for these actors. 7 Each system’s institutional arrangement to set Evaluating the impact of education policies and programs performance goals for school principals is In all the selected OECD systems, impact evaluation falls under the purview of the Min- described in Strengthening the quality assurance istry or Department of Education, and impact evaluation studies are increasingly being system for basic and commissioned to educational experts and researchers from universities or think tanks secondary education: Comparison of education because of their relative independence to conduct such analysis as well as their capacity quality assurance systems to do so. and institutions in selected countries (The World Bank In the Netherlands, the Ministry of Education, Culture and Science has gone beyond the 2009a). In Ireland and Scotland, performance commissioning of independent impact evaluation studies, by adopting a particularly inno- standards for school vative structure to manage, coordinate and integrate the results of these studies. Before principals are set by the Department of Education the Knowledge Chamber was established, education policy decisions did not always have (in Scotland, these were an evidence- base and, when they did, they were usually based on one single study of the first established in 2002). 78 In New Zealand, broad topic or policy issue under discussion. This situation was identified as problematic by the performance goals were Ministry of Education, the Education Council and the national government, all of which established in 1997 in conjunction with working recommended a strengthening of the relationship between education policy and educa- groups that included tion knowledge. To address this concern, the Knowledge Chamber was created in 2006 associations of school principals, the New by the Ministry of Education. It consists of a platform for regular consultations between Zealand Educational top-rank government officials and the directors of knowledge institutions. Together, they Institute, the New establish the knowledge agenda and discuss the policy implications of research. From the Zealand School Trustees Association, the Teacher government side, participants include representatives from several agencies: the Ministry Registration Board, and of Education, the Inspectorate, the Education Council, the Netherlands Bureau for Eco- the Education Review Office. A school principal’s nomic Policy Analysis, SenterNovem (an agency of the Ministry of Economic Affairs), and standards are specified other government agencies that conduct or commission education research. The Knowl- by the School Board that edge Chamber enables these government agencies to share knowledge on education employs him/her. In the Netherlands, there are policy; decide collectively on research priorities for the education sector; and participate no standards for school in policy debates informed by different –and sometimes conflicting- research on the same principals. topic. Table 24. Characteristics of the Knowledge Chamber within the Dutch Ministry of Education Number and type of A core of 14 regular participants: the Ministry of Education’s Secretary-General, the four Directors-General (the participants Deputy Secretary-General, the Director for Primary and Secondary Education, the Director for Vocational and Higher Education and Science, and the Director for Culture and Media), plus the directors of the Education Council, the Advisory Council for Science and Technology Policy, the Council for Culture, the Scientific Council for Government Policy, the Netherlands Bureau for Economic Policy Analysis, the Social and Cultural Planning Office, the Netherlands Organization for Scientific Research, SenterNovem (agency of the Ministry of Economic Affairs), the Consultative Committee of Sector Councils for Research and Development, and the Education Inspectorate. A varying number of researchers are always present. Who participates depends on the agenda. Frequency of meetings Twice a year, once in spring (the knowledge agenda for the following year is established) and once in autumn (the policy implications of previously commissioned research are discussed). Type of meetings Initial presentation by a scientist followed by a discussion and the drawing of conclusions. Knowledge to be generated Trends and developments in the educational field. Behavior and perspective of education-sector stakeholders. and discussed Efficiency of education institutions. Ex-ante evaluation of newly designed policy instruments. Ex-post evaluation of the effectiveness of policies. Source: Stegeman and Rouw (2007). In Chile, research centers, think tanks and universities are actively involved in educa- tion research, and both the Ministry of Finance and the Ministry of Education regularly commission research to independent education experts. Institutionalizing the relationship between policy-makers and knowledge-makers would contribute to increase the extent to which research findings feedback into the design and implementation of education policy. Moreover, a consultation platform similar to the Knowledge Chamber would al- low government agencies to share their knowledge on education policies and programs; 79 make collective decisions about research priorities for the sector; and promote policy de- bates informed by the research findings of several studies on the same issue. The Dutch experience provides important lessons that may inform the institutionalization of this relationship, which are described in Figure 7. Figure 7. Strengths, Weaknesses, Opportunities and Threats (SWOT) Analysis of installing a Knowledge Chamber Positive Negative Strengths Weaknesses Regular and institutionalized consultation between The Knowledge Chamber per se does not guarantee that the policymakers and education experts improves the link between results of research will affect education policy. Internal origin policy and knowledge. Flexible design does not help to encourage scientists’ Policy is better informed when contrasting several researchers’ commitment to the purpose of the chamber. findings on a topic. Flexible structure allows the content and form of discussions to vary as necessary. Contributes to knowledge sharing between different units within the Ministry. Low management cost. Opportunities Threats Incentives may be developed to encourage policymakers Education experts’ and policymakers’ interests may not to use research in policy design (e.g., awarding a quality coincide. External origin hallmark to policy proposals validated by research; judging the Scientists and experts may be reluctant to express strong sustainability of major policy projects through the Knowledge opinions or controversial ideas about an issue. Chamber). If the policy implications of research are not clear, Creativity in the way meetings are organized may make the policymakers are less likely to make decisions based on debate more lively and honest (e.g., having students, parents research. or teachers participate in a meeting). Strategies to regularly disseminate knowledge among Knowledge Chamber participants may contribute to commitment. In sum, two main lessons that emerge from the international experience regarding the exercise of the impact evaluation function include: • A policy that systematically links education research and policymaking may be useful in distilling lessons from an increasing volume of information and pre- venting useful information from being lost or compartmentalized. 80 • Individual researchers may have their own vested interests. Conclusions and pol- icy decisions based on a single study should be avoided. At the very least, these conclusions should be validated by several researchers. Setting requirements to operate In the countries selected for this study, requirements to operate in the education sector are set either by the Ministry of Education alone (in the Netherlands) or by the Ministry of Education and the Teacher Council (in Ireland, New Zealand and Scotland). In countries that have a Teacher Council, requirements to enter and remain in the teaching profession are set by that institution, and requirements to operate as a school or school owner are established by the Ministry of Education. In countries that do not have a Teacher Council, such as the Netherlands, the Ministry sets requirements to operate for teachers, schools and school owners. In Chile, the Ministry of Education would set these requirements, but the Superintendency of Education would also set some requirements for schools and school owners. Providing and/or funding technical-pedagogical support Different mechanisms exist across countries to ensure that teachers and schools have ac- cess to the support they need to improve their education services. In the Netherlands, the Ministry of Education facilitates and funds access to technical-pedagogical support, but it does not directly provide this type of support. Technical-pedagogical support is provided by one central government institution in New Zealand, by two in Ireland and by three in Scotland. In these three countries, the Teacher Council provides technical-pedagog- ical support to teachers. In addition, Ireland and Scotland have established specialized agencies for the provision of technical-pedagogical support. Also, Scotland’s Inspectorate actively disseminates best practices that it identifies through the external evaluation of schools. In Chile, responsibility for the provision of technical-pedagogical support would fall under the Ministry of Education in Scenario 1, or under the National Education Service in Scenario 2. The international experience highlights the importance of technical-pedagogical support as a driver of educational improvement and suggests the following lessons: • Providers of technical-pedagogical support tend to specialize in one or two areas of educational improvement, such as the teaching of the curriculum, the use of ICTs in education, student assessment, school administration, or school self- evaluation. • It is important to ensure that the support services available respond to the spe- cific needs of teachers and schools. Shortage areas should be identified and initiatives to encourage the provision services in those should be put forward. • In the particular case of private-sector services to support the development of school plans and the implementation of school self-evaluation processes, a risk should be anticipated –the self-evaluation processes promoted by these private- 81 sector agencies may be perceived as the norm, and lead to a questioning in the role of external inspecting agencies. Ensuring adequate funding and its equitable distribution In Ireland, New Zealand and Scotland, the Ministry of Education would be in charge of ensuring the availability and distribution of adequate and equitable resources for the provision of education. In the Netherlands, the Ministry of Education is in charge of administering central government funding for education, but the actual channeling of funds from the central government to school owners is managed by the Central Funding of Institutions Agency. In Chile, the Ministry of Education would continue to administer and channel funds from the central government to school owners. Funds are allocated on a per-student basis. Since the introduction of the Preferential School Subsidy in 2008, schools that serve students from disadvantaged backgrounds are eligible to receive ad- ditional funds and are held accountable for performance. Also, in the selected systems, the financial relationship with schools is through their own- er (the school board, the competent authority, etc.). Funds are allocated on a per-student basis and sent to a school owner, who then decides how to distribute these funds across the schools it manages. Applying performance-based accountability measures In the selected OECD systems, the choice and implementation of performance-based consequences is typically a shared responsibility that at the very least involves the Ministry of Education and the Education Inspectorate. 82 Table 25. Consequences of repeated low performance in external evaluations Possible consequences for repeated failure of Ireland Netherlands New Scotland performance evaluations Zealand For students Denial of secondary school diploma n n n n For teachers Professional development and support for individual n teachers Professional development and support for the school’s n n n n teaching staff as a whole Denial of teaching certificate to teachers on probation n n n Suspension/ Removal from the classroom Removal of teaching certificate n n n For school Professional development and support n n n n principals Removal from duties For schools Reputational sanction (school report made available n n n n and school to the general public) owners Withholding of public funds n n n Reduction of public funds n n n Intervention of the school n n n School closure n n n As already mentioned in the section devoted to the Agency for Education Quality, in the selected systems, students are required to pass a national assessment to be able to graduate from secondary school. Whether this would be the case in Chile remains to be determined. The external evaluation of teachers’ and school principals’ performance is part of the school evaluations conducted by Inspectorates. The quality of teaching is a crucial com- ponent of these evaluations, but performance is assessed for the entire teaching staff, without identifying – much less sanctioning – individual under-performing teachers. Ac- cordingly, the consequences of low teaching quality are applied to the group as a whole, usually in the form of intensified professional development and support for the teachers in that school. The only exception applies to beginning (first-year) teachers. In this case, in most systems an external evaluation of individual new teachers is carried out during the 83 probationary period, before a teacher is confirmed as fully qualified for the profession. In contrast, individual performance evaluations of fully qualified teachers are administered by the authority that hired them (a school principal, school owner, school board, munici- pality or local educational authority). In countries that have a Teacher Council (Ireland, New Zealand and Scotland), this institution may withdraw teacher certifications based on proof of serious misconduct or severe incompetence. In Chile, continuous unsatisfactory performance as measured by the National System of Teacher Performance Evaluation may lead to teacher dismissals. Schools are subject to performance-based consequences that vary in nature and intensity across the selected systems. In Ireland, sanctions for under-performing schools are only of a reputational nature. In the Netherlands, New Zealand and Scotland, the Ministries or Departments of Education apply some kind of disciplinary action to under-performing schools, such as intervention of the school, a decrease in financial resources, the with- drawal of certain rights, the imposition of fines, or school closure. In the Netherlands, New Zealand and Scotland, schools and school owners are sanc- tioned when they fail to comply with statutory regulations or when they repeatedly fail to meet standards regarding student outcomes. Inspectorates apply additional criteria to evaluate the quality of schools and school owners (e.g., they look at teaching prac- tices, management practices, etc.). However, failure to comply with these criteria is usu- ally not subject to disciplinary sanctions. The reason is that, although these criteria rep- resent what is generally understood as “quality� education, there is a strong conviction that schools must remain autonomous to decide how to provide education as long as they comply with the statutory regulations and show proof that their students are meeting the learning outcomes. On the other hand, information about these additional criteria is included in the individual school reports that are made available by the Inspectorate to the general public. As mentioned previously, there is evidence that the publication of these reports affects the behavior of educational providers and promotes educational improvement. In other words, even if some evaluation criteria are not tied to disciplinary measures, informing the public about how a school does in relation to these criteria may inflict a reputational type of sanction or reward and influ- ence the school’s behavior. In Chile, the Agency for Education Quality, the Superintendency of Education and the Ministry would all play a role in holding schools accountable for performance. It is most likely that sanctions would be applied based only on schools’ repeated under-perfor- mance in national assessments of student achievement, or where schools have failed to comply with statutory regulations. This is in line with the experience of the Netherlands, New Zealand and Scotland. The evidence in those countries suggests that, even if school evaluation frameworks do not have a statutory recognition, the publication of perfor- mance reports derived from the application of those frameworks can affect teachers’ and school principals’ behaviors and contribute to educational improvement. 84 Implementation of institutional reforms: Potential obstacles and lessons Human resources management An important challenge to be addressed as Chile plans for the implementation of the proposed reforms is the management of human resources and, in particular, of staff cur- rently working at the Ministry of Education and who will be affected by the reforms. In this regard, the profound institutional reforms that took place in New Zealand beginning in 1989 included the following elements: • All positions were created anew. Former employees of the Department of Education had to apply to at least one position if they wished to remain in the public sector. • Several options were provided to staff. Employees could choose to some extent where they wanted to belong during and after the transition. In line with New Zealand’s experience, options could include the ability to: (i) apply to one or more positions at the Ministry, the Agency and/or the Superintendency, and when applying to several positions, do so through only one application; or (ii) accept a financially attractive exit option (“redundancy�) for staff who do not wish to be part of the reform. • New positions were assigned giving priority to former staff from the Min- istry and, among these, on a merit basis. Past performance, the applicants’ cover letter and motivation, and the match between the applicants’ qualifica- tions and the job requirements were all taken into account to make hiring deci- sions. • Training was provided to staff who wished to apply to a position that was not aligned with their qualifications or experience. • Staff had access to counseling services on issues such as how to prepare the CV and cover letter, or how to prepare for an interview, as well as on financial planning, especially for staff considering the “redundancy� package. iv. National Education Service Institutional design The lessons for the National Education Service (NES) coincide with those outlined under the section devoted to the Ministry of Education, in particular in the discussion about the provision of technical-pedagogical support. Specifically, the proposed NES would play an important role in supporting schools and teachers to improve teaching and learning. In this sense, the ability to recruit, motivate, and retain staff who are knowledgeable 85 in evaluating and raising school quality will be critical to the success of this proposed agency. At first, focusing on specific subject areas may be desirable, in order to develop a culture of working closely with school administrators and teachers to improve teaching and learning. One practical option is to focus on the same subject areas as does SIMCE, so as to also contribute to a culture of using student assessment information to inform school and classroom practices. As the NES gains experience and identifies shortcomings in schools, it can also play a role in managing external technical assistance to support pedagogic improvements in schools. Implementation of institutional reforms: Potential obstacles and lessons As mentioned in Section 4.a.i., it seems wise for Chile to maintain the number of quality assurance institutions at four, as would be the case under scenario 1. The implementation challenges under this scenario would already be considerable. The creation of the Na- tional Education Service, a fifth institution which would be part of the quality assurance system under scenario 2, could become overwhelming given the additional demands it would place on the sector. International experience suggests that institutions that provide technical-pedagogic sup- port tend to take a more active and directive role in under-performing schools. Schools whose performance is up to standard are generally granted greater autonomy in deciding if they wish to use part of their resources to obtain support services for their teaching staff and, if they do, they enjoy autonomy in deciding what type of support to obtain. Schools that are not up to standard are required to obtain support. In general, the decision re- garding the type of support a school should receive results from a conversation between school staff and staff from the Inspectorate and the Ministry or Department of Education. In this sense, international experience suggests the desirability of providing differential support to schools based on their performance, while at the same time ensuring that the supply of support is not restricted to those who are below standard but, instead, is avail- able to any school that wants external support to improve education services. 86 87 88 5 Conclusion A review of the quality assurance institutions and reforms of high-performing education systems with strong private sector participation indicates that the proposed reforms in Chile are likely to contribute to improving education quality and equity. Nevertheless, as the international experience in education reforms as well as the experience of local re- forms in other sectors, the design of the reforms is only a first step. Implementation mat- ters. Therefore, in this final report, we have summarized not only the key features of the four selected high-performing education systems (Ireland, the Netherlands, New Zealand and Scotland), but strived to draw lessons from the evolution of these systems that may shed light on the design and implementation of the proposed reforms in Chile. In the rest of this concluding section, we highlight several key lessons related to the design and implementation of education quality assurance reforms. Regarding the design of education quality assurance systems, four central lessons emerge from the review of the selected OECD systems. First, a single institution is formally re- sponsible for oversight of the education sector in these systems whereas in Chile the current proposals establish two distinct agencies to share oversight responsibilities – the Superintendency and the Agency for Education Quality. This separation of oversight func- tions between two agencies results in coordination challenges that the selected OECD countries have not needed to address. Second, none of the OECD systems reviewed carry out the performance evaluation func- tion within a single agency. Typically, the evaluation of schools is performed by the Inspec- torate, whereas student assessments are developed and applied by a different agency. In contrast, the proposals in Chile imply that the Agency would be responsible for evaluat- ing both schools and students. While this design may make sense for Chile today because it implies more manageable implementation efforts, it is worth noting this difference to inform future directions in institutional reform. Third, most selected OECD systems have put in place a teacher council to regulate the standards of competence and conduct necessary to enter and remain in the teaching pro- 89 fession, and to maintain a registry of qualified teachers. Often, teacher councils may also be responsible for accrediting initial teacher education programs and may provide profes- sional development and technical-pedagogical support to teachers and school principals. Teaching councils were introduced relatively recently and respond to years of demand by teachers to regulate their own profession. The reforms that have been proposed for Chile do not include a teacher council. Again, this likely makes sense for Chile today, because it maintains the implementation efforts at a fairly reasonable level. But it is worth noting that the selected systems have evolved toward creating teaching councils, as this may inform future directions in institutional reform. FFinally, in two of the selected systems (Ireland and Scotland), specialized agencies are charged with the provision of technical-pedagogical support for teachers and schools. In Chile, the possible creation of a National Education Service would provide for such a specialized institution. Having agencies that specialize on a few quality assurance func- tions implies a trade-off. While specialized agencies are likely to carry out their assigned functions more effectively than a single institution with responsibility for all functions, having a relatively larger number of institutions implies greater implementation efforts to install these agencies and, subsequently, to ensure coordination between them. When many agencies are involved in education quality assurance, a fair amount of institutional maturity is required for the system to function effectively as a whole. At this point, it seems wise for Chile to maintain the number of quality assurance institutions at four (the Ministry, the National Education Council, the Agency and the Superintendency). The implementation challenges of this reform scenario are already considerable, and a gradual implementation of the changes is advisable. Three critical lessons emerge from the international experience regarding the implemen- tation of education quality assurance reforms. First, the experience of the selected sys- tems highlights the need to carry out regular and broad-based consultations with key stakeholders. Consultations are an essential component of these education systems’ in- stitutional culture. For example, to set performance standards for schools, teachers, and students, extensive consultations take place early on with a broad range of stakeholders including representatives of students, parents, teachers, school principals, schools, school owners, education inspectors, education researchers and experts, knowledge institutions, local educational authorities, and the range of government agencies that are part of the education quality assurance system. But the consultative nature of the education sec- tor goes beyond the process of setting standards: consultations are an integral part of the process of inspecting and evaluating schools, as well as the process of designing an institutional reform. As Chile moves forward with the implementation of institutional re- forms, it will be critical for the Ministry, the Agency and the Superintendency to introduce early on a regular consultative process with the different actors affected by the reforms. Indeed, the implementation of the proposed reforms presents an opportunity to intro- duce a cultural shift within the Chilean education sector. Second, the experience of the selected OECD systems emphasizes the importance of building constructive relationships between inspectors and school staffs. Indeed, the ex- ternal evaluation process presents an opportunity for inspectors to promote discussions 90 between a school’s teaching and non-teaching staff about different dimensions of the school’s quality; build their capacity to use evidence to identify the school’s strengths and weaknesses; encourage them to think about ways in which they could address their weaknesses; and provide examples of how schools with a similar context have dealt with similar challenges. For example, instead of simply applying a “checklist� to evaluate the quality of education in a particular school, inspectors also take the time to meet with teachers and administrators, provide them with feedback, and listen to their views and reactions. In addition, all systems have mechanisms in place to measure the effective- ness of the evaluation process –from “customer satisfaction� surveys that are filled in by schools to assess the extent to which the inspection process has contributed to their work, to more rigorous impact evaluations that are carried out by the Inspectorate or commissioned to independent researchers. As Chile moves forward with the implementa- tion of the functions assigned to the Agency for Education Quality, it will be imperative to acknowledge the Agency’s role to “guide educational improvements�, which is rec- ognized in Article 6 of Law Proposal No. 609-356. Fulfilling this role will require several steps, from recruiting the right leadership for the Agency to ensuring extensive training of inspectors and installing self-audit mechanisms. Third, the experience of the selected systems shows that political leadership and commit- ment are a key determinant of a reform’s success. The success of institutional reforms of the magnitude proposed for Chile’s education sector requires commitment at the highest political level and, with that, a commitment of the resources necessary to implement the reform in a gradual manner. The proposed reforms provide an opportunity to introduce modernization-of-the-public-sector features that would contribute to the effectiveness and efficiency of government institutions. Even if this opportunity is not fully seized, implementing the reform will require a substantial investment of public resources. Mak- ing a commitment to the education sector is usually not politically attractive, because the short-term costs are likely to outweigh the short-term benefits, as the bulk of benefits would arise in the longer term. The successful implementation of the proposed reforms will require a political leader who has the vision and willingness to assume the costs that are necessary to overhaul the quality of education in Chile. Further, the range and scope of the proposed reforms in Chile surpasses those of the selected OECD systems, suggest- ing the critical role that leadership will play in ensuring the success of the reforms. Given the magnitude of the reform, its implementation will likely take several years. Indeed, the international experience suggests the need for planning for a gradual, multi-year implementation process. Last, although the quality of education in Chile compares favorably to that of other Latin American countries, it lags well behind that of OECD countries. While, over time, the proposed quality assurance reforms will likely contribute to reduce this gap with OECD, other education reforms will also be necessary. Some important reforms have recently been introduced to contribute to educational improvement, including the General Law of Education and the Preferential School Subsidy. But challenges remain in important areas, such as in the quality of initial teacher education programs as well as in strengthening the systems of professional development for in-service teachers. 91 Bibliography Background reports Education Counts (New Zealand). Education Statistics of New Zealand 2007. Available from: The World Bank. 2009a. Strengthening the http://www.educationcounts.govt.nz/publica- quality assurance system for basic and second- tions/ece/2507/27456/4 ary education: Comparison of education quality assurance systems and institutions in selected Ley 23.370 (Chile). August 2009. Ley General de countries. Educación. The World Bank. 2009b. Implementation of Mineduc & Organización de Estados Iberameri- education quality assurance reforms: Lessons canos (eds.). 2009. ¿Qué nos dice PISA sobre la for Chile from selected OECD systems and local educación de los jóvenes en Chile? reforms in other sectors. Ministry of Education, Culture and Science (the Bitrán, Ricrado, Paulina Gómez and Marcos Netherlands). Key Figures 2004-2008. Avail- Vergara. 2009. Lecciones aprendidas en base a able from: http://www.minocw.nl/documenten/ la implementación de reformas sectoriales en key%20figures%202004-2008.pdf Chile. La reforma de salud. Murnane, Richard, Lindsay Page and Emiliana Ve- Vargas Viancos, Juan Enrique. 2009. Reforma gas. 2009. The Preferential School Subsidy (SEP) procesal penal. Lecciones como política pública. in Chile. Early evidence on its effects. Unpublished manuscript. Washington, DC: The World Bank. references Propuesta de Ley No. 609-356 (Chile). July 2008. Barrera-Osorio, Felipe. 2009. Education System Formula indicación sustitutiva del proyecto de Reform: An analysis using relative position in in- ley que crea la Superintendencia de Educación. ternational tests over time. Unpublished manu- script. Propuesta de Ley No. 1151-356 (Chile). Decem- ber 2009. Fortalece la educación pública. Butterworth, Graham and Susan Butterworth. 1998. Reforming Education: The New Zealand Scottish Government. School Education Statistics Experience 1984-1996. Dunmore Press and the 2008. Available from: http://www.scotland.gov. New Zealand Ministry of Education. uk/Topics/Statistics/Browse/School-Education Department of Education and Science (Ireland), Standaert, Roger. 2000. Inspectorates of Educa- Education Statistics 2007/2008. Available from: tion in Europe: A critical analysis. http://www.education.ie/home/home.jsp?pcate Stegeman, Hans and Rien Rouw. 2007. The gory=17216&ecategory=17321&language=EN Knowledge Chamber (the Netherlands). In Evi- de Wolfe, Inge F. and Frans J.G. Janssens. 2007. dence in education: linking research and policy, Effects and side-effects of inspections and ac- published by the OECD and the Center for Edu- countability in education: an overview of em- cational Research and Innovation. pirical studies. Oxford Review of Education. Vol. The World Bank. 2007. Institutional design for 33, no. 3, pp. 379-396. an effective education quality assurance. Ehren, Melanie, Frans Leeuw and Jaap Schee- Treviño, Ernesto, Franscisca Donoso and y Ma- rens. 2005. On the Impact of the Dutch Educa- carena Bonhomme. 2009. ¿Cómo las escuelas tional Supervision Act Analyzing Assumptions chilenas pueden mejorar el aprendizaje en Cien- Concerning the Inspection of Primary Education. cias? In Mineduc & Organización de Estados Ib- American Journal of Evaluation. Vol. 26, no. 1, eramericanos (eds.). ¿Qué nos dice PISA sobre la pp. 60-76. educación de los jóvenes en Chile? 92 The World Bank Human Development Department The Latin American and Caribbean Region