RBF EDUCATION EVIDENCE RESULTS-BASED FINANCING JAKARTA Can Self-Evaluations and Soft Performance Contracts Help Schools Achieve Education Standards? DECEMBER 2019 REACH supported a study in Jakarta to explore ways to strengthen school- based management and help schools use funding more effectively to improve performance. The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and Schools with more implementation of successful performance-based programs. autonomy perform better on student assessments. In an effort to improve education perform better on international outcomes, many countries student assessments.1 But evidence have introduced school-based also shows that for school-based management reforms to decentralize initiatives to be effective, institutional decision making to the school level. reforms should accompany increases The idea is that by shifting decision- in authority and resources to schools. making authority to schools, where teachers and administrators are Indonesia began to decentralize REFORM better informed about student its education services in 2003, and needs, funds for education will be today schools are responsible for used more effectively to improve the drawing up their own development Institutional reforms teaching and learning environment. plans and managing funds they should accompany While school-based management receive through national and local increases in authority and school grant programs. However, resources to schools. reforms are not uniform and vary significantly across countries, evidence suggests that schools research suggests that countries continue to struggle to implement where schools have more autonomy school-based management This note was adapted from Fasih, Tazeen, Kevin Macdonald, Unika Shrestha, Bapak Santoso, and Wisno Harto Adi Wijoyo. 2019. “Managerial Capacity and the Effectiveness of School-Based Management Reforms: Evidence from a Randomized Evaluation in Jakarta.” 2 RFB EDUCATION | EVIDENCE mechanisms and to use the funds coupled with a soft performance capacity in the schools. The self- they receive effectively. To address contract between schools and the evaluation tool only benefitted these issues, the Results for Jakarta government. While this schools with low capacity and Education for All Children (REACH) contract committed the school to had no effect on schools with high Trust Fund provided a grant to achieve targets associated with capacity. The self-evaluation tool Jakarta, Indonesia’s capital city and education, they were not linked to coupled with the soft performance a special administrative region, to any financial rewards or sanctions. contract had a positive impact on explore ways to strengthen school- The randomized control trial explored standards achievement in high- based management. The project how these interventions affected capacity schools but a negative tested two interventions at the a school’s rating, as defined by its effect on low-capacity schools. school level: a) a self-evaluation progress towards achieving the The findings demonstrate the exercise based on a set of national national education standards. importance of managerial capacity education standards to help schools in determining the effectiveness of prioritize funding allocations; and The results of the interventions school-based management reforms.   b) the same self-evaluation exercise varied with the level of managerial CONTEXT governments. Approximately 60 Recent studies have shown percent of districts and 45 percent however that schools in Jakarta of provinces provide schools with were not allocating their resources The Indonesian government has grants for their budgets.3 Schools effectively towards meeting the significantly increased its investment are meant to use the funding to help national standards. A World Bank in education, and by constitutional them meet two sets of standards: public expenditure review found mandate it must allocate 20 percent the Minimum Service Standards schools were spending little on of its annual budget to education. (MSS) and the National Education teacher professional development This increase in expenditure Standards (NES), the latter of but overspending on hiring non-civil has coincided with significant which are relevant to this study. servant teachers. Schools also decentralization of education The NES consist of 157 standards spent significant amounts on school service delivery over the past two divided into eight domains: content/ maintenance, despite that Jakarta decades.2 Decentralization has curriculum, process, student/ government already implements a put local governments, particularly graduate competency, HRD/ large school maintenance program. district administrations, at the center educator, facilities/infrastructure, And government audits found many of basic education service delivery. management, financing, and schools failed to produce expenditure District responsibilities include the evaluation (Table 1). These are plans, a requirement for receiving overall management of the education calculated for every school at least both types of grants. system as well as management of once every four years and are used public financing for primary and In an effort to improve the quality of for accreditation purposes. junior secondary schools. education, the Jakarta government Schools receive two types of per capita grants to support the achievement of national education standards. The Bantuan Operasional Jakarta Sekolah (BOS) grants are provided by Schools do not allocate the Government of Indonesia while resources effectively the Biaya Operasional Pendidikan towards meeting national (BOP) grants are provided by local education standards. JAKARTA 3 introduced an electronic system Table 1. National Education Standards Structure to help schools align their funding Domains Examples of topics covered by standards No. of standards allocations with the national 1. Content/ Curriculum development and implementation; 18 standards and channel more funding Curriculum implementation of self-development programs; to areas they were weak in. This use of educational calendar system, called Rencana Kegiatan 2. Process Link between curriculum, syllabus, and lesson 11 dan Anggaran Sekolah berbasis implementation plans; principal’s supervision of learning process and evaluation of teachers Elektronik (e-RKAS), features a built-in goods and services catalogue 3. Student/Graduate Students gain learning experience related to 20 Competency curricular goals (natural and social phenomena, to help compute costs. It allows environment, art, character development, etc.) schools to produce an activity plan  RD / Educator 4. H Qualifications of teachers, principals and other 26 classified according to the National staff members Education Standards, a spending plan on monthly expenditures, and 5. Facilities/ Safety of buildings; availability of facilities 16 Infrastructure a budget expenditure document that allows schools to monitor their 6. Management School vision and setting goals; establishing 20 expenditure. The system has enabled and implementing planning; staff job descriptions the Jakarta government to make its and review grant distributions conditional on the 7. Financing Use of RKAS planning tools in resource allocation 25 submission of school expenditure plans and allows education officials 8. Evaluation Classroom evaluation practices 21 to monitor the design, progress, and implementation of these plans. WHY WAS THE INTERVENTION CHOSEN? To support the use of the new effects of school self-evaluations on management has largely been electronic planning and resource- education outcomes, while research studied from the perspective of how allocation system, the government on performance-based contracts has well participatory accountability wanted to explore different ways shown mixed results.4 The REACH- mechanisms, such as the involvement to help schools identify their funded project sought to explore of parents in schools, work.5 What has weaknesses and use the system to the effects of both by examining been less studied is how the capacity allocate resources in a more effective how self-evaluations, and how self- of the school leadership itself can way. Under the REACH grant, a evaluations plus soft performance alter school-based management self-evaluation tool was developed contracts, affected Jakarta schools’ outcomes. Presumably school-based to help schools identify their efforts to achieve the National management reforms would be less strengths and weaknesses, while a Education Standards. effective if school leaders lack the performance contract mechanism ability to use their autonomy and enabled school leaders to define and The effectiveness of school-based available information to make the best work towards specific goals. Schools reforms also depends on how decisions. To address this question, could then make better decisions decision-makers, particularly school the REACH project also looked at on how to plan and allocate their principals, use their autonomy and how school management capacity, resources using the online system. available information. The capacity measured at baseline, altered the Little research has been done on the of such agents in school-based effects of the two interventions. 4 RFB EDUCATION | EVIDENCE HOW DID THE INTERVENTION WORK? A randomized control trial was schools were randomly assigned evaluation, school principals also set up to test two school-based to one of three groups, with 50 signed soft performance contracts management interventions in Jakarta schools in each group. Schools with Jakarta’s education department. schools: an electronic self-evaluation, in the first group completed the These performance contracts were and an electronic self-evaluation electronic self-evaluation and used “documents of promise” in which school principals identified the NES plus soft performance contract. the information to create their annual domains where they would seek A total of 150 junior secondary school plan. Those in the second improvements for their schools and group completed the self-evaluation, specified the target levels they would used the information to complete aim for, although the results were their annual plan, and signed soft Treatment Group 1 performance contracts with the not linked to either monetary rewards or penalties. 50 schools Jakarta government. The remaining Electronic self-evaluation used to 50 schools served as a control group, The evaluation ultimately considered help create annual school plan creating their school plans without 132 of the 157 NES standards, and any interventions. No school received did not include standards that were Treatment Group 2 the performance contract alone. either not collected as part of the 50 schools survey or where there were a lot Electronic self-evaluation used to For the first group, school of missing values. Because of the help create annual school plan, managers used an electronic self- high number of potential outcome coupled with soft performance evaluation tool to rate their school’s variables, the evaluation grouped contract with government performance against the National the standards into four categories. Education Standards’ 157 standards. Three were based on the ease with Control Group These included such items as which school management teams 50 schools principal’s supervision of the could affect them: standards that the No interventions used learning process and evaluation of management team could achieve teachers (under the Process domain) alone (e.g., curricular standards); and curriculum development and standards that required action implementation (under Content/ by school stakeholders beyond Curriculum). This self-evaluation the management team (e.g., exercise helped the managers to student competency standards); identify where their schools needed and standards that likely required to improve and where they needed intervention from stakeholders more resources—information they outside the school including could then use when drawing up their government authorities (e.g., staffing school development plans through qualifications). The fourth category the new online planning system included the management standards e-RKAS. The school managers domain (one of the NES’s eight participated in an intensive, three-day domains and which incorporated training on how to rate schools standards addressing such things based on the NES indicators. as school vision and goal-setting). This category was used to provide a For the second group, in addition measure of the management capacity to completing an electronic self- of each school at baseline, so that JAKARTA 5 Table 2: Descriptive Statistics by NES Categorization NES standards by locus of autonomy NES School All school School and other management management team stakeholders stakeholders standards Number of NES standards included in the category 45 34 34 19 Average score of NES standards within the category at baseline 3.70 3.50 3.52 3.75 Proportion of NES standards fully achieved within the category 76.0% 67.4% 66.3% 80.3% at baseline for schools with low baseline management capacity 74.1% 64.7% 66.0% 75.0% for schools with high baseline management capacity 78.8% 71.3% 66.7% 87.7% schools could be classified as low and from 3.5 to 3.75 for each standard. at baseline. Because of ties in high management capacity schools. And the average proportion of the index, 41 percent of schools standards rated as fully achieved in were categorized as having high The outcome variable for the each category totaled more than half, management capacity. evaluation was the proportion of with higher proportions for schools standards within each of the four with high baseline management While a randomized selection of categories that were “fully achieved.” capacity (Table 2). Because of some schools allowed for effects of the The evaluation determined the imbalance in the average proportion two interventions to be unbiased impact of the two interventions by of standards that were fully achieved estimates for the underlying estimating how much the outcomes within each category at baseline, a population of 292 junior schools in changed for each treatment group difference-in-differences approach Jakarta, the school management and comparing these differences was used to measure the effects of capacity was not exogenous in with those of the control group. the two treatments. the evaluation and could possibly Tests at baseline of differences in the represent other underlying factors. reporting and levels of the outcome To differentiate the effects of the To check if differences between variable between the treatment interventions between schools high and low management and control groups were generally with high and low management capacity schools can be attributed statistically insignificant but a capacity, the average score of to management capacity rather difference-in-differences approach the management standards (for than other underlying factors, the was used to measure the effects of those without missing values) at differences in several background different treatments. baseline was used as an index. variables (such as school’s number Management standards included of teachers, number of students, The team collected baseline data having institutional vision and goals, total school expenditures, etc.) on NES achievement in May and having work plans and adhering to between high and low management June 2016 before the interventions them, having guidelines and clear capacity schools were measured. started in July, the beginning of the job descriptions for staff as well No statistically significant difference school year. Endline data collection as performance evaluations of between high and low management occurred in January 2018 to allow for staff, among others. High and low capacity schools was found for any one full fiscal year cycle between the management capacity schools were of those variables. collections. Schools’ average scores defined by being above and below at baseline were quite high, ranging the median index value, respectively, 6 RFB EDUCATION | EVIDENCE WHAT WERE THE RESULTS? Receiving the self-evaluation the self-evaluation, schools with contract was added to the self- increased the proportion of fully low capacity saw a statistically evaluation, schools with low achieved standards under direct significant positive effect in terms managerial capacity saw a control of the school management of achieving National Education statistically significant negative team and of those that required Standards (NES) directly under their impact on the proportion of action by both school and non-school influence (0.04 SD) (Table 3). This NES standards they achieved actors, regardless of whether the result was not surprising because that were under the influence of school principal signed a performance managerial capacity at baseline the school management team contract. For the standards that the was determined by how effective (-0.03 SD). This negative effect school management team had direct schools were at managerial tasks neutralized any positive effect of control over, positive effects were such as having institutional vision, the self-evaluation. One potential found from either receiving the self- goals, and work plans. Thus, explanation for this finding is that evaluation alone or receiving it with schools categorized as having low-capacity schools may have the performance contract (0.03 SD high managerial capacity were set unrealistic or hard-to-achieve for both). For standards controlled arguably already good at managerial targets in terms of meeting the by school and non-school actors, a tasks and did not benefit from National Education Standards. positive effect was found only when undertaking the self-evaluation. schools received both interventions In contrast, schools with low However, performance contracts (0.03 SD). Receiving the self-evaluation managerial capacity gained new had a positive impact on schools alone was also found to improve knowledge about their strengths and with high management capacity. the proportion of fully achieved weaknesses that could help school In the case of schools with high management standards (0.03 SD). leaders to prioritize and plan better management capacity, adding the and ultimately achieve a higher soft performance contract to the Self-evaluations had a positive proportion of the national standards. self-evaluation had a statistically effect on schools with low significant positive effect on management capacity at baseline. Adding soft performance achievement of NES domains While schools with high managerial contracts had a negative effect directly under the influence of the capacity at baseline did not see a on schools with low management management team (0.04 SD). statistically significant impact from capacity. When the performance JAKARTA 7 Table 3: Effect (Difference-In-Differences) of Interventions on Proportion of Standards Fully Achieved by Baseline Management Standards and NES Categorization NES standards by locus of autonomy NES School All school School and other management management team stakeholders stakeholders standards For schools with low management standards at baseline 0.04* -0.02 0.02 0.04* Effect of receiving self-evaluation only (0.02) (0.05) (0.02) (0.02) Effect of receiving both self-evaluation and 0.01 0 0.02 0.01 performance contract (0.02) (0.04) (0.02) (0.02) Difference in effects (effect of adding the -0.03* 0.02 0 -0.03 performance contract) (0.02) (0.05) (0.02) (0.02) For schools with high management standards at baseline 0.01 -0.08 0.02 0.01 Effect of receiving self-evaluation only (0.02) (0.05) (0.02) (0.01) Effect of receiving both self-evaluation and 0.05*** -0.04 0.03 -0.02 performance contract (0.02) (0.05) (0.02) (0.02) Difference in effects (effect of adding the 0.04* 0.04 0.01 -0.02 performance contract) (0.02) (0.05) (0.02) (0.02) Number of observations: 150 schools, 50 in each treatment and control group Standard errors denoted in parentheses, statistical significance at the 1, 5, and 10 percent levels denoted by ***, ** and *, respectively. WHAT WERE challenging for schools to identify The use of a web-based filing which issues they should prioritize in system can be especially helpful in THE LESSONS their school plan. large education systems. Jakarta already has an online platform LEARNED? Both the local government and that allows schools to submit school leadership need to have their development plans, while at a good understanding of any the national level, all public and Lessons on strengthening school- new school-based initiatives private schools in Indonesia can based management were drawn for them to be effective. Local access a new electronic quality from the results of the intervention governments play a central role in assurance instrument to conduct and the team’s experience overseeing and disbursing funding self-evaluations. Both the Jakarta supporting the Jakarta government. to schools in Indonesia. The central and national governments are These included: government can support this role moving towards linking school by providing good information to planning and self-evaluation tools Developing a simple, objective, and local authorities on new initiatives into unified electronic platforms. effective self-evaluation tool for to improve implementation. In the With more than 250,000 schools in schools can make it easier for them case of the National Education Indonesia, the development of linked to create a strong school plan. The Standards, school leaders should online platforms for conducting self- current self-evaluation tool is quite understand the importance of evaluations and submitting school complicated since it covers more creating strong school plans and plans will make it easier for school than 150 standards. Even though it budgets so schools can effectively management and government helped low-capacity schools improve use government grants to achieve authorities alike to monitor school their performance, a complicated the education standards. progress and results. self-evaluation can make it more Overall, these CONCLUSION impact high-capacity schools. The use of soft performance contracts, results support the Many countries are implementing in contrast, had a positive effect on idea of expanding school-based management reforms high-capacity schools but a negative impact on low-capacity schools. the use of electronic and giving more authority to principals, teachers, parents, and Overall, these results support the idea self-evaluations to local communities to try and improve of expanding the use of electronic self-evaluations to all schools, and education services. Indonesia began all schools. decentralizing its education system in fact, the Jakarta government almost 20 years ago, but conferring is moving to link such a tool with schools with more authority and its existing online school planning funding hasn’t automatically led and budgeting system. The mixed to higher-quality service delivery result of the performance contracts or improved learning. Institutional underscores the important role reforms are also needed, and the of school management capacity Special Administrative Region of in school-based management Jakarta has been a pioneer of such reforms. For such reforms to work, efforts. Supported by REACH, Jakarta principals, teachers, parents, and tested two school-based initiatives: the local community must have the electronic self-evaluations and soft information and skills needed to do performance contracts. The use their jobs, whether they are running of self-evaluations helped schools a school, teaching students, or with low managerial capacity in their providing oversight—to achieve the efforts to achieve National Education common goal of giving students a Standards, while it did not negatively better learning experience. 1 Fuchs, Thomas, and L. Woessmann. 2007. “What Accounts for International Differences in Student Performance? A Re-Examination Using PISA Data.” Empirical Economics 32 (2–3): 433–64. Woessman, Ludger. 2003. “Schooling Resources, Educational Institutions, and Student Performance: The International Evidence.” Oxford Bulletin of Economics and Statistics 65 (2): 117–70. 2 World Bank. 2013. Local Governance and Education Performance: A Survey of the Quality of Local Education Governance in 50 Indonesian Districts. Washington, DC: World Bank. 3 World Bank. 2015. Assessing the Role of the School Operational Grant Program In Improving Education Outcomes in Indonesia. Washington, DC: World Bank. 4 Ladd, Helen F. 1996. Holding Schools Accountable: Performance-Based Reform in Education. Washington, DC: Brookings Institution. Stucker, J. P., and G. R. Hall. 1971. The Performance Contracting Concept in Education. Santa Monica, CA: RAND. 5 Bardhan, P., and D. Mookherjee. 2006. “Decentralization and Accountability in Infrastructure Delivery in Developing Countries.” Economic Journal 116 (508): 101–27. Blimpo, M. P., and D. K. Evans. 2011. School-Based Management and Educational Outcomes: Lessons from a Randomized Field Experiment. Washington, DC: World Bank. Bruns, B., D. Filmer, and H. Patrinos. 2011. Making Schools Work: New Evidence on Accountability Reforms. Washington DC: World Bank. Duflo, Ester, Pascaline Dupas, and Michael Kremer. 2015. “School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools.” Journal of Public Economics Vol. 123: 92–110. Galiani, S., P. Gertler, and E. Schargrodsky. 2008. “School Decentralization: Helping the Good Get Better, but Leaving the Poor Behind.” Journal of Public Economics Vol. 92, Issues 10–11: 2106–20. Parker, C. 2005. “Teacher Incentives and Student Achievement in Nicaraguan Autonomy Schools.” In Incentives to Improve Teaching: Lessons from Latin America, edited by E. Vegas, 359–88. Washington, DC: World Bank. PHOTO CREDITS: Cover: Students at a secondary school in Jakarta. Photo courtesy of World Bank. Page 4: Students at a secondary school in Jakarta. Photo courtesy of World Bank. Page 6: School managers participate in a three-day training on how to rate schools based on NES indicators. Photo courtesy of World Bank. RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.