76460 School Spending and Student Performance: BOOST Analysis of Resource Allocation in Serbian Primary Education1 The World Bank June 2012 1 This report was authored by Igor Kheyfets (Economist, ECSH2) under the Serbia Education Technical Assistance (TA) engagement led by Ivana Aleksic (Human Development Specialist, ECSH2). Table of Contents Acknowledgements ........................................................................................................................ iii Executive Summary ....................................................................................................................... iv I. Context .................................................................................................................................... 1 II. Data Description ...................................................................................................................... 1 III. Analytical Methods ................................................................................................................. 5 IV. Findings and Results ............................................................................................................... 6 V. Next Steps.............................................................................................................................. 11 References ..................................................................................................................................... 13 Annex 1. “BOOST: Using data analysis to inform better policies in education� Presentation .... 15 Annex 2. Contents of the Serbia BOOST Education Module Dataset ......................................... 28 Annex 3. Cross-tabs of Equity and Efficiency of Resource Utilization by Primary Schools ....... 31 Annex 4. Descriptive Statistics of Regression Samples ............................................................... 34 Annex 5. Regression Results on the Determinants of Student Performance ................................ 37 List of Tables and Figures Table ES1. Estimated Effects of Various Inputs on Student Exam Scores (All Schools) .............. v Figure 1. Serbia BOOST Education Module Screenshot................................................................ 2 Figure 2. Distribution of Average Student Performance by SES of 8th Grade Students ................ 4 Figure 3. Distribution of Per-Student Spending and Exam Results by School Size and Location. 8 Table 1. Estimated Effects of Various Inputs on Student Exam Scores (All Schools) .................. 9 Table A2.1. Complete List of School-Level Variables................................................................. 28 Figure A3.1. Distribution of Educational Inputs by Urban-Rural Location and Quintiles........... 31 Figure A3.2. Distribution of Teacher Characteristics by Urban-Rural Location and Quintiles ... 32 Figure A3.3. Distribution of Student Performance by Urban-Rural Location and Quintiles ....... 33 Table A4.1. Descriptive Statistics (All Schools) .......................................................................... 34 Table A4.2. Descriptive Statistics (Urban Schools) ..................................................................... 35 Table A4.3. Descriptive Statistics (Rural Schools) ...................................................................... 36 Table A5.1. Determinants of Student Performance (All Schools)................................................ 37 Table A5.2. Determinants of Student Performance (Urban Schools) ........................................... 38 Table A5.3. Determinants of Student Performance (Rural Schools) ............................................ 39 ii Acknowledgements This report has been prepared under the Serbia Education TA (P118286) in support of the efforts of the Ministry of Education and Science to work on improving education quality despite the context of a difficult financing reform and the overall fiscal austerity linked to the economic crisis. The report benefited from support of a large group of institutions and individuals who have cooperated on the Serbia Education TA over the past three years, many of whom were critical for both getting access to data and for providing guidance for the analysis. We are grateful to Tinde Kovac Cerovic, State Secretary of the Ministry of Education and Science and her staff for their overall guidance; Marko Paunovic, Executive Director of the Center for Liberal-Democratic Studies for the work on Serbia BOOST databases; Aleksandar Baucal, Professor at Belgrade University and Gordana Capric from Institute of Education Quality and Evaluation for technical advice on student achievement results; Predrag Lazetic from the Center for Education Policy for technical advice on student characteristics; Jelena Radisic from Belgrade University for technical advice on teacher characteristics; Andrijana Zaric for technical advice on school characteristics. We are especially grateful to the Treasury Department of the Ministry of Finance of the Republic of Serbia for providing detailed data on public expenditures for 2005-2010 that made possible the development of the BOOST databases for Serbia. From the World Bank, special thanks go to Igor Kheyfets, who produced this convincing report; Cristian Aedo, for his thoughtful comments, guidance, and advice; Loup Brefort,Country Manager for Serbia; and Alberto Rodriguez, Sector Manager for Education for Europe and Central Asia. Ivana Aleksic Serbia Education TA Task Team Leader iii Executive Summary 1. In November 2011, the World Bank convened a workshop to discuss with Serbia’s policymakers and researchers the country’s analytical agenda in the area of education financing and student performance. The discussion centered on using existing education sector data to provide a deeper insight into guiding education policy. Specifically, the policymakers expressed a desire to be able to study the equity and efficiency of resource allocation in the education sector and to identify schools that achieve results. In partnership with local stakeholders, the World Bank (under its ongoing Serbia Education TA engagement and the World Bank’s BOOST initiative) set out to gather existing data in order to illustrate the potential of new analytical approaches to these policy questions. 2. The data collected for this analysis come from a variety of national sources and are compiled in the Serbia BOOST education module dataset. The Ministry of Education and Science (MOES); Ministry of Finance (MOF); Ministry of Culture, Media and Information Society (MCMIS); and the Republican Statistical Office (RSO) were among the agencies providing the necessary data. Specifically, the World Bank gathered data in the following categories:  School-level expenditures (compiled in a World Bank BOOST database);  School characteristics (school location, enrollment, teacher characteristics, etc.);  Student performance data (8th grade examinations administered in June 2011);  Student family background (several measures of student socioeconomic status);  Municipal welfare index (measures relative welfare of Serbia’s municipalities). 3. Using these data, the analysis employs several methods to illustrate the relationship between educational inputs and student learning outcomes. Various two-way cross- tabulations show the distribution of educational inputs—such as per-student spending, class sizes, and teacher characteristics—across different categories of schools. The allocation of these inputs across urban and rural schools, small and large schools, and schools located in well-off and less well-off municipalities, are informative for gauging the equity of resource distribution. Additionally, multivariate regression analysis models learning outcomes as a production function of school-level inputs and student background characteristics. The results provide a clearer picture of which inputs correlate closely with student performance at school level, holding constant the background factors that are beyond policymakers’ control. 4. The findings show a progressive system of education financing that targets financial resources in a generally pro-poor and pro-rural manner. On average, rural schools are smaller and have lower class sizes and student-teacher ratios than urban schools, leading to high unit costs in small rural primary schools. However, schools in rural and less well-off areas are shown to be staffed with teachers who are less well-educated—only 53 percent of teachers in communities with the lowest municipal welfare have university degrees, compared to 79 percent iv in communities with highest municipal welfare. As a result of flat teacher salary profiles, higher levels of financing for small rural schools generally translate into smaller class sizes rather than more qualified teachers. The resulting learning outcomes show large gaps between the performance of 8th grade students from small rural schools and those from large urban ones. 5. The results of the multivariate regression analysis further demonstrate the importance of teacher and school effects in determining student performance. Holding constant the student background and community characteristics, an increase in the share of teachers with a university degree by one standard deviation (SD) is shown to correspond to a 0.13 to 0.25 SD increase in average 8th grade exam results. Teacher experience levels also correlate positively with student performance, as does school size and the share of the school budget that is spent on non-staff inputs. The overall levels of per-student spending and average class sizes appear to have no statistically significant effect on learning outcomes, once other factors have been taken into account.(See Table ES1below for a complete summary of the determinants of student performance.) Table ES1. Estimated Effects of Various Inputs on Student Exam Scores (All Schools) Unit Unit change Associated change change Associated change in input in exam scores in input in exam scores School-level expenditures Average annual expenditure per student +1 SD Negligible effect +10% Negligible effect Salary as percent of total expenditure +1 SD No effect to –0.07** SD +10% No effect to –0.13** SD Non-teaching staff salary as percent of total staff salary +1 SD –0.07* to –0.09** SD +10% –0.26* to –0.33** SD School and community characteristics Average years of experience of teaching staff +1 SD +0.06* to +0.14*** SD +1 year +0.03* to +0.06*** SD Percent of teaching staff with a university degree +1 SD +0.13*** to +0.25*** SD +10% +0.11*** to +0.20*** SD +1 Average class size in grade 8 +1 SD Negligible effect student Negligible effect Small school (400 students vs. large or fewer) +1 SD –0.12* to –0.24*** SD school –0.24* to –0.48*** SD Municipal welfare index +1 SD No effect to +0.08** SD +10% No effect to +0.05** SD Students’ socioeconomic status (SES) Percent of students whose mothers have secondary ed. +1 SD +0.34*** to +0.35*** SD +10% +0.16*** to +0.16*** SD Percent of students whose fathers are employed +1 SD Negligible effect +10% Negligible effect Notes: Asterisks denote effects that are statistically significant at 10%(*), 5%(**), and 1%(***). Negligible effects are those for which the majority of estimated coefficients is not statistically different from zero. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. v 6. The report recommends several next steps to be taken to follow up this initial analytical work. First, Serbia’s policymakers can build on the findings presented here to further refine the relevant policy questions of interest to the education sector. Second, the Ministry of Education and Science should take necessary steps to improve the education sector data systems in Serbia. A functioning comprehensive Education Management Information System (EMIS) and a well-developed student assessment system would go a long way in reducing the costs of collecting comparable school-level data necessary for future analyses. Third, the Ministry can work with other stakeholders to build demand and technical capacity that will enable future sector analysis. By partnering with other agencies and independent research institutions, the Ministry can foster domestic capacity that will be used to answer additional policy questions with more sophisticated analytical techniques. 7. The report is divided into five main sections. Section I presents the context and motivation for this work. Section II describes the data collected for the analysis and its sources. Section III outlines the analytical methods employed. Section IV discusses the findings and results of the analysis. Section V suggests the appropriate next steps to guide future analytical work. vi I. Context 1. In November 2011, the World Bank convened a workshop in Belgrade with a small group of representatives from the Serbian Ministry of Education and Science, Ministry of Finance, as well as other agencies and independent research institutes.2 The aims of the workshop included agreeing on a research agenda under Serbia’s participation in the World Bank BOOST initiative and increasing the transparency and timely provision of education sector data for the design of the per capita funding formulae. Following the presentation titled “BOOST: Using data analysis to inform better policies in education� (see Annex 1), the discussion centered on the third objective—how existing education sector data can be used to provide deeper insight into educational policymaking in Serbia. Chaired by the State Secretary for Education, the workshop concluded that issues of equity, efficiency, and quality of education were of primary importance to Serbian policymakers. In particular, the critical focus was put on laying the groundwork for creating a system in which policymakers can identify which schools perform better than others in adding value to student learning given their levels of financial resources and other characteristics. The results of such analyses would be used as a basis for qualitative research on school factors that contribute to better student achievement and to support the design of the school improvement policies. It was agreed that the World Bank would compile data from various sources currently available in Serbia and help launch a discussion of the analytical methods that can be used to inform the policy questions of interest. 2. The objective of this work, therefore, is two-fold. The first aim is to collect and use the data currently available in the Republic of Serbia to examine the equity and efficiency of resource allocation in compulsory primary education. This is done by bringing together detailed data on school-level expenditures; school, teacher, student, and community characteristics; and information on student performance for the first time in the Serbian context. The analytical approach follows similar studies conducted in other countries within the framework of the World Bank’s BOOST initiative.3 3. Second, this exercise aims to illustrate the potential of new analytical approaches for informing policymaking that can be used more widely once the data systems that collect information on Serbia’s education sector develop further. Though Serbia’s education sector data systems have room to mature, the demand for analytical work in the sector should encourage the process. This note presents examples of analytical questions that can be answered with presently available data and proposes additional avenues for analysis that can be pursued once further data on student achievement and school characteristics become available. II. Data Description 4. The analysis conducted in this work uses data compiled from a variety of national sources. These include the Ministry of Education and Science (MOES), Ministry of Finance 2 The workshop was attended by the representatives of the Ministry of Education and Science; Ministry of Finance; Institute for Education Quality and Evaluation; Social Inclusion and Poverty Reduction Unit (SIPRU) of the Government of Serbia; Center for Education Policy in Belgrade; and the University of Belgrade. 3 Kheyfets, et al. (2011). 1 (MOF), Ministry of Culture, Media and Information Society (MCMIS), and the Republican Statistical Office (RSO), among others. The school-level indicators are assembled into a dataset known as the “Serbia BOOST education module�, which contains data on several dimensions of the education process in Serbia’s primary schools presented in an easy-to-use Excel PivotTable interface(see screenshot in Figure 1; a full list of indicators contained in this dataset can be found in Annex 2). Figure 1. Serbia BOOST Education Module Screenshot 5. Although a true value-added analysis would require multiple years of data in order to observe changes in student performance over time, the data availability at the time of writing this report limited the scope of the analysis to a cross-section of Serbian primary education. Nearly all of the indicators compiled for this analysis were collected in 2010 or 2011; the only exception was the data on school-level expenditures, which was available on an annual basis from 2005 to 2010. The results that follow, therefore, are meant to be illustrative of the various analytical approaches that shed light on the correlations between resource allocation and student performance in primary education at one point in time but do not attempt to present any causal links for past or future years.4 4 In order to assess the causal impact of specific interventions on learning outcomes, randomized evaluations or quasi-experimental techniques would need to be applied to a sample of students over a period of several years. 2 6. School-level expenditures: Information on school-level spending is derived from the Serbia BOOST government expenditure database of indirect budget beneficiaries. Constructed by the World Bank on the basis of detailed Treasury data, the database contains spending by all publicly financed schools whose expenditures are reported to the MOF.5For the purpose of this analysis, annual expenditure data were extracted for each primary school from 2005 to 2010. This information was then used to compute average annual per-student expenditure amounts over the last six calendar years. The per-student expenditures are reported in aggregate, as well as by financing sources and economic expenditure categories (e.g., salaries, goods and services, etc.). 7. School characteristics: Information on various aspects of the school environment was compiled from several data sources. These include data collected in 2010 by the RSO, MOES, MCMIS, and anonymous payroll data of school staff from the MOF Treasury system. School- level indicators constructed for this analysis include, among others:6  School location  School size (total enrollment)  Student-teacher ratio  Average class size (8th grade only)  Average teacher age, experience, and salary  Percent of teachers with a university degree  Percent of school employees who are non-teaching staff 8. Student performance data: In June 2011, Serbian students completing compulsory education after 8th grade were required for the first time to pass a standardized national examination. The results of these exams, aggregated by school, are used in the analysis as the principal measure of student performance. Specifically, indicators of learning outcomes measured through the average scores on the Serbian language and mathematics exams are linked with financial and non-financial inputs into the education process through a “production function� that illustrates to what extent various input measures correlate with learning outcomes. 9. Student family background: During the course of exam administration in 2011, schools submitted data on the socioeconomic status (SES) of 8th grade students. As a result, indicators of student performance are supplemented with information about the educational attainment and employment status of the students’ parents. Numerous studies have shown that parents’ educational attainment is one of the strongest predictors of student academic performance.7 Therefore, this analysis benefits from the presence of SES information linked to student 5 The Serbia BOOST government expenditure database of indirect budget beneficiaries (IBBs)—and the complementary BOOST government expenditure database of the consolidated national public budget —was constructed under the World Bank’s BOOST initiative in collaboration with the Serbian Ministry of Finance. Version 0.5 of the database, which was used to extract school-level financial indicators for the Serbia BOOST education module, is a large Excel file (80 MB / more than 800,000 rows) with a PivotTable interface that allows the user to quickly and easily prepare reports on the expenditures of publicly financed budget organizations (such as primary schools) broken down along a range of detailed budget classification categories. 6 A full list of school-level indicators, arranged by data source, can be found in Annex 2. 7 See, for example: Lockheed, et al. (1989); Carneiro, et al. (2007); and Engin-Demir (2009); among others. 3 achievement data by facilitating a study of the correlations between school-level funding allocations and student performance, while taking into account the differences in students’ backgrounds. The scatter plots in Figure 1 below show a strong positive correlation between the average exam scores in Serbian language and mathematics and the percent of 8th grade students whose mothers had completed secondary education or higher. The relationship is particularly evident in the upper end of the SES distribution, signifying a stronger link between mother’s educational attainment and student performance for children who come from the most advantaged backgrounds. Figure 2.Distribution of Average Student Performance by SES of 8th Grade Students Source: Author’s calculations based on data from MOES. 10. Municipal welfare index: In addition to SES data, the analysis uses a composite welfare index of municipalities to account for the differences in underlying conditions of the communities in which the students live and study. The welfare index, developed in 2010 to identify underdeveloped municipalities, relies on multiple measures of “material welfare� (consisting of production and consumption) and “non-material welfare� (health and education) to rank Serbia’s municipalities according to their levels of development.8 This index is used in the analysis to supplement students’ SES data as a control for uneven starting conditions with which children from different parts of Serbia enter school. 11. Sample description: The sample of schools used in the analysis is dictated by the availability of data from the sources described above. The main analytical sample (“Sample 1�) is made up of 959primary schools in Serbia—three-quarters of the nation’s 1,313 primary schools—for which data on expenditures, school characteristics, and exam results were available. These exclude all schools located on the territory of Kosovo and Metohija, all primary school students who took the 2011 exams within secondary schools or in minority languages, and a small number of schools that were identified as having doubts about the validity of 8th grade exam results.9 A sub-sample of 675 schools (“Sample 2�) excludes from the main sample the 8 See: Radosavljevic (2010). The welfare index was developed by the Center for Advanced Economic Studies (Centar za visoke ekonomske studije, CEVES) in Belgrade, Serbia; today, CEVES is transformed into the Foundation for the Advancement of Economics (Fond za razvoj ekonomske nauke, FREN). 9 Per advice of Aleksandar Baucal, professor at Belgrade University, excluded are school administrations in which there were credible indications that the administration of the final exam was not performed in line with 4 schools that did not report students’ SES data along with the results of June 2011 8th grade exams. III. Analytical Methods 12. A variety of analytical approaches can be employed to study the distribution of educational resources and their ties to students’ learning outcomes. Two such techniques are described in this section; their results are presented in the Findings and Results section below and in the corresponding annexes. Additional techniques—including some described in the Next Steps section—can also be applied by the Government of Serbia and by independent researchers to study the equity, efficiency, and effectiveness of education sector inputs. Further study can help broaden the knowledge base about the manner in which the country’s education sector utilizes public resources to achieve better learning outcomes. 13. Cross-tabs between measures of resource allocation and school characteristics: The MOES expressed a strong desire to enhance their understanding of the various aspects of educational equity among Serbia’s primary schools. One way to do this is by analyzing two-way relationships (“cross-tabulations� or “cross-tabs�) between measures of educational inputs across different types of schools and communities. Annex 3 presents the detailed results of cross-tabs between various measures of resource utilization—such as per-student expenditures, average class sizes, student-teacher ratios, and so on—across schools divided into quintiles by school size, municipal welfare, and student SES. The implications of these results are discussed in the Findings and Results section below. 14. Multivariate regression analysis of student performance:10 At the core of this study is a set of statistical analyses that illustrate how variations in different inputs into the education process are associated with students’ learning outcomes. The dependent variables of interest— average scores on the 8th grade exam in Serbian language and mathematics administered in June 2011—are regressed on independent variables that measure the school’s level and distribution of financial resources, the qualifications of its teachers, the school’s location and other characteristics, as well as parent and community socioeconomic factors. The ordinary least squares (OLS) regression model is specified as follows: Average exam score = α + β * SCHOOL + γ * SES + δ * MUNI + ε, where:  SCHOOL is a vector of school characteristics, which include: average annual spending per student from 2005 to 2010; salary expenditures as a percent of all school spending in 2010; non-teaching staff salaries as a percent of all salary the prescribed procedures/protocols. For this reason, the results in those school administrations may not adequately reflect the level of student achievement at the exam. The affected school administrations are Leskovac, Vranje and Novi Pazar. 10 This analytical approach has been applied in several other countries under the World Bank’s BOOST initiative to analyze the relationship between the allocations of school-level inputs and student performance. World Bank (2011) provides an example of one such study from Armenia. 5 spending in 2010; teachers’ average years of experience in 2010; percent of teachers with a university degree in 2010; average 8th grade class size in 2011 and average class size squared (to allow for a potential quadratic relationship between class size and learning outcomes);and indicators for rural schools and small schools (i.e., schools with 400 students or fewer), as well as an interaction term between the two indicators (all specifications);  SES is a set of controls for the socioeconomic status of the school’s students in 2011, expressed in percent of 8th grade students whose mothers have completed at least secondary education and percent of 8th grade students whose fathers are employed (specifications 3 and 4 only);  MUNI is a set of controls for community characteristics either through the use of the 2010 Municipal Welfare Index (specifications 1 and 3) or through municipal fixed effects (specifications 2 and 4). 15. Each set of four specifications is modeled separately for two dependent variables (average exam scores in Serbian language and mathematics) and for all schools in the sample, as well as separately for urban and rural schools. The next section describes the results of these regressions in detail. Descriptive statistics for all variables can be found in Annex 4, while full regression results are reported in Annex 5. IV. Findings and Results 16. Cross-tabs between measures of resource allocation and school characteristics: As described in the previous section, a detailed analysis of two-way relationships between measures of educational inputs and school and student characteristics sheds light on the equity of resource allocation across Serbia’s primary schools. The full results of these cross-tabs are presented in a series of bar charts of enrollment-weighted sample means, which can be found in Annex 3. Several key policy questions are answered below. 17. Are important educational inputs—such as per-student financing and numbers of teachers and classes—distributed in an equitable manner across Serbia’s primary schools? How well are these resources targeted toward the neediest students, schools, and communities? (See Figure A3.1inAnnex 3forcomplete results.)  Small schools spend twice as much per student as large schools; the distribution of spending is generally equitable and progressive, targeted toward rural schools and schools that serve students from lower socioeconomic backgrounds.  On average, urban schools are twice as large as rural schools; class sizes and student- teacher ratios are also higher in urban areas by an average of three students per teacher and per class, which indicates a greater targeting of resources toward rural areas. 6  Average class sizes and student-teacher ratios increase in line with school size—the increase is sharpest in urban areas where the student-teacher ratio in the largest schools (13.7) is two and a half times higher than in the smallest schools (5.4).  Among urban schools, a positive correlation is observed between class sizes/student- teacher ratios and the level of municipal welfare/student SES; this again indicates a pro- poor targeting of resources in the form of smaller class sizes in less well-off parts of Serbia. 18. What types of teachers teach in different kinds of primary schools? Are the most qualified and most experienced teachers employed in urban schools and in areas with more well- off residents or are they more likely to be found in Serbia’s more disadvantaged communities? (See Figure A3.2 in Annex 3forcomplete results.)  Urban schools have better educated and more experienced teachers than rural schools; only 61 percent of rural teachers have a university degree, compared to 68 percent of urban teachers. Urban teachers also have an average of one to two years more of teaching experience than their rural counterparts, a statistically significant difference.  Teachers’ educational levels are closely tied to municipal welfare in both urban and rural areas; the differences are most pronounced among rural schools, where only 53 percent of teachers in the least developed municipalities are university educated versus 79 percent in the most developed municipalities. A similar trend is observed with respect to better education teachers working in schools with higher student SES.  Teachers’ salaries are remarkably flat across Serbia’s schools and municipalities; on average, urban teachers are paid only one percent more than rural teachers as a result of higher educational qualifications. 19. How is student performance—as measured by the results of the June 2011 8th grade exam—distributed across Serbia’s primary schools? Do students in large and urban schools perform better than their counterparts in rural areas or small schools? (See Figure A3.3 in Annex 3for complete results.)  Students in urban schools, large schools, and schools with higher average levels of maternal education score consistently higher on the Serbian language and mathematics exams than students in rural schools, small schools, and schools with lower levels of parental education.  The gap in performance of 8th grade students between large urban schools and small urban schools was remarkably large—about 66 points on the standardized exam scale, or approximately 1.5 standard deviations. This difference exceeded even the gap between schools with students in the highest SES quintile and those in the lowest SES quintile. 7  This was true despite the fact that students in smaller schools were slightly more likely to benefit from school-organized exam preparation sessions than those in larger schools; overall, about 85 percent of 8th grade students participated in such sessions. 20. The overall picture that emerges from this analysis is one of pro-poor and pro-rural targeting of financial resources that still produces educational outcomes that benefit urban and more affluent areas. Perhaps this is not surprising given the strong predictive power that students’ background has on their academic performance. However, it shows that the targeting of financial resources toward small schools and schools with more disadvantaged students does little to counteract the advantage enjoyed by large schools, especially those in urban areas (see Figure 3). Figure 3.Distribution of Per-Student Spending and Exam Results by School Size and Location Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 21. One finding of note is the strong correlation between municipal welfare and the level of education of the teachers who work in that municipality. It suggests that although less affluent areas get slightly more funding per student, these financial resources translate into smaller class sizes and not into more qualified teachers—after all, flat teacher salaries dictate that greater levels of financing are more likely to be used to increase the quantity of teachers, rather than attract those of higher quality. But which of these decisions is more likely to contribute to better learning outcomes for primary school students? In order to answer this question, we need to isolate the effects of individual policy decisions on the learning process by using multivariate regression analysis. 22. Multivariate regression analysis of student performance: This section describes the findings of a statistical analysis that disaggregates the variation in average exam scores by school into their respective components that correlate with student performance. The variables used in this analysis were chosen from the data compiled in the Serbia BOOST education module (discussed in the Data Description section above). Complete results of all regressions can be found in Annex 5, while Annex 4reports the descriptive statistics for the corresponding regression samples. 8 23. As discussed above, average exam scores are modeled as a production function of school characteristics, municipal characteristics, and students’ socioeconomic status in order to identify the independent effects of each input on students’ academic performance.Table1 presents a summary of each independent variable’s estimated effects, which were calculated on the basis of the regression results for the full sample of schools presented in Table A5.1. Separate regression results for samples of urban and rural schools can be found in Tables A5.2. and A5.3., respectively. Table 1. Estimated Effects of Various Inputs on Student Exam Scores (All Schools) Unit Unit change Associated change change Associated change in input in exam scores in input in exam scores School-level expenditures Average annual expenditure per student +1 SD Negligible effect +10% Negligible effect Salary as percent of total expenditure +1 SD No effect to –0.07** SD +10% No effect to –0.13** SD Non-teaching staff salary as percent of total staff salary +1 SD –0.07* to –0.09** SD +10% –0.26* to –0.33** SD School and community characteristics Average years of experience of teaching staff +1 SD +0.06* to +0.14*** SD +1 year +0.03* to +0.06*** SD Percent of teaching staff with a university degree +1 SD +0.13*** to +0.25*** SD +10% +0.11*** to +0.20*** SD +1 Average class size in grade 8 +1 SD Negligible effect student Negligible effect Small school (400 students vs. large or fewer) +1 SD –0.12* to –0.24*** SD school –0.24* to –0.48*** SD Municipal welfare index +1 SD No effect to +0.08** SD +10% No effect to +0.05** SD Students’ socioeconomic status (SES) Percent of students whose mothers have secondary ed. +1 SD +0.34*** to +0.35*** SD +10% +0.16*** to +0.16*** SD Percent of students whose fathers are employed +1 SD Negligible effect +10% Negligible effect Notes: Asterisks denote effects that are statistically significant at 10%(*), 5%(**), and 1%(***). Negligible effects are those for which the majority of estimated coefficients is not statistically different from zero. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 24. In light of the Ministry of Education and Science’s interest in studying issues that relate to the equitable provision of education in Serbia’s primary schools, several findings are noteworthy. The single strongest driver of variation in students’ exam results is their socioeconomic status—an increase in one standard deviation (SD) in the average level of students’ maternal education is associated with an increase in exam results of up to 0.35 SD (about 15 point on the standardized exam scale).While this result is not surprising, policymakers can typically do little to directly affect students’ SES. Instead, they can focus on the factors within the Ministry’s control that are associated with better student performance, as highlighted below: 9  Teacher qualifications and experience play a strong role in determining students’ learning outcomes. A one-year increase in the average teacher experience level corresponds to a statistically significant increase in exam results of up to 0.06 SD. Meanwhile, a 10 percent increase in the share of teachers who have a university degree correlates with a substantial increase in exam scores of up to 0.20 SD. In practice, this implies that bringing average teacher qualifications in rural schools to the urban school average may be associated with a small but significant narrowing of the performance gap between urban and rural students (closing about one-sixth of the current performance gap).  Small schools are found to underperform larger ones. On average, a school with 400 or fewer students11 is predicted to perform up to 0.48 SD lower on the 8th grade exam than a large school with more than 400 students, even when controlling for a variety of other relevant factors. This effect is evident for both urban and rural schools, though the small school “penalty� is larger for urban schools. This relationship is independent of the effects of class sizes, teacher and student characteristics, etc. Although this relationship requires further study, the substantial magnitude of the performance gap between small and large schools found here should urge policymakers to give immediate attention to examining the quality of education provided in Serbia’s smaller primary schools.  The allocation of school-level expenditures matters in predicting student performance… Schools that spent a larger share of their budgets on teaching and learning inputs, rather than on staff salaries—and, in particular, on salaries of non-teaching staff—are seen to have better student performance on the 8th grade exam. An average primary school in the regression sample spent 62 percent of its budget on staff salaries, 13 percent of which went to non-teaching staff. Decreasing either of those proportions by 1 SD (5.3 and 2.7 percentage points, respectively) is seen to correlate with exam scores that are 0.07 to 0.09 SD higher, on average. This finding may suggest a need for refocusing the discussion on greater school-level financial autonomy in Serbia, giving school directors the flexibility to allocate funds toward inputs they view as most needed in their particular schools (e.g., away from staff salaries and toward other teaching and learning inputs).  …but the total level of per-student spending does not. After the characteristics of students, teachers, and schools are taken into account, the absolute level of per-student financing does not appear to play a significant role in predicting academic performance. The same is true for average class size—a key driver of per-student costs—which has a negligible effect on exam scores. As has been suggested by numerous studies on the subject: “What matters in education finance is how money is spent, not simply how much.�12 Therefore, policymakers should be careful in avoiding the often appealing argument that higher levels of overall financing and smaller classes are a panacea to improving educational quality. A comprehensive discussion of how best to achieve the education sector goals with the resources currently available ought to focus on 11 According to the Law on School Rationalization, schools with 400 students or fewer are subject to merger or rationalization if there is another school within 2 km in the same municipality. 12 World Bank (2012a). 10 considerations of the allocation of funds among different educational inputs (teachers, textbooks, technology, etc.) and across different parts of the country (targeting disadvantaged students and communities). V. Next Steps 25. The analytical findings presented above should not be treated as a conclusive set of prescriptions for policy reform. Rather they should be viewed as a starting point for further analytical work aimed at providing recommendations to the Government of Serbia to continue improving equity and efficiency of education spending for the increased quality and relevance of the country’s education system. The possible next steps can be grouped into three main areas: (i) refining the relevant policy questions to be answered through future analytical work; (ii) improving the data systems that collect timely and reliable information on educational resource allocation and learning outcomes at school level; and (iii) building technical capacity and demand for future analysis using new techniques. The World Bank is ready to partner with the Government of Serbia, upon request, to work together toward progress in any or all of these areas. 26. Refining the relevant policy questions: The policymakers in the Ministry of Education and Science, as well as other agencies in the Government of Serbia, can build on the findings presented above by setting out a broader research agenda aimed at enhancing the knowledge base about the sector. The analysis presented here focuses on the equity of resource allocation and the quality of education provision in Serbia’s primary schools. Future analytical work can broaden the relevant policy questions to a number of areas, including: (i) measuring the relative efficiency of education service delivery by different schools and subnational units; (ii) examining the quality of education provision to particular demographic groups—such as students who come from disadvantaged backgrounds, linguistic or ethnic minorities, or boys versus girls; (iii) investigating how teachers with particular types of qualifications add value to student learning; and so forth. However, to conduct any of these analysis, better data will be required than what is currently available in Serbia’s education sector. 27. Improving education sector data systems: To enable researchers to conduct more thorough analyses of education sector performance, the data collection practices in Serbia will need to mature. At present, the Ministry of Education and Science does not have a functioning Education Management Information System (EMIS). The analysis in this note, therefore, was complicated by the need to collect and combine detailed data relating to the education sector and related financial data from at least five different sources. Unfortunately, school-level databases compiled by different ministries (e.g., MOES, MOF, MCMIS) do not use the same unique codes to identify schools, thus further complicating the task of linking enrollment numbers with test results, teacher characteristics, and financial data. A comprehensive and flexible EMIS that collects and reports data on wide range of school-level indicators in a manner consistent with international best practices would go a long way to facilitating the kinds of analyses described in this note. 28. Moreover, Serbia’s student assessment system will need to develop further in order to supply policymakers and researchers with information on student learning that is comparable 11 across schools and over time. Because the nationwide examination of 8th graders was implemented for the first time in a standardized fashion in June 2011, it suffered from some early setbacks. In its first iteration, the testing exercise produced a large amount of useful information—particularly the detailed data on student socioeconomic status—but saw issues of data validity that arose from non-standard exam implementation or errors in data recording, as well as incomplete information on socioeconomic indicators for a large number of schools. As the process moves forward, the quality of data is likely going to improve to allow for more reliable measures of student performance.13 29. Building demand and technical capacity for future analysis: The analysis presented in this note can serve as a stepping stone to a more rigorous study of resource allocation and student performance. Value-added (VA) analysis, in particular, can be used to measure the contributions of schools—and even individual teachers—to changes in students’ learning outcomes. However, multiple years of performance data generated through regular student assessments—as well as more detailed data on school, teacher, and student characteristics—are needed in order to implement this analysis.14Additionally, relative efficiency analysis can be used to measure the degree to which schools convert educational inputs into the desired outputs and outcomes. Relative efficiency scores for school districts or individual schools can be computed using a variety of techniques, including data envelopment analysis (DEA), stochastic frontier analysis (SFA), or regression residual analysis.15Further study can focus on identifying individual schools that do better (or worse) than expected at improving student performance given their available resources and student characteristics. 30. In order to make these and other analyses possible—and, in fact, routine—the improvement of data systems must be accompanied by enhanced capacity among education sector analysts, both within the sector Ministry, its two technical institutes, and elsewhere. The Government of Serbia can partner with universities and independent research organizations that already possess this type of capacity to build the skills of civil servants in response to anticipated analytical needs. For their part, policymakers should demand rigorous analytical evidence on which to base their policy decisions. Meanwhile, individual citizens and civil society organizations should demand a public sector that is efficient and accountable in delivering educational services. All of this will require the new techniques to be applied to analyzing the data that will soon become available and a clear strategic direction for which policy questions should guide future analysis of Serbia’s education sector. 13 See World Bank (2012c) for a summary of what makes for an effective student assessment system. 14 See Hanushek and Rivkin (2010) for a brief discussion of value-added models of teacher effectiveness. See World Bank (2012b) for a summary of what makes for effective teacher policies. 15 See Cordero Ferrera, et al. (2005) for an example of a school-level application of data envelopment analysis in a European country (Spain). See World Bank (2009) for an example of stochastic frontier analysis applied in the Serbian context to measure the efficiency of primary health care facilities. 12 References Carneiro, Pedro, Costas Meghir, and Matthias Parey. 2007. “Maternal Education, Home Environments and the Development of Children and Adolescents.� IZA Discussion Paper No. 3072 (September 2007). IZA: Bonn, Germany. Available online at: http://ftp.iza.org/dp3072.pdf. Cordero Ferrera, José Manuel, Francisco Pedraja Chaparro, and Javier Salinas Jiménez. 2005. “Eficiencia en educación secundaria e inputs no controlables: sensibilidad de los resultados ante modelos alternativos.� Hacienda Pública Española / Revista de Economía Pública, Vol. 173 (February 2005), pp. 61-83. Engin-Demir, Cennet. 2009. “Factors influencing the academic achievement of the Turkish urban poor.� International Journal of Educational Development, Vol. 29, No. 1 (2009), pp. 17-29. Available online at: http://www.sciencedirect.com/science/article/pii/S0738059308000278. Hanushek, Eric A. and Steven G. Rivkin. 2010. “Generalizations about Using Value-Added Measuresof Teacher Quality.� American Economic Review: Papers & Proceedings, Vol. 100 (May 2010), pp. 267–271. Available online at: http://www.aeaweb.org/articles.php ?doi=10.1257/aer.100.2.267. Kheyfets, Igor, Massimo Mastruzzi, Dino Merotto, and Lars Sondergaard. 2011. “A New Data Tool to BOOST Public Spending Efficiency.� Europe and Central Asia Knowledge Brief Vol. 43 (September 2011). World Bank: Washington DC. Available online at: http://go.worldbank.org/S14MC5Z9L0. Lockheed, Marlaine E., Bruce Fuller, and Ronald Nyirongo. 1989. “Family Effects on Students’ Achievement in Thailand and Malawi.�Sociology of Education, Vol. 62, No. 4 (October 1989), pp. 239-256. Available online at: http://www.jstor.org/stable/10.2307/2112829. Radosavljevic, Goran. 2010. “Local Development and Roma Policies in Serbia.�Mimeo.(April 2010). World Bank. 2009. “Serbia: Baseline Survey on Cost and Efficiency in Primary Health Care Centers before Provider Payment Reforms.� Report No. 45111-YF (January 2009). World Bank: Washington DC. Available online at: http://documents.worldbank.org/curated/en/2009/01/10468570/serbia-baseline-survey- cost-efficiency-primary-health-care-centers-before-provider-payment-reforms. World Bank. 2011. “Republic of Armenia: Fiscal Consolidation and Recovery.� Vols. 1 and 2. Report No. 62587-AM (November 2011). World Bank: Washington DC. Available online at:http://documents.worldbank.org/curated/en/docsearch/report/62587. 13 World Bank. 2012a. “Education Finance: It’s How, Not Simply How Much, that Counts.� World Bank Education Notes (February 2012), by Emiliana Vegas and Chelsea Coffin. Available online at: http://documents.worldbank.org/curated/en/2012/02/15940645/education-finance-not- simply-much-counts. World Bank. 2012b. “Learning from the Best: Improving Learning Through Effective Teacher Policies.� World Bank Education Notes (February 2012), by Emiliana Vegas, Alejandro Ganimian, and Analia Jaimovich. Available online at: http://documents.worldbank.org/curated/en/2012/02/15940672/learning-best-improving- learning-through-effective-teacher-policies. World Bank. 2012c. “Measuring Learning: How Effective Student Assessment Systems Can Help Achieve Learning for All.� World Bank Education Notes (February 2012), by Marguerite Clarke. Available online at: http://documents.worldbank.org/curated/en/2012/02/15940684/measuring-learning- effective-student-assessment-systems-can-help-achieve-learning-all. 14 Annex 1. “BOOST: Using data analysis to inform better policies in education� Presentation 15 16 17 18 19 20 21 22 23 24 25 26 27 Annex 2. Contents of the Serbia BOOST Education Module Dataset Table A2.1. Complete List of School-Level Variables Data Variable Name Variable Description Source(s)1 School Identification Variables school_id1 School ID number from BOOST data [1] school_id2 School ID number from Treasury payroll data [2] school_id3 School ID number from MOES schools catalog [3] school_id4 School ID number from student achievement data [4] region Statistical region [6] district Administrative district [6], [3] municipality Municipality [2], [1], [6] location Location of school headquarters [3], [4] school_admin School administration authority [4], [3], [2] school_name School name [1], [3], [4] school_type School type: 13701 = primary; 13702 = secondary [1], [2] school_func School functional classification [2] school_urban_rural School location: 1 = urban; 2 = rural [2] Municipal Welfare Index (2010)2 municipal_welfare_index Municipal welfare index [5] municipal_welfare_index_rank Municipal welfare index rank [5] School Variables (2010) students_total Total number of students in school [3] school_satellite_objects Number of school satellite objects reporting to school [3] students_satellite_pct Percent of school's students enrolled in satellite school objects [3] Grade 8 Learning Outcome Variables (2011) grade8_classes Number of grade 8 classes [4] grade8_students Number of grade 8 students [4] grade8_avg_class_size Average class size in grade 8 [4] grade8_students_taking_exam_lang Number of grade 8 students who took the exam in June 2011: Serbian language [4] grade8_students_taking_exam_math Number of grade 8 students who took the exam in June 2011: Mathematics (in Serbian language) [4] grade8_exam_avg_score_lang Average score on grade 8 exam -- Serbian language [4] grade8_exam_avg_score_math Average score on grade 8 exam -- Mathematics (in Serbian language) [4] grade8_exam_stdev_lang Standard deviation of scores on grade 8 exam -- Serbian language [4] grade8_exam_stdev_math Standard deviation of scores on grade 8 exam -- Mathematics (in Serbian language) [4] grade8_students_exam_all_lang Number of grade 8 students who took the exam in June 2011 in all languages [4] grade8_students_exam_other_lang Number of grade 8 students who took the exam in June 2011 in languages other than Serbian [4] grade8_students_taking_exam_iep Number of grade 8 students who took the exam in June 2011 according to IEP [4] grade8_students_in_exam_prep Number of grade 8 students who participated in exam preparation [4] grade8_exam_prep_org_by_school Did the school organize exam preparation for students? 0 = no; 1 = yes [4] grade8_students_exam_alllang_pct Percent of grade 8 students who took the exam in June 2011 in all languages [4] grade8_students_exam_othlang_pct Percent of grade 8 students who took the exam in June 2011 in languages other than Serbian [4] grade8_students_taking_exam_pct Percent of grade 8 students who took the exam in June 2011 in Serbian language [4] grade8_students_exam_iep_pct Percent of grade 8 students who took the exam in June 2011 according to IEP [4] grade8_students_in_exam_prep_pct Percent of grade 8 students who participated in exam preparation [4] Grade 8 Student SES Variables (2011) grade8_students_mother_ed_1 Number of students whose mothers have incomplete primary education [4] grade8_students_mother_ed_2 Number of students whose mothers have primary education [4] grade8_students_mother_ed_3 Number of students whose mothers have secondary education [4] grade8_students_mother_ed_4 Number of students whose mothers have higher or university education [4] grade8_students_mother_ed_noinfo Number of students with no data on maternal education [4] grade8_students_mother_employed Number of students whose mothers are employed [4] grade8_students_mother_not_emp Number of students whose mothers are not employed [4] grade8_students_mother_noempinfo Number of students with no data on maternal employment [4] grade8_students_father_ed_1 Number of students whose fathers have incomplete primary education [4] grade8_students_father_ed_2 Number of students whose fathers have primary education [4] grade8_students_father_ed_3 Number of students whose fathers have secondary education [4] grade8_students_father_ed_4 Number of students whose fathers have higher or university education [4] grade8_students_father_ed_noinfo Number of students with no data on paternal education [4] grade8_students_father_employed Number of students whose fathers are employed [4] grade8_students_father_not_emp Number of students whose fathers are not employed [4] grade8_students_father_noempinfo Number of students with no data on paternal employment [4] grade8_students_mother_ed_se_pct Percent of students whose mothers have at least a secondary education [4] grade8_students_mother_ed_hi_pct Percent of students whose mothers have higher or university education [4] grade8_students_mother_emp_pct Percent of students whose mothers are employed [4] 28 grade8_students_father_ed_se_pct Percent of students whose fathers have at least a secondary education [4] grade8_students_father_ed_hi_pct Percent of students whose fathers have higher or university education [4] grade8_students_father_emp_pct Percent of students whose fathers are employed [4] Staff Payroll Variables (2010) staff_total Total number of staff in school [2] staff_teaching Total number of teaching staff in school [2] staff_nonteaching Total number of non-teaching staff in school [2] staff_total_student_ratio Students to total staff ratio [2], [3] staff_teaching_student_ratio Students to teaching staff ratio [2], [3] staff_nonteaching_student_ratio Students to non-teaching staff ratio [2], [3] staff_nonteaching_pct Percent of total staff who are non-teaching staff [2] staff_teaching_uni_degree_pct Percent of teaching staff with a university degree [2] staff_teaching_avg_age Average age of teaching staff [2] staff_teaching_avg_experience Average years of experience of teaching staff [2] staff_teaching_avg_education Average level of education (1-10) of teaching staff [2] staff_teaching_avg_paygrade Average salary grade (I-VII) of teaching staff [2] staff_teaching_avg_basepay Average base salary of teaching staff [2] staff_teaching_avg_totalpay Average total salary of teaching staff [2] staff_total_workload Total staff workload [2] staff_teaching_workload Teaching staff workload [2] staff_nonteaching_workload Non-teaching staff workload [2] staff_total_workload_per_student Total staff workload per student [2], [3] staff_teach_workload_per_student Teaching staff workload per student [2], [3] staff_nonteach_wrkld_per_student Non-teaching staff workload per student [2], [3] staff_total_pay Total staff salary [2] staff_teaching_pay Teaching staff salary [2] staff_nonteaching_pay Non-teaching staff salary [2] staff_nonteaching_pay_pct Percent of total staff salary which is non-teaching staff salary [2] staff_total_pay_per_student Total staff salary per student [2], [3] staff_teaching_pay_per_student Teaching staff salary per student [2], [3] staff_nonteach_pay_per_student Non-teaching staff salary per student [2], [3] staff_total_pay_budget_pct Percent of total staff salary financed from budget [2] Primary Education Expenditure Variables (2005-2010)3 exp_total Average annual expenditure 2005-2010 (thd 2010 RSD) -- Total [1] exp_type1_salary Average annual expenditure 2005-2010 (thd 2010 RSD) -- Salary [1] exp_type2_other_personnel Average annual expenditure 2005-2010 (thd 2010 RSD) -- Non-salary personnel costs [1] exp_type3_goodsandservices Average annual expenditure 2005-2010 (thd 2010 RSD) -- Goods and services [1] exp_type4_other_recurrent Average annual expenditure 2005-2010 (thd 2010 RSD) -- Other recurrent costs [1] exp_type5_capital Average annual expenditure 2005-2010 (thd 2010 RSD) -- Capital costs [1] exp_fin1_republic_budget Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from Republican budget [1] exp_fin2_province_budget Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from AP Vojvodina budget [1] exp_fin3_municipal_budget Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from municipal budgets [1] exp_fin4_social_ins_funds Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from social insurance funds [1] exp_fin5_donations Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from donations [1] exp_fin6_other_sources Average annual expenditure 2005-2010 (thd 2010 RSD) -- Financed from other sources [1] exp_type1_salary_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Salary [1] exp_type2_other_personnel_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Non-salary personnel costs [1] exp_type3_goodsandservices_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Goods and services [1] exp_type4_other_recurrent_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Other recurrent costs [1] exp_type5_capital_pct Average annual expenditure 2005-2010 (thd 2010 RSD) –% Capital costs [1] exp_fin1_republic_budget_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from Republican budget [1] exp_fin2_province_budget_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from AP Vojvodina budget [1] exp_fin3_municipal_budget_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from municipal budgets [1] exp_fin4_social_ins_funds_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from social insurance funds [1] exp_fin5_donations_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from donations [1] exp_fin6_other_sources_pct Average annual expenditure 2005-2010 (thd 2010 RSD) -- % Financed from other sources [1] exp_total_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Total [1], [3] exp_type1_salary_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Salary [1], [3] exp_type2_oth_person_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Non-salary personnel costs [1], [3] exp_type3_goods_svcs_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Goods and services [1], [3] exp_type4_oth_recurr_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Other recurrent costs [1], [3] exp_type5_capital_per_student Average annual expenditure per student 2005-2010 (thd 2010 RSD) -- Capital costs [1], [3] exp_salary_per_staff_2010 Average salary expenditure per staff member in 2010 (thd 2010 RSD) [1], [3] 29 Notes: 1 Data were compiled from the following sources: Data Source Description Unit / Date(s) of Reporting [1] Expenditure data (BOOST) BOOST government expenditure database of indirect budget School / 2005-2010 (annually) beneficiaries constructed by the World Bank on the basis of Treasury data from the Ministry of Finance [2] Payrolldata (MOF) Payroll data for staff of primary and secondary schools from the Employee engagement / March 2010 Treasury database of the Ministry of Finance [3] Schoolsdata (MCMIS) School object data from the Digital Agenda database of the School object / July-September 2010 Ministry of Culture, Media and Information Society [4] Examdata (MOES) Student achievement data and information on students' Class / June 2011 socioeconomic status from the results of the 8th grade exam administered by the Ministry of Education and Science [5] Welfareindex (MOF) Municipal welfare index calculated by the Ministry of Finance for Municipality / April 2010 the Ministry of Education and Science based on data from 2001 to 2008 [6] Statistical data (RSO) Statistical data and information on administrative-territorial n/a divisions from the website of the Republican Statistical Office 2 Municipal welfare index was calculated by the Ministry of Finance in April 2010 from statistical sources covering 2001-2008 years according to the methodology outlined in Radosavljevic (2010). 3 Primary education expenditure variables were constructed on the basis of data, which were extracted from the Serbia BOOST v0.5 database of indirect budget beneficiary expenditures according to the budget classification rules listed below. Average annual expenditure amounts were then computed by deflating past years’ expenditures into 2010 RSD terms using CPI data from the IMF's World Economic Outlook database. Administrative classification: Primary and secondary education {admin3 = 13701 OR 13702}. Economic classification: Salary {econ3 = 411000}; Other personnel {econ2 = 410000 AND NOT econ3 = 411000}; Goods and services {econ2 = 420000}; Other recurrent {econ1 = 400000 AND NOT econ2 = 410000 AND NOT econ2 = 420000}; Capital {econ1 = 500000}. Source of financing: Republican budget {fin_source = 1}; AP Vojvodina budget {fin_source = 2}; Municipal budgets {fin_source = 3}; Social insurance funds {fin_source = 4}; Donations {fin_source = 5}; Other sources {fin_source = 6}. Years: {year = 2005 through 2010}. 30 Annex 3. Cross-tabs of Equity and Efficiency of Resource Utilization by Primary Schools Figure A3.1. Distribution of Educational Inputs by Urban-Rural Location and Quintiles (i) School Size Quintiles (ii) Municipal Welfare Quintiles (iii) Student SES Quintiles (a) Average annual expenditure per student, 2005-2010 (b) Average school size, 2010 (c) Average 8th grade class size, 2011 (d) Student-teacher ratio, 2010 Note: Reported sample means are weighted by school size. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 31 Figure A3.2. Distribution of Teacher Characteristics by Urban-Rural Location and Quintiles (i) School Size Quintiles (ii) Municipal Welfare Quintiles (iii) Student SES Quintiles (a) Percent of teachers with a university degree, 2010 (b) Average teacher age, 2010 (c) Average teacher years of experience, 2010 (d) Average teacher pay, 2010 Note: Reported sample means are weighted by school size. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 32 Figure A3.3. Distribution of Student Performance by Urban-Rural Location and Quintiles (i) School Size Quintiles (ii) Municipal Welfare Quintiles (iii) Student SES Quintiles (a) Percent of 8th grade students taking the compulsory education completion exam, 2011 (b) Percent of 8th grade students participating in school-organized exam preparation, 2011 (c) Average score on the 8th grade Serbian language exam, 2011 (d) Average score on the 8th grade Mathematics exam, 2011 Note: Reported sample means are weighted by school size. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 33 Annex 4. Descriptive Statistics of Regression Samples Table A4.1. Descriptive Statistics (All Schools) Sample 1 Sample 2 (n = 959) (n = 675) Mean SD Mean SD Student performance variables Average score on the 8th grade Serbian language exam (2011) 483.242 45.800 483.034 42.502 Average score on the 8th grade Mathematics exam (2011) 481.757 46.190 480.720 44.735 School expenditure variables Average annual expenditure per student (2005-2010 in thd 2010 RSD) 128.276 100.321 125.045 66.038 Staff salary as percent of total expenditure (2010) 61.946 5.342 62.328 5.178 Non-teaching staff salary as percent of total staff salary (2010) 13.286 2.729 13.389 2.73 School and community characteristics Average years of experience of teaching staff (2010) 15.954 2.217 15.902 2.278 Percent of teaching staff with a university degree (2010) 66.011 12.027 64.749 12.036 Average class size in 8th grade (2011) 18.738 5.361 18.926 5.182 Small school indicator (0 if 400 students or fewer, 1 if more than 400) 0.455 0.498 0.467 0.499 Rural school indicator (0 if urban, 1 if rural) 0.423 0.494 0.425 0.495 Municipal welfare index (2010) 46.261 14.347 45.472 14.415 Students’ socioeconomic status (SES) Percent of students whose mothers have at least a secondary education (2011) -- -- 72.218 21.739 Percent of students whose fathers are employed (2011) -- -- 71.251 24.568 Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 34 Table A4.2. Descriptive Statistics (Urban Schools) Sample 1 Sample 2 (n = 553) (n = 388) Mean SD Mean SD Student performance variables Average score on the 8th grade Serbian language exam (2011) 498.881 43.965 498.601 39.990 Average score on the 8th grade Mathematics exam (2011) 497.000 44.167 496.510 41.360 School expenditure variables Average annual expenditure per student (2005-2010 in thd 2010 RSD) 111.906 111.279 106.334 52.202 Staff salary as percent of total expenditure (2010) 61.409 5.468 61.648 5.455 Non-teaching staff salary as percent of total staff salary (2010) 12.523 2.567 12.709 2.627 School and community characteristics Average years of experience of teaching staff (2010) 16.441 2.062 16.471 2.092 Percent of teaching staff with a university degree (2010) 68.793 11.058 67.358 10.909 Average class size in 8th grade (2011) 20.451 4.717 20.667 4.400 Small school indicator (0 if 400 students or fewer, 1 if more than 400) 0.184 0.388 0.193 0.395 Rural school indicator (0 if urban, 1 if rural) 0.000 0.000 0.000 0.000 Municipal welfare index (2010) 51.167 16.361 50.443 16.712 Students’ socioeconomic status (SES) Percent of students whose mothers have at least a secondary education (2011) -- -- 81.848 16.075 Percent of students whose fathers are employed (2011) -- -- 79.612 21.469 Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 35 Table A4.3. Descriptive Statistics (Rural Schools) Sample 1 Sample 2 (n = 406) (n = 287) Mean SD Mean SD Student performance variables Average score on the 8th grade Serbian language exam (2011) 461.941 39.224 461.990 36.333 Average score on the 8th grade Mathematics exam (2011) 460.995 40.479 459.373 40.073 School expenditure variables Average annual expenditure per student (2005-2010 in thd 2010 RSD) 150.572 77.877 150.342 73.979 Staff salary as percent of total expenditure (2010) 62.678 5.082 63.247 4.631 Non-teaching staff salary as percent of total staff salary (2010) 14.325 2.599 14.309 2.597 School and community characteristics Average years of experience of teaching staff (2010) 15.292 2.250 15.133 2.297 Percent of teaching staff with a university degree (2010) 62.223 12.271 61.223 12.595 Average class size in 8th grade (2011) 16.404 5.310 16.573 5.238 Small school indicator (0 if 400 students or fewer, 1 if more than 400) 0.823 0.382 0.836 0.371 Rural school indicator (0 if urban, 1 if rural) 1.000 0.000 1.000 0.000 Municipal welfare index (2010) 39.577 6.665 38.753 5.739 Students’ socioeconomic status (SES) Percent of students whose mothers have at least a secondary education (2011) -- -- 59.198 21.638 Percent of students whose fathers are employed (2011) -- -- 59.948 23.995 Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 36 Annex 5. Regression Results on the Determinants of Student Performance Table A5.1. Determinants of Student Performance (All Schools) Dependent Average score on the Average score on the variable1: 8th grade Serbian language exam, 2011 8th grade Mathematics exam, 2011 Specification2: (1) (2) (3) (4) (1) (2) (3) (4) Average annual expenditure per student (2005-2010 in thd 2010 RSD) 0.012 -0.004 0.047* 0.022 0.009 -0.012 0.070** 0.027 [0.019] [0.022] [0.028] [0.027] [0.017] [0.018] [0.032] [0.032] Staff salary as percent of total expenditure (2010) -0.585** -0.598** -0.508* -0.243 -0.467* -0.232 -0.464 0.053 [0.252] [0.291] [0.288] [0.314] [0.265] [0.295] [0.312] [0.339] Non-teaching staff salary as percent of total staff salary (2010) -1.412** -1.435** -1.191** -1.394** -1.216** -1.308** -1.181* -1.477** [0.560] [0.658] [0.551] [0.656] [0.550] [0.623] [0.632] [0.734] Average years of experience of teaching staff (2010) 2.496*** 2.344*** 1.260* 1.378* 2.868*** 2.587*** 1.991*** 2.518*** [0.713] [0.847] [0.743] [0.822] [0.716] [0.789] [0.761] [0.815] Percent of teaching staff with a university degree (2010) 0.902*** 0.714*** 0.546*** 0.517*** 0.945*** 0.756*** 0.579*** 0.662*** [0.124] [0.180] [0.143] [0.184] [0.126] [0.180] [0.143] [0.192] Average class size in 8th grade (2011) 2.343 4.079* -1.348 0.562 0.007 1.922 -2.934 -1.332 [2.327] [2.335] [1.805] [1.697] [2.041] [2.196] [1.862] [1.723] Average class size squared (2011) -0.016 -0.062 0.065 0.007 0.042 -0.014 0.107** 0.048 [0.057] [0.056] [0.045] [0.042] [0.050] [0.053] [0.046] [0.043] Small school (400 students or fewer in 2010) -21.966*** -20.380*** -19.273*** -17.108*** -17.345*** -16.378*** -12.109** -10.989* [5.772] [5.916] [5.260] [5.184] [5.761] [5.801] [5.856] [5.755] Rural school -19.074*** -15.510*** -7.923* -4.742 -21.658*** -17.769*** -13.443** -11.006* [3.946] [4.229] [4.726] [5.250] [4.507] [4.418] [5.741] [6.076] Small school * Rural school 20.714*** 17.073** 12.908* 10.329 21.395*** 18.182** 13.031* 12.874* [7.171] [7.233] [6.727] [6.572] [7.423] [7.364] [7.897] [7.726] Municipal welfare index (2010) 0.175 -- -0.006 -- 0.253** -- 0.097 -- [0.116] -- [0.116] -- [0.121] -- [0.127] -- Municipal fixed effects -- YES*** -- YES*** -- YES*** -- YES*** Percent of students whose mothers have at least a secondary education (2011) -- -- 0.715*** 0.716*** -- -- 0.706*** 0.704*** -- -- [0.108] [0.118] -- -- [0.106] [0.115] Percent of students whose fathers are employed (2011) -- -- 0.071 0.067 -- -- 0.021 -0.002 -- -- [0.065] [0.064] -- -- [0.069] [0.068] Constant 401.974*** 392.255*** 421.151*** 384.809*** 399.471*** 376.519*** 411.507*** 356.191*** [34.467] [37.919] [33.002] [36.943] [33.804] [38.768] [36.903] [41.061] Observations 959 959 675 675 959 959 675 675 R-squared 0.350 0.531 0.456 0.644 0.313 0.540 0.394 0.620 Notes: 1 Robust standard errors in brackets; asterisks denote statistical significance at 10% (*), 5% (**), and 1% (***). 2 Regression specifications 1 and 2 are applied to Sample 1, specifications 3 and 4 are applied to Sample 2. 3 Estimated coefficients for municipal fixed effects are not shown, though asterisks denote their joint significance. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 37 Table A5.2. Determinants of Student Performance (Urban Schools) Dependent Average score on the Average score on the variable1: 8th grade Serbian language exam, 2011 8th grade Mathematics exam, 2011 Specification2: (1) (2) (3) (4) (1) (2) (3) (4) Average annual expenditure per student (2005-2010 in thd 2010 RSD) 0.065*** 0.058** 0.077* 0.042 0.053* 0.039 0.158*** 0.154*** [0.025] [0.028] [0.043] [0.045] [0.029] [0.028] [0.044] [0.055] Staff salary as percent of total expenditure (2010) -0.470 -0.367 -0.297 0.040 -0.390 -0.164 -0.409 0.227 [0.319] [0.390] [0.344] [0.390] [0.339] [0.403] [0.362] [0.394] Non-teaching staff salary as percent of total staff salary (2010) -2.484*** -1.750** -2.572*** -2.888*** -2.435*** -1.506* -3.091*** -2.882*** [0.687] [0.842] [0.669] [0.836] [0.721] [0.862] [0.733] [0.904] Average years of experience of teaching staff (2010) 3.768*** 2.761** 1.633* 1.939* 4.536*** 2.814*** 3.171*** 2.972*** [0.949] [1.097] [0.913] [0.997] [0.960] [1.072] [0.996] [1.101] Percent of teaching staff with a university degree (2010) 0.941*** 0.581** 0.587*** 0.476* 1.064*** 0.729*** 0.681*** 0.627** [0.178] [0.275] [0.187] [0.268] [0.172] [0.263] [0.176] [0.262] Average class size in 8th grade (2011) 9.921* 13.468*** 2.385 2.595 7.664* 10.613** -0.410 -1.720 [5.322] [5.062] [2.851] [2.857] [4.411] [4.689] [2.790] [2.954] Average class size squared (2011) -0.171 -0.256** -0.024 -0.040 -0.127 -0.201* 0.037 0.051 [0.125] [0.116] [0.066] [0.065] [0.103] [0.107] [0.064] [0.068] Small school (400 students or fewer in 2010) -13.738*** -13.563** -11.138** -9.277 -8.912 -12.712* -8.213 -14.507** [5.074] [5.689] [5.526] [6.117] [5.477] [6.468] [6.131] [7.297] Rural school -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Small school * Rural school -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Municipal welfare index (2010) 0.157 -- -0.008 -- 0.226* -- 0.097 -- [0.124] -- [0.121] -- [0.131] -- [0.130] -- Municipal fixed effects -- YES*** -- YES*** -- YES*** -- YES*** Percent of students whose mothers have at least a secondary education (2011) -- -- 1.065*** 1.101*** -- -- 1.021*** 1.098*** -- -- [0.180] [0.236] -- -- [0.177] [0.221] Percent of students whose fathers are employed (2011) -- -- 0.095 0.061 -- -- 0.060 0.022 -- -- [0.089] [0.088] -- -- [0.085] [0.087] Constant 291.982*** 272.883*** 344.181*** 334.190*** 287.068*** 273.403*** 346.454*** 327.749*** [67.339] [74.750] [41.520] [50.604] [59.256] [70.330] [46.688] [56.359] Observations 553 553 388 388 553 553 388 388 R-squared 0.362 0.624 0.498 0.743 0.315 0.616 0.435 0.722 Notes: 1 Robust standard errors in brackets; asterisks denote statistical significance at 10% (*), 5% (**), and 1% (***). 2 Regression specifications 1 and 2 are applied to Sample 1, specifications 3 and 4 are applied to Sample 2. 3 Estimated coefficients for municipal fixed effects are not shown, though asterisks denote their joint significance. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 38 Table A5.3. Determinants of Student Performance (Rural Schools) Dependent Average score on the Average score on the variable1: 8th grade Serbian language exam, 2011 8th grade Mathematics exam, 2011 Specification2: (1) (2) (3) (4) (1) (2) (3) (4) Average annual expenditure per student (2005-2010 in thd 2010 RSD) -0.090* -0.096 0.024 0.017 -0.074* -0.066 0.013 -0.014 [0.051] [0.073] [0.030] [0.047] [0.039] [0.046] [0.028] [0.038] Staff salary as percent of total expenditure (2010) -0.770** -0.234 -0.891* 0.084 -0.563 0.360 -0.643 0.449 [0.381] [0.639] [0.473] [0.690] [0.402] [0.624] [0.522] [0.794] Non-teaching staff salary as percent of total staff salary (2010) 0.473 -0.578 0.557 -0.192 0.758 -0.729 1.088 -0.426 [0.808] [1.071] [0.862] [1.261] [0.824] [1.060] [1.004] [1.429] Average years of experience of teaching staff (2010) 0.924 1.961 0.647 1.264 0.829 1.937 0.316 2.259 [0.963] [1.292] [1.106] [1.466] [0.994] [1.212] [1.108] [1.457] Percent of teaching staff with a university degree (2010) 0.812*** 0.766*** 0.590** 0.489 0.765*** 0.643** 0.540** 0.575* [0.194] [0.272] [0.231] [0.305] [0.207] [0.292] [0.242] [0.338] Average class size in 8th grade (2011) -0.394 1.953 -2.516 0.827 -3.825** -0.673 -4.803* -0.788 [1.942] [2.557] [2.412] [2.894] [1.887] [2.657] [2.453] [2.950] Average class size squared (2011) 0.013 -0.047 0.080 -0.008 0.119** 0.031 0.155** 0.043 [0.055] [0.074] [0.068] [0.082] [0.053] [0.075] [0.068] [0.084] Small school (400 students or fewer in 2010) -4.798 -4.600 -12.400** -8.671 -0.124 -1.821 -3.715 0.192 [4.877] [6.080] [5.044] [5.988] [5.063] [6.045] [5.986] [7.282] Rural school -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Small school * Rural school -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Municipal welfare index (2010) 0.175 -- -0.765* -- 0.469 -- -0.591 -- [0.331] -- [0.449] -- [0.355] -- [0.480] -- Municipal fixed effects -- YES*** -- YES*** -- YES*** -- YES*** Percent of students whose mothers have at least a secondary education (2011) -- -- 0.494*** 0.590*** -- -- 0.520*** 0.650*** -- -- [0.117] [0.162] -- -- [0.123] [0.187] Percent of students whose fathers are employed (2011) -- -- 0.069 0.111 -- -- 0.020 -0.054 -- -- [0.096] [0.114] -- -- [0.111] [0.125] Constant 451.816*** 387.435*** 484.872*** 346.600*** 445.165*** 386.760*** 471.557*** 349.858*** [43.050] [50.277] [49.972] [67.507] [45.061] [51.152] [55.583] [72.694] Observations 406 406 287 287 406 406 287 287 R-squared 0.137 0.413 0.193 0.487 0.106 0.451 0.152 0.504 Notes: 1 Robust standard errors in brackets; asterisks denote statistical significance at 10% (*), 5% (**), and 1% (***). 2 Regression specifications 1 and 2 are applied to Sample 1, specifications 3 and 4 are applied to Sample 2. 3 Estimated coefficients for municipal fixed effects are not shown, though asterisks denote their joint significance. Source: Author’s calculations based on data from BOOST, MOF, MOES, MCMIS, RSO. 39