WPS8669 Policy Research Working Paper 8669 Comprehensive Private School Model for Low-Income Urban Children in Mexico Lucrecia Santibañez Juan E. Saavedra Raja Bentaouet Kattan Harry Anthony Patrinos Education Global Practice December 2018 Policy Research Working Paper 8669 Abstract In low-income countries, private schools are perceived as statistically outperformed those not selected in math, but superior alternatives to the public sector, often improving the gains were more modest. Relative to the control group, achievement at a fraction of the cost. It is unclear whether parents of selected students were more satisfied with their private schools are as effective in middle-income countries school and had higher educational expectations for their where the public sector has relatively more resources. To children. Unlike findings from low-income countries, address this gap, this paper takes advantage of lottery-based these gains came at increased cost—twice as much on a admissions in first grade for a Mexico City private school per pupil basis relative to public schools. Additional anal- that targets and subsidizes attendance for low-income chil- yses indicate gains made by the lowest income students in dren. Over three years, selected students via lottery scored the sample help explain the school’s impact. This suggests 0.21 standard deviation higher than those not selected in private schools could bring down persistent achievement literacy tests, corresponding to a normalized gain of one- gaps in these countries, but puts into question the validity half of a grade level every two years. Lottery winners also of implementation at scale. This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at hpatrinos@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Comprehensive Private School Model for Low-Income Urban Children in Mexico1 Lucrecia Santibañez Claremont Graduate University Juan E. Saavedra University of Southern California and NBER Raja Bentaouet Kattan Harry Anthony Patrinos The World Bank JEL codes: I20, I25, I28, O15 1 Acknowledgments here. Corresponding author: L. Santibañez. Email: lucrecia.santibanez@cgu.edu. Support from the World Bank’s Strategic Impact Evaluation Fund (SIEF) is gratefully acknowledged. The views expressed here are those of the authors and should not be attributed to their institutions. 1. Introduction In low-income countries, private schools are perceived as superior alternatives to the public sector, often improving achievement at a fraction of the cost (e.g., Patrinos, Barrera-Osorio and Guaqueta 2009; Muralidharan and Sundararaman 2015; Andrabi, Das and Khwaja 2008). Many of these countries have weak public school systems, since public sector capacity is strongly correlated with national income (Rauch and Evans 2000; DeJanvry and Dethier 2012). It is unclear whether private schools are as effective in middle-income countries where the public sector has relatively more resources. This paper analyzes the academic impact and cost of an innovative private schooling model in Mexico. We take advantage of lottery-based admissions in first grade for a Mexico City private school that targets and subsidizes attendance for low-income children. Over three years, students selected via lottery score 0.21 SD higher than students not selected in literacy tests, corresponding to a normalized gain of about one- half of a grade level every two years. Lottery winners also statistically outperform those not selected in math, but gains are more modest. These gains come at an increased per- pupil cost--more than double than traditional public schools. The school we study targets children in poverty. Our findings confirm that this student population benefits the most. To arrive at this result we re-weight the sample to resemble an average Mexico City household in terms of income and other socio-economic characteristics. Other results suggest parents of children attending the private school are more satisfied and more likely to expect that their children will go to college than parents in traditional public schools. There are only a handful of rigorous studies exploring the effects of private schooling in middle-income countries. The most well-known school choice program in 2 the region, the Chilean national voucher reform, significantly increased student sorting and stratification by socioeconomic status with mixed results on test scores (see, for a review, Epple, Romano and Urquiola 2017), but positive effects on subsequent labor market outcomes (Bravo, Mukhopadhyay and Todd 2010; Patrinos and Sakellariou 2011). In Colombia, a voucher program which allowed low-income students to attend private secondary schools showed positive effects on high school graduation, college access and the labor market (Angrist, Bettinger and Kremer 2006; Bettinger, Kremer and Saavedra 2010; Bettinger et al. 2018). Most of these studies, however, focus on private school vouchers rather than private schools, and on student outcomes in secondary school. The majority of students in middle-income countries, however, do not successfully transition from primary into secondary school. In Mexico, for example, only half the students who complete elementary school ever attend high school (Kattan and Szekely 2015). To understand the potential benefits of private schools in these countries, it is important to look at early grades. Establishing a good literacy and numeracy foundation in the early grades will make the transition to secondary school less risky and eventually lead to long-term academic success (Duncan et al. 2007; Hernandez 2011; Cunningham and Stanovich 1997; Watts et al. 2014). To the best of our knowledge, there are no published experimental studies that investigate the academic effects of private schools and costs in middle-income countries in the earlier grades. Our results meet high evidence standards. Overall and differential sample attrition in the study are low and well within acceptable benchmarks for experimental studies, such as the Institute of Education Sciences' What Works Clearinghouse (Deke, Sama- 3 Miller and Hershey 2015). Our findings are robust to various empirical strategies, including randomization inference and re-weighting for increased external validity. In addition to leveraging the experimental, lottery-based research design to demonstrate academic impacts, we conduct additional observational analyses to understand mediating mechanisms. Using a strategy that matches on baseline covariates, we find that the length of the school day partially mediates the school's effects on achievement. Our findings are based on the results of a single school and families applying to Christel House de Mexico (CHM) do not resemble the average family in Mexico City. To account for this, we re-estimate all models using a method of entropy weights, which calibrates each observation based on their income and other socio-economic characteristics. Using this information, we generate two re-weighted samples: (1) one that resembles the average Mexico City household and (2) one that resembles a low-income Mexico City household (i.e., those with incomes below the median). This analysis indicates that CHM effects are no longer significant when the population resembles an average family in Mexico City, but are significant when the study sample resembles a low-income family. In other words, re-weighting the units so that study students look more like the average student in Mexico City mitigates our findings and highlights the fact that CHM is a model that is particularly beneficial for the most disadvantaged children. Our findings are consistent with similar studies conducted in the United States, which suggest private schools are especially favorable for low-income students with poor public schooling alternatives (Chabrier, Cohodes and Oreopoulos 2016; Angrist et al. 2012 for KIPP Schools; Dobbie and Fryer 2011 for Harlem Children's Zone). Our 4 contribution to the debate about the value of private schools in developing countries is to suggest that, at least in the context of middle-income countries like Mexico, expanding private schools could improve outcomes for the most vulnerable children, but will not necessarily save taxpayer money. The results presented in this paper indicate that significantly improving learning outcomes for low-income children in these countries requires substantial additional resources. 2. Background Close to 90 percent of elementary and lower secondary students in Mexico attend public schools (OECD 2014). Learning outcomes in public schools are quite poor: close to 60 percent of 15-year old students in the country place below basic proficiency levels in the Math PISA tests (OECD 2015). Results on national standardized tests paint a similar picture: two-thirds of 9th graders test below basic standards in mathematics and one-third test below basic standards in Spanish (INEE 2018).2 Results for students in rural schools (community schools and "telesecundarias") are particularly dismal: more than 70 percent of students in rural middle schools test below the basic proficiency level in math, and close to 50 percent tests below this level in Spanish. Most public schools in Mexico meet for half a day (4.5 hours) and follow the Ministry of Education curricula. Most offer some kind of preventive health services, remedial and extracurricular activities. None of these services are required by law. There are no contractual parental commitments, although public school parents in Mexico routinely perform volunteer cleaning and maintenance services (Santibañez, O'Donoghue and Abreu 2014). Teachers are hired centrally by the federal (or state) government and 2 Results refer to the "PLANEA" test. This test was introduced in 2015 to replace "ENLACE" which was administered from 2006-2013. 5 are granted tenure after 2 years on the job. Over 90 percent of national expenditures on education go toward paying staff payroll (OECD 2013). To become a public school teacher, teachers must have a teaching credential, which is granted upon completion of a B.A. teacher education program. Public schools in Mexico can be elementary (grades 1- 6), middle (grades 7-9) or high schools (grades 10-12). In 2007, owing to declining demographic trends in many large cities, the Ministry of Education began converting half-day elementary schools into full day schools. In 2016, there were over 24,000 full-time schools operating across Mexico. About 10 percent of these new full-time schools are located in Mexico City and its metropolitan area. Full time schools operate as 6-hour schools without lunch ("Escuelas de Jornada Ampliada") or 8-hour schools with lunch ("Escuelas de Tiempo Completo"). Research on full-day schools in Latin America finds positive effects on students, particularly those in rural and low-income communities (Cabrera-Hernandez 2016; Padilla-Romo 2016; Holland, Alfaro and Evans 2015). To enroll in a public school in Mexico City, parents submit an application to the Administración Federal de Servicios Educativos del Distrito Federal (AFSEDF), Mexico City's office of education. Parents must choose schools in their residential or work area. Charter schools are not legally allowed, and neither are vouchers at any educational level. Privatization efforts are highly controversial in Mexico due to formidable opposition from powerful teachers' unions (Santibañez and Jarillo 2008; Hecock 2014). This is not unique to Mexico. In low- and middle-income countries public delivery dominates education provision and accounts for over 85 percent of total primary school, and 75 percent of secondary school enrollment (World Bank 2018). 6 Christel House de Mexico (CHM) CHM is part of the Christel House International Organization, a philanthropic educational initiative that provides educational services to socioeconomically disadvantaged children. Christel House International operates two schools in India, one in South Africa, four charter schools in the U.S., and one school in Mexico. For this study we collaborated with the school located in Mexico City. During the time of our analysis, the school served grades 1 to 9, but has since expanded to serve grades 1 to 12. The school provides free or heavily subsidized education to over 400 low-income children. Most children belong to families with disadvantaged circumstances that might include infrequent employment, disability, incarcerated parents, mental illness, homelessness, and single parenthood. In 2013, the last year that Mexico held a national standardized exam testing all students in all grades, CHM 3rd graders scored 661 points in Mathematics, and 598 points in Spanish. This compared favorably to the average for Mexico City public schools of 619 points in Mathematics (std. dev. = 56) and 586 points in Spanish (std. dev. = 71).3 A small proportion of CHM families pay tuition on a sliding scale according to household income. In our sample, 10 percent of the families who won the lottery and enrolled in CHM paid tuition. Parents who reported paying tuition paid US$10-$20 per month on average.4 The remaining 90 percent of the families do not pay any tuition. Information from a parent cost survey administered as part of this study revealed that Christel House International covers CHM's administrative and fundraising expenses. The 3 Own calculations based on ENLACE Mexico City data for 2013. Available at: http://www.enlace.sep.gob.mx/ba/resultados_historicos_por_entidad_federativa/. 4 At 2016 exchange rate of 20 pesos per US$. As a comparison, average elementary private school tuition in Mexico City is about $150 per month. See: http://izq.mx/noticias/31/2018/01/cual-es-el-costo-de-tener-a- tus-hijos-en-una-escuela-privada-son-gastos-millonarios/. 7 school's operation, including teacher salaries, scholarships, nutrition, extracurricular activities, school uniforms, etc., is funded through local business and individual tax- deductible contributions and charitable donations. The school receives a small amount of public subsidies to pay for school lunches. Like many charter schools in the U.S., CHM requires strong parental involvement and commitment. Parents sign an agreement at the beginning of the year agreeing to attend 12 sessions of parenting education and donate 40 hours a year to school duties. For the most part, these duties include cleaning and maintenance. Students commit to an attendance rate of at least 95 percent a year. CHM has high levels of compliance with parent-student-school agreements and attendance is very high. Few students have ever been asked to leave for non-compliance reasons. CHM's educational model targets cognitive (academic) and non-cognitive development. Relative to most public schools in Mexico City, CHM offers a longer school day—9 versus 4.5 hours in most public schools (see Appendix Table A1). Close to six hours per day are spent in academic activities and close to two hours per day are spent on enrichment and other activities (sports, music, art, character development). CHM follows the curriculum and content standards set by the Ministry of Education in Mexico. They use publicly provided textbooks—which are provided free of charge to all schools (public and private) in Mexico. CHM purchases additional textbooks and resources from commercial curriculum vendors (i.e., Pearson). The school offers a wide range of academic and enrichment activities including sports, arts, English as a second language, a tutoring program for students needing remediation, counseling services for career- and college-readiness, and computer skill development. Consistent 8 with CHM’s "holistic" pedagogical approach, students receive preventative health services—mental health services, annual medical and dental check-ups—and a comprehensive nutritional package that includes daily breakfast and lunch following healthy dietary guidelines. All children receive school uniforms and school materials free of charge. To ensure high-quality teaching, CHM carefully selects teachers and dismisses them if they do not perform up to their standards. Teachers are not unionized and do not have tenure --as they do in public schools. Teachers receive professional development and are monitored and supported by the elementary school coordinator. In addition, the school employs literacy and mathematics coordinators to coach and support teachers in these areas. All students are expected to finish high school and go to college. Table 1 presents a contrast with curricular and program offerings at Mexico City public schools. CHM offers a much longer school day than traditional public schools, and a slightly longer day than full-time public schools. The school offers more nutrition, enrichment and other services. Overall, teacher seniority is lower than at public schools and teachers can be hired/fired at will. Teacher salaries are also lower at CHM than in public schools. Class sizes are the same. 9 Table 1. Christel House de Mexico and Public Schools - Main Features CHM Traditional Extended Full-Time Public Day Public Public Schools Schools Schools "Jornada "Escuelas Ampliada" Tiempo Completo" Length of school day (hours) 9 4.5 6 8 Curriculum National + National National National other /1 English as a second language, Yes Yes Yes Yes Breakfast Yes Yes/2 Yes/2 Yes/2 Lunch Yes No No Yes Art, music, computer skills Yes In some Yes Yes schools Autonomy over teaching Yes No No No hiring/firing Teacher salaries + benefits (average $16,829 $18,119 $18,119* $18,119* monthly, 1st-3rd grade only) in Mexican Pesos/3 Teacher Seniority (average, 1st-3rd 2 8 8 8 grade) (average) Parenting programs, psychological Yes No No No services for students/parents Extended day program, tutoring, Yes In some Yes Yes socio-emotional development, schools health preventive services. Class Size (average) 27 27 27 27 /1 CHM uses the national curriculum provided by SEP, but supplements with other textbooks, instructional materials and resources. /2 Only in schools that participate in the National Free Breakfast Program. These are usually schools that enroll a majority of low-income students. /3 Salaries are for a 4.5 hour (traditional) teaching position. In schools where teachers teach an extended day, or multiple positions, salaries will be higher. Source: For CHM, cost interviews conducted as part of this study. For public schools, various published sources and interviews with a small sample of schools. See cost section for details. 3. Data and Research Design Lottery Procedures and Data Every year CHM has more eligible applicants than open seats for its 1st grade cohort. Before this study began, CHM agreed to hold a public lottery for available first- 10 grade seats. We designed the lottery and carried it out. Figure 1 shows a timeline of the study, including baseline data collection, lottery, and testing. Figure 1. Timeline of Main Evaluation Events We collected data on lottery results (including wait list assignments), and enrollment. Each year, we updated the data for both CHM and public schools to account for any sample attrition. Between 2013 and 2015, 242 students participated in the lottery as part of three cohorts.5 CHM carried out separate lotteries for boys and girls, effectively stratifying by gender, to preserve a balanced gender mix. Through the lottery mechanism, 100 students, belonging to three cohorts, were offered a seat in CHM's 1st-grade cohort and 142 were not offered admission. Of the students offered a spot, 50 percent were girls and 50 percent were boys. In each lottery cohort, CHM also randomly selected 10 students (5 boys, 5 girls) for a wait-list. Children in the random wait list were offered a 5 Applicants with siblings already enrolled in CHM have guaranteed admission and did not participate in the lottery. They are not reflected in these numbers. Two parents/students refused informed consent to participate in this study. While they still participated in the lottery, they are not included in this sample. 11 spot whenever a lottery winner did not enroll in CHM or dropped out. For the analysis, we classify wait-listed children as treated. Overall, treatment compliance was high. Of those originally offered a 1st grade spot (and not on the waitlist), 94 percent took up the offer and enrolled in CHM. Of those originally on the waitlist and subsequently offered a spot, 9 enrolled in CHM. This represents 6.5 of the control group. However, as previously noted, we count waitlisted children—regardless of subsequent enrollment in CHM—as treated. Data were collected at baseline (before the lottery) and at the end of each academic year. We followed each cohort from the time of the lottery until the last data collection round in 2015. This means that for the first cohort we have three years of data (i.e. data for 1st, 2nd and 3rd grades), for the second cohort we have two years of data (1st and 2nd grades), and for the third cohort we have one year of data (1st grade). In our tables and discussion, we refer to each of these points as 1st, 2nd, and 3rd follow-ups. In 2015 the Mexico City office of education (AFSEDF) did not allow test administrators access into control group public schools. During this year control group students were tested in their homes, while CHM students were tested at the school. Varying testing conditions could by themselves explain some of the differences in test outcomes between treatment and control. To account for this potential source of bias, we run all of our analyses for three samples: (1) “One-Year Sample," which includes first- year follow-up results only, (2) "Same Testing Conditions Sample," which includes only observations that were tested under the same conditions (excludes 2015); and (3) "All Observations," which includes the full sample of observations in all follow-up years. 12 Lottery Sample CHM families tend to earn about half what those in the average Mexico City household sample earn on a monthly basis—about 3,700 pesos or US $185 per month compared to 7,800 pesos or US $390 for the Mexico City sample (exchange rate 20 pesos/dollar). Overall, CHM mothers have slightly less schooling than the household sample, but in both cases the average level of schooling completed is lower secondary school (9th grade). It should be noted that Mexico City has one of the highest levels of average educational attainment in the country. More students live in single-parent homes in the CHM sample, than in the Mexico City household sample, about 50 and 20 percent respectively. Lastly, CHM families are more likely than the average Mexico City family to rent their home, to not have a kitchen, to not be hooked to a public utility source for electricity, and to not have a landline (see Table 2). After the lottery was conducted, students enrolled in either CHM or a public school. Of the total sample, 43 percent enrolled in CHM, 26 percent enrolled in full-day public schools, and 30 percent enrolled in half-day public schools. An additional two percent enrolled in schools of unknown type (i.e., full-day or half-day). 13 Table 2. Descriptive Statistics and Balance Mexico CHM- CHM- CHM p-val of Standardi City HH Control Treatment Diff. (C- C-T Diff. zed Diff. Sample /3 Variable (1) (2) T) (3) (4) (5) (6) Age (incoming 1st graders)/1 5.216 5.297 -0.081 0.223 -0.16 -- (0.047) (0.046) (0.066) Gender (incoming 1st graders) 0.577 0.508 0.069 0.289 0.14 -- (0.047) (0.044) (0.065) Raven Global Score (incoming 1st graders) 15.599 16.445 -0.846 0.101 -0.21 -- (0.377) (0.350) (0.515) WISC Global Score (incoming 1st graders) 25.087 23.844 1.244 0.361 0.11 -- (1.082) (0.851) (1.360) Monthly mean income (in MX pesos)/2 3,664 3,389 274 0.412 0.12 7,812 (226.23) (242.95) (335.35) (419.12) Mother's education (Secondary=4) 4.144 4.219 -0.075 0.426 -0.07 4.35 (0.072) (0.061) (0.094) (0.073) Mother has elementary school or less 0.177 0.142 0.036 0.449 0.10 0.22 (0.036) (0.031) (0.047) (0.030) Mother has high school or more 0.360 0.398 -0.038 0.547 -0.08 0.45 (0.046) (0.043) (0.063) (0.036) Both parents are present in the home 0.495 0.528 -0.033 0.613 -0.07 0.75 (0.047) (0.044) (0.064) (0.031) Home is rented 0.374 0.456 -0.082 0.197 -0.17 0.24 (0.045) (0.044) (0.063) (0.030) Home has a kitchen 0.681 0.598 0.083 0.179 0.18 0.95 (0.044) (0.043) (0.062) (0.016) Home is hooked to public electricity 0.811 0.787 0.024 0.644 0.06 1.00 (0.036) (0.036) (0.052) (0.000) Home has landline 0.373 0.339 0.034 0.576 0.07 0.46 (0.045) (0.042) (0.062) (0.035) Observations 111 128 239 270,000 *** p<0.01, ** p<0.05 ** p<0.10 Notes: Source for Mexico City sample is ENIGH-2014 (tradicional). HH with children in grades 1-3 in public schools. Sample represents over 270,000 HHs in Mexico City. Source for CHM treatment and control is lottery sample. F-value tests the hypothesis that the coefficients on key demographic variables measured at baseline (age, Raven, WISC) on winning the lottery (treatment status) are all jointly zero. F-value from joint test: 0.71. p-value from F-test 0.54. /1 Student Age for CHM is collected between March-June in the school year previous to entering 1st grade. /2 Average monthly household income from wages and other fixed sources. /3 Standardized with respect to the control group standard deviation. 14 Balance in Baseline Covariates Baseline differences between lottery winners (treatment) and losers (control) in baseline demographic and cognitive measures are not significant for any of the variables used in the analysis (see Table 2, Columns 3-4). The F-value for the joint test of balance for all these observable characteristics has a p-value of 0.54. All the differences were less than the WWC standard for baseline equivalence of 0.25 SD of the control group mean (see Table 2, Column 5). Together, these results strongly suggest the study sample was balanced prior to the lottery rendering the experimental design highly internally valid. Differential Attrition Overall sample attrition in the study was low. The sample contains 502 observations over three data collection waves. We are missing data for 18 of these observations (3.5 percent attrition rate). These 502 observations represent 242 students. We are missing some follow-up data for 16 of these students, or 6.6 percent. Most of these students missed only one test application but have data for the other applications. This means attrition by wave is lower than the overall reported attrition rate. Student attrition was 1.2 percent of the sample at the first follow-up, 4.9 percent of the sample at the second follow-up, and 6.4 percent of the sample at the third follow-up. Differential attrition between treatment and control groups can generate bias for the treatment effect estimates if those who leave the treatment group differ from those who leave the control group along dimensions related to the outcome of interest. In such a case, estimated program impact can be biased even if the lottery successfully generates comparable treatment and control groups at baseline. 15 We follow current practice in the analysis of field experiments (Duflo, Glennerster and Kremer 2006) and use two tests to determine whether attrition is likely to generate bias in program impact estimates. The first examines whether the fraction of applicants with missing outcome data differs between lottery winners and losers. For each applicant cohort, we propose to regress , an indicator random variable for whether outcome data are missing for a given applicant, on lottery status and other previously defined covariates, as follows: (1) The null hypothesis of no difference in the probability of dropping out of the evaluation sample between lottery winners and losers corresponds to : 0. Note that this analysis needs to be conducted for each outcome at each follow-up point since attrition patterns can vary across outcomes and over time. The second test examines whether the factors that predict attrition are the same between lottery winners and lottery losers. To implement this test, we modify equation (1) to include interactions between and the vector of covariates : (2) and test the null joint hypothesis that all the parameters in the vector are zero: : ⋯ 0, where is the number of baseline characteristics in the vector X. Rejection of the null hypothesis on these two tests raises concerns about attrition bias. Table 3 shows results of these two tests of differential attrition. The first test (Panel A) finds significant differential attrition of -0.04 for the pooled sample. An analysis of differential attrition by year of the study finds statistically significant differential attrition at the 2nd follow-up (coefficient of -0.06), and no statistically 16 significant differential attrition at the 1st and 3rd follow-ups (coefficients of -0.01, and - 0.06). As previously mentioned, during the 2nd year of the study we were denied access into control group schools by the Mexico City educational authorities, which negatively impacted our ability to reach as many control students as we did in the other two years of the study. Table 3. Differential Sample Attrition Pooled 1st 2nd 3rd Follow-up Follow-up Follow-up Test 1: Differential Attrition Test Coefficient on Lottery Admission variable (model with lottery strata fixed -0.04** -0.01 -0.06** -0.06 effects only) (0.016) (0.017) (0.028) (0.046) Intercept (control group mean) 31.02* -12.52 92.66 0.13** (18.645) (9.514) (59.339) (0.059) Coefficient on Lottery Admission variable (model with lottery strata fixed -0.04** -0.02 -0.06** -0.06 effects + student covariates) (0.016) (0.021) (0.026) (0.046) Intercept (control group mean) 25.03 -33.68 100.73 0.00 (25.081) (27.353) (67.873) (0.219) Test 2: Predictors of Differential F-test p-value Attrition Test F-test (attrition predictors do not vary 0.256 0.923 0.582 0.643 across treatment and control - model with lottery strata FE + student covariates + interactions) N 520 242 184 94 *** p<0.01, ** p<0.05, * p<0.1 Note: Lottery strata fixed effects included for gender and cohort. Student-level covariates include RAVEN and WISC test score, age, mother's education, mean monthly income, both parents present in the home, home is rented, home has a kitchen. Test 1: OLS regression with sample attrition as dependent variable, and lottery admission as the key independent variable (to test for differential attrition between treatment and control group). Test 2: Same as Test 1 but adding interactions between lottery status and covariates. We report the joint f-test of significance that attrition predictors do not vary across treatment and control. This model is estimated using OLS regression. 17 A second test of differential attrition examines whether factors that jointly predict attrition vary among lottery winners and losers. This test cannot be rejected at conventional levels (see Panel B of Table 3). Together, these results suggest that, if anything, the missing treatment and control group data in the pooled sample and second follow-up appear to be missing at random. These low differential attrition rates are well within the benchmarks issued by the Institute of Education Sciences' What Works Clearinghouse standards for evidence in randomized experiments.6 Outcome Measures - Instruments Mexico has no national or state standardized tests in grades 1-3. For this reason, we use the Early Grade Reading Achievement test (EGRA; RTI International 2009) to assess student skills in Spanish literacy and the Early Grade Mathematics Achievement test to assess student skills in mathematics (EGMA; Mejia and Pflepsen 2012). For this study we used RTI-approved Spanish translations of EGRA, which we adapted to Mexican-style Spanish and validated in a pilot administration prior to the first year of data collection. Results from the validation exercise deemed both instruments to be reliable with a Cronbach's alpha of 0.88 for EGRA and 0.87 for EGMA (Barrios, 2014). EGRA has been used to assess reading skills of children at grades 1-3 in more than 50 countries and 70 languages (Gove and Wetterberg 2011). Prior studies showed it is a valid and reliable tool to measure reading achievement at early grade levels (RTI International 2009).7 EGRA includes eight subtests, including letter name knowledge, phonemic awareness, letter sound knowledge, listening comprehension, unfamiliar non- 6 The WWC sets the maximum acceptable rate of differential attrition at around 7-7.5 percent for studies with overall attrition lower than 10 percent (Deke, Sama-Miller and Hershey 2015). 7 The tests have been criticized on the grounds that they measure a narrow set of skills (e.g. Bartlett et al., 2015). Dubeck and Gove (2015) argue the tests measure foundational skills that are important in the early grades and have shown to correlate with later indicators of reading in standardized tests. 18 word reading, familiar word reading, passage reading and comprehension, and dictation. Trained examiners orally administer each test to children one-on-one. Completing the test takes 15-20 minutes. EGMA measures quantitative reasoning skills of students at grades 1-3. Prior studies showed that EGMA is a reliable measure of quantitative reasoning skills in primary schools (USAID 2010). EGMA has seven subscales, including number identification, quantity discrimination, missing number, word problem, addition/subtraction problem, shape recognition, and pattern extension. Trained examiners administer these tests to individual students orally. It takes 15-20 minutes to complete the test. Gains in EGRA/EGMA are measured by comparing results at the end of each corresponding grade per cohort. EGRA/EGMA are designed to capture the full range of abilities in 1st-3rd grade. Consequently, the average first grader is expected to score lower on the same test than the average third grader. Parent Perception Data A survey was administered to Cohort 2 parents to measure differences in opinions about their child's schooling and expectations. The survey was administered two years after baseline (in 2016). Of 91 parents who consented to respond to the parent survey at baseline, 83 took the survey two years later, a 91 percent response rate. Research Design and Empirical Strategy This evaluation follows an experimental design that takes advantage of lottery- based admissions into CHM. To estimate the causal effect of attending CHM on literacy and numeracy test scores, we estimate ANCOVA models that have the form: 19 (3) where is an outcome for applicant i in applicant cohort c (2013, 2014, 2015), measured in post-treatment year t (2014, 2015, 2016). This notation implies that we observe Cohort 1 over 3 waves of data collection, Cohort 2 over two waves, and Cohort 1 over one wave; are cohort fixed effects; is an indicator that takes the value of one if applicant i wins the admissions lottery and zero otherwise; is the vector of student background characteristics (including baseline cognitive scores and age); Because lotteries were stratified by gender, we include a gender dummy . To account for the fact that we have repeated post-treatment outcomes for the 2013 and 2014 applicant cohorts, we cluster the standard errors at the applicant level. Estimates of capture the regression-adjusted intent-to-treat (ITT) effect of being offered a first-grade slot at CHM at the lottery stage.8 Because we observe different cohorts for different years, we follow Angrist et al. (2010) to convert each cohort's estimates into a normalized average per-year effect. We do this by modeling the impact of CHM on outcomes as a function of time spent in CHM: (4) where are cohort fixed effects; is the number of years applicant i from cohort c has spent in CHM at the time of the post-treatment measurement year t; is the vector of student background characteristics (including baseline cognitive assessment scores); is a residual. We estimate equation (2) using instrumental variables in a two-stage least 8 Other research has shown initial offers are consistent and asymptotically normal estimators of the LATE, even if they have higher variance than other estimators (De Chaisemartin and Behaghel 2017). 20 squares (2SLS) framework in which the randomly assigned lottery outcome serves as an instrument for . Standard errors are clustered at the applicant level. A possible issue with randomization is that results may be sensitive to just a few outliers or clusters of observations. As described in Young (2018) and Athey and Imbens (2017), model-based regression designs may render cluster and robust variance estimates more unstable for a given regression design generating incorrect tests of significance. To counter this potential issue, we run randomization inference tests for each literacy and numeracy outcome as well omnibus tests of overall significance. EGRA/EGMA yield results in multiple domains (eight subscales for EGRA, and seven for EGMA). To ease presentation and interpretation we pool all subscale score/effects using a fixed-effects average effect size (Cooper, Hedges and Valentine 2009; Bettinger, Kremer and Saavedra 2010; Katz, Kling and Liebman 2001). To produce a summary score for early grade literacy, and a summary score for early grade numeracy, we first standardize all domain score results relative to the control group mean and standard deviation (SD) in the first follow-up. Next, we calculate a precision- weighted average effect size. This is obtained using the inverse of the estimate's variance and calculating its t-statistic. This technique assumes estimating a single population parameter under a fixed-effects model (Cooper, Hedges and Valentine 2009). Reweighting for External Validity Families applying to the CHM lottery look very different from the average family in Mexico City. One concern is that our results cannot be easily extrapolated to the average family in Mexico City due to "participation bias" (Andrews and Oster 2018). To account for this, we use the method of "entropy weights" as proposed by Hainmuller 21 (2012) and Hainmueller and Xu (2013) to re-weight the sample to resemble the average Mexico City family. The method resembles propensity score matching. To the extent that most of the potential participation bias is explained by observable characteristics, results will be externally valid (Andrews and Oster 2018). To estimate the weights, we use Mexican household data from the Encuesta Nacional de Ingreso y Gasto de los Hogares 2014 (ENIGH-MCS). For this estimation we select households in Mexico City with children attending public schools in 1st-3rd grade. We use ENIGH variables that overlap with socioeconomic data collected from the CHM sample at baseline. The resulting weights closely replicate the first and second moments of the Mexico City household distribution. Results are shown in Table A.2. in the Appendix. Cost Data and Analysis To compare the costs of providing education at CHM compared to traditional public schools attended by control students, we use a version of the ingredients method, a simple yet rigorous procedure to guarantee inclusion of all relevant costs (Dhaliwal et al. 2011; Levin and McEwan 2001, 2000). We collected cost data about CHM from school staff using the ingredients method excel worksheet in the fall of 2016. We collected costs only for the elementary school portion of CHM's operations. We pro-rated overlapping or other costs that serve the entire school by elementary school enrollment. To estimate costs in control schools we used information provided by the Mexico City Office of Education, the Mexican Ministry of Education and other published 22 sources. To fill in gaps and cross-reference the information provided by the Mexico City office of education we collected cost data from a sample of five control group schools. To estimate the average teacher cost in each of the control schools, we obtained data on the number of teachers and their seniority from the 2016-17 school census ("Forma 911"). Using these data, we identified each of the teachers and other staff working in control group schools by type of position. We used published salary schedules to estimate compensation for each teacher and staff position, taking into account base salary and benefits, as well as any bonuses received by teachers participating in incentive programs such as "Carrera Magisterial." Most public schools in Mexico City received their buildings as donations from private individuals or foundations, and these donations go back many decades. The Mexican office of education was unable to provide information that would allow us to estimate infrastructure costs except in the case of one school, which was being rented at market rates. We pro-rated the rent paid by this one school on the basis of enrollment and applied that rate to other control schools based on enrollment. Utility costs (water, electricity) were obtained using data provided by AFSEDF for 83 public control group schools in our sample. To estimate the cost of providing breakfast and lunch in schools where this service is available, we use data contained in the school, teacher and student census of 2013 ("CEMABE 2013") as well as budget information for the National Free Breakfast Program. We also use information from CEMABE 2013 to identify control group schools that participate in the national breakfast program. Costs were pro-rated by school enrollment. To estimate the cost of providing lunch in full-day schools, we use data 23 gathered from interviews with five control group schools to calculate per-student lunch costs. We gathered educational supply costs for textbooks and school supplies. Textbook costs were obtained using the published cost of producing these textbooks by the National Commission on Free Textbooks ("CONALITEG"). Mexico has a national free textbook program, and all students in public elementary schools receive textbooks free of charge. To estimate school supplies costs, we use published prices from one large supplier in Mexico City. The Mexican Ministry of Education publishes an approved "school supply" list that schools in Mexico City adhere to. This list was used as a benchmark to estimate costs for all control group schools based on their enrollment. Both CHM and traditional public schools receive donated goods and services. In addition, parents pay out-of-pocket costs. We ask questions in the parent survey about these expenses and report them below. One potentially important cost that we do not include is parental time, which is required at CHM and frequently donated at public schools across the country (Santibañez, O' Donoghue and Abreu 2014). We do not have data to estimate the monetary value of parental time. Given that over 90 percent of the cost of public schools in Mexico covers teaching and non-teaching staff salaries we feel confident our estimation is close to the real cost of providing schooling to students in our sample. 4. Results Effects on Early-Grade Literacy and Mathematics Table 4 shows intent-to-treat (ITT) effects of winning the admissions lottery. Results for the full sample (Table 4, Column 3) show that CHM students who won a seat 24 in the school's first grade cohort via lottery scored about 0.21 SD higher in early grade literacy, and 0.09 SD higher in early grade numeracy than students who did not win the lottery. These results represent an average for lottery Cohort 2013 students observed three times up to third grade; lottery Cohort 2014 students observed two times up to second grade, and lottery Cohort 2015 students observed once in first grade. Table 4. Effects of Private School on Learning Outcomes (1) (2) (3) (4) (5) (6) Panel A. Early Grade Literacy (EGRA) Won admissions lottery 0.26** 0.19** 0.21** 0.30** 0.17** 0.18** (0.042) (0.036) (0.031) (0.042) (0.037) (0.031) Panel B. Early Grade Numeracy (EGMA) Won admissions lottery -0.03 0.13** 0.09** 0.06* 0.10** 0.06 (0.047) (0.044) (0.038) (0.038) (0.043) (0.038) Comparable Comparable 1st grade testing 1st grade testing Sample/1 only conditions All Obs. only conditions All Obs. Demographic and cognitive baseline controls/2 No No No Yes Yes Yes N 239 328 502 239 328 502    *** p<0.01, ** p<0.05, * p<0.1. Robust standard errors in parentheses. /1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing values imputed, and missing dummies included for imputed cases. Note: All regressions include lottery strata fixed effects (gender and cohort). These are intent-to-treat effects with lottery offer as the key explanatory variable, and test outcome as the dependent variable. Column 1 of Results shown in Column 2 of Table 4 show that students who won the lottery for admission into CHM scored 0.19 SD higher than those who did not win the lottery, over the average time spent in CHM without taking into account 2015 – the year 25 where control students were tested in their homes. Unless otherwise noted, all of these effects are statistically significant at the 95 percent confidence level. The magnitude of the effects diminishes only slightly when student controls are included, to 0.17-0.30 SD in literacy, and 0.06-0.10 in mathematics. Figure 2 shows test score growth curves for each grade observed in the study. Results are pooled for all cohorts. They suggest that CHM improved learning outcomes in literacy in every grade except 2nd grade. In mathematics, CHM improved outcomes every grade, although the absolute levels are lower than they are for literacy. Results are statistically significant at the 95 percent confidence level, except for the 1st grade estimate for math. 26 Figure 2. Grade Level Growth Curves, All Cohorts Note: These growth curves represent intent-to-treat with lottery offer as the key explanatory variable, and test outcome as the dependent variable. All regressions include lottery strata fixed effects (gender and cohort) and robust standard errors. Effect sizes are averaged using the meta-analytic technique discussed in the text. See text for more detail. 27 To account for differences across cohorts in the time spent at CHM, we model the causal effect of attending CHM on outcomes as a function of how much time students spend at the school. To do this we instrument time spent in CHM with lottery offers using a 2SLS approach. Table 5 shows the results. Table 5. Average Annualized Achievement Gains in Private School (1) (2) (3) (4) (5) (6) Panel A. Effect of Winning the Lottery on Number of Years of Private School Attendance Won admissions lottery 1.70** 1.86** 1.87** (0.104) (0.138) (0.131) Panel B. Average Annualized Achievement Gain in Literacy Years in CHM (instrumented) 0.15** 0.10** 0.11** 0.17** 0.09** 0.10** (0.024) (0.019) (0.017) (0.023) (0.019) (0.016) Panel C. Average Annualized Achievement Gain in Math Years in CHM (instrumented) 0.02 0.07** 0.05** 0.05* 0.05** 0.03 (0.028) (0.024) (0.021) (0.027) (0.023) (0.020) Comparable Comparab 1st grade testing 1st grade le testing Sample only conditions All obs. only conditions All obs. Demographic and baseline controls/1 No No No Yes Yes Yes N 239 328 502 239 328 502 *** p<0.01, ** p<0.05, * p<0.1. Robust standard errors, clustered at the student level, are reported in parentheses. /1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing values imputed, and missing dummies included for imputed cases. Note: These results are from regressions that instrument time spent in CHM with lottery offers. Models are estimated using 2SLS. All regressions include lottery strata fixed effects (gender and cohort). As expected, the first-stage results, shown in Panel A, indicate that winning an offer of admission results in about two additional years spent in CHM (full sample, Table 28 5, Column 3). The 2SLS results using the full sample indicate that students gain an average of 0.11 SD in literacy for each year spent in CHM and 0.05 in numeracy (Column 3, Table 5). When we restrict the sample to only the first follow-up (at the end of 1st grade), each additional year in CHM raises literacy scores by 0.15 SD and mathematics scores by 0.02 (not significant). When we take out 2015, the year when students were not tested under comparable conditions, each additional year in CHM raises literacy scores by 0.10 SD and mathematics scores by 0.07. Unless otherwise noted, all reported results are statistically significant at the 95 percent confidence level. Results change only slightly when demographic and cognitive baseline controls are included. To put this per-year effect in context, we calculate the average growth of the control group over the period under study. In EGRA, the control group grew an average of 0.45 SD per year in literacy and 0.71 SD per year in math. Relative to the control group, after two years the average student at CHM would be about one-half of a year ahead in literacy, and on par in math. The results presented above are robust to randomization inference in all samples except the 1st grade only sample in mathematics. Table 6 shows results from randomization inference omnibus tests performed on the joint significance of all pooled test score outcomes. The omnibus tests for randomization inference are significant at the 95 percent confidence level for all samples of the literacy outcomes. The randomized p- value for the literacy average effect is 0.005, 0.001, and 0.001 for all three samples respectively. For the mathematics outcomes, the values are 0.14 for first grade only, and 0.012 and 0.024 for the comparable testing conditions and all student samples. These 29 results imply that our findings are not sensitive to just a few outliers or clusters of observations as can sometimes be the case with smaller-sample studies. Table 6. Randomization Inference Tests of the Average Effect of Private School on Learning Outcomes Pooled Min Max Randomized Omnibus Tests p-value p-value p-value N Early Grade Literacy Sample of 1st grade students 0.000 0.001 0.005 239 Sample of students in comparable testing conditions 0.000 0.001 0.001 328 All students 0.000 0.001 0.001 502 Early Grade Numeracy Sample of 1st grade students 0.136 0.138 0.137 239 Sample of students in comparable testing conditions 0.012 0.013 0.012 328 All students 0.022 0.024 0.024 502 Note: This table presents results of randomization inference tests over the average effect size for the intent-to-treat estimates presented in Table 4. We use the randcmd in STATA to conduct these tests (Young, 2017). For each test, we conduct 1,000 permutations. Early-grade Gains Accounting for Differences in Length of School Day CHM's positive impact on early grade literacy and numeracy can be explained by a number of factors: length of the school day, better teachers, more parental involvement, health prevention services, better leadership, etc. We do not have data to unpack everything driving CHM's positive impact, but we do have data on one key factor: length of the school day. Our data allow us to identify whether Mexico City public schools attended by the control group are on a half-day (4-hour) or on a full-day (6 or 8 hour) schedule. With this information we can estimate the effects of CHM relative to other full- day schools and separate out the effect of the longer school day from the overall impact. One issue with this estimation is that while students enter CHM via lottery, parents do not follow a random process to enroll their students in public schools. Parents 30 could self-select into full-time public schools in ways that could bias an estimation of program effects. To explore this potential bias, we run descriptive statistics on students by whether they attend school full- or half-day. Results are shown in Table 7 and suggest balance on most characteristics, except for two. First, the Raven baseline score is statistically higher in the full-day school sample. This is not the case for the verbal WISC score. The magnitude of the Raven score difference, however, is small: about 0.02 SD. Second, single parents are more likely to seek out full time schools for their children. All other demographic characteristics show no significant differences among students attending full- and part-time public schools. Table 7. Characteristics of Control Group Students in Full-time vs. Half-day Public Schools Half-Day Full-Time Variable Public School Public School Diff. p-val Age (incoming 1st graders)/1 6.102 5.882 0.22 0.068 (0.079) (0.080) (0.113) Raven Global Score (incoming 1st grade) 14.656 16.394 -1.739 0.001 (0.343) (0.337) (0.484) WISC Global Score (incoming 1st grade) 24.63 25.123 -0.493 0.839 (1.088) (0.865) (1.419) Monthly mean income/2 3636.54 3370.08 266.46 0.208 (187.47) (228.39) (292.85) Mother's education (Secondary=4) 4.18 4.14 0.040 0.480 (0.067) (0.097) (0.095) Both parents are present in the home 0.59 0.37 0.221 0.001 (0.041) -0.043 (0.06) Home is rented 0.412 0.348 0.064 0.333 (0.041) (0.043) (0.060) Home has a kitchen 0.678 0.638 0.04 0.381 (0.039) (0.043) (0.058) N 137 119 256 1 Student Age for CHM is collected between March-June in the school year previous to entering 1st grade. Mexico City population, is collected between September-November of the corresponding school year. /2 Average monthly household income from wages and other fixed sources. Notes: Logistic regression. Indicator Full time schools = 1. Robust standard errors in parentheses. 31 To account for any bias that may arise from parental self-selection into full-time schools based on observable characteristics, we use propensity score matching. First, we restrict the sample to students who attend CHM and full-time public schools and fall within the common support region. To estimate the common support region and predict the propensity score we estimate a probit model of lottery status on baseline student covariates and lottery strata. We do this for the subset of schools that include CHM and full time public schools only. We then use the min/max method to eliminate observations that fall outside the common support region. Next, we estimate ITT using a matching estimator that accounts for the propensity to be selected for the program for students at CHM or full day public schools. We estimate the LATE using instrumental variables with the propensity score as an independent variable. Results are shown in Table 8. Table 8. Length of School Day as Possible Mediating Mechanism (1) (2) Panel A. Early Grade Literacy (EGRA) Won admissions lottery 0.10* 0.20** (0.055) (0.043) Years in CHM (instrumented) 0.03 0.05** (0.022) (0.018) Panel B. Early Grade Numeracy (EGMA) Won admissions lottery 0.11 0.07 (0.068) (0.056) Years in CHM (instrumented) -0.02 -0.01 (0.028) (0.023) Comparable testing Sample conditions All Observations Demographic and cognitive baseline controls/1 Yes Yes N 198 307 *** p<0.01, ** p<0.05, * p<0.1. Robust standard errors in parentheses. Note: To obtain these results we first match CHM students to control-group students in full-time schools using propensity score matching. We keep students in the common support region. This eliminated 16 observations or 5% of the matched sample.. All regressions include lottery strata fixed effects (gender and cohort). 32 /1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing values imputed, and missing dummies included for imputed cases. The literacy ITT on the matched full sample is similar as it was for the main model (0.20 SD vs. 0.21 SD). The normalized (instrumented) effect of attending CHM is much lower in the model that adjusts for length of school day, 0.05 SD vs. 0.11 SD for all students. In math, none of the effects are significant for the matched sample. Results are similar when we filter out the year when students were not tested under comparable conditions—except that now the literacy coefficient is only significant at the 10 percent confidence level, and the instrumented coefficient is no longer significant. Taken together, these findings indicate that the positive impact of CHM was largely mediated by the longer school day--the only mediating school-level variable that we could test. A portion of the effect does remain in literacy, suggesting that in this case other CHM characteristics remain important predictive factors.9 Re-Weighted Results - External Validity To explore whether our findings are sensitive to whether or not CHM lottery applicants are observationally similar to the average family in Mexico City, we re- estimate all models using entropy weights as described in the methods section. We use two sets of weights: (1) "average weights" based on the entire Mexico City household sample and (2) "low-income weights" using only households below the median income. Results are shown in Table 9. When the study sample is weighted to resemble the "average student" both the ITT and IV literacy effects of CHM are either negative 9 We conducted a small qualitative study of CHM and two control schools to understand what drove the differences between literacy and math. Results from classroom observations using the CLASS rubric indicate higher classroom quality scores in Spanish at CHM relative to control schools. In math, however, there was little observed difference (Razo 2017). 33 ("comparable testing conditions" sample) or not statistically significant (full sample). The effects on math are not statistically significant across the board. When the study sample is weighted to resemble a low-income student (see Table 9, Columns 3 and 4), however, the results are once again positive and statistically significant for literacy. Results remain non-significant for math. Table 9. Effects of Private School Attendance on Achievement Re-weighted for External Validity (1) (2) (3) (4) Panel A. Early Grade Literacy (EGRA) Won admissions lottery (ITT) -0.18** 0.06 0.14** 0.28** (0.057) (0.074) (0.058) (0.048) Years in CHM (instrumented) (LATE) -0.10** 0.03 0.07** 0.12** (0.033) (0.042) (0.027) (0.023) Panel B. Early Grade Numeracy (EGMA) Won admissions lottery 0.00 -0.12 0.09 0.01 (0.086) (0.076) (0.069) (0.058) Years in CHM (instrumented) (LATE) 0.01 -0.06 0.05 0.01 (0.049) (0.044) (0.036) (0.030) Comparable Comparable testing testing Sample conditions Full Sample conditions Full Sample Calibrated to Calibrated to Calibrated to Calibrated to Type of Weight/1 Average HH Average HH low-income HH low-income HH Demographic and cognitive baseline controls/2 Yes Yes Yes Yes Observations (N) 328 502 328 502 *** p<0.01, ** p<0.05, * p<0.1 Note: All Samples includes only observations that are in the common support region attending CHM and other full- /1 Weights are calculated using ENIGH 2014 data. Variables include: income, rents, pub_elec, landphone, mother's education, and both parents present. Models are estimated using the ebalance command in STATA per Hainmuller & Xu (2013). See text for more details. 34 /2 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing values imputed, and missing dummies included for imputed cases. This result is striking for two reasons. First, even though our results come from a single school, they can be replicated for a larger population of students who are observationally similar (i.e., students below the median income distribution in Mexico City). Second, CHM's model of education is most beneficial to low-income students-- their intended targeted population. When we give more weight to slightly better-off students so that the study sample resembles the average Mexico City household, CHM test scores are no longer above the control group test scores (and are even lower under some specifications). However, when we re-weight the study sample so that the lowest income students get a higher weight, the results are once again significant. Other authors have arrived at similar conclusions using charter school evidence from the United States (Chabrier, Cohodes and Oreopoulos 2016). Parent perceptions High parental involvement and expectations can positively influence academic outcomes for low-income students (Domina 2005; Marschall 2006; Yamamoto and Holloway 2010). Most research in the U.S. finds that charter school parents tend to be more satisfied and more involved in school than public school parents (Gleason et al. 2010; Cheng and Peterson 2017). This is also the case at CHM (see Table 10). Parents of students who win the lottery of admission at CHM think more highly of their school than control parents, 0.97 points above the control group mean of 2.9 on a Likert-type scale where 1 is "Bad" and 4 is "Excellent" (Table 10, Column 2). CHM parents think their child's school is more demanding academically than parents whose 35 kids attend public schools (coefficient of 0.70). There does not appear to be a different in perception among parents with regard to how strict (discipline-wise) their child's school is--not surprising since public schools in Mexico are notoriously strict when it comes to enforcing order and discipline (Ochoa Cervantes and Diez-Martinez 2013). Table 10. Effects of Private School Attendance on Parent Perceptions Effects of Annualized Control Private Achievement Mean School (ITT) Gains (IV model) (1) (2) (3) My school is very demanding academically (1-4) 2.7 0.70*** 0.44*** (0.188) (0.095) I think my school is (1=Bad, 4=Excellent) 2.9 0.97*** 0.62*** (0.152) (0.071) My school is very strict (discipline) (1-4) 3.8 0.71 0.45 (1.192) (0.690) On a scale of 1-10, my school is a…. 8.2 1.51*** 0.96*** (0.275) (0.141) On a scale of 1-10, the probability my child will complete high school is .... 9.2 0.35* 0.22* (0.207) (0.116) On a scale of 1-10, the probability my child will complete college is .... 8.6 1.03*** 0.65*** (0.276) (0.150) Sample Cohort 2 Parents Only Demographic and baseline controls/1 Yes Yes N 83 83 *** p<0.01, ** p<0.05, * p<0.1. Note: All regressions include lottery strata fixed effects (gender and cohort). Robust standard errors are reported in parentheses. /1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing values imputed, and missing dummies included for imputed cases. CHM parents also tend to have higher expectations for their child's education than parents in the control group. On a scale of 1-10, control group parents set the probability of their child completing high school at 9.2. CHM parents set this probability 0.35 points higher, although the difference is statistically significant only at the 10 percent level. 36 Parents who win the lottery to attend CHM have significantly higher expectations than control group parents that their child will complete college, 1.03 points higher (on a scale of 1-10) over the control group mean of 8.6. 7. Cost Analysis Table 11 shows operational costs of CHM relative to full- and half-day public schools in Mexico City. The monthly average cost of educating a student at CHM in US dollars is $240, or about US$2,400 per year. The estimated monthly cost of educating a student at a control group public school is US$120 or about US$1,200 per academic year. Table 11. Operating Costs of Private School and Control Public Schools (US $) /1 Monthly Average Cost per Student Grades 1-6 (in $US) Enrollment Payroll Other Total Type of School Teachers Total Costs Cost Treatment CHM 337 31.7 111.9 128.4 240.3 Control All Public Schools 361 43.3 69.0 53.7 122.7 Half-Day 377 36.5 55.9 26.7 82.6 Full-Day (no lunch) 323 46.5 75.9 26.6 102.6 Full-Day (lunch) 392 56.9 93.4 63.0 156.4 Note: Exchange rate is 18 pesos per dollar. Costs are at current prices. /1The control group includes 83 public schools with available data. Source for CHM staff interviews and documentation provided by institution. Source for public schools is AFSEDF, the school census for 2016-17 (Forma 911), and CEMABE 2013-14. See text for more details. Per-student costs are much higher at full-day than at half-day public schools. We estimate average per-student monthly costs to be US$156 (US$1,560 annually) at full- day schools offering lunch. Per-student average monthly cost at half-day schools is $82 or about $800 per year. These estimated costs are in line with published sources. The national evaluation institute in Mexico, INEE, calculates an average annual (10 month) per-student cost of $900 for elementary public schools in the country (INEE, 2017). Teacher salaries do not drive the higher cost of CHM. In fact, the average teacher 37 payroll at CHM is about 27 percent lower than the teacher bill for control group public schools. At CHM, the vast majority of the per-student cost pays for infrastructure, equipment, materials and services offered at the school (including rent, utilities, extracurricular activities, books, school uniforms, lunch, breakfast, etc.). Thirty-three percent covers non-teaching staff (including administrators, psychologists, and support staff). Only 13 percent of the monthly per-student cost goes to pay teaching salaries. In the case of control public schools, 35 percent of the monthly average cost goes toward teacher salaries, 21 percent pays for non-teaching staff, and 44 percent covers infrastructure, utilities, equipment, materials and other services. Table A3 in the Appendix shows household expenses for CHM and control schools. Parents in CHM spend less out-of-pocket than parents in public schools in fees, school uniforms (which are required for public schools and CHM), textbooks, school supplies, and nutrition. The only category in which CHM parents spend significantly more money than public school parents is in transportation to and from school. 8. Conclusion In low-income countries, private schools are perceived as superior alternatives to the public sector. The evidence suggests private schools can deliver high-quality education, at a fraction of the cost of public schools, in low-income countries (Patrinos, Barrera-Osorio and Guaqueta 2009; Muralidharan and Sundararaman 2015; Andrabi, Das and Khwaja 2008). We know less about the advantages private schools offer in middle- income countries to students who in some cases have a plethora of public school options. We use a lottery-based admissions process for one comprehensive private school model in Mexico City to test the claim that private schools benefit low-income children. 38 We find that children living in urban poverty in Mexico City perform better on academic tests of early grade literacy and mathematics when they attend a full-day, resource-rich private school as opposed to traditional public schools. Results are robust to various empirical strategies, including randomization inference. Parents of children in private school express higher levels of satisfaction with the school and are more likely to expect that their child will attend college than parents of children in public school. The private school model that CHM provides benefits low-income children the most--perhaps not surprising given that this is the intended target population of CHM. The costs of providing this education are significant: The school costs almost double what public schools in Mexico City cost, on a per pupil basis. This suggests private schools can be effective, but costly alternatives to improve the academic performance of vulnerable children in middle-income developing countries, such as Mexico. A test of whether these results hold if study participants are re-weighted to resemble the average household in Mexico City did not replicate these results. On the contrary, it suggests that private schools such as CHM may not provide any advantages to better-off children (or less poor children) above what traditional public schools provide. This study aims to fill an important gap in the literature examining private school impact on early-grade outcomes. One limitation of the study is that it focuses on early- grade basic literacy and mathematics skills. Although these foundational skills predict future academic outcomes (Duncan et al. 2007), they may not be able to capture the full range of benefits offered by a comprehensive school like CHM. Efforts to prepare children for college are likely to bear fruition in high school, one of the riskiest transitions for students in Latin America (Busso, Bassi and Muñoz 2013). Following this 39 sample of students over a longer period of time and collecting data on cognitive and non- cognitive outcomes, as well as on additional mediating factors that are policy relevant would greatly add to our understanding of whether and how private schools can benefit low-income populations in the region. 40 REFERENCES Andrabi, T., Das, J., and Khwaja, A. I. (2006). A dime a day: The possibilities and limits of private schooling in Pakistan. The World Bank. Andrews, I., and Oster, E. (2018). Weighting for External Validity (No. w23826). National Bureau of Economic Research. Update: January, 2018. Angrist, J. D., Dynarski, S. M., Kane, T. J., Pathak, P. A., and Walters, C. R. (2012). Who benefits from KIPP? Journal of Policy Analysis and Management, 31(4), 837-860. Angrist, J., Bettinger, E., and Kremer, M. (2006). Long-term educational consequences of secondary school vouchers: Evidence from administrative records in Colombia. American Economic Review, 96(3), 847-862. Athey, S. and Imbens, G.W., 2017. The econometrics of randomized experiments. Handbook of Economic Field Experiments, 1, pp.73-140. Barrera-Osorio, F., Blakeslee, D. S., Hoover, M., Linden, L. L., Raju, D., and Ryan, S. (2017). Delivering Education to the Underserved through a Public-Private Partnership Program in Pakistan. Policy Research Working Paper No. 8177. World Bank. Barrios, J. J. (2014) Informe de la prueba de confiabilidad aplicada a los instrumentos EGRA y EGMA aplicados a México. CIASES, Nicaragua. Bravo, D., S. Mukhopadhyay, and P. E. Todd (2010). "Effects of school reform on education and labor market performance: Evidence from Chile's universal voucher system." Quantitative Economics 1, no. 1 : 47-95. Busso, M., Bassi, M. and Muñoz, J.S. (2013). Is the Glass Half Empty or Half Full? School Enrollment, Graduation, and Dropout Rates in Latin America. IDB Working Paper. Bettinger, E., M. Kremer, and J.E. Saavedra (2009). Education Vouchers in Colombia. Emerging Evidence on Vouchers and Faith-Based Providers in Education:71. Bettinger, E., M. Kremer, and J.E. Saavedra (2010). Are Educational Vouchers Only Redistributive? The Economic Journal 120 (546): F204-F228. Baker, R., Bettinger, E., Jacob, B., and Marinescu, I. (2018). The effect of labor market information on community college students’ major choice. Economics of Education Review. Chabrier, J., Cohodes, S., and Oreopoulos, P. (2016). What can we learn from charter school lotteries? Journal of Economic Perspectives, 30(3), 57-84. Cheng, A., and Peterson, P. E. (2017) How Satisfied are Parents with Their Children’s Schools? New evidence from a U.S. Department of Education survey. Education Next, Spring 2017. Vol 17, no. 2. Cooper, H., Hedges, L.V. and Valentine, J.C. eds. (2009). The handbook of research synthesis and meta-analysis. Russell Sage Foundation. Cunningham, A. E., and Stanovich, K. E. (1997). Early reading acquisition and its relation to reading experience and ability 10 years later. Developmental Psychology, 33(6), 934. de Chaisemartin, C. and Behaghel, L. (2015). Next please! A new definition of the treatment and control groups for randomizations with waiting lists. Working Paper. 41 Deke, J., Sama-Miller, E., and Hershey, A. (2015). Addressing Attrition Bias in Randomized Controlled Trials: Considerations for Systematic Evidence Reviews.” Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2015. Dhaliwal, I., E. Duflo, R. Glennerster, and C. Tulloch. (2011). Comparative Cost- Effectiveness Analysis to Inform Policy in Developing Countries: A General Framework with Applications for Education. Abdul Latif Jameel Poverty Action Lab (J-PAL), MIT. Dobbie, W., and Fryer Jr, R. G. (2011). Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem Children's Zone. American Economic Journal: Applied Economics, 3(3), 158-87. Domina, T. (2005). Leveling the home advantage: Assessing the effectiveness of parental involvement in elementary school. Sociology of Education, 78 (3), 233-249. Duflo, E., R. Glennerster, and M. Kremer. (2007). Using randomization in development economics research: A toolkit. Handbook of development economics 4:3895- 3962. Duncan, G.J., Dowsett, C.J., Claessens, A., Magnuson, K., Huston, A.C., Klebanov, P., Pagani, L.S., Feinstein, L., Engel, M., Brooks-Gunn, J. and Sexton, H. (2007). School readiness and later achievement. Developmental Psychology, 43(6), p.1428. Epple, D., Romano, R.E. and Urquiola, M. (2017). School vouchers: A survey of the economics literature. Journal of Economic Literature, 55(2), pp.441-92. Gleason, P., Clark, M., Tuttle, C., and Dwoyer, E. (2010). The Evaluation of Charter School Impacts (NCEE 2010-4029). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Glewwe, P.W., E.A. Hanushek, S.D. Humpage, and R. Ravina. (2011). School Resources and Educational Outcomes in Developing Countries: A Review of the Literature from 1990 to 2010. National Bureau of Economic Research. Gove, A., and A. Wetterberg. (2011). The Early Grade Reading Assessment: An Introduction. The Early Grade Reading Assessment:1. Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25-46. Hainmueller, J., and Xu, Y. (2013). Ebalance: A Stata package for entropy balancing. Hecock, R. D. (2014). Democratization, Education Reform, and the Mexican Teachers' Union. Latin American Research Review, 49(1), 62-82. Hernandez, D. J. (2011). Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation. Annie E. Casey Foundation. INEE (2015). Resultados PLANEA. http://www.inee.edu.mx/index.php/planea/bases-de- datos-planea INEE (2017). Panorama Educativo 2016. Mexico D.F. INEE (2018). PLANEA. Resultados nacionales 2018. Educación Básica. Noviembre, 2018. INEE: Cd. de México. Available at: 42 http://planea.sep.gob.mx/content/general/docs/2018/RESULTADOS_NACIONA LES_PLANEA2018_INEE.pdf Kattan, R. B., and Székely, M. (2014). Dropout in Upper Secondary Education in Mexico: Patterns, Consequences and Possible Causes. The World Bank. Katz, L. F., Kling, J. R., and Liebman, J. B. (2001). Moving to opportunity in Boston: Early results of a randomized mobility experiment. The Quarterly Journal of Economics, 116(2), 607-654. Levin, H.M., and P.J. McEwan. (2000). Cost-effectiveness analysis: Methods and applications. Vol. 4: Sage Publications, Incorporated. Marschall, M. (2006). Parent involvement and educational outcomes for Latino students. Review of Policy Research, 23(5), 1053-1076. Mejia, J. and Pflepsen, A. (2012). Administering EGRA and EGMA: Rules and tips for ensuring quality administration and results. Retrieved from https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=406 Muralidharan, K., & Sundararaman, V. (2015). The aggregate effect of school choice: Evidence from a two-stage experiment in India. The Quarterly Journal of Economics, 130(3), 1011-1066. OECD, 2015. Mexico Country Note. Patrinos, H. A., Barrera-Osorio, F. and Guáqueta, J. (2009). The role and impact of public-private partnerships in education: World Bank Publications. Patrinos, H. A., and C. Sakellariou. "Quality of Schooling, Returns to Schooling and the 1981 Vouchers Reform in Chile." World Development 39, no. 12 (2011): 2245- 2256. Rauch, J. E. and Evans, P. B. (2000) “Bureaucratic Structure and Bureaucratic Performance in Less Developed Countries.” Journal of Public Economics, January, 75(1), pp. 49–71. Razo, A. (2016). Evaluation of a low-cost private school model in Mexico: Qualitative Report. Working Paper. December, 2016. RTI International. (2009). Manual para la evaluación inicial de la lectura en niños de educación primaria [Early Grade Reading Assessment toolkit, Spanish adaptation]. Adaptation by J. E. Jimenez. Prepared for USAID. Research Triangle Park, North Carolina: RTI International. Santibañez, L., and Jarillo, B. (2008). Muscle, Timing and Priorities: Teacher Unions and Education Quality in Mexico. Well-Being and Social Policy, 3 (2), pp. 21-40. Santibañez, L., O’Donoghue, J. L. & Abreu, R. (2014). School based management effects: Resources or governance change? Evidence from Mexico. Economics of Education Review, 39, 97-109. USAID (2012). EGRA EGMA: Tools to measure and improve student learning. Presentation made December 15th for the World Bank. Watts, T. W., Duncan, G. J., Siegler, R. S., & Davis-Kean, P. E. (2014). What’s past is prologue: Relations between early mathematics knowledge and high school achievement. Educational Researcher, 43(7), 352-360. World Bank (2003). Making Services Work for Poor People: World Bank and Oxford University Press. World Bank (2018). World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank 43 Yamamoto, Y., and Holloway, S. D. (2010). Parental expectations and children's academic performance in sociocultural context. Educational Psychology Review, 22(3), 189-214. Young, A. (2017). Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results. Working Paper. August 2017. Young, A. (2018). Consistency without Inference: Instrumental Variables in Practical Application. Working Paper. June 2018. 44 APPENDIX Table A1. Use of Time at CHM and Public Schools Jornada Tiempo Traditional CHM Ampliada Completo (half-day) (full-day) (full-day) Daily schedule 7:00-17:00 8:00-12:30* 8:00-14:30 8:00-16:00 Weekly Hours Spent on Various Activities Academic Activities (Total) 29 18 26 26 Spanish 10 9 11 11 English as a second language 7 0 3 3 Math 8 6 8 8 Science 2 2 3 3 Civics and Ethics 1 1 1 1 Computers 1 0 0 0 Art education (Total) 2 1 2 2 Physical Education (Total) 2 1 2 2 Other activities (Total) 11 0 0 0 Character/Leadership 1 0 0 0 Hygiene/good habits 10 0 0 0 Breakfast/Lunch (Total) 3.5 0 0 7.5 Recess (Total) 2.5 2.5 2.5 2.5 Total Hours per Week 50 22.5 32.5 40 * Morning shift. Afternoon shift meets from 14:00 to 18:30 hrs. 45 Table A2. Descriptive Statistic Replication - Re-Weighted Sample All Weights (all HHS) Treated units: 221 total of weights: 221 Control units: 241 total of weights: 221 Before: without weighting Treatment - HH Data Control - CHM Data Mean Variance Skewness Mean Variance Skewness rents 0.249 0.188 1.162 0.421 0.240 0.324 bothpresent 0.765 0.181 -1.248 0.506 0.246 -0.025 ing_fijo 7870 38100000 1.338 3515 6593485 1.267 mother_pri~s 0.217 0.171 1.372 0.156 0.130 1.912 kitchen 0.941 0.056 -3.750 0.633 0.229 -0.558 moth_educ_~m 4.348 1.028 -0.082 4.186 0.517 -0.023 pub_electr~y 1.000 0.000 . 0.797 0.159 -1.494 landphone 0.421 0.245 0.321 0.350 0.224 0.634 After: eweight_0 as the weighting variable Treatment - HH Data Control - CHM Data Mean Variance Skewness Mean Variance Skewness rents 0.249 0.188 1.162 0.249 0.188 1.163 bothpresent 0.765 0.181 -1.248 0.764 0.181 -1.244 3810000 ing_fijo 7870 38100000 1.338 7861 0 0.390 mother_pri~s 0.217 0.171 1.372 0.218 0.171 1.367 kitchen 0.941 0.056 -3.750 0.941 0.056 -3.734 moth_educ_~m 4.348 1.028 -0.082 4.347 1.028 0.313 pub_electr~y 1.000 0.000 . 0.999 0.001 -27.120 landphone 0.421 0.245 0.321 0.420 0.245 0.323 Low-income Weights Treated units: 112 total of weights: 112 Control units: 242 total of weights: 112 Before: without weighting Treatment - HH Data Control - CHM Data Mean Variance Skewness Mean Variance Skewness rents 0.277 0.202 0.998 0.421 0.240 0.324 bothpresent 0.768 0.180 -1.269 0.506 0.246 -0.025 ing_fijo 3499 4774546 -0.461 3515 6593485 1.267 mother_pri~s 0.259 0.194 1.101 0.156 0.130 1.912 kitchen 0.929 0.067 -3.328 0.633 0.229 -0.558 moth_educ_~m 4.116 0.914 0.202 4.186 0.517 -0.023 pub_electr~y 1.000 0.000 . 0.797 0.159 -1.494 landphone 0.304 0.213 0.854 0.350 0.224 0.634 46 After: eweight_low as the weighting variable Treatment - HH Data Control - CHM Data Mean Variance Skewness Mean Variance Skewness rents 0.277 0.202 0.998 0.277 0.201 0.998 bothpresent 0.768 0.180 -1.269 0.767 0.180 -1.262 ing_fijo 3499 4774546 -0.461 3496 4776145 0.387 mother_pri~s 0.259 0.194 1.101 0.260 0.193 1.098 kitchen 0.929 0.067 -3.328 0.928 0.067 -3.305 moth_educ_~m 4.116 0.914 0.202 4.114 0.916 0.764 pub_electr~y 1.000 0.000 . 0.999 0.001 -34.440 landphone 0.304 0.213 0.854 0.304 0.212 0.854 Note: These are estimated using the entropy weights method and the ebalance command in STATA per Hainmuller & Xu (2013). See text for more details. Table A3. Parent Out-of-Pocket Expenses, (US $) Public (Control) CHM Schools Lump-sum costs (one time) At the beginning of the school year, how much did you pay in matriculation fees? $0 $4 For this school year, how much did you pay for school uniforms? $0 $37 At the beginning of the school year, how much did you pay for textbooks and school materials? $0 $21 For this school year, how much did you pay for other school expenses? $10 $16 Recurrent costs Lunch and/or breakfast at school? $1 $142 Transportation to and from school? $217 $153 School materials, supplies, copies, etc.? $20 $60 School tuition, and/or fees? $75 $84 Total Per-Student Average Out-of-Pocket Expenses (per year) $324 $519 Notes: These are responses for the parents of Cohort 2014 students. Survey was administered in June of 2016 (end of the school year). Parents were asked to refer to the study sample child only, when responding questions. Parent responses were coded in the column (CHM or Public School) corresponding to their actual child's enrollment (not lottery-status). This was done in order to get accurate estimates of average costs in CHM vs. public schools. As explained in the text, the vast majority (94%) of students in our sample complied with their assigned lottery status, so the numbers here accurately reflect what parents in treatment/control groups were paying out of pocket. All expenses were converted to US$ using an exchange rate of 20 Mexican Pesos per US$. Monthly expenses such as transportation, lunch, school materials, tuition and fees, were converted to yearly costs using a 10-month academic school year. Other school expenses include: school events and festivities, field trips, school carnival, school pictures, extended day fees, and music lesson fees. 47