Technical Assessment ARGENTINE REPUBLIC: IMPROVING INCLUSION IN SECONDARY AND HIGHER EDUCATION PROGRAM (P168911) The World Bank April 26, 2019 Strategic Relevance 1. Rationale for Bank Engagement and Bank’s value added. There is a strategic rationale for the Bank to engage in the proposed Operation, as it is consistent with the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by supporting programs targeted towards students at risk of dropout and repetition, who overwhelmingly come from vulnerable backgrounds. The Operation is aligned with the social spending floor agreed with the International Monetary Fund (IMF), as well as with the commitment to ensuring sustainability and improved efficiency of priority education programs. Additionally, the Bank has a history of engagement with Argentina, through which it has acquired in-depth knowledge of both the education sector and the federal nature of the country. Given that this federal nature could present bottlenecks to implementation due to varying political will and implementation capacity among provinces, the Bank is well-positioned to rapidly respond to the Borrower’s request for support to programs that are carried out (or expected to be scaled up) nationwide by drawing on and incorporating lessons learned from previous and ongoing operations into their design. Furthermore, the Bank’s vast experience on improving information management systems, evaluation, and decision-making in the sector can be a great value added in the form of technical assistance, as it can draw from international experience and best practices to improve government systems. 2. No additional partners are expected. However, the Inter-American Development Bank (IDB) will partially support the activities under Results Area 1 until June 2019, through its project Programa de Apoyo a la Equidad y Efectividad del Sistema de Protección Social en Argentina (AR-L1302). The objectives of the IDB support are to fund PROGRESAR scholarships and contribute to the monitoring and information system of the program, to support (i) the strengthening of academic trajectories of students supported by the PROGRESAR scholarships; (ii) a more efficient incorporation of former AUH beneficiaries to the PROGRESAR Scholarships system; and (iii) to contribute to early dropout alert systems. The total amount of support of the IDB component related to PROGRESAR Scholarships is US$320 million and the support is expected to end before the World Bank support for PROGRESAR Scholarships becomes effective (June 2019). The objectives of the IDB operation are aligned with the development objectives of the World Bank support, and there would be dialogue and articulation between both institutions. Rationale for Public Sector Provision/Financing 3. Three factors provide a strong rationale for public sector financing of the interventions supported by the Program. First, individual acquisition of education generates positive externalities on society by making individuals more engaged and responsible citizens and increasing the overall level of productivity and growth in the economy. To the extent that individuals do not take these benefits into account when making education investment decisions, public sector investments can lead to efficiency gains. Second, there is imperfect information that makes individuals unaware of the importance of education in future labor opportunities and, even if they are aware, financial constraints might prevent them from financing these investments in the credit market. Third, the education system has the potential to promote equality of opportunities. Since dropout through secondary and higher education is particularly relevant for students from disadvantaged households, any policy aimed at reducing financial and non-financial barriers at these educational levels will benefit vulnerable groups and is therefore desirable from an equity perspective. Technical Soundness 4. Technical Soundness. This section presents a review of the available literature to provide further evidence that the Operation’s design is technically sound, based on international best practices, and in line with evidence-based interventions. 5. Financial support (Conditional Cash Transfers -CCTs, merit- and need-based scholarships) for school completion: a large body of evidence finds that scholarship programs almost always improve school participation outcomes (enrollment, attendance, completion and dropout) and can, in some cases, improve student learning. The 2018 improvements and modifications to PROGRESAR Scholarships, which promote more inclusion, merit-based benefits, and strategic careers in higher education, have also been identified in the literature as effective interventions for education outcomes. Evidence suggests that interventions that increase the benefits of attending school (such as scholarships) increase time in school,1 while those that increase the benefits of higher effort and better academic performance (such as merit- based scholarships) improve learning outcomes. The significantly positive effects on school participation are found in Brazil,2 Colombia,3 Cambodia,4 China,5 Honduras,6 Malawi,7 Mexico,8 Nicaragua,9 and Pakistan.10 In Malawi and Nicaragua, estimates calculated several years after the end of the program show that scholarship beneficiaries had higher cognitive skills, as measured by test scores. In contrast, in other countries (Cambodia, China and Colombia), there is no evidence of impacts on learning. Only one of the interventions to increase demand for education in developing countries has consistently worked to increase test scores across multiple contexts: merit-based scholarships.11 Overall, the findings indicate that merit-based scholarships can effectively increase test scores and attendance, while need-based scholarships tend to affect mainly enrollment and attendance, but not test scores. However, there can be an equity issue, as merit-based scholarships typically reach students who are already better off, which highlights the importance of combining this merit-based scholarship with strong targeting mechanisms. Evidence on the effect of merit-based scholarships for higher education students is also promising, although the results are more mixed.12 In particular, a few rigorous recent evaluations have shown large impacts. For instance, Angrist et al (2017) show that scholarships significantly boosted college enrollment and persistence. Four years after award receipt, randomly-selected scholarship winners were 13 percentage points more likely to be enrolled in college. 6. Non-financial support is also critical to strengthen the transition to higher education. This is particularly relevant for disadvantaged students who are the first generation in their family to have access to higher education and may lack support to make informed decisions and basic skills to succeed in higher education. Information-based programs that provide key data to students and parents have been shown to improve career choices in higher education. A recent randomized control trial (RCT) in Chile shows that providing information to future higher education students, through an online platform, on the potential earnings (and fee costs) corresponding to different enrollment choices, helped 1 Glewwe & Muralidharan (2015,), Damon et al (2018) 2 De Janvry et al (2012) 3 Barrera-Osorio et al (2011) 4 Barrera-Osorio & Filmer (2013) 5 Mo et al (2013) 6 Galiani and McEwan (2013) 7 Baird et al (2011) 8 Schultz (2004;); Behrman et al, 2009, 2011 9 Braham et al (2013) 10 Chaudhury and Parajuli (2010) 11 Kremer, Miguel and Thornton (2009), Friedman et al (2011), Blimpo (2014) 12 Dynarski (2008), Castelman (2014), Scott-Clayton (2011b), Scott-Clayton & Zafar (2016), Bettinger et al (2016); DesJardins & McCall (2014), DesJardin et al, (2010), Sjoquist & Winters (2012, 2015) particularly low-socioeconomic students enroll in degrees with higher returns.13 The introduction of additional incentives for “strategic� careers (most high in demand) aimed at guiding the decision of new students could reinforce the effects of providing this information. Studies that analyze the role of financial aid in supporting strategic career (STEM) attainment in the U.S. find that eligibility for need-based financial aid increased STEM credit completion by 20 to 35 percent among academically-ready students in a large, public higher education system.14 Evidence from basic education also highlights the potential of information-based interventions. An RCT in the Dominican Republic finds that providing students with information on earnings differences by education reduced dropout by 7 percent in the subsequent year, and increased school completion by 0.2 years.15 Similarly, informing fourth grade students and their parents of earning differences by education levels in Madagascar increased average attendance by 3.5 percentage points.16 7. Socioemotional support and low-cost behavioral interventions are also essential to support students struggling with the more complex curriculum and environment of higher education. During the transition to a new education level, students can face frequent social setbacks and feelings of anxiety and isolation. Interventions that provide self-administered short programs on motivation and belonging have shown that college students benefit when they understand that challenges in the transition to college are common and improvable, and thus, that early struggles need not portend permanent lack of belonging or potential.17 In addition, a RCT showed that providing services that helped students set goals and acquire financial aid between their transition from high school to college increased college enrollment by 3 percentage points (even more for low-income students).18 Recent experimental evidence in the U.S. found that brief, one-time messages to students increased recurring tutoring attendance, leading to relevant changes in students’ study habits.19 8. Standardized student assessments are a key ingredient to shift the focus to learning and support students’ trajectories. The benefits of education—cultural, economic, and social—accrue to society only when learning occurs (OECD, 2010). For example, an increase of one standard deviation in scores on international assessments of reading and mathematics achievement levels has been linked to a 2 percent increase in annual growth rates of Gross Domestic Product (GDP) per capita (Hanushek and Woessmann, 2007, 2009). For governments to learn what policies worked to improve learning and to avoid “flying blind,� they need credible data (World Bank, 2018). The Operation would support the strengthening of the student evaluation system. In a recent paper, Bergbauer, Hanushek and Woessmann (2018) find that the sole fact that countries adopt a standardized student assessment and make results of the evaluations public is associated with improvements in student achievement. As part of the Operation, the SEE would improve the report on the results of the assessment that each school receives to provide principals with information on what students mastered and what competencies they need to improve. A recent RCT in La Rioja, Argentina found that providing schools with such information (together with a short training) improved student achievement between 0.28 and 0.38 of a standard deviation, which represents around one additional academic year of learning (de Hoyos et al, 2018). 9. The assessment of teachers’ professional development is the second cornerstone of the 13 Neilson et al (2018) 14 Castelman et al (2018) 15 Jensen (2010) 16 Nguyen (2008) 17 Yaeger et al (2016;); Walton & Cohen (2011) 18 Castleman & Page (2015) 19 Pugach & Wilson (2018.) education evaluation system. Improving teacher pre-service education is crucial to improve student learning. Teachers are the most important school-based determinant of student learning (Hanushek and Rivkin, 2010; World Bank, 2018). Research in Peru and Ecuador shows that teacher subject content knowledge (Metzler and Woessmann, 2012) and pedagogical practices (Araujo et al, 2016) in the classroom are among the most important factors to explain teachers’ value added. The Operation would strengthen the evaluation system of soon-to-be teachers in Argentina in all these dimensions, by (a) consolidating the current content and pedagogic knowledge assessment (ENSEÑAR); and (b) complementing it with an assessment of what the teachers do in the classroom. This information would be used to provide feedback to teacher training institutes to improve their performance, to develop a suite of tools for self-evaluation, and to inform the design of tools and strategies for external evaluation initiatives in the future. Finally, as part of the technical assistance, the Operation would develop and pilot a strategy to integrate the use of student assessments and observations of what teachers do in the classroom with the systems of professional development principals and teachers. This is in line with current research on best practices in professional development (Popova, Evans, & Arancibia, 2016; Yoshikawa et al., 2015; Bambrick-Santoyo, 2012; Fryer, 2017). 10. The current design of ASISTIRÉ is aligned with best international practices, and the planned complementary interventions would reinforce its effectiveness. Extensive research has shown that it is more cost-effective to act early to prevent dropout than to try to reengage out-of-school youth in the educational system. EWS that rely on administrative data, even when they are based on basic algorithms, have proven to have high predictive power of dropout, allowing identification of children at risk and early action.20 A recent RCT in the U.S. showed that an EWS that relies on basic data (chiefly absences, grades, and behavioral problems), coupled with close follow-up of students at risk by the school team (case workers), reduced chronic absence and course failure by almost 20 percent after only one year of implementation.21 The improvements in the program’s design, which include strengthening the evidence- base of key interventions (such as closing skills gaps, teenage pregnancy prevention, and reinforcing the channels for family-school communications) would further boost the effectiveness of the program. At the secondary level, programs targeting key socioemotional skills22 with academic tutoring, as well as other supportive interventions (such as remedial education23 and teenage pregnancy prevention programs) are starting to show substantial effects on dropout for students in at-risk communities. For instance, a series of well-identified studies has shown that programs with a focus on student socioemotional skills (e.g. counseling programs,24 cognitive behavioral therapy, coaching and tutoring) improve mental health and, consequently, improve school engagement and student achievement. Interventions as diverse as life- planning skills, HIV prevention education,25 and vocational training, combined with reproductive health education, have shown promise in reducing adolescent pregnancy and pregnancy-related school dropout.26 Similarly, evidence has also shown that providing information to parents on their children’s absenteeism and grades (through automated text messages, emails or phone calls) improves student outcomes and reduces absenteeism.27 20 Knowles (2015); Adelman et al (2016) 21 Faria et al (2017) 22 Cook et al., 2014, Heller el al (2016), Wang et al (2016). Other studies on effective coaching/tutoring interventions in K –12 education include Dobbie & Fryer (2013), Fryer (2014), Kraft (2015), Kosse et al. (2016), and Ander, Guryan, & Ludwig (2016). 23 Banerjee et al. (2007) 24 Carrell & Carrell (2006) 25 Baird et al (2011), Duflo et al, 2011, 2015; Dupas (2011) 26 Duflo et al (2015), Cristia et al. (2014) 27 Bergman (2015), Rogers & Feller (2018), Berlinski et al (2016), Bennett & Bergman (2018) Assessment of PROGRESAR’s Management and Information System28 11. Description. The PROGRESAR Scholarship Program involves the following steps related to: i) scholarship enrollment; i) application assessment; ii) scholarship approval and award; and iv) payment and accountability. The Program’s enforcement authority is the Ministry of Education, Culture, Science and Technology (Ministerio de Educación, Cultura, Ciencia y Tecnología, MECCyT) through the National Directorate for Educational Scholarships (Dirección Nacional de Becas Educativas, DNB), which interacts with other agencies, including the National Administration of Social Security (Administración Nacional de la Seguridad Social, ANSES), which was the enforcement authority until early 2018, and with which the MECCyT signed a cooperation agreement for participation at certain stages. Below is a detailed description of each process and sub-process. 12. Scholarship Enrollment. The enrollment period opens annually from February 1st to March 31st. The application process begins when a potential recipient enrolls as a program applicant. Two different enrollment mechanisms exist, depending on whether the application is submitted for the Basic Education scholarship, for the Technical and Professional Education scholarship, or for the Higher Education scholarship. For Basic Education Scholarships, most applications are submitted in person at the Comprehensive Service Units (UDAI) of ANSES, but they can also be filed before the Mobile Service Units (UDAM) of the same agency, or online. To complete the application at the UDAI, the applicant must first make an appointment either online or by phone (line 130). The application process requires filling in a form, which can be downloaded previously or be handed to the applicant in person. Together with the form, the applicant must submit a regular student certificate duly signed and stamped by the educational institution where he/she attends primary or elementary school. The data is manually entered into an ANSES platform. The ANSES staff checks the applicant’s personal information, identity card, identity information in the People’s Data Administration System (ADP), and that the stamp on the student certificate matches the authorized Single Institution Code (CUE). Once these checks are made, the process is approved. For more details on this process, see Figure A3.1. For enrollment in Higher Education scholarships, enrollment is through the MECCyT’s online platform, where the applicant must log in with a user name and password. This user name will be valid throughout the scholarship award process, since the Program will use this platform to communicate with the applicant (for instance, to notify the applicant of the need for additional information, the granting or denial of the scholarship, and the ways to file claims). By entering the user name and password, the applicant can fill in a form with his/her personal data and information on the area and institution where he/she is enrolled and/or studying. For more details on this process, see Figure A3.2. 13. Application Assessment. The application assessment process ensures the fulfillment of the terms and conditions set forth under the executive order that initiated the program (No. 90/2018). For this purpose, the MECCyT requests certain information from ANSES and Universities and Tertiary Institutions to determine the eligibility of the enrolled applicants. The socioeconomic assessment process aims to verify the applicant’s identity and validate personal and family economic status. At this point, the decision is made as to whether the applicant falls within the eligibility criteria established under the executive order that created the program (No. 90/2018). The socioeconomic assessment is performed by ANSES, taking into account the income level reported by the applicant and his/her family. The academic requirement verification is as follows for the different scholarships. For Basic Education scholarships, the regular student certificate submitted by the applicant at the enrollment stage is deemed as sufficient. For 28Based on an assessment carried out by the IDB in the context of the Programa de Apoyo a la Equidad y Efectividad del Sistema de Protección Social en Argentina (AR-L1302) and information provided by MECCyT. this reason, no additional step is taken at this point. The information requested from Higher Education scholarships applicants through the enrollment form allows grouping of applicants by educational institution. In order to complete the applicants’ academic assessment for scholarships, MECCyT sends each university a TXT file for them to certify the applicant’s current career, enrollment year and progress. The tertiary institutions can access the information posted on the MECCyT’s web platform and certify the academic status of their applicants (year of entry, career and approved subjects). Finally, in the case of Teacher Training scholarships, the applicants must pass an exam administered by the SEE. Those who were granted this scholarship in the past must have passed 20 percent of the curricula subjects each year covered by the scholarship. 14. Scholarship approval and award. MECCyT receives information on the socioeconomic and academic assessments and, based on this information, initiates an information consolidation step to conclude the process. At this point, the decision is made to grant or deny the scholarship or to identify inconsistencies for later clarification. The scholarship amount is determined through a ministerial resolution and is determined based on the incentives scheme of the scholarships. 15. Payment and Accountability. The funds from the scholarships are paid throughout 10 months (from March to December) and are collected monthly in arrears. The recipients for Basic and for the first year of higher education receive 80 percent of the scholarship amount. The remaining 20 percent is withheld and paid the following year, once the requirements established under executive order No. 90/2018 are fulfilled. 16. Pre-settlement. MECCyT keeps the recipient database at the NB. Every month, it performs the pre-settlement process, which generates the list of scholars for the respective payment period, creating a TXT file that contains the personal data of the scholar, the payment period, and the items to settle. The pre-settlement process is sent to ANSES via an internal note (no amount indicated) and the TXT format file. Once this information is received, ANSES checks enrollment validity, existence of labor ID code (CUIL), and vital status of the individual according to the data contained in the People’s Data Administration system (ADP). ANSES updates its own file of PROGRESAR scholars (called Z2- PROGRESAR 2) based on the information submitted by MECCyT. Once these checks are completed, ANSES creates an issuance and payment order file, and assigns a payment terminal for scholars who have not yet been assigned one. Assignment is performed based on the individual’s address recorded in ANSES. The possible payment terminal options include: Banks (Nación or others), Correo Argentino, Rural offices or PIM (virtual wallet from Nación Servicios, via cell phone). Non-bank payment methods are available when scholars do not have easy access to a bank branch due to geographical location. ANSES sends the file to the scholar’s Payment area for a preliminary estimation of bank fees and notifies MECCyT of the amount to pay. This includes the amount to pay to the scholars and the applicable fees charged by the bank. 17. Payment. MECCyT then conducts the permanent updates and formalizes the payment order by sending a Note to the General Administration Directorate (DGA). The DGA generates an e-SIDIF file and submits it to the MH to allocate budget and send the funds to the Banco de la Nación Argentina account number 3793/42 (exclusive account for PROGRESAR Scholarships). With a second Note, it sends the updated file to the Fund Payment Department, which is responsible for processing the payment to scholars. Once the funds have been transferred from MECCyT to ANSES, the latter uses an account in the Central Bank of the Republic of Argentina (BCRA), which receives the funds coming from the already mentioned Banco Nación account, and from where it makes the payments to the payment network (mainly banks and Correo Argentino). The monthly payment operation lasts approximately 40 days. In a large number of cases, the funds are credited to the scholars’’ savings accounts on day 1 of the payment operation; however, in some cases, the funds are not directly credited, so the payer institution keeps the funds until the scholar collects them at any time during that period. Likewise, Correo Argentino keeps the funds available for collection during that period of time. 18. Accountability of funds. Once that period is completed, the monthly payment operation is deemed as closed, and both the Banks and Correo Argentino must account for the received funds before the BCRA. For this purpose, they must submit a file stating the paid funds and the funds due for payment together with a note, which will be valid as a sworn statement for 100 percent of the funds sent. Likewise, the funds due for payment must be sent back to the BCRA account that originally made the transfer. At that point, the BCRA checks that the entities have reported the total number of cases they account for, the paid funds, the paid fees, the due amounts, and the unpaid fees. This accountability process before the BCRA can take up to 10 business days (or more in the case of Correo Argentino). The BCRA consolidates the information received by banks, posts the reported accounts into the system, and submits the information to ANSES. ANSES then consolidates the information on the results of the payment process, i.e., it makes a summary of the total funds transferred to the BCRA, how much was paid, how much was not paid and the amount of paid and unpaid fees. Based on this information, an electronic file is created, and a Final Report is sent to the MECCyT, including the summary data for the payment process with a table attached, specifying this information per payer institution and some additional data. Finally, ANSES takes the necessary steps to return the funds by instructing its Finance area to approve the return of funds. Thus, the Accounting area generates a Payment Order for the Banco Nación account 4982/45 (the account selected by the MECCyT for the return of funds). The Treasury area then executes the Payment Order and makes the payment to the MECCyT’s bank account. Figure A3.3 illustrates the scholarship payment process. Figure A3.1. Basic Education Scholarship Enrollment Process Performed in ANSES – MECCyT receives the final information directly from ANSES Figure A3.2. Higher Education Scholarships Enrollment Process Applicant ANSES MECCyT Universities Tertiary institutions Applicant enters Updates Progresar Web personal information Platform and sends TXT into the Progresar with applicant’s details Web Platform for validation Receives information and sends for SIU to Makes socio- update academic economic information assessment Updates academic path by CUIL number in Progresar Web Platform SIU updates data by CUIL number and reports the University Updates TXT and sends results to Updates TXT (only MECCyT those found in the system) and sends to MECCyT Consolidates data and issues Final Analysis Base Updates Web Platform to Notifies and, if communicate the approval, necessary, rejection or request for updates additional information. information or files claim Files claim or updates Yes data No End Figure A.3.3. Payment Process Assessment of the Program Expenditure Framework 19. The scope of the proposed Operation would be to support a subset of Government Programs (GP) from 2019 to 2023, clustered around two results areas under a Program for Results (PforR) Operation (US$300 million) and a Technical Assistance Operation (US$40.1 million) using the Investment Project Financing (IPF) instrument. The total cost of the activities under the Operation is estimated at about US$1.35 billion, of which US$341 million would be financed through a Bank loan. 20. The PforR’s operating and investment budget totals US$1.113 billion, out of which US$300 million (24.73 percent) would be funded by the Bank. The scope of the PforR Program supports two Results Areas selected as priorities by MECCyT to foster access and completion at the secondary and higher education levels and to strengthen the student, teacher and institutional evaluation systems in the education sector. To reach these objectives, MECCyT is implementing two programs, which together conform the PforR Program: (i) the PROGRESAR Scholarship program for 18 to 24-year-olds; and (ii) the APRENDER student learning assessment system and the ENSEÑAR teacher training evaluation for recent graduates. 21. Results Area 1. PROGRESAR Scholarships29 (IBRD: US$290 million, Total: US$1.20 billion). This Results Area would finance the provision of education scholarships to beneficiaries, reimbursing approximately 23 percent of the total program spending over the proposed Operation duration. 22. Results Area 2. Supporting the implementation of the national standardized student learning assessment APRENDER,30 the ENSEÑAR assessment,31 and the evaluation of students applying to scholarships for teacher training (IBRD: US$10 million, Total: US$13 million). 29 PROGRESAR was established in 2014 as subsidy transfer to foster access and completion at all education levels for students aged 18-24 that belonged to the poorest households. The program was managed by ANSES and was conceived as a continuation of the AUH, which covers up to 17-year-olds. 80 percent of the monthly transfer was contingent on age and income, and 20 percent on regular attendance to a public education institution at any level. By 2017, the program reached 1 million beneficiaries, but monitoring of eligibility and attendance was carried out poorly. In 2018, the program was transferred to the MECCyT and was modified from a transfer to a scholarship program - PROGRESAR Scholarships- and significant improvements of the program have been introduced over the last two years. The main objectives of these changes were to align benefits with education outcomes, to promote more inclusion and merit-based benefits, and close the gap between education and labor markets by promoting strategic careers in higher education. 30 APRENDER was established in 2016 to obtain and to disseminate timely and quality information to better understand student learning, via a census-based standardized student learning assessment for primary (6th/7th grade) and secondary students (year 5/6) to : (i) measure trends in the quality of education over time; (ii) use the information to design strategies focused on improving quality; and (iii) share the results with provincial authorities, school principals, and the public to report on performance. The results are reported at the national, provincial and school levels and the SEE has actively adopted several strategies for the widespread dissemination of the results of APRENDER. They have done that both through a password-access online platform that provides each school with a tailor report analyzing the results in comparative perspective, and through an open online platform that allows for basic statistical analysis, disaggregated to the provincial level. 31 ENSEÑAR is an evaluation system for teacher training institutes. In 2017, the SEE launched ENSEÑAR to assess teacher training institutes’ quality through a student content knowledge assessment and a toolkit for self -institutional assessment. The SEE plans to improve the tools to assess the teacher training institutes and generate reports based on the findings from ENSEÑAR, including strategies for improvement to advise teacher training institutes and to advance towards a quality accreditation process based on relevant international experiences. Table A3.1. World Bank Financing – Expenditure Framework US$ Program Component (Bank funds) Jun 2019 2020 2021 2022 Jul 2023 Total Component 1: Support Government Programs 240 244 244 244 122.5 1093 (PforR) (70) (117) (64) (49) (0) (300) Results Area 1: 240 240 240 240 120 1080 PROGRESAR Scholarships (70) (115) (60) (45) (0) (290) Results Area 2: 0 4 4 4 1 13 National Student and Teacher Evaluation Systems (0) (2) (4) (4) (0) (10) Component 2: IPF to Improve Design and 5 10 20 3 2 40.1 Implementation (5) (10) (20) (3) (2) (40.1) 0 0 0 0 0 0 Front-end fee (0.85) (0) (0) (0) (0) (0.85) 245 254 264 247 123 1133 Total (75) (127) (84) (52) (2) (341) 23. The Operation would be carried out through the Ministry of Finance (Ministerio de Hacienda, MH) and MECCyT. The MH, through the Directorate of Programs and Special Projects with Wide Sectoral Scope (Programas y Proyectos Especiales con Enfoque Sectorial Amplio, DPPEESA), would oversee overall implementation of the Operation and would be responsible for all actions related to disbursements and withdrawals. The MECCyT, through the DNB, would be responsible for the implementation of activities related to PROGRESAR Scholarships and for the coordination of the Operation, for fiduciary aspects, for technical and operational decision-making, and for monitoring and evaluation. The Secretariat of Education Evaluation (Secretaría de Evaluación Educativa, SEE) would be responsible for student and teacher evaluations, including activities under APRENDER and ENSEÑAR. The Secretariat for Educational Management (Secretaría de Gestión Educativa, SGE) would be responsible for all activities related to ASISTIRÉ through its national coordination team. Further specifications for these roles and responsibilities are detailed in the Project Operations Manual (POM). Financial sustainability 24. In the context of the IMF Stand-By Arrangement (SBA), the federal government committed to achieving a primary fiscal balance in 2019 and a 1 percent of GDP surplus in 2020. This represents a fiscal effort as high as 3 percentage points of GDP, given the current deficit of 2.7 percent of GDP and the impact of past measures on the 2019 balance (e.g. pensions indexation, tax cuts). Achieving fiscal consolidation targets is key for the Government of Argentina’s (GoA) fiscal program to stabilize and reduce its debt-to- GDP ratio, which reached close to 88 percent in 2018. Fiscal room will be thus limited at the federal level. An additional reduction in public investment and subsidies and a transitory increase in the export taxes are expected to contribute the most to fiscal consolidation. These efforts will be accompanied by decentralization of spending responsibilities to provinces of part of electricity, transport, and water subsidies. A stable macroeconomic environment and a recovery in economic activity, with a positive impact on public revenues, will thus be key to achieving fiscal targets without further expenditure adjustments. 25. In this context, the proposed program would add up to about 0.3 percent (US$250.6 million) of total government expenditures, and most of the Bank funding would not represent additional resources to the existing budgetary lines of the MECCyT, but rather a change in the funding source. In a context of strong fiscal consolidation, there is a risk that a number of programs including the proposed operation could be affected unless there is a commitment by the authorities to prioritize social programs involving cash transfers and ringfence these expenditures. Results Chain 26. Table A3.2 summarizes the Operation’s results chain, and how the Disbursement-Linked Indicators (DLIs) plan to contribute to achieve the expected results. Table A3.2. Results Chain Outcomes/intermediate Planned activities Products Objective outcomes PROGRESAR Scholarships Financial support - scholarships  Finance PROGRESAR scholarships  PROGRESAR Scholarships are  Basic financial needs of  Develop nationwide information system delivered to beneficiaries (by beneficiaries are met (*) to allow schools to certify enrollment and education level)  % of higher education attendance of students of PROGRESAR  Information system for scholarship candidates Basic Education academic certification is whose academic certification Increase implemented (DLI 5) is verified by their academic enrollment in Studies to inform improvements to the institution (DLI 1) basic program  % of applicants that comply education for  Study of the estimation of the PROGRESAR  Study of the program coverage with eligibility requirements vulnerable scholarships’ coverage and prioritization strategy by education level (Basic, youth  Diagnosis of the barriers to completion non-university higher faced by students  National helpdesk for scholars education and University) implemented  % of scholars’ re-enrolment Increase Tackle Information barriers (DLI 2) access to  Provide relevant information to potential  % of first-year scholars of higher beneficiaries in the scholarship higher education that study education for application process strategic careers (DLI 3) the most  National helpdesk to support scholars  Online toolkit for students to  Information system to certify vulnerable support their application, attendance of basic Tackle academic and motivational barriers academic progression and education is used by schools  Produce materials for academic and motivation is implemented (DLI (*) motivation support for students based on 4)  Students have access to the the diagnosis online toolkit  Pilot a mentorship program to provide academic, personal and motivational support to students Support the Student and Teacher Evaluation Systems Implementation and Dissemination of APRENDER  Develop the student assessments, including additional modules to widen its scope  Train jurisdictional personnel on the implementation of the assessment  Produce materials for the assessment, including developing new modules to add  APRENDER is implemented in new measures (e.g. on SES, household schools (DLI 6) education levels, etc.)  % of jurisdictional units that  Analyze assessment outcomes to produce received the training and Strengthening school reports awareness process for the  School principals use the student,  Coordinate strategies and logistics to assessment report to improve teacher and disseminate assessment outcomes pedagogical practices (DLI 7) institutional  Adapt and pilot “Teach� classroom  APRENDER results are evaluation observation tool to Argentina disseminated system  School principals’ access and Improve APRENDER reports use enhanced reports and  Collect international experiences on supplementary pedagogical reporting results from standardized materials assessments  APRENDER coverage increase  Focus groups with school directors on how to improve the APRENDER Summary  New APRENDER Summary report. Report uploaded in the platform  Conduct new analysis on APRENDER for school Directors to access it assessments, optimizing the use of  Studies produced cumulative databases.  Teach Argentina is developed  Analyze additional APRENDER data in and piloted technical schools to characterize the educational dynamics of its students.  Analyze the human capital, infrastructure and other supplies of primary and secondary schools, with the purpose of building clusters. Evaluation of teacher institutes  International workshop on experiences on evaluation and self-evaluation of teacher  Self-evaluation of Institutes is training institutes.  Diagnostic evaluation system for implemented  Set up a committee of experts on teacher training plan (DLR 8.1)  Proposal for an external evaluation and self-evaluation of teacher  ENSEÑAR evaluation is evaluation system agreed training institutes implemented with stakeholders  Develop a suite of tools for self-evaluation  % institutes that receive  Diagnostic evaluation system including the adaptation of ¨Teach¨ customized reports with for teacher training is  Disseminate the suite of tools diagnostic of teacher practices implemented (DLR 8.2)  Implement ENSEÑAR (*)  Develop a proposal for an external evaluation system Discuss the proposal with relevant stakeholders Support the scale-up of ASISTIRÉ Scale-up of ASISTIRÉ in selected vulnerable schools nationwide  Development of key documents to scale- up the program (facilitators profile and  Number of Provinces with screening, training plan, training manual signed agreements for the for schools, diagnostic tools, intervention implementation of ASISTIRÉ in a protocols comprising key evidence-based group of vulnerable schools interventions on socio-emotional skills—  Documents are finalized and  Schools implementing e.g. motivation, study skills, etc.— printed ASISTIRÉ (and meeting teenage pregnancy prevention, and  ASISTIRÉ facilitators are trained minimum requirements) remedial education) in the revamped training  Hire and train ASISTIRÉ facilitators  Schools receive the program  Acquisition of program inputs (including inputs ICT technology, internet connectivity,  School teams are trained in the etc.) new online attendance system  Follow-up interventions Reduce  Train schools in use of the new online targeted to “at risk� students dropout rates attendance system in schools implementing in secondary ASISTIRÉ education in Revamp of ASISTIRÉ ASISTIRÉ  Improve accuracy of EWS schools in the  Analysis of main dropout causes selected  Improve evidence-based content of Provinces facilitators training (focus on key barriers  Implementing the improved  Improvement in students’ such as key socio-emotional skills, version of the EWS attendance, motivation, and teenage pregnancy prevention and  New training modules are performance (*) remedial education) finalized and printed  Develop dropout prevention toolkit  Dropout prevention toolkit (learning materials to support the distributed to schools evidence-based interventions covered in  ASISTIRÉ Familia implemented in the training) a number of provinces  Develop cell phone app ASISTIRÉ Familia to track real-time attendance and other key data in a sample of schools) (*) Expected outcome not measured by the Operation Lessons Learned 27. The Operation’s design is informed by lessons learned from other Bank projects , in particular the Colombia Access and Quality in Higher Education Project - PACES, which provided lessons on strategies to increase the enrollment of students from disadvantaged socioeconomic backgrounds in quality tertiary education programs, as well as on processes and institutional agreements that promote quality assurance systems in tertiary institutions. Other Bank projects including components to support vulnerable students’ trajectories towards higher education helped the design process, including the Ecuador Transformation of the Tertiary Technical and Technological Institutes Project. 28. Regarding improvements towards a quality assurance system, previous experiences indicate that governance issues and the capacity of training institutions to carry out the reforms are crucial elements for success. As sustained in PACES, if institutions do not have autonomy for decision-making or do not have the managerial or financial capacity to carry out the necessary changes for accreditation, quality is unlikely to improve. Building consensus and capacity within tertiary institutions and designing a sensible and organized transition towards a national unified strategy, as learned in the task Design of a Monitoring and Evaluation Strategy for the Higher Education Reform in Chile Project (P157268), is therefore a provision included in the technical assistance for a quality assurance system. 29. The support for M&E systems conducted in the framework of the Analytical Services and Advisory Supporting Evidence-Based Policy Interventions in Argentina, which supported dissemination and knowledge-sharing to school principals in schools in vulnerable contexts in the provinces of Buenos Aires, Salta and La Rioja, and previous collaboration with the SEE in the ongoing PROMER II. It also highlighted good practices in the empowerment of educational institutions’ self-evaluation capacity and provided lessons on political economy towards the construction of a solid national evaluation culture. 30. Previous experience and knowledge generated by projects that promoted access of vulnerable youth to higher education show financial incentives alone cannot support students’ trajectories, unless they are also supported by initiatives that promote resilience as a key socioemotional skill, as well as various kinds of remedial education tools. The Operation builds on extensive previous experience from Bank analytical and advisory tasks building knowledge and containing initiatives to foster socioemotional skills, such as Brazil, Skills and Jobs (P133162), Peru New umbrella for Education, Skills & Employment (P147264), and the ongoing pilot initiative in PBA within the framework of PROMER II. 31. Preparation will also benefit from the support provided by the Bank to the AUH national program through the ongoing Children and Youth Protection, which has helped close coverage gaps for vulnerable populations by supporting the ANSES in obtaining missing information on beneficiaries, thus allowing more children to enter the eligibility screening process. The Project aimed at exploiting new technologies to reduce paperwork to prove compliance, and reduced barriers through early alerts directed to beneficiaries and service providers, innovations aligned with the objectives of Components 1 and 2. Economic Justification 32. This section presents the economic and financial analysis of the Project . The analysis will focus on: i) financing and strengthening PROGRESAR Scholarships, and ii) promoting the expansion of ASISTIRÉ – the most important program for dropout prevention in lower secondary education. These activities concentrate most of the Project resources (92 percent).32 33. The Operation is expected to yield substantial economic and social benefits through its contribution to increased educational attainment, but also through an improvement in educational quality. The Operation is expected to reduce dropout in Secondary and Higher Education –especially among the poorest households – and therefore to increase the number of years of education in the population. From the individual point of view, the Operation would have a direct positive impact on lifetime earnings of the beneficiaries through an increase in their labor productivity. Other individual labor outcomes are also expected to be affected, since higher educational attainment facilitates labor market entry for youth and increases the probability of being employed and having a better job. From a social perspective, the Operation would raise human capital endowment, having a positive impact on growth33 and therefore reducing poverty rates. Moreover, since it is targeted to disadvantaged populations, it would potentially influence the reduction of income inequality. Its impact on the society goes beyond the individual’s outcomes, since a more educated population has better health, less crime, higher democratic participation, more environmental consciousness, etc.34 Since the programs involved also aim to improve educational quality, all these effects are largely augmented.35 34. The analysis will focus on the monetary private benefits deriving from an increase in the number of years of education of the beneficiaries. Despite all the individual and social gains resulting from the Operation, many of them cannot be captured in the economic analysis since the available data do not allow it. Both PROGRESAR and ASISTIRÉ should lead to an increase in the number of years of education. This approach would only capture the most readily quantifiable benefits, as better educated individuals also benefit from improved health and greater life satisfaction, and society enjoys the multiple positive externalities provided by a more educated population. Moreover, the positive effect of the Operation on education quality translates into an increase in the returns to schooling that is more difficult to predict and quantify and is therefore not included in the economic analysis. Bearing this in mind, the economic benefits that are computed are only a lower bound of the Operation’s potential returns. 35. Efficiency. The Operation’s efficiency is evaluated with a cost-benefit analysis that compares the estimated costs of each of the Components of the Program with the expected benefits associated to the PDOs. The net effect of the Project on beneficiaries is estimated using a present discounted value (PDV) approach. This approach entails estimating the stream of benefits and costs of schooling over the lifetime of a representative student with and without the Operation. Given the structure of benefits and costs presented, an Internal Rate of Return (IRR) is calculated, which is defined as the discount rate that equates the net present value of the investment per student with the net present value of all earnings differential over a student’s life cycle. Both the PDV and the IRR of the Project allow us to assess whether it is a good investment from an economic point of view. 32 The remaining portion correspond to the student and teacher evaluation activities and are not included in the estimations. 33 As in an augmented Solow model. 34 See for instance Acemoglu and Angrist (2000) and Moretti (2004) for more on social returns to education. 35 See for instance Patrinos and Saellariou (2008) and Bertschy et al. (2009) for more on the impact of quality on earnings and Hanushek and Woessmann (2008) for more on its effect on growth. 36. Costs. The total estimated cost of ASISTIRÉ is US$14.4 million, US$4.4 million of which are expected to be devoted to an improvement in the design of the Program,36 while the other 10 are expected to finance its expansion. In effect, while the program reached 220 schools and around 50,000 students in 2018, it is expected to substantially expand in the following years. ASISTIRÉ is expected to be implemented in 800 new schools over the Program’s timeframe. If the average ASISTIRÉ beneficiaries per school are preserved in this expansion, 181,818 new students will be added to the 50,000 that benefited in 2018. If the number of students served by this Program expands linearly over the years until reaching 231,818 and distributing the cost of the improvement and the expansion accordingly among the projected beneficiaries gives as an average annual cost per student of US$6 for the improvement in the design and US$55 for the implementation of the expansion, totalizing an average cost per student of US$61. This value will be used as the estimated average cost per year and beneficiary of ASISTIRÉ. 37. The cost per scholar of the PROGRESAR Scholarships in the five-year period requires several assumptions. While there is a fixed budget of US$240 million each year, the students are distributed among 3 modalities: youth aged 18-24 in Basic education and Technical professional training, students aged 18-30 in Tertiary non-university studies, and individuals aged 18-30 pursuing a University degree. Assuming that the PROGRESAR students remain constant during the five years of the Operation in the level achieved in 2018, as well as their distribution across the different modalities of the program, 259,380 students would receive the scholarship in Basic education and Technical professional training, 151,000 in higher non-university education, and 195,000 in university level each year.37 Under this scenario, the yearly cost per scholar of the PROGRESAR Scholarships remain fixed at US$350 per scholar in Basic Education and Technical professional training, while the cost per scholar ranges between US$420 and US$620 with an even higher cost for teacher training in higher non-university institutes, and between US$420 and US$1300 in university. Due to a lack of data on the latter two modalities, we estimate the average cost per scholar for the latter two modalities by dividing all the program resources not devoted to Basic Education and Technical professional training (US$149.2 millions) by the number of students in these two modalities (346,000) and the resulting cost is US$431 per scholar each year. 38. Benefits. The benefits from the Operation are summarized in Figure A3.4. Another expected benefit in the framework includes improved management capacity through the Strengthening of the MECCyT (Subcomponent 4.4), which is not incorporated in the economic analysis. If it were, even higher returns would be expected, as future reforms would also be affected by this increase in efficiency. As previously explained, the social benefits and improvements in quality affecting the returns to schooling are also not considered. Thus, the numbers presented in this economic analysis should be considered as a lower-bound estimate of the Operation’s benefits. 36This amount includes the cost of the impact evaluation, which is added to the cost to be conservative. 37These assumptions imply a political decision of maintaining the amount of the scholarship constant in US dollars across the 4- year period instead of increasing coverage at the expense of a reduction in the amount of money that is transferred. Figure A3.4. Framework of Project Benefits 39. Estimating the impact of PROGRESAR Scholarships on short-term outcomes. We rely on rigorous evaluations of the impact of education scholarships to get a sense of the range of the effect that PROGRESAR can have on education attainment and other short-term outcomes.38 A review of the literature that evaluates the impact of scholarships is summarized in Table A3.3. Typically, the studies assess the effects of either merit-based or poverty-based scholarships on educational attainment (via enrollment and grade progression) and/or learning outcomes (captured by test scores).39 Poverty-based programs tend to deliver less-promising results in terms of learning outcomes but are still effective to increase the educational attainment of beneficiaries. Table A3.3. Summary of Evaluations of the Impact of Scholarships on short-term outcomes Effect on Effect on Study Type of scholarship Level of intervention educational test scores attainment Filmer, D. & Shady, N. (2009). "School Enrollment, Basic education 0.24 years of Selection and Test Scores". Development Research Poverty-based not significant (secondary) schooling Group, The World Bank. Blimpo, M. P. (2014) “Team Incentives for 0.66 years of Education in Developing Countries: A Randomized Basic education Merit-based - schooling (0.27 Field Experiment in Benin.� American Economic (secondary) sd) Journal: Applied Economics 6(4): 90-109 Behrman, J., Parker, S., Todd, p., and Wolpin, K. (2015). “Aligning Learning Incentives of Students 0.41 years of Basic education and Teachers: Results from a Social Experiment in Merit-based - schooling (0.17 (secondary) Mexican High Schools.� Journal of Political sd) Economy 123 (2): 325-364. Angrist, J., Autor, D., Hudson, S., and Pallais, A. "Evaluating Post-Secondary Aid: Enrollment, Poverty and Merit- Higher education 0.52 years of - Persistence, and Projected Completion Effects." based (college) schooling NBER working paper 23015. 38 There is not a rigorous impact evaluation of PROGRESAR Scholarships. 39 In the Table, we express the estimated effects on learning outcomes (always reported as standard deviations of the exam score) in terms of years of schooling. To that end, we use the PISA equivalence between learning outcomes and school attainment (0.41 standard deviations equals the effect of 1 year of schooling). Duflo, E., Dupas, P. and Kremer, M. (2017). The 0.32 years of 0.09 years of Poverty and Merit- Basic education Impact of Free Secondary Education: Experimental schooling schooling (0.15 based (secondary) Evidence from Ghana. (annualized) sd annualized) 40. There are limitations to using this literature review to estimate the impact of PROGRESAR on short-term outcomes. First, the experiments evaluated in the literature have limited external validity. Second, PROGRESAR scholarships are different to the programs evaluated in the literature in several dimensions. For instance, the age group that is covered is different, and more importantly, the majority of PROGRESAR beneficiaries attend either tertiary university education or tertiary non-university. To the best of our knowledge, however, no rigorous evidence exists in developing countries on the impact of scholarships at the Tertiary level, while the existing evidence for developed countries is not conclusive.40 In an effort to avoid possible upward biases in the estimation of the benefits, we chose to use the lowest effect found in the literature. Under this conservative assumption, the impact of the financial support of PROGRESAR scholarships is estimated as 0.24 years of schooling per year of exposure to the program. 41. This effect does not include the expected impact of the complementary activities included in the Operation. The effect of these complementary activities comes from a literature review of the effect of selected interventions on students that is averaged and multiplied by a correction factor of 0.1 trying to capture the relatively lower intensity of these activities in PROGRESAR Scholarships with respect to those found in the literature. This additional effect is included in the sensitivity analysis. 42. To estimate the impact of ASISTIRÉ on short term outcomes, we rely on the limited empirical evidence on EWS along with targeted interventions. A recent RCT in the U.S. showed that an early warning system that relies on basic data (chiefly absences, grades, and behavioral problems), coupled with close follow-up of students at-risk by the school team (case workers), reduced chronic absenteeism and course failure by almost 20 percent after only one year of implementation (Faria et al, 2017). The statistically significant difference of 20 percent (or 5 percentage points) that the paper finds in course failure between students in treatment and control schools would translate into 0.05 years of education for the average treated student if course failure would be equivalent to grade failure. Since only a fraction of students who fail a course end up failing the grade, we assume half of the effect find in this paper for ASISTIRÉ (0.025).41 However, ASISTIRÉ has distinctive features that differ from the evaluated program. Because of this, the alternative that is explored in the sensitivity analysis section is the use of an average effect among the impacts that are found in the literature on interventions on at-risk students that affect the same channels: support to the students (mainly remedial education and socioemotional skills) and communication with parents. A correction factor of 0.1 is applied to the estimated effects since ASISTIRÉ has a significant lower intensity intervention with respect to the reviewed programs. 40 Evidence from the U.S. on merit-based scholarships suggest either a positive effect on academic achievement (Dynarski 2008, Castelman 2014, Scott-Clayton 2011, Scott-Clayton and Zafar 2016, Bettinger at al. 2016) or no effect at all (DesJardins and McCall 2014, DesJardin at al. 2010, Sjoquist and Winters 2012, Sjoquist and Winters 2015). Quasi-experimental identification strategies on poverty-based scholarships find either positive or null effects (see Deming and Dynarski 2009 (US), Nielsen at al. 2010 (Denmark), Dearden et al. 2014 (UK) for positive effects and Baumgartner and Steiner 2005 (Germany) for null effects). 41 We would obtain a similar estimate if we use instead the only rigorous evaluation of an EWS in a developing country (the USAID- funded School Dropout Prevention Pilot Program (SDPP) in Cambodia, India, Tajikistan, and Timor-Leste). This program had a statistically significant favorable impact on dropout rates and grade progression in Cambodia, where 63.3 percent of students in EWS + Engagement schools enrolled in the ensuing grade, while only 61.5 percent did so in the control group. This statistically significant difference of 1.8 percent points can be translated into an increase in 0.018 years of education for the average treated student, which is similar to the 0.025 chose to use in our baseline estimation. 43. To translate the expected increase in years of schooling for a student into wage improvements, we will use the standard labor economics model to estimate the returns of education, namely the Mincer (1974) equation: 𝑊𝑖 = 𝛼 ̂1 𝑆𝑖 + 𝑋𝑖 ′𝜃 ̂ + 𝛽 ̂ + 𝜀̂𝑖 Where Wi is the logarithm of the hourly wage of individual i; Si is the years of schooling of i; X is a vector of controls; 𝛼 ̂ , and 𝜃 ̂ , 𝛽 ̂ are the estimated parameters; and 𝜀̂ is the error term. 44. The yearly monetary benefit of the Project can be captured by the increase in the years of schooling times the estimated returns to education, as described in the following equation: ̂ ∗ ∆𝑆𝑡 𝐵𝑡 = 𝑌𝑡 ∗ 𝛽 where 𝐵𝑡 are the monetary benefits of the Project at time t, ∆𝑆𝑡 captures the impact in years of schooling; ̂ corresponds to the returns to education, and 𝑌𝑡 stands for the expected labor income in year t for a 𝛽 student without the Project. 45. The Mincer equation for Argentina is estimated using microdata from EPH corresponding to the second semester of 2017. For the estimation, the set of controls we include comprises a sex dummy, potential experience, potential experience squared, regional dummies, a wage-earner dummy, and six sectorial variables. The parameters in the equation are estimated by Heckman full maximum likelihood using individuals aged 25–55. The selection equation includes the same covariates in the wage equation plus number of children, number of children interacted with the gender dummy, a marriage indicator and a school attendance binary variable. However, alternative estimations of the returns to schooling in Argentina are used to check the robustness of the results. We use the yearly labor income per capita (US$) ̂ and the as the reference income Yi.42 Multiplying this Yi by the estimated returns in the Mincer equation 𝛽 estimated impact on short-term outcomes ∆𝑆𝑡 we obtain the yearly benefits of the Project per student 𝐵𝑡 43, which are summarized in the Table below. The yearly benefits of ASISTIRÉ amount to US$15 per year, while the benefits of PROGRESAR total US$186 per year. Table A3.4. Benefits per student each year by Program Component and Results Area Expected Return to Impact in Yearly labor income education years of benefits without the schooling of the project project Yt β Δ St Bt ASISTIRÉ + PROGRESAR Scholarships $9,915 0.0814 0.265 $ 214 ASISTIRÉ $9,915 0.0814 0.025 $ 20 PROGRESAR Scholarships $9,915 0.0814 0.240 $194 Sources: (1) Expected labor income without the project computed as the average monthly labor income in nominal LCU (computed from SEDLAC microdata - second semester 2017) * 12 * average nominal exchange rate in second semester of 2017: $14386.84*12*(1/17.413). (2) Return to education estimated from a Mincer equation estimated by Heckman full maximum likelihood. (3) Impact in years of schooling of ASISTIRÉ comes from the only rigorous evaluation of an Early Warning System with impact in dropout rates, while the impact of PROGRESAR comes from the minimum effect found in the literature of scholarships. 42 Other alternatives are explored in the sensitivity analysis section. 43 An implicit assumption in this estimation is that the hours of work are not affected by the Operation. 46. Cost-benefit analysis. The last step of the analysis is to compare costs and benefits. Using a discount rate, the yearly cost per student is brought to the present, as well as the whole stream of yearly future benefits 𝐵𝑡 . We assume that students benefited from the Operation enter the workforce at age 21 and retire at age 61 (after 40 years of work) earning US$ 𝐵𝑡 per year more than their peers. For the PROGRESAR scholarships, we maintain the 40-year timeframe but delay the entrance to the workforce until the age of 25.44 With a conservative discount rate (5 percent), the Net Present Value of the income benefits of the average Operation beneficiary are compared with its cost, and the IRR is computed. The Table below summarizes the results. Results from the economic analysis suggest that benefits from the expansion of ASISTIRÉ are 4 times its cost, while benefits from PROGRESAR scholarships in Basic Education and Technical Professional Training exceed 7 times its cost. The Benefit/Cost ratio for improving PROGRESAR scholarships in higher education is a bit lower than in the other level (5.8), which is explained by the relatively higher cost of these types of scholarships. However, the robustness of this result to the possible presence of heterogeneous impacts on short-term outcomes (i.e. a non-linear return to education) is explored in the sensitivity analysis. Table A3.5. Results of the economic analysis Benefit/Cost Internal Rate of Ratio Return (IRR) PROGRESAR scholarships Basic Education 7.1 19 % PROGRESAR scholarships Higher education 5.8 17 % Scale-up of ASISTIRÉ 4.0 13 % 47. Sensitivity analysis. The analysis is based on several parameters and assumptions that can be relaxed or modified to check the robustness of results. We test the consistence of the results to: (i) different discount rates; (ii) different estimations of the impact of the program on schooling; (iii) different estimations of the returns to schooling (in particular, allowing for non-linearity in the returns to additional years of education at basic, tertiary non-university, and university education); (iv) different reference incomes; (v) different profiles of insertion in the labor market (varying the entrance and retirement age, or the timeframe). The results are presented in the following tables. Table A3.6. Benefit/Cost ratios with different discount rates 5% 7% 10% PROGRESAR scholarships Basic Education 7.1 4.9 3.1 PROGRESAR scholarships Higher education 5.8 4.0 2.5 Scale-up of ASISTIRÉ 4.0 2.7 1.7 44More specifically, we assume that the benefits of the project start after 7 years for ASISTIRÉ and after 8 years for PROGRESAR. Moreover, all the computations are expressed per year of investment to avoid having to make additional assumptions regarding the number of years of exposure to the program. Table A3.7. Benefit/Cost ratios with alternative estimations of the impact of the program on schooling45 With impact estimations from programs Baseline affecting the same channels (revamped/improved programs) PROGRESAR scholarships Basic Education 7.1 7.4 PROGRESAR scholarships Higher education 5.8 6.0 Scale-up of ASISTIRÉ 4.0 6.3 Table A3.8. Benefit/Cost ratios with non-linear returns to schooling46 With non-linear Baseline returns to schooling PROGRESAR scholarships Basic Education 7.1 3.9 PROGRESAR scholarships higher non-University education 5.8 6.1 PROGRESAR scholarships University 5.8 8.0 Scale-up of ASISTIRÉ 4.0 2.2 Table A3.9. Benefit/Cost ratios with the same return to education (in %) but a lower reference income47 With lower Baseline reference income PROGRESAR scholarships Basic Education 7.1 3.2 PROGRESAR scholarships Higher education 5.8 2.6 Scale-up of ASISTIRÉ 4.0 1.8 45 The assumed impacts of ASISTIRÉ and PROGRESAR on schooling in these estimations are 0.044 and 0.255, respectively. The impact of ASISTIRÉ results from the 0.15 years of schooling from interventions on students coming from Heller et al. (2017) and Oreopoulos et al. (2017) scaled-down by a factor of 0.1, plus the intervention on parents in Berlinski et al. (2016) equivalent to 0.029 years of schooling. The impact of PROGRESAR results from adding to the baseline estimation the impact of complementary activities, assumed to have the same impact as the intervention on students in ASISTIRÉ (0.15*0.1=0.015). Since we assume students served in the first year of the Operation would not perceive the benefit from the improved programs, a correction factor of 3/4 was also applied to the additional yearly benefits. 46 Returns to an additional year of schooling at basic, tertiary and university education are separately estimated by including degree completion dummies in the Mincer equation and dividing the estimated effect of an additional degree by the theoretical number of years necessary to complete each level. Estimated returns per year of education are therefore 0.0450 for basic education, 0.0857 for tertiary and 0.1136 for university education. 47 The reference income in these estimations is the per capita labor income of the bottom 50 percent of the income distribution. Table A3.10. Benefit/Cost ratios postponing 5-years the stream of benefits With delayed Baseline insertion in the labor market PROGRESAR scholarships Basic Education 7.1 5.6 PROGRESAR scholarships higher education 5.8 4.5 Scale-up of ASISTIRÉ 4.0 3.1