FOR OFFICIAL USE ONLY Report No: PAD3177 PROGRAM APPRAISAL DOCUMENT ON A PROPOSED LOAN IN THE AMOUNT OF US$341 MILLION TO THE ARGENTINE REPUBLIC FOR IMPROVING INCLUSION IN SECONDARY AND HIGHER EDUCATION June 7, 2019 Education Global Practice Latin America and Caribbean Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective May 31, 2019) Currency Unit = Argentine Peso AR$44.56 = US$1 FISCAL YEAR January 1 - December 31 Regional Vice President: Axel van Trotsenburg Practice Group Vice President: Jaime Saavedra Chanduvi Country Director: Jesko S. Hentschel Practice Manager: Emanuela Di Gropello Helena Rovner, Francisco Haimovich Paz, Marcelo Task Team Leader(s): Becerra ABBREVIATIONS AND ACRONYMS ANSES National Administration of Social Security (Administración Nacional de la Seguridad Social) APE Pregnancy Allowance (Asignación por Embarazo) AUH Universal Child Allowance (Asignación Universal por Hijo) BCRA Central Bank of the Republic of Argentina (Banco Central de la República Argentina) CABA Autonomous City of Buenos Aires (Ciudad Autónoma de Buenos Aires) CCT Conditional Cash Transfers CONEAU Comisión Nacional de Evaluación y Acreditación Universitaria CPF Country Partnership Framework CUE Single Institution Code (Código Único del Establecimiento) DIEE Directorate of Information and Education Statistics DLI Disbursement-linked Indicator DNB National Directorate for Educational Scholarships (Dirección Nacional de Becas Educativas) DPPEESA Directorate of Special Programs and Projects with Wide Sectoral Scope (Programas y Proyectos Especiales con Enfoque Sectorial Amplio) EEP Eligible Expenditure Program EPH Permanent Household Survey (Encuesta Permanente de Hogares) ESA Environmental and Social Assessment ESCP Environmental and Social Commitment Plan ESF Environmental and Social Framework ESSA Environmental and Social Systems Assessment EWS Early Warning System GDP Gross Domestic Product GoA Government of Argentina GP Government Programs HCI Human Capital Index IBRD International Bank for Reconstruction and Development ICESI Index of Social Conext of Education (Índice de Contexto Social de la Educación) ICT Information and Communications Technology IDB Inter-American Development Bank IERR Internal Economic Rate of Return IMF International Monetary Fund IPF Investment Project Financing IRR Internal Rate of Return IT Information Technology LAC Latin America and the Caribbean LGBTIQ Lesbian, Gay, Bisexual, Transgender, Intersex, and Queer M&E Monitoring and Evaluation MECCyT Ministry of Education, Culture, Science and Technology (Ministerio de Educación, Cultura, Ciencia y Tecnología) MH Ministry of Finance (Ministerio de Hacienda) NAT National ASISTIRÉ Team NPV Net Present Value OECD Organisation for Economic Co-operation and Development ONE National Evaluation Operative (Operativo Nacional de Evaluación) PAP Program Action Plan PBA Province of Buenos Aires PDO Project Development Objective PDV Present Discounted Value PforR Program for Results PISA Programme for International Student Assessment PIU Project Implementation Unit POM Project Operations Manual PROMER II Argentina Second Rural Education Improvement Project (Segundo Proyecto de Mejoramiento de Educación Rural) RA Annual Survey (Relevamiento Annual) RCT Randomized Control Trial SBA Stand-by Arrangement SCD Systematic Country Diagnostic SD Standard Deviation SEE Secretariat of Education Evaluation (Secretaría de Evaluación Educativa) SES Socioemotional Skills SGE Secretariat for Educational Management (Secretaría de Gestión Educativa) SIU University Information System (Sistema de Información Universitaria) SMVM Minimum Vital and Mobile Salary (Salario Mínimo Vital y Móvil) SRFI Subsecretariat of International Financing Relations (Subsecretaría de Relaciones Financieras Internacionales) STEM Science, Technology, Engineering and Mathematics TOR Terms of Reference UDAI Comprehensive Service Units of ANSES (Unidad de Atención Integral de ANSES) UDAM Mobile Service Units of ANSES (Unidad de Atención Móvil de ANSES) UEPEX System of Financial Administration and Control for Agencies Executing Foreign Loans (Sistema de Administracion y Control Financiero para las Unidades Ejecutoras de Préstamos Externos) VA Verification Agent ARGENTINA IMPROVING INCLUSION IN SECONDARY AND HIGHER EDUCATION TABLE OF CONTENTS I. STRATEGIC CONTEXT ...................................................................................................... 8 A. Country Context .................................................................................................................. 8 B. Sectoral and Institutional Context .................................................................................... 10 C. Relationship to the CPF and Rationale for Use of Instrument .......................................... 14 II. PROGRAM DESCRIPTION............................................................................................... 16 A. Government Program .................................................................................................... 16 B. PforR Program Scope ..................................................................................................... 20 C. PDO and PDO Level Results Indicators .............................................................................. 22 D. Disbursement Linked Indicators and Verification Protocols ............................................ 23 E. IPF Component .................................................................................................................. 27 III. PROGRAM IMPLEMENTATION ...................................................................................... 28 A. Institutional and Implementation Arrangements ............................................................. 28 B. Results Monitoring and Evaluation ................................................................................... 29 C. Disbursement Arrangements ............................................................................................ 30 IV. ASSESSMENT SUMMARY ................................................................................................ 31 A. Technical (including program economic evaluation) ........................................................ 31 B. Fiduciary ............................................................................................................................ 36 C. Legal Operational Policies ................................................................................................. 37 D. Environmental and Social ................................................................................................. 37 E. Risk Assessment................................................................................................................. 40 ANNEX 1. RESULTS FRAMEWORK MATRIX ........................................................................... 42 ANNEX 2. DISBURSEMENT LINKED INDICATORS, DISBURSEMENT ARRANGEMENTS AND VERIFICATION PROTOCOLS .................................................................................................. 64 ANNEX 3. TECHNICAL ASSESSMENT ..................................................................................... 77 ANNEX 4. FIDUCIARY SYSTEMS ASSESSMENT ....................................................................... 99 ANNEX 5. SUMMARY ENVIRONMENTAL AND SOCIAL SYSTEMS ASSESSMENT ..................... 129 ANNEX 6. PROGRAM ACTION PLAN .................................... ERROR! BOOKMARK NOT DEFINED. ANNEX 7. IMPLEMENTATION SUPPORT PLAN .................................................................... 135 ANNEX 8. DETAILED DESCRIPTION OF IPF COMPONENT ..................................................... 138 ANNEX 9. RATIONALE AND THEORY OF CHANGE OF THE GOA’S PROGRAM ........................ 144 ANNEX 10. GENDER GAPS AND ACTIONS............................................................................ 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) BASIC INFORMATION Is this a regionally tagged project? Financing Instrument No Program-for-Results Financing Bank/IFC Collaboration Does this operation have an IPF component? No Yes Environmental and Social Risk Classification (IPF Component) Moderate Proposed Program Development Objective(s) The objectives of the Operation are to (i) reduce dropout rates in basic and higher education among the most vulnerable; and (ii) strengthen the Borrower’s education evaluation systems. Organizations Borrower : Argentine Republic Implementing Agency : Ministerio de Educacion, Cultura, Ciencia y Tecnologia COST & FINANCING FIN_SUMM_WITH_IPF SUMMARY (USD Millions) Government program Cost 1,133.00 Total Operation Cost 341.00 Total Program Cost 300.00 IPF Component 40.15 Other Costs 0.85 Total Financing 341.00 Financing Gap 0.00 Page 1 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Financing (USD Millions) International Bank for Reconstruction and Development (IBRD) 341.00 Expected Disbursements (USD Millions) Fiscal Year 2020 2021 2022 2023 Absolute 108.35 122.50 66.00 44.15 Cumulative 108.35 230.85 296.85 341.00 INSTITUTIONAL DATA Practice Area (Lead) Education Contributing Practice Areas Climate Change and Disaster Screening Yes PRI_PUB_DATA_TBL Private Capital Mobilized No Gender Tag Does the program plan to undertake any of the following? a. Analysis to identify Project-relevant gaps between males and females, especially in light of country gaps identified through SCD and CPF Yes b. Specific action(s) to address the gender gaps identified in (a) and/or to improve women or men's empowerment Yes Page 2 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) c. Include Indicators in results framework to monitor outcomes from actions identified in (b) Yes SYSTEMATIC OPERATIONS RISK-RATING TOOL (SORT) Risk Category Rating 1. Political and Governance ⚫ Substantial 2. Macroeconomic ⚫ High 3. Sector Strategies and Policies ⚫ Moderate 4. Technical Design of Project or Program ⚫ Moderate 5. Institutional Capacity for Implementation and Sustainability ⚫ Moderate 6. Fiduciary ⚫ Moderate 7. Environment and Social ⚫ Moderate 8. Stakeholders ⚫ Low 9. Other ⚫ Moderate 10. Overall ⚫ Substantial COMPLIANCE Policy Does the program depart from the CPF in content or in other significant respects? [ ] Yes [✔] No Does the program require any waivers of Bank policies? [ ] Yes [✔] No Page 3 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ESStandards Environmental and Social Standards Relevance Given its Context at the Time of Appraisal E & S Standards Relevance Assessment and Management of Environmental and Social Risks and Impacts Relevant Stakeholder Engagement and Information Disclosure Relevant Labor and Working Conditions Relevant Resource Efficiency and Pollution Prevention and Management Not Currently Relevant Community Health and Safety Not Currently Relevant Land Acquisition, Restrictions on Land Use and Involuntary Resettlement Not Currently Relevant Biodiversity Conservation and Sustainable Management of Living Natural Not Currently Relevant Resources Indigenous Peoples/Sub-Saharan African Historically Underserved Traditional Relevant Local Communities Cultural Heritage Not Currently Relevant Financial Intermediaries Not Currently Relevant ESStandardsNote NOTE: For further information regarding the World Bank’s due diligence assessment of the Project’s potential environmental and social risks and impacts, please refer to the Project’s Appraisal Environmental and Social Review Summary (ESRS). Legal Covenants Sections and Description Schedule 2. Section I.A Operation Institutions. The Borrower shall carry out the Operation through MH and MECCyT (including SGE, SEE and DNB), with the distribution of roles and responsibilities laid out in the Loan Agreement and detailed in the Operational Manual. Sections and Description Schedule 2. Section I.B Operational Manual. Without limitation to the provisions of Article V of the General Conditions, the Borrower shall carry out, and cause the Operation to be carried out, in accordance with the Operational Manual. Sections and Description Schedule 2. Section I.C DLR Verification Arrangements. The Borrower, through MECCyT, shall appoint a verification Page 4 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) agent for the Program that is acceptable to the Bank, in accordance with the terms of reference acceptable to the Bank. Sections and Description Schedule 2. Section I.D Program Action Plan. The Borrower, through MECCyT and MH, shall undertake the actions set forth in the Program Action Plan in a manner satisfactory to the Bank. Sections and Description Schedule 2. Section I.E 1 Environmental and Social Standards for the Project. The Borrower, through MECCyT, shall ensure that the Project is carried out in accordance with the Environmental and Social Standards, in a manner acceptable to the Bank. Sections and Description Schedule 2. Section I.E 2 ESCP. The Borrower, through MECCyT, shall ensure that the Project is implemented in accordance with the Environmental and Social Commitment Plan (“ESCP”), in a manner acceptable to the Bank. Sections and Description Schedule 2. Section I.E 4 Grievance Mechanism. The Borrower, through MECCyT, shall maintain and publicize the availability of a grievance mechanism, in form and substance satisfactory to the Bank, to hear and determine fairly and in good faith all complaints raised in relation to the Project and take all measures necessary to implement the determinations made by such mechanism in a manner satisfactory to the Bank. Sections and Description Schedule 2. Section I.F Participation Agreements. Prior to carrying out any activities under Parts 2.2 and 2.3 of the Project, the Borrower, through MECCYT shall enter into an agreement (the Participation Agreement) with each Participating Province, in accordancewith the Operational Manual. Sections and Description Schedule 2. Section III. Monitoring, Reporting and Evaluation. The Borrower, through MH, shall furnish to the Bank each Project Report and Program Report not later than forty-five days after the end of each calendar semester, covering the calendar semester. Conditions Type Description Disbursement (Schedule II. Section IV.B. Program Withdrawal Conditions. Notwithstanding the provisions of Part A above, no withdrawal shall be made for purposes of Section Page 5 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 2.03 of the Program General Conditions, for DLR achieved prior to the Signature Date, except that withdrawals up to an aggregate amount not to exceed $70,000,000 may be made for such DLRs, as further described and detailed in the Disbursement and Financial Information Letter, achieved prior to this date but on or after October 1, 2019 (but in no case more than one year prior to the Signature Date). Type Description Disbursement Schedule II. Section IV.B. Program Withdrawal Conditions (2). Notwithstanding the provisions of Part A above, no withdrawal shall be made for any DLR (as further described and detailed in the Disbursement and Financial Information Letter) under Categories (1) to (8) until and unless the Borrower has furnished evidence satisfactory to the Bank that said DLR has been achieved, as further detailed in the Verification Protocols. Type Description Disbursement Schedule II. Section IV.B. Program Withdrawal Conditions (3). Notwithstanding the provisions of Part A above, no withdrawal shall be made under Categories (1), (2), (3), (6), and (7), until after the date specified in the Disbursement and Financial Information letter for the respective DLR under each DLI. Type Description Disbursement (d) Schedule II. Section IV.C. Project Withdrawal Conditions. Notwithstanding the provisions of Part A above, no withdrawal shall be made for payments under the Project made prior to the Signature Date, except that withdrawals up to an aggregate amount not to exceed $5,000,000 may be made for payments made prior to this date but on or after November 8, 2018 (but in no case more than one year prior to the Signature Date), for Eligible Expenditures under Category (9). TASK TEAM Bank Staff Name Role Specialization Unit Team Leader(ADM Helena Rovner GED04 Responsible) Francisco Haimovich Paz Team Leader GED04 Marcelo Becerra Team Leader GED04 Procurement Specialist(ADM Martin Ariel Sabbatella GGOPL Responsible) Miguel-Santiago da Silva Financial Management GGOLF Oliveira Specialist(ADM Responsible) Antonella Novali Team Member GED04 Page 6 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Claudio Luis Daniele Environmental Specialist GENLC Elba Lydia Gaggero Environmental Specialist GENLC Eleonora Beatriz Camalli Social Specialist GSU04 Ezequiel Molina Team Member GEDGE Florencia Pilar Chaves Team Member LCC7C German Nicolas Freire Team Member GSU04 Luciana Garcia Team Member LCC7C Maria Jose Vargas Mancera Team Member GED04 Maria Virginia Hormazabal Team Member WFACS Paula Agostina Di Crocco Team Member Financial Management Analyst GGOLF Pilar Elisa Gonzalez Counsel LEGLE Rodriguez Santiago Scialabba Social Specialist GSU04 Extended Team Name Title Organization Location Page 7 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) I. STRATEGIC CONTEXT A. Country Context 1. Argentina has been engaged in a reform agenda to transform its economy, while confronting the challenge of unwinding macroeconomic imbalances and mitigating the social costs of the transition. In December 2015, the Government of Argentina (GoA) faced the challenge of pervasive macroeconomic imbalances, large microeconomic distortions, and a weakened institutional framework; large fiscal deficits, financial repression, monetization of the deficit, high inflation, and low investment were accompanied by price controls, large and regressive subsidies, trade restrictions, and rationing of foreign currency. The GoA eliminated foreign exchange controls and moved to a flexible exchange rate regime, put in place an inflation-targeting framework, initiated the process of realigning utility prices and reducing subsidies, and improved official statistics. The GoA also initiated reforms to strengthen the competitiveness of the economy and remove distortions holding back private sector-led growth, including reducing export taxes and easing import controls. However, the country remained vulnerable to market sentiment and changes in global financial conditions. 2. An unprecedented increase in public expenditure was the main driver of Argentina’s macroeconomic woes. The federal spending-to- Gross Domestic Product (GDP) ratio almost doubled between 2006 and 2016 (from 13.7 percent to more than 26 percent of GDP), while general government spending (federal government plus provinces) grew from 26 percent to more than 41 percent of GDP in the same period. The increase in federal spending was dominated by a rise in current expenditures, of which more than two-thirds were due to an expansion of pensions and economic subsidies, followed by an increase in the wage bill. In this context, Argentina’s overall fiscal deficit has progressively widened, reaching 6.7 percent of GDP in 2017 (of which the federal deficit was 5.9 percent of GDP), making fiscal consolidation unavoidable. Following an initial expansion, the new administration curbed the primary deficit into a decreasing path. 3. Despite the reforms put in place, the tightening of global financial conditions in the first part of 2018 brought to the fore Argentina’s underlying vulnerabilities. Continued macroeconomic imbalances, with a primary deficit of 4.2 percent of GDP and inflation of 24.8 percent at end-2017, combined with high external financing needs given a current account deficit of 4.7 percent of GDP, made Argentina vulnerable to global financial turmoil. Accordingly, Argentina’s financial markets came under sudden pressure in April 2018, with a large depreciation of the peso and a rise in the sovereign risk premium. The severe drought’s impact on agricultural production and exports, together with a tightening of global financial conditions (U.S. dollar appreciation and an upward shift in U.S. interest rates) and a heightened increase in the perceived riskiness of the more vulnerable emerging markets were the initial drivers behind this turmoil. These events prompted the GoA to seek financial support from the International Monetary Fund (IMF) in May 2018. 4. In June 2018, the GoA reached an agreement with the IMF for a Stand-by Arrangement (SBA) of US$50 billion, which was expanded in October 2018, based on a strengthened set of economic policies. After a short period of relative calm following the initial SBA approval, global financial volatility returned in August 2018, leading to a widespread depreciation of emerging market currencies, which the Argentine Page 8 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Peso was hit hard by. In this context, the GoA requested a revision of the SBA program to restore market confidence. The First Review of the SBA in October 2018 increased financing to US$56.3 billion and frontloaded disbursements from 2020-2021 to 2018-2019. The revised SBA includes: (i) the acceleration of fiscal consolidation to achieve a primary balance in 2019 and a primary surplus in 2020; and (ii) the shifting from inflation targeting to a tight targeting of the monetary base and a commitment to a floating exchange rate within a wide non-intervention zone. 5. In the context of declining federal spending, federal education spending is being reduced. Although the reduction in federal expenditure is mainly driven by subsidies and public investment, expenditure in education is also being affected, falling from 1.6 percent of GDP in 2015 to 1.4 in 2018. The Federal Budget for 2019 foresees further reductions in education spending to 1.1 percent of GDP, a reversion to 2006 levels. The declining trend is mirrored at the subnational level as well, for an overall fall of general government expenditure in education from 6.1 percent of GDP in 2015, the maximum since the approval of the Educational Financing Law, to 5.6 percent in 2017.1 6. Although social investment commitments are at the heart of the GoA’s priorities, fiscal consolidation could put pressure on programs not explicitly protected under the SBA. The SBA includes a social safeguard that protects the level of social spending (1.3 percent of GDP) and grants resources to prioritized social assistance programs. The focus will be on programs that protect children through the universal child allowance (Asignación Universal por Hijo, AUH, and Asignación por Embarazo, APE), as they have proven effective in producing positive socioeconomic outcomes and are well-targeted to those in need.2 The context of fiscal consolidation calls for measures to ensure sustainability and efficiency of social expenditure. Efforts are focused on integrating social services to avoid overlapping and duplication, reducing administrative costs, and improving targeting. 7. Moving forward, accumulation of human capital in Argentina will be key to increase productivity, innovation and job creation.3 Argentina performs below its potential in the Human Capital Index (HCI), with special lags in education quality, allowing children to achieve only 66 percent of their potential productivity as future workers.4 Strengthening human capital will be particularly important in view of the additional pressures that climate change will exert on human capital and economic growth. The country’s combined vulnerability and readiness ranking according to the Notre Dame Global Adaptation Initiative Index is 69 out of 181 ranked countries.5 Natural hazards are a concern, as 85 percent of all economic losses over that last 50 years resulted from hydro-meteorological hazards like floods and droughts.6 Such hazards have increased with climate change and stronger El Niño Oscillation events.7 1 Last data available. 2 Empirical evidence shows that AUH improved school attendance rates, in particular among eligible males aged 7 to 17 years old (6 percentage points) and had a smaller significant impact on boys aged 12 to 14 years old. It also reduced intra-annual dropout among girls aged 12 to 14. For both boys and girls, AUH improved primary school completion rates. See Edo and Marchioni (2018). 3 World Bank “Argentina - Systematic Country Diagnostic.” Washington, D.C.: World Bank Group (2018) 4 World Bank Human Capital Index, available online [http://www.worldbank.org/en/publication/human-capital] 5 The ND-GAIN Country Index summarizes a country’s vulnerability to climate change and other global challenges, in combination with its readiness to improve resilience. It aims to help governments, businesses, and communities better prioritize investments for a more efficient response to the immediate global challenges ahead. 6 Swiss-Re (2016), “Staying Afloat: Flood Risk in Argentina.” 7 Capriolo, A.D. and O.E. Scarpati (2010), “Extreme Hydrologic Events in North Area of Buenos Aires Province (Argentina),” Page 9 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Slow-onset impacts of climate change are also a concern for the accumulation of human capital in Argentina through the effects on nutrition, health, migration,8 and structural employment shifts. In this line, the institutional strengthening of the education programs to efficiently boost human capital are central for long-term growth and shared prosperity. B. Sectoral and Institutional Context 8. A federal system shapes education policy in Argentina. Provinces are responsible for the financing and management of initial, primary, secondary, and higher non-university levels. The Federal Government is responsible for the university level, formulating education guidelines and norms, and providing financial and technical assistance to the 23 provinces to support policies, reforms and compensatory programs.9 By 2017, the system had 11.4 million students in the Educación Común modality,10 1.2 million in Adult education, and 1.9 million in University.11 By 2015, there were 2,239 institutions in the higher non-university level, of which 47 percent were public and 59 percent offered teacher training courses. Most of the enrollment in higher education involves teacher training courses. Argentina’s public higher education is free with unrestricted admission to high school graduates. Public universities are characterized by a high level of institutional autonomy. In terms of quality assurance, the Comisión Nacional de Evaluación y Acreditación Universitaria (CONEAU) is responsible for external evaluation and accreditation of programs and institutions at the university level. There is no quality assurance system for non-university higher education institutes. 9. Education spending in Argentina, currently at nearly 6 percent of its GDP, is well above those of its regional peers. However, outcomes are below what would be expected, which can be partly explained by weaknesses in the system’s service delivery, such as poor governance and lack of transparency, which have hindered the implementation of effective policies that would help reduce dropout and increase quality. Argentina stands out in the Latin America and Caribbean (LAC) region in terms of public expenditure in education. Between 2006 and 2017, the Education Finance Law (2005) resulted in a budget increase from 4.5 to approximately 6 percent of GDP. This amount was above the regional average of about 5 percent, and similar to investments made by members of the Organization for Economic Co- operation and Development (OECD), also averaging about 6 percent of GDP. While coverage rates in education are in line with these fiscal efforts, key outcomes like dropout and learning are well below the predicted levels, indicating important inefficiencies in the implementation of education policies. Despite large investments in education, a large share of students does not achieve the expected outcomes in terms of timely graduation or math skills. (http://dx.doi.org/10.5402/2012/145081) 8 World Meteorological Organization (2014). The Impact of Climate Change: Migration and Cities in South America. https://public.wmo.int/en/resources/bulletin/impact-of-climate-change-migration-and-cities-south-america 9 2006 National Education Law 10 Students in Initial: 1.8 million; Primary: 4.8 million; Secondary: 3.8 million; and higher education non-university: 0.9 million. 11 Initial education is compulsory for 4 and 5-year-olds. Primary school is compulsory for all children and comprises 6 grades in 12 jurisdictions, and 7 grades in the rest. Since 2006, secondary is compulsory and lasts 6 years in the former and 5 years in the latter. Secondary school is divided into lower (3 years, Ciclo Básico) and upper (2 or 3 years depending on the jurisdiction, Ciclo Orientado). Basic education comprises primary and secondary. The non-university higher education level is composed by institutes for teacher training (Institutos de Formación Docente), and technical-professional institutions. Page 10 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 10. A significant share of youth drop out of school without basic cognitive skills. Although access to secondary is almost universal, the enrollment rate starts to fall significantly after 15 years of age —when the opportunity cost of staying in school rises— especially for boys and low-income students.12 At this age, the learning deficits are deep, with more than two-thirds of youth not being able to solve simple math problems, and half unable to interpret basic texts.13 Around half of students that start secondary do not complete it on time, and only one third of the poorest do so, leaving the educational system without key competences. One in five youth aged 15-24 are not in employment, education or training, and one in five study and work simultaneously, which rises to 25 percent for women, one of the highest rates among peer countries.14 11. While economic issues stand out as the main reasons for dropout, other interacting factors affect the decision to leave school. The 2014 National Youth Survey reports that 33.7 percent of 15 to 29- year-olds secondary dropouts needed to work or did not have the resources to attend school; 20 percent did not like to study, nor did they consider it useful; 15.8 percent left due to pregnancy; and 15.4 percent self-reported learning deficits. For girls, the main reason for dropping out of school is pregnancy (29.8 percent), in line with an adolescent fertility rate three times higher than in the OECD. Overall, offering financial aid, ensuring high private returns to education (e.g. by boosting quality and relevance of education), preventing teenage pregnancy, and providing comprehensive support are critical to tackle the dropout challenge in Argentina. In addition, the multidimensional nature of these challenges often requires the strong articulation of multisectoral policies to address them properly. 12. Global evidence shows that it is more cost-effective to tackle these challenges early on, and that administrative data can be used to accurately target interventions to at-risk students. The decision to drop out of school often results from a process of frustration and disengagement, which usually begins years before the actual decision to drop out is made.15 Evidence indicates that this process shows “early warnings” in terms of poor academic performance, high absenteeism, and behavioral problems; the dynamic is similar in Argentina. National standardized tests show that approximately one third of children that finish primary education lack basic math and Spanish skills. Many of these students struggle with the more complex curriculum in secondary, and repetition and overage rates spike in the first grades of this level. This is usually accompanied by a continuous growth in the absenteeism rate and behavior problems. Evidence shows that all these variables can be used to accurately predict the risk of dropout, act early on, and improve the cost-effectiveness of preventive interventions.16 13. While most school dropouts enter the informal labor market or face unemployment or 12 Dropout is structurally higher for students from vulnerable groups identified through the social assessment: (i) indigenous peoples, (ii) afro-descendants, (iii) people with disabilities and (iv) LGBTIQ people (in particular trans students). In all cases, completion rates both for secondary and higher education are 30 percent lower than for the general population. 13 By age 15, according to PISA, the median Argentine performs in math an equivalent of 2.5 years below the average of OECD countries, and within Argentina, low-income students are 2 years behind high-income ones. Females outperform males in language (1 additional year of education). Test scores have stagnated at the same time as spending was increasing, indicating important inefficiencies. Learning outcomes are below the expected for Argentina’s GDP per capita. 14 World Bank (2018) 15 Oreopoulous (2016), Adelman (2015) 16 Knowles, 2015; Adelman et al, 2016. Page 11 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) inactivity, some return to education systems through adult education programs. In Argentina, this modality is offered through “adult education programs,”17 designed for adults and out-of-school youth completing their basic education studies or attending technical professional training. Most adult schools offer night shift programs and alternatives to complete pending subjects for graduation. Public adult schools served 1.1 million students in 2017 (13 percent in primary, 44 percent in secondary, and 43 percent in technical professional training18). Despite having lower school dropout rates, women account for 55 percent of adult education students; 83 percent of the students completing secondary in adult schools are between 17 and 19 years old.19 14. A large share of students who manage to complete secondary education has access to higher education, but efficiency indicators are quite modest, particularly for the most vulnerable. A large expansion of higher education during the last 15 years boosted access, reaching an enrollment of 32 percent (of which 70 percent corresponds to university education). The expansion of coverage in higher education disproportionately benefited students from the poorest quintiles; currently, around 40 percent of higher education students come from the poorest 50 percent of the population. However, the dropout rates are particularly high in the first years, and the system has one of the lowest completion rates in the region (just 29 graduates out of 100 new entrants), and one of the largest time-to-degree ratios. These indicators are also highly correlated with socioeconomic background; for instance, completion rates for students coming from the richest quintile almost triple those from the poorest quintile.20 Around 66 percent of non-university higher education students and 78 percent of university students attend public institutions. 15. Vulnerable students face financial and non-financial barriers to complete higher education, including limited economic resources, inadequate support to make informed career decisions, and critical skill deficits. Students from poor backgrounds face important economic challenges to study in this level. They often need to work to make a living and are more vulnerable to climate shocks.21 Furthermore, the opportunity costs of being out of the market grows at the fastest rate at this age. More than 20 percent of those who manage to keep studying have full or part-time jobs, and work on average more than 24 hours per week.22 Furthermore, many of these students are the first generation in their family to reach higher education, and hence have limited help to make informed career decisions.23 Finally, a large share is academically unprepared and lacks key socio-emotional skills needed to succeed in this level. 16. Furthermore, there is limited variety in the selection of degrees and fields, reflecting some 17 Educación Permanente de Jóvenes y Adultos. 18 Technical education programs 19 Relevamiento Anual 2017. 20 Own estimations based on Encuesta Permanente de Hogares 2017 (EPH). The calculations ignore the potential “sorting” income effects associated with completing the level, but similar patterns are observed when taking this potential into account. 21 According to World Bank (2016), climate change affects food security for the poor in LAC by affecting their food prices by 3-12 percent. Before enrolling in the Program, about 67,448 student families are below the extreme poverty line (defined as a set of minimum food expenditures) and face major nutritional challenges. Their food consumption and nutrition outcomes are affected by climate change impacts such as droughts and uncertain rain patterns. 22 Own estimations based on EPH. Back of the envelope calculations suggest that access to scholarships reduces part-time work and hours worked by 25 percent. 23 World Bank (2016) Page 12 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) extent of uninformed decision-making by students. Only one third of higher education students choose a non-university degree, one of the lowest rates among regional peers, despite the larger completion rates for this degree, and the often higher economics returns, particularly for women and the poor.24 Among those who chose a university degree, traditional careers predominate: 46 percent of higher education students graduate from Humanities, Arts, Social Sciences, Business and Law programs, and only around 14 percent graduate in Science, Technology, Engineering and Mathematics (STEM) fields, limiting the transition to the labor market. The skew towards humanities and law degrees also generates shortages for some skills in the labor market, especially for technicians, skilled trades and engineers.25 While women outperform men in terms of access (by 12 percentage points) and graduation rates (by 15 percentage points), they are much less likely to enroll in and graduate from STEM programs. 17. The GoA has a comprehensive strategy to tackle exclusion and dropout, especially for the most vulnerable students, and to improve their learning trajectories (Table 1). The strategy is multidimensional and includes financial aid throughout primary and higher education (AUH and PROGRESAR scholarships), programs to improve the marginal benefit of staying in school (e.g., by remediating learning deficits with Escuelas FARO, or improving the relevance of the curriculum with Secundaria 2030), an early dropout prevention program (ASISTIRÉ), and programs to support vulnerable students in their transition to higher education (e.g. Nexos).26 The strategy is carried out by the Ministry of Education, Culture, Science and Technology (Ministerio de Educación, Cultura, Ciencia y Tecnología, MECCyT). In 2016, the GoA created the Secretariat of Education Evaluation (Secretaría de Evaluación Educativa, SEE), within the MECCyT, with the objective of strengthening evaluation and monitoring of both students and teachers. In 2016, the SEE launched APRENDER—a census-based student learning assessment that covers the last grades of primary and secondary education in public and private institutions. In 2017, it launched ENSEÑAR, a one-time standardized noncompulsory assessment of candidates for the teaching career.27 In 2018, a self-evaluation tool for teacher training institutes was launched in 17 Provinces, data collection instruments were applied, and the process will continue with analysis of outcomes and workshops on improvement strategies in 2019 (see Table 1 for a summary of the GoA’s strategy).28 18. Within this strategy, the PROGRESAR Scholarships and the national education evaluation system reflect the GoA’s efforts to institutionalize and strengthen service delivery capacity and effective policy-making in the education system—two critical institutional bottlenecks. First, PROGRESAR Scholarships is a key instrument to provide financial aid and incentives for the most vulnerable youth to 24 Non-university graduation rates more than double that of university degrees for students in the bottom 40 percent, and the average returns for completing a Non-university degree are (at least) 50 percent larger than the ones corresponding to an incomplete university degree for women and the poorest students. 25 INET (2016) 26 These programs cover students attending public institutions. 27 The Second Rural Education Improvement Project (PROMER II, P133195, Loan 8452-AR) provided support to the pilot of Secundaria 2030 in 300 rural schools and included initiatives to enhance school trajectories for indigenous students. APRENDER and ENSEÑAR have been technically and financially supported by PROMER and by the analytical work under the Supporting Evidence-Based Policy Interventions task (P164061), including the support for workshops on the use of APRENDER for 1,500 school principals in the Province of Buenos Aires. The workshops integrated lessons learned through activities in the provinces of Salta and La Rioja, under the analytical tasks Effects of Student Assessments on Learning Outcomes in La Rioja and Salta (P157235 and P157237). 28 A full description of the rationale and theory of change for the GoA’s strategy is available in Annex 9. Page 13 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) continue their education trajectories. It serves over 600,000 students (scholars, hereafter) from the poorest socioeconomic households. It migrated from a monetary transfer to an education scholarship with incentives to enter and complete education programs. This was paired with an increase in transparency through the establishment of strong monitoring and evaluation mechanisms. Second, with the institutionalization of evaluation in the education sector, the GoA established a foundation for tracking, monitoring and improving student and teacher learning trajectories. These two initiatives constitute an important cornerstone to strengthen the governance, transparency, and accountability of the education system. In addition, the project will reinforce the service delivery and effectiveness of the system by improving the procurement and financial management within the MECCyT, as well as their anticorruption polices. 19. This Operation would provide financial and technical assistance to strengthen the policy implementation capacities of the GoA. Support would be focused on the following programs: PROGRESAR Scholarships, the education evaluation system and ASISTIRÉ. Sustaining these three programs and improving their implementation would be central to support the most vulnerable in their education progression and graduation and would support the GoA’s efforts to strengthen the education system’s capacity and accountability. Table 1. GoA National Programs to Support Education Trajectories and Learning Objective/ World Bank Program Scope Description Rationale Support through Conditional cash transfer to the most vulnerable households with co- P158791 AUH29 Nationwide Provide financial aid responsibilities in health and education (though basic education) and P167851 from primary to higher PROGRESAR Financial support to students from the poorest socioeconomic households This operation Nationwide education Scholarships to complete basic education, and to access and complete higher education (PforR and IPF) Provides intensive teacher and principal training aimed at closing gaps in Improve the marginal Escuelas Provincial basic skills and reducing repetition, particularly in lower secondary in the benefit of staying in - FARO most vulnerable schools. Represent 30 percent of secondary public schools. school Secundaria Curricular reform aiming to revamp secondary education by updating the Improve relevance of Provincial P133195 2030 content and structure of the secondary level curriculum Early warning system to identify at-risk students and intervene to prevent Comprehensive support This operation ASISTIRÉ Provincial dropout at the secondary level to prevent dropout (IPF) Improve transition Interventions to support vulnerable students in the transition to higher NEXOS Nationwide between secondary and - education through information, remedial education, and mentoring higher education This operation National Standardized Student Test in the last years of primary (6th / 7th APRENDER Nationwide Improve evaluation (PforR and IPF) grade) and secondary education (5th / 6th grade) and P133195 Includes a i) National Standardized Test for soon-to-be graduates of the This operation ENSEÑAR Nationwide teacher training institutes; ii) a Self-Evaluation Institutional Assessment of Improve evaluation (PforR and IPF) teacher training institutes. and P133195 C. Relationship to the CPF and Rationale for Use of Instrument 20. The proposed Operation is fully aligned with the priorities laid out in the Argentina Country Partnership Framework (CPF) for the period FY19-22 (Report No. 131971–AR, discussed by the Executive Directors on April 25, 2019). The Operation supports the CPF’s Focus Area 2: Addressing Key Institutional 29AUH covers 3.9 million children in the most vulnerable households. Eligibility: (i) both parents unemployed, informal, or not benefitting from other ANSES programs; (ii) proof of compliance with co-responsibilities of health and education. Page 14 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Constraints for Better Governance and Service Delivery, specifically Objective 6: Building Skills for the Future. Argentina’s human capital stock needs to be expanded to attract long-term private financing, which would require an educated labor force that can competitively work with new technology in high productivity sectors. These objectives are pressing given that completion rates and learning in secondary school remain low. The CPF aims to improve both learning outcomes and completion at the secondary and higher education levels to foster employability of youth from vulnerable backgrounds by improving the institutional setting for service delivery. The proposed interventions respond to these needs by tackling sources of these imbalances. The Operation also intends to strengthen the adaptability of vulnerable students, especially female students, to climate change by improving their nutritional outcomes. It would also contribute to mitigation efforts by fostering strategic careers for mitigation and adaptation such as renewable energy, landscape planning, sustainable urban development, renewable natural resources engineer for arid areas. 21. The Operation is also consistent with the findings from the 2018 Systematic Country Diagnostic (SCD, Report No. 130104), which identified the need to improve the quality and relevance of education as a key policy priority for Argentina. Furthermore, the technical assistance activities that will focus on evidence-based interventions to tackle dropout are in line with the SCD’s finding that it will be essential to foster the development of critical basic cognitive and (21st century) soft-skills in secondary school. 22. Choice of Financing Instrument. The use of a hybrid operation responds to the GoA’s interest in a full package of interventions: a program based on results, accompanied by robust technical assistance, a results framework to monitor its objectives, and support that focuses on technical discussions. The Bank would use this hybrid operation to support the GoA’s efforts to: (i) improve the institutional quality of existing programs such as PROGRESAR Scholarships, which aims to foster retention and completion for youth aged 18-24 at the secondary and higher education levels; (ii) continue to support student and teacher evaluation; and (iii) reduce dropout rates in secondary education through the expansion of the pilot program ASISTIRÉ. 23. The Operation would combine Program-for-Results (PforR) and Investment Project Financing (IPF). The PforR instrument would be used for 85 percent of the loan and would finance the improvement of existing programs: PROGRESAR Scholarships and student and teacher evaluations. This instrument would allow the dialogue during implementation to focus on institutional strengthening and capacity- building within the implementing institutions to not only improve the execution of current programs, but also ensure that they achieve sustainable results. Providing resources under a PforR scheme provides clear incentives for a government program to achieve concrete results. The PforR also helps create an external framework under which the MECCyT can set and follow certain objectives for its existing programs, which is further strengthened by oversight from the Ministry of Finance (Ministerio de Hacienda, MH). Most importantly, this also sets up an intra-governmental relationship of accountability among the two entities that will create greater commitment to development of the set objectives. 24. The IPF component is the most suitable for a subset of activities under the Operation, particularly those requiring implementation by the Provinces, especially considering that this is the first use of the PforR instrument in the country. Firstly, this component would include support to the implementation of key activities for APRENDER at the Provincial level already being supported by the Bank through the Argentina Second Rural Education Project – PROMER II (Loan 8452-AR, P133195), an IPF, and switching Page 15 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) those to PforR processes would hinder their implementation and entail lengthy processes for such a low- cost activity. Certain activities related to the expansion of ASISTIRÉ would also require implementation at the provincial level. Providing incentives for provincial authorities is challenging and would require significant structural changes, but it would be difficult to push for these changes through low-cost activities. Second, the selection of high-quality consultants and/or technical assistance has historically been more successful using the Bank’s, rather than the Borrower’s, procurement systems. The IPF would also allow the financing of capacity-building activities that do not have existing budget lines. II. PROGRAM DESCRIPTION A. Government Program PROGRESAR Scholarships 25. Established in 2014, PROGRESAR was conceived mainly as a subsidy transfer to foster access and completion of basic and higher education for disadvantaged youth but monitoring of eligibility for this transfer was weak.30 The program was managed by the National Administration of Social Security (Administración Nacional de la Seguridad Social, ANSES) and conceived as a continuation of the AUH, which can be received up to 17 years of age. A total of 80 percent of the monthly transfer was contingent on age and income, and 20 percent on regular attendance to a public education institution at any level. By 2017, the program reached 1 million beneficiaries, but monitoring of eligibility and attendance was carried out poorly, and around 40 percent of the beneficiaries were not attending any education institution. The structure of the benefit was not aligned with education outcomes, as no incentives for progression completion were in place and benefits were fixed across education levels. Moreover, the program was focused on the financial support to scholars, with no follow-up on their academic trajectory. 26. In 2018, the program was transferred to the MECCyT with the objective of increasing transparency and accountability and institutionalizing the national scholarship system and was modified from a transfer to a scholarship program – PROGRESAR Scholarships – with significant improvements introduced to the program. The National Directorate for Educational Scholarships (Dirección Nacional de Becas Educativas, DNB), within MECCyT, is in charge of the national scholarship system, consolidated in PROGRESAR Scholarships. The program offers four types of scholarships (Figure 1) that cover overaged students completing primary and/or secondary education and students accessing and completing higher education. The main objectives of the technical improvements introduced to the scholarships were to align the benefits with education outcomes to improve human capital, to promote more inclusive and merit-based benefits, and to close the gap between education and labor markets by promoting strategic careers in higher education. This was all paired with stronger management and monitoring of the program to increase its efficiency and transparency. 27. In 2018, for the first time, PROGRESAR Scholarships introduced the following institutional and technical modifications to the program: i. Application process: The program eliminated the automatic renewal of the scholarship. The application for Basic Education and Technical Professional Training is carried out in person through 30 Students aged 18-24 with a household income up to three times the Minimum Vital Moving Salary ( Salario Mínimo Vital y Móvil, SMVM). Beneficiaries cannot receive any other national, provincial or municipal benefit, but can receive AUH or AUE. Page 16 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANSES and through an online website for higher education scholarships.31 ii. Academic requirements: The academic requirements to receive the scholarship increased, requiring the scholar to pass 50 percent of the theoretical study plan to be eligible (in higher education). A proficiency test was also included as a condition of eligibility for students applying to the scholarship for teacher training programs (Compromiso Docente). iii. Monitoring of eligibility requirements: The program increased the monitoring of eligibility requirements, particularly the certification of regular attendance to an education institution. The first year these eligibility conditions were enforced, beneficiaries decreased from 1 million to 606,000. The income eligibility requirement is verified by ANSES for all the scholarships. Regular attendance for PROGRESAR Basic Education is also certified by ANSES during the application process, in which the applicant needs to present the certification issued by the school. For PROGRESAR Higher Education, the MECCyT carries out the certification in direct communication with the higher education institutions and universities (Figure 1). iv. Financial incentives: The Program offers financial incentives for Strategic Careers,32 increasing benefits on progression, and eligibility up to 30 years of age for higher education to cover students who are well into their education trajectory (advanced students). The selection of Strategic Careers was based on the demand for careers by area of study and geographical region, as defined by the Secretariat of University Policies. v. Merit-based component: Top performing scholars can receive a merit-based component equivalent to the total scholarship received during the immediate previous year. The eligibility requires passing 50 percent of the theoretical study plan and a minimum grade point average of 8. 28. The DNB carries out all the activities related to the design and implementation of the application process for the scholarship, monitoring of eligibility conditions, scholarship approval, and granting of awards based on the incentives scheme. 29. The 2018 PROGRESAR scholars are the first cohort of the revamped program, with positive early results on the increased transparency and accountability and the effects of targeting and incentives. The 2018 PROGRESAR Scholarship call received over 979,000 applications, of which around 40 percent did not meet the age, income, or academic requirements. Of the total scholarships, 43 percent supported Basic Education, 25 percent went to non-university higher education institutions, and 32 percent to university scholars. The targeting to the most vulnerable was effective, as a high share of the scholars’ households are below the poverty line (Figure 2): 79 percent of Basic Education scholars’ households are below the poverty line.33 64 percent of the higher education scholars were advanced students (between 25 and 30 years old) who were already progressing in their studies. 63 percent of the scholars are women. The location of scholars mirrored the education enrollment geographical distribution, with 30 percent in the Autonomous City of Buenos Aires (Ciudad Autónoma de Buenos Aires, CABA), 22 percent in the Central region, and only 5 percent in the Southern region. Following the introduction of economic incentives for Strategic Careers, the number of scholars in these fields increased by 36 percent relative to the 2017 cohort. However, these only represent 55 percent of the scholars in non-strategic careers. The strategic 31 https://www.argentina.gob.ar/becasprogresar 32 Health, Engineering, Architecture & Design, IT, Agriculture, among others. 33 Estimations of the effect of the scholarship on income show that by receiving the scholarship, 12,000 households overcame the “indigence line” (around US$200 monthly) and 15,000 went out of poverty. Page 17 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) areas with more scholars are health (nursing), engineering, architecture and design, and Information Technology (IT). 30. Important challenges remain in the management and monitoring of the program: i. Improve monitoring and availability of timely information on eligibility requirements, mainly the academic requirements, including attendance to education institutions and academic performance. In the short term, the objective would be to increase the frequency of attendance control (quarterly report by the institutions of enrollment, attendance, and subjects passed).34 In the medium run, it would be to develop a “nominal” information system where each scholar’s attendance can be monitored in real time; ii. Develop alternative eligibility options in view of possible fiscal constraints by ensuring targeting to the poorest students; iii. Monitor and evaluate the program, including the effects of the incentives , such as scholarship amount, priority programs, continuation, minimum academic requirements, etc.; iv. Develop strategies, jointly with provinces, universities and institutions, to support scholars’ success through non-financial support. Figure 1. PROGRESAR Scholarships, 2018 34 For higher education. Page 18 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Figure 1. Process of verification of eligibility requirements for applicants YES: NO: Eligible Meet academic YES: YES: requirement? Certification of New student? NO: YES: regular student in YES: Not eligible Household an education Eligible income < 3 institution? NO: 18 - 24 years? SMMLV Not eligible NO: NO: Not eligible Not eligible Note: There are no academic requirements for Basic Education scholarships. Scholarships for teacher training include the approval of the Compromiso Docente. Advanced students’ age limit is up to 30 years old. National student and teacher evaluation system 31. Created in 1993, the national student evaluation system in Argentina (Operativo Nacional de Evaluación, ONE) faced technical and implementation weaknesses. Its comparability and technical robustness were questioned by scholars and experts alike, particularly because of the scarce dissemination of its results.35 For instance, in the last ONE implementation in 2013, data for the primary level was not disseminated, and information was aggregated only at the regional level, with no statistics by public/private sector or urban/rural schools. 32. In an effort to increase transparency and accountability of the education system, the SEE was created in 2016 to obtain and disseminate timely, quality information to better understand student learning at different education levels. The SEE introduced the standardized student evaluation APRENDER to replace ONE and consolidate the national student evaluation system. The APRENDER evaluation: (i) measures trends in the quality of education over time; (ii) uses the information to design strategies focused on improving quality; and (iii) shares the results of the student assessment with provincial authorities, school principals, and the public at large. Since its first implementation in 2016, it has increased coverage of schools (from 88 to 94.4 percent in 2018) and of students within those schools (71 to 78.7 percent in 2018). APRENDER results are reported at the national, provincial and school levels and the SEE has actively adopted strategies for the widespread dissemination of its results through: (i) a password-protected online platform that provides each school with a tailored report detailing school performance and benchmarking the results with similar schools; and (ii) an open online platform that allows for basic statistical analysis, disaggregated at the provincial level. APRENDER has promoted proactive communication with school principals, who have sole access to their specific institution’s report, and its results have been widely disseminated through national media and created public debate on education outcomes. The results reports are also accompanied by general pedagogical and technical recommendations to improve teaching practices. Based on these recommendations and results, schools 35See Ganimian, A., 2013. El INDEC educativo, El Bastión Digital, March 25, 2013 and Ganimian, A. J., 2015. "El termómetro educativo: Informe sobre el desempeño de Argentina en los Operativos Nacionales de Evaluación (ONE) 2005-2013," Working Paper 257736, Harvard University OpenScholar. Page 19 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) and Provinces decide how to use this data and information for improvement. 33. At the higher non-university education level, SEE’s actions are focused on the evaluation of teacher training to improve the quality of and build an evaluation culture in the teaching profession. The ENSEÑAR assessment, implemented for the first time in 2017, is voluntary for students in their final year in teacher training institutes. Its objective is to generate critical information for policymakers on the performance of these institutes and the areas for improvement. In 2018, the SEE launched the design and implementation of ENSEÑAR self-evaluation in 17 Provinces, comprised of toolkits for teacher training institutes based on the results of the ENSEÑAR assessment, which in 2019 will be expanded to three more Provinces. In 2019, the SEE will also design a strategy to implement external assessments of teacher training institutes to complement and improve the self-evaluation processes based on agreements at the federal level. 34. The SEE is also in charge of the evaluation of teacher training scholarships. This merit-based scholarship – Becas Compromiso Docente – is granted to prospective teacher training students based on the evaluation carried out by the SEE. 35. The GoA’s education evaluation system managed by SEE has been successful in several dimensions, but several challenges remain. i) The institutionalization of the APRENDER evaluation by: (i) strengthening and extending the coverage of its implementation; and (ii) strengthening its dissemination strategies. While the coverage of student evaluation has increased at the national level, several Provinces continue to show participation rates as low as 49 percent. ii) Technical strengthening of the APRENDER evaluation through: (i) new assessment instruments; and (ii) data analysis to understand the impact of multiple dimensions on the learning process. iii) In terms of teacher evaluation, there is also a need to institutionalize evaluation processes and consolidate them into a cohesive system. This includes the institutionalization of the ENSEÑAR assessment, and its future implementation. iv) Evaluation processes have traditionally been hindered by resistance from certain political sectors. Support the technical and political dialogue among relevant education actors, stakeholders and Provincial authorities to build a strategy for an external quality assessment for teacher training institutes. B. PforR Program Scope 36. Policy Context. Improving inclusion and quality of secondary and higher education are key priorities for the GoA, as stated within the objectives of the MECCyT’s core documents Plan Estratégico Nacional 2016-2021, Argentina Enseña y Aprende, and the Marco de Organización de los Aprendizajes (MOA) Aprendizaje 2030. Long-term objectives in these strategic policy papers are: (i) universalization of completion of the secondary level, giving greater opportunity to poorest students; (ii) improvements in completion rates and internal efficiency in higher education; (iii) improvements in the quality and relevance of learning; and (iv) promotion of higher productivity for sustainable growth through enhanced access to employment for youth and lifelong learning. These objectives seek a substantial decrease in existing outcome inequalities among Provinces and socioeconomic groups, and among genders. Page 20 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 37. Operation Scope and Expenditure Framework. The scope of the proposed Operation would be to support a subset of Government Programs (GP) from 2019 to 2023, clustered around two results areas under a PforR component (US$300 million) and a Component that would use the IPF instrument (US$40.1 million). The total cost of the activities under the Operation is estimated at about US$1.133 billion, of which US$341 million would be financed through the Bank loan. Table 2 presents the Operation’s cost by component. The results chain (Table A3.2 in Annex 3) illustrates how the Operation’s activities and products would contribute to achievement of the Program Development Objectives (PDO) and maps the theory of change behind the programs. Table 2. Cost by Component US$ Program Component (Bank funds) Jun 2019 2020 2021 2022 Jul 2023 Total Component 1: Support Government Programs 240 244 244 244 121 1093 (PforR) (70) (119) (63) (48) (0) (300) Results Area 1: 240 240 240 240 120 1080 PROGRESAR Scholarships (70) (115) (60) (45) (0) (290) Results Area 2: 0 4 4 4 1 13 National Student and Teacher Evaluation Systems (0) (4) (3) (3) (0) (10) Component 2: IPF to Improve Design and 5 10 20 3 2 40.1 Implementation (5) (10) (20) (3) (2.15) (40.15) 0 0 0 0 0 0 Front-end fee (0.85) (0) (0) (0) (0) (0.85) 245 254 264 247 123 1133 Total (75.85) (129) (83) (51) (2.15) (341) 38. The PforR Program Scope. The scope of the PforR Program supports two Results Areas of the GP, selected as priorities by the MECCyT, to foster access and completion at the secondary and higher education levels and to strengthen the student and teacher evaluation systems in the education sector. To reach these objectives, MECCyT is implementing two programs, which together conform the PforR Program: (i) Results Area 1: the PROGRESAR Scholarship program for 18 to 24-year-olds (US$290 million); and (ii) Results Area 2: the APRENDER student learning assessment system and the ENSEÑAR teacher training evaluation for recent graduates (US$10 million). These programs are financed under MECCyT’s core budget, currently representing around half of the total. The analysis of the current program budgetary codes, based on 2019 budget figures, shows that the projected amount of eligible expenditures under these codes (US$1.133 billion) is enough to reconcile the loan funds (US$300 million) by Program closing. A continuation of their implementation is envisaged for at least the next five years. Component 1: Support to PROGRESAR Scholarships, and the National Student and Teacher Evaluation Systems (PforR) 39. Results Area 1. Improving the management and monitoring of PROGRESAR scholarships (IBRD: US$290 million, Total: US$1.1 billion). This Results Area would improve the targeting, management, monitoring and efficiency of PROGRESAR through the financing of PROGRESAR scholarships, and the carrying out of program of activities to: (i) ensure that higher education institutions submit on time the Page 21 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) academic certifications for scholarship applicants, so that their eligibility can be assessed during the inscription period and the conditionality on enrollment can be enforced; (ii) improve the progression of students in their careers; (iii) improve the effectiveness of the strategic career incentives in place in Higher Education; (iv) design and implement an action plan to overcome non-financial barriers for education progression and completion, including the dissemination of tools through a website; and (v) develop an action plan for a phased implementation of an academic certification system for Basic Education scholars. 40. The management and monitoring activities include the coordination of the nationwide annual call for applicants, the application process for the scholarships, the monitoring of eligibility in coordination with ANSES and the education institutions, approval of scholarships, granting of awards based on the incentives scheme, and the coordination of the strategy on non-financial support to scholars. Activities under Component 2 of the Operation would complement this financing to strengthen the design and implementation of the program. 41. Results Area 2. Supporting the National Student and Teacher Evaluation Systems (IBRD: US$10 million, Total: US$13 million). This Results Area would support the Borrower’s national evaluation systems for students, teachers and teacher training institutes, through the financing of annual census- based APRENDER student evaluations for the period 2020-2023, teacher evaluations for teacher training scholarships, and teacher and teacher training institute evaluations under ENSEÑAR, and the carrying out of a program of activities to: (i) increase the coverage of APRENDER, focusing on the Participating Provinces that report lowest coverage; (ii) support the use of the APRENDER summary reports by school principals to enhance pedagogical practices; and (iii) implement an evaluation system for teacher training institutes. 42. The activities supported include, both for ENSEÑAR and APRENDER assessments, the design of the evaluation instruments, the production and distribution of evaluation materials, training of staff that support and implement evaluations at the national level, software design and development for databases and data processing of the evaluations, design and production of general and specific evaluation reports, design of communication and advocacy materials for dissemination of evaluation outcomes and implementation of that dissemination. C. PDO and PDO Level Results Indicators 43. The objectives of the Operation are to (i) reduce dropout rates in basic and higher education among the most vulnerable, and (ii) strengthen the Borrower’s education evaluation systems. 44. To track progress towards the PDO, the Operation would include the following Results Indicators: i) Average dropout rate of ASISTIRÉ schools in selected provinces. ii) Persistence in the first two years of higher education for the most vulnerable youth. iii) Percentage of students assessed through APRENDER evaluation nationwide. - Percentage of students assessed through the APRENDER evaluation in the Provinces with lowest coverage. iv) Implementation of a diagnostic evaluation system for teacher training institutes. Page 22 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 45. The Operation is expected to reduce dropout rates in basic and higher education through PROGRESAR Scholarships and the implementation of ASISTIRÉ. The strengthening of the evaluation systems would be achieved through the support to APRENDER and ENSEÑAR, and the implementation of an evaluation system for teacher training institutes. D. Disbursement Linked Indicators and Verification Protocols 46. The results-based financing instrument (US$300 million) would disburse resources conditional on satisfactory achievement of a series of DLIs described in Table 3, as defined in the verification protocols (Annex 2). Each DLI would reflect intermediate outcomes, outputs, or institutional changes that are critical to achievement of the PDO. These were selected and weighted based on their technical relevance and likelihood of achievement. Most DLIs are designed to be disbursed proportionally to the quantitative achievement of the results (this is, scalable). 47. Rationale for DLIs under Results Area 1. DLI 1 reflects the GoA’s efforts to strengthen the Program’s monitoring and evaluation (M&E) and it is the foundation of the transition from a transfer program with no monitoring to a merit-based education scholarship program. Verification of eligibility is now the foundation of the system, without which further outcomes cannot be achieved. This DLI would aim to ensure that all education institutions provide the academic information of scholarship applicants so that their eligibility can be assessed during the inscription period and the conditionality can therefore be enforced. Before 2018, recipients would obtain 80 percent of the transfer without verification of eligibility requirements. In 2018, the DNB received the academic information of 75 percent of applicants to PROGRESAR Higher Education to assess their elegibility. The remaining 25 percent of applicants was not eligible for the scholarships as a result of this lack of information. This serves to show that the DLI would also have a direct impact on applicants, as stronger verification ensures that all applicants are assessed to obtain the scholarship. 48. DLI 2 is strongly linked with the PDO, capturing the progression of scholars in their careers, measured by the reenrollment of students in both the scholarship and their studies. This DLI measures retention and progression given that the renewal of the scholarship requires advancing in the career trajectory by passing at least 50 percent of the theoretical study plan. 49. DLI 3 would focus on following the effectiveness of the strategic career incentives that were put in place in 2018, which are expected to have an impact on the students’ labor market outcomes. Strategic Careers36 include basic and applied sciences, engineering, and nursing and were established after a supply and demand gap analysis by region. They seek to meet specific professional needs in each region to support priority areas for development. 50. DLI 4 is key to broaden PROGRESAR Scholarships’ support to the most vulnerable students by providing not only financial support, but a comprehensive package that includes non-financial tools for education progression. This DLI would be linked to the implementation of key interventions to increase and improve non-financial support to scholars. It uses diagnostic studies of scholars’ barriers to progress and completion of their studies and implements an action plan to give them specific tools to overcome 36 A complete list of the Strategic Careers is included in the Project’s Operations Manual. Page 23 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) them. These evidence-based tools would be disseminated through the PROGRESAR Scholarships website, which is currently the direct communication channel with scholars. 51. DLI 5 would follow the implementation of an academic certification information system for basic education scholars, which is currently carried out directly by the applicants to PROGRESAR Basic Education. The applicants complete their application and academic certification themselves and present it to a local ANSES office. This limits the M&E of eligibility requirements for these scholars. The new system would streamline the application process, and facilitate further engagement between the DNB and the applicants. 52. It is worth noting that several of these DLIs involve climate co-benefits related to shock- mitigation, poverty alleviation, and climate mitigation and adaptation. DLIs 1 and 2 bring about climate adaptation co-benefits for low-income recipients who are particularly vulnerable to climate, as the scholarships improve their food security (about 12,586 families were lifted from extreme poverty in 2018 due to PROGRESAR scholarships). DLI 3 brings about climate mitigation and adaptation benefits, as Strategic Careers include sectors such as renewable energy, landscape planning, sustainable urban development, renewable natural resources engineering for arid areas, etc. All Strategic Careers are high- skill careers and diversify away from livelihoods that are threatened by climate change impacts such as agriculture. Overall, PROGRESAR scholarships build the long-term resilience of targeted beneficiaries. 53. Rationale for DLIs under Results Area 2. DLI 6 supports the consolidation of the APRENDER evaluation by increasing its coverage in the Provinces that report the lowest participation and in the secondary level assessment, which also reports the lowest coverage. 54. DLI 7 is relevant as it focuses on the dissemination and use of the student evaluation results to improve learning. Specifically, it focuses on the use of student learning evaluation results to enhance pedagogical practices and, therefore, learning. It follows the share of school principals and teachers that report using the APRENDER Summary Report to enhance pedagogical practices in the school. By focusing on the use of the results, it also captures access to the results by school principals and teachers. 55. DLI 8 follows the implementation of a diagnostic evaluation system for teacher training institutes. The SEE is leading a process of self-evaluation for the teacher training institutes, launched in 2018, which represents the first phase towards establishing an institutionalized evaluation system. The results supported under this DLI, namely the design of an action plan for evaluation and dissemination of results and the piloting of the evaluation, constitute the next key steps in this institutionalization. Page 24 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Table 3. Disbursement-Linked Indicators, by expected calendar year of completion Indicative timeline for DLI achievement (DLIs eligible for partial achievement are marked ¨Scalable¨) Total DLI DLI as DLI DLR137 DLR2 DLR3 DLR4 DLR5 DLR6 DLR7 Amount % of Baseline (US$M) Total Oct 2019 Apr 20 Oct 20 Apr 21 Oct 21 Apr 22 Oct 22 Results Area 1: Improving the Management and Monitoring of PROGRESAR Scholarships 80% 85% 90% 95% DLI 1: Percentage of higher education scholarship candidates whose academic $100 M 33% 75% $35 M $25 M $25 M $15M information has been provided by their (Scalable, (Scalable (Scalable (Scalable academic institution min: 75%) min: 80%) min: 85%) min: 90%) 30.7% 32.7% 34.7% 36.7% DLI 2: Percentage of higher education scholars that have renewed the $80 M 27% 28.7% $25 M $20 M $20 M $15 M scholarship (Scalable, (Scalable (Scalable (Scalable min: 28.7% min: 30.7%) min: 32.7%) min: 34.7%) 31.3% 33.3% 35.3% 37.3% DLI 3: Percentage of first-year higher education scholars enrolled in Strategic $40 M 13% 29.3% $10 M $10 M $10 M $10 M Careers (Scalable (Scalable (Scalable (Scalable min: 29.3%) min: 31.3%) min: 33.3%) min: 35.3%) Website with Diagnostic study information to DLI 4: Tools to diagnose and overcome and action plan guide the non-financial barriers to education published $60 M 20% No application for progression and completion designed and scholarships implemented $20 M $40 M Results report of Action plan for the pilot of the the certification certification DLI 5: Academic certification system for system for basic system for basic $10 M 3% No education education basic education scholars implemented $5 M $5 M 37 Disbursement-linked Result (DLR) is the target of a DLI for the dates indicated. Page 25 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Results Area 2: Supporting the National Student and Teacher Evaluation Systems 65% 50% 67% 56% DLI 6: Percentage of basic education primary secondary primary secondary (2018) students assessed through the APRENDER $4.5 1% evaluation in the Participating Provinces 47.8% $2.5 M $1 M $1 M with the Lowest Coverage secondary (Scalable (Scalable (Scalable (2017) min: 47.8 %) min: 65%) min: 50 %) 45% 55% DLI 7: Percentage of school principals using the APRENDER Summary Report to $3 1% 0% $2 M $1 M enhance pedagogical practices (Scalable) (Scalable) Action plan for Pilot of the evaluation and evaluation carried DLI 8: Diagnostic evaluation system for dissemination of out $2.5 1% No results teacher training institutes implemented $1.5 M $1 M TOTAL (US$ Million) $300 M 100% $70 M $22.5 M $96.5 M $6 M $57 M $6 M $42 M Page 26 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) E. IPF Component 56. Component 2: Strengthen the Borrower’s Institutional Capacity for the Design, Implementation, and M&E of PROGRESAR, ENSEÑAR, APRENDER, and ASISTIRÉ (US$40.1 million). The objective of this Component would be to secure the long-term implementation sustainability of the GP supported in Component 1, and to support the scale-up and evaluation of a dropout prevention program. This component would provide support to the MECCyT’s existing budget, with the possibility of crowding in resources during implementation. The Component would improve the quality of implementation and support improvements in the programs’ design to increase their effectiveness. 57. Subcomponent 2.1: Strengthen PROGRESAR Scholarship Program (US$11.0 million): This subcomponent would strengthen PROGRESAR through activities to improve its information management system, support scholarship beneficiaries in their education trajectories with tailored interventions for the most vulnerable, and evaluate results, including but not limited to: i) Improving the information and management systems to enhance the application process, and to monitor eligibility requirements, and the certification of regular attendance of students in Basic Education receiving support from PROGRESAR; ii) Introducing a national helpdesk to support students and institutions in the application process; iii) Carrying out studies to estimate PROGRESAR scholarships’ coverage and develop a prioritization strategy to select beneficiaries; iv) Designing and implementing interventions to close information gaps; v) Carrying out a diagnosis of the main non-financial barriers that students face to progress in their studies and graduate; vi) Developing tools for students to close critical cognitive and non-cognitive skills gaps through evidence-based interventions; and vii) Piloting a mentoring program, jointly with NEXOS. 58. Subcomponent 2.2: Support the SEE (US$13.0 million): The Subcomponent would include institutional strengthening of the evaluating units of Participating Provinces that are responsible for local implementation, through technical assistance for, inter alia, the following activities: i) Implementing APRENDER by SEE and Participating Provinces; ii) Adding new modules to APRENDER to measure, inter alia, SES and household and family- related variables; iii) Conducting studies to analyze APRENDER data; iv) Developing tools to assess teaching practices, including tools for self-evaluation and improvement, and the adaptation and piloting of the TEACH Tool for class observation; and v) Designing instruments for self and external evaluation of teaching training institutes. 59. Subcomponent 2.3: Strengthen the expansion and implementation of ASISTIRÉ (US$14.4 million) through, inter alia: Page 27 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) i) Scaling up ASISTIRÉ in selected vulnerable schools nationwide by: (a) strengthening the capacity of the national and provincial-based teams, including hiring and training key staff; and (b) acquiring key inputs for ASISTIRÉ’s implementation, such as tablets with connectivity; and ii) Revamping ASISTIRÉ by: (a) improving accuracy of early warning systems; (b) strengthening the delivery and evidence-based content of ASISTIRÉ interventions; (c) strengthening the institutional capacity of stakeholders and SEE staff; (d) reinforcing the channels for family- school communication; and supporting impact and process evaluations. 60. Subcomponent 2.4: Strengthen the Institutional Capacity of MECCyT (US$1.7 million). This Subcomponent would strengthen the institutional capacity of MECCyT to manage, administer, and supervise the implementation of the Operation, including, but not limited to, support on: (a) monitoring and evaluation; (b) development of a beneficiary feedback mechanism; (c) the carrying out of independent audits for the Operation; and (d) carrying out DLR compliance verification. 61. A detailed description of the activities under this Component is provided in Annex 8. III. PROGRAM IMPLEMENTATION A. Institutional and Implementation Arrangements 62. The Program would be implemented over a four-year period, with effectiveness expected in July 2019 and closing in June 2023. 63. The Operation would be carried out through the MH and MECCyT. The MH, through the Directorate of Special Programs and Projects with Wide Sectoral Scope (Programas y Proyectos Especiales con Enfoque Sectorial Amplio, DPPEESA), would oversee overall implementation of the Operation and would be responsible for all actions related to disbursements and withdrawals for the PforR Component. The MECCyT, through the DNB (who will act as the PIU of the Operation), would be responsible for the implementation of activities related to PROGRESAR Scholarships and for the general coordination of the Operation, including fiduciary aspects, technical and operational decision-making, and monitoring and evaluation. The SEE would be responsible for student and teacher evaluations, including activities under APRENDER and ENSEÑAR. The Secretariat for Educational Management (Secretaría de Gestión Educativa, SGE) would be responsible for all activities related to ASISTIRÉ through its national coordination team (Equipo Nacional de ASISTIRÉ, NAT). Further specifications for these roles and responsibilities are detailed in the Project’s Operations Manual (POM). 64. Project Operation Manual (POM). The Borrower has prepared and adopted an acceptable POM that covers detailed aspects of project design, implementation and processes. It also defines stakeholders’ roles and responsibilities and clear monitoring, reporting and evaluation mechanisms. 65. Implementation Arrangements. The POM describes the structure of the PIU, responsible for oversight and coordination of the Operation. Additionally, the technical units responsible for the implementation of program activities will be strengthened through the Operation lifespan in line with the Page 28 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) implementation requirements. The DNB national team will include new roles to strengthen implementation capacity. The SEE will include five additional professionals to the team to cover technical, communications and administrative tasks. Finally, the NAT will increase the national team with one national coordinator, six coordinators to oversee Provincial implementation, one professional in IT, and three monitoring and evaluation professionals. B. Results Monitoring and Evaluation 66. Monitoring and evaluation. The M&E of the education system in Argentina builds on two pillars: (i) the main source of statistical information for the education sector is the Relevamiento Anual (RA), an annual school census that gathers administrative data for all schools in the country and collects information on enrollment, internal efficiency, teachers and school organization, which is managed by the Directorate of Information and Education Statistics (DIEE); and (ii) APRENDER, the annual national learning assessment that covers all basic and secondary schools of Educación Común, managed by the SEE. At the higher education level, some basic administrative data is collected in the RA, but there is no centralized information on student trajectories. For the university level, the University Information System (Sistema de Información Universitaria, SIU)38 centralizes information systems available to universities. PROGRESAR Scholarships use some of these systems (SIU-Guarani) to monitor eligibility criteria of recipients. 67. National household surveys are a reliable source of information on education trajectories and outcomes. The Permanent Household Survey (Encuesta Permanente de Hogares, EPH) is the principal household survey in Argentina, carried out quarterly since 1970 by the National Institute of Statistics and Censuses (INDEC). By design, the EPH is representative at the urban level, and gathers demographic, educational, labor and socioeconomic data. In absence of a longitudinal survey, the EPH provides relevant information on the education outcomes the Operation aims to impact. The information is public and comparable over time. 68. Most of the selected PDO and intermediate results indicators and DLIs are currently tracked by DIEE, DNB, SEE, or SGE, and one PDO indicator is based on national household surveys. However, most of the indicators are not part of the usual M&E system at the national level and require the establishment of a regular process of requesting and compiling these indicators to report them to the project. This would be a central task of the M&E unit of the PIU. 69. MECCyT information systems provide credible, robust data of the supported programs. Annex 3 presents a technical assessment of the information system of PROGRESAR Scholarships, which will be used to track DLI 1, DLI 2 and DLI3. SEE’s DLI 6 and DLI 7 are tracked through the APRENDER assessment information system, whose assessment is also presented in Annex 3. 70. DLI verification. Verification of the progress towards achieveing the Operation’s objectives and DLIs would be reviewed and assessed by a Verification Agent (VA) according to the verification protocols (Annex 2). Verification will follow a phased approach: (i) For DLR1, individual consultants would be hired to ensure timely verification; (ii) for DLR2 to DLR7 a selected agency will carry out verification. 38 The SIU was created through the support of a World Bank loan between 1996 and 2003. Page 29 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 71. Reporting. The Borrower, through the PIU, would monitor and evaluate the progress of the Operation and prepare periodical advance reports, in line with the provisions of Section 5.08 of the General Conditions and on the basis of indicators agreed in the results framework and the POM. As explained above, The PIU would submit the DLI achievement Report to the MH, who will review it and submit it to the Bank. C. Disbursement Arrangements 72. Disbursement of Bank loan proceeds will be made at the request of the Borrower upon achievement of Disbursement-Linked Indicators (DLIs). Some DLIs (see DLI matrix) are scalable, thus allowing for disbursements to be proportional to the progress towards achieving the targeted DLI value. Other DLIs are not scalable, as the indicators relate to actions that are either achieved or not. 73. Verification protocols. Verification of the Prior-Results DLIs will be undertaken by individual consultants. The Borrower will then either contract a consulting firm (to be paid from the proceeds of the IPF Operation) or a government agency (Auditoría General de la Nación, AGN, or Sindicatura General de la Nación, SIGEN, for example), that has the capacity to provide independent verification for the remaining DLIs, to ensure credible verification of their achievement. The Verification Agents selected will need to be acceptable to the Bank and will thus be assessed by the Bank to ensure it/they, has/have the necessary independence, experience and capacity of ensuring credible verification. 74. Results achievement notification. The MH will notify the Bank of DLI achievement, supported by the relevant evidence and documentation. Following the Bank's review of the complete documentation, including any additional information considered necessary to confirm the achievement of the DLI results, the Bank will confirm the achievement of the DLI(s) and the level of Program financing proceeds available for disbursement against each DLI. 75. Disbursement requests (Withdrawal Applications) will be submitted to the Bank by the MH using the Bank's e-disbursement (Client Connection) system. A copy of the Bank's official communication, confirming the DLI achievement, should be attached to the disbursement requests. 76. Under the PforR Operation, funds will be disbursed from the Bank to the Borrower, in U.S. dollars into an account indicated by the Borrower and acceptable to the Bank, based on the achieved DLIs. An amount of US$70 million of the PforR operation will be disbursed on account of the DLIs met by the Borrower between the date of the Program Concept Review and the date of the Legal Agreement (Prior Results). The Operation does not intend to use the functionality of PforR Advances. 77. The Bank may decide, without formally extending the Closing Date, to disburse or approve the use of proceeds of the loan for Withdrawal Applications received within six months after the Closing Date for DLIs achieved by the Borrower before the Closing Date. Upon the Borrower’s request, the Bank may decide to extend the period for receipt of such Withdrawal Applications. 78. Disbursements under IPF Component. The disbursement of funds from the Bank to DNB, will be processed in accordance with Bank procedures as stipulated in the Legal Agreement and Disbursement and Financial Information Letter (DFIL). The primary disbursement method will be Advances. The Project Page 30 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) will be also able to process Reimbursements and Direct Payments under the Project, if required. All payments will be made using the e-SIDIF and UEPEX systems, once payment obligations have been committed and verified. The implementation of Subcomponent 2.2 of the IPF, would also involve the transfer of funds to Provincial authorities and decentralized implementation of certain activities. Provincial Executing Units (UEPs) already established under the Bank’s previous Rural Education Improvement Project–PROMER I (P070963) and current Rural Education Improvement Project-PROMER II (P133195), will be responsible for the provincial implementation, under the coordination/supervision of DNB. 79. The General Conditions state that if after the Closing Date the Borrower fails to provide the Bank evidence satisfactory to the Bank, that the withdrawn loan balance does not exceed the total amount of Program Expenditures (payments made on or after the Signing Date but before the Closing Date), the Borrower shall, upon notice from the Bank, promptly refund to the Bank such excess amount of withdrawn balances.39 IV. ASSESSMENT SUMMARY A. Technical (including program economic evaluation) Economic Analysis 80. The Operation’s benefits are substantial. An economic analysis of PROGRESAR Scholarships and ASISTIRÉ shows positive results in both the benefit/cost analysis and the Internal Economic Rate of Return (IERR) under several scenarios.40 A detailed Economic and Financial Analysis is available in Annex 3. 81. Cost-benefit analysis. Using estimations of costs and expected benefits of each component and a discount rate, the yearly cost per student is brought to the present, as well as the whole stream of yearly future benefits. With a conservative discount rate (5 percent), the Net Present Value (NPV) of the income benefits of the average Operation beneficiary are compared with its cost, and the IERR is computed. Table 4 summarizes the results, which suggest that benefits from the PROGRESAR Scholarship exceed 7 times its cost in Basic Education and Technical Professional Training and reaches almost 6 times its cost in higher education, while benefits from the expansion of ASISTIRÉ are more than 4 times its cost. All programs yield positive and significant IERR (between 13 and 19 percent). Both the cost/benefit analysis and the IERR remain positive under several sensitivity analyses. Table 4. Results of the Economic Analysis Benefit/Cost Ratio Internal Rate of Return (IRR) PROGRESAR scholarships Basic Education 7.1 19 % PROGRESAR scholarships Higher education 5.8 17 % Scale-up of ASISTIRÉ 4.0 13 % 39 Alternatively, in the last year of the Program, the MH, in coordination with the Bank, may reconcile the audited Program Expenditure with the DLI amounts disbursed by the Bank to date, and any shortfall in the Program expenditure in relation to DLI disbursement could be adjusted from the final DLI claim. 40 These represent 92 percent of the Project’s cost. Results Area 2 was not included in the analysis. Page 31 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Technical Analysis 82. Technical Soundness. This section presents a review of the available literature to provide further evidence that the Operation’s design is technically sound, based on international best practices, and in line with evidence-based interventions. A detailed analysis can be found in Annex 3. 83. A large body of evidence finds that scholarship programs tend to improve school participation outcomes (enrollment, attendance, completion and dropout), and can, in some cases, improve student learning. The 2018 improvements and modifications to PROGRESAR Scholarships, which promote more inclusion, merit-based benefits, and Strategic Careers in higher education, have also been identified in the literature as effective interventions for education outcomes. Evidence suggests that interventions that increase the benefits of attending school (such as scholarships) increase time in school,41 while those that increase the benefits of higher effort and better academic performance (such as merit-based scholarships) improve learning outcomes. Only merit-based scholarships, however, have consistently worked to increase test scores across multiple contexts. Evidence on the effect of merit-based scholarships for higher education students is also promising, although the results are mixed.42 In particular, a few rigorous recent evaluations have shown large impacts. For instance, Angrist et al (2017) show that scholarships significantly boosted college enrollment and persistence. Four years after award receipt, randomly- selected scholarship winners were 13 percentage points more likely to be enrolled in college. 84. Non-financial support is also critical to strengthen the transition to higher education. This is particularly relevant for disadvantaged students who are the first generation in their family to have access to higher education and may lack support to make informed decisions and basic skills to succeed in higher education. Information-based programs that provide key data to students and parents have been shown to improve career choices in higher education. A recent randomized control trial (RCT) in Chile shows that providing information to future higher education students, through an online platform, on the potential earnings (and fee costs) corresponding to different enrollment choices, helped students from particularly low-socioeconomic contexts enroll in degrees with higher returns.43 The introduction of additional incentives for “strategic” careers (most high in demand) aimed at guiding the decision of new students could reinforce the effects of providing this information. Evidence from basic education also highlights the potential of information-based interventions. An RCT in the Dominican Republic finds that providing students with information on earnings differences by education reduced dropout by 7 percent in the subsequent year, and increased school completion by 0.2 years.44 85. Socioemotional support and low-cost behavioral interventions are also essential to support students struggling with the more complex curriculum and environment of higher education. During the transition to a new education level, students can face frequent social setbacks and feelings of anxiety and isolation. Interventions that provide self-administered short programs on motivation and belonging have 41 Glewwe & Muralidharan (2015,), Damon et al (2018) 42 Dynarski (2008), Castelman (2014), Scott-Clayton (2011b), Scott-Clayton & Zafar (2016), Bettinger et al (2016); DesJardins & McCall (2014), DesJardin et al, (2010), Sjoquist & Winters (2012, 2015) 43 Neilson et al (2018) 44 Jensen (2010) Page 32 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) shown that college students benefit when they understand that challenges in the transition to college are common and improvable, and thus, that early struggles need not portend permanent lack of belonging or potential.45 In addition, a recent RCT showed that providing services that helped students set goals and acquire financial aid between their transition from high school to college increased college enrollment by 3 percentage points (even more for low-income students).46 New experimental evidence in the U.S. found that brief, one-time messages to students increased recurring tutoring attendance, leading to relevant changes in students’ study habits.47 86. Standardized student assessments are a key ingredient to shift the focus to learning and support students’ trajectories. The benefits of education—cultural, economic, and social—accrue to society only when learning occurs.48 For governments to learn what policies worked to improve learning and to avoid “flying blind,” they need credible data.49 The Operation would support the strengthening of the student evaluation system. Recent research found that the sole fact that countries adopt a standardized student assessment and make results of the evaluations public is associated with improvements in student achievement.50 As part of the Operation, the SEE would improve the report on the results of the assessment that each school receives to provide information to principals on what students mastered and what competencies they need to improve. A recent RCT in La Rioja, Argentina found that providing schools with such information (together with a short training) improved student achievement between 0.28 and 0.38 of a standard deviation, which represents around one additional academic year of learning.51 87. The assessment of teachers’ professional development is the second cornerstone of the education evaluation system. Improving teacher pre-service education is crucial to improve student learning. Teachers are the most important school-based determinant of student learning.52 Research in Peru and Ecuador shows that teacher subject content knowledge53 and pedagogical practices54 in the classroom are among the most important factors to explain teachers’ value added. The Operation would strengthen the evaluation system of soon-to-be teachers in Argentina in all these dimensions, by (i) consolidating the new content and pedagogic knowledge assessment; and (ii) complementing it with an assessment of what the teachers do in the classroom. This information would be used to provide feedback to teacher training institutes to improve their performance, to develop a suite of tools for self-evaluation, and to inform the design of tools and strategies for external evaluation initiatives in the future. Finally, as part of the technical assistance, the Operation would develop and pilot a strategy to integrate the use of student assessments and observations of what teachers do in the classroom with the systems of professional development of principals and teachers. This is in line with current research on best practices in professional development.55 45 Yaeger et al (2016;); Walton & Cohen (2011) 46 Castleman & Page (2015) 47 Pugach & Wilson (2018) 48 OECD (2010) 49 World Bank (2018) 50 Bergbauer, Hanushek and Woessmann (2018) 51 De Hoyos et al. (2018) 52 Hanushek and Rivkin (2010); World Bank (2018) 53 Metzler and Woessmann (2012) 54 Araujo et al. (2016) 55 See Popova, Evans, & Arancibia (2016); Yoshikawa et al. (2015); Bambrick-Santoyo (2012); Fryer (2017) Page 33 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 88. The current design of ASISTIRÉ is aligned with best international practices, and the planned complementary interventions would reinforce its effectiveness. Extensive research has shown that it is more cost-effective to act early to prevent dropout than to try to reengage out-of-school youth in the educational system. EWS that rely on administrative data, even when they are based on basic algorithms, have proven to have high predictive power of dropout, allowing identification of children at risk and early action.56 A recent RCT in the U.S. showed that an EWS that relied on a few variables routinely collected (chiefly absences, grades, and behavioral problems), coupled with close follow-up of students at risk by the school team (case workers), reduced chronic absenteeism and course failure by almost 20 percent after only one year of implementation.57 The improvements in the program’s design, which include strengthening the evidence-base of key interventions (such as closing skills gaps and teenage pregnancy prevention) would further boost the effectiveness of the program.58 Similarly, evidence has also shown that interventions reinforcing the channels for family-school communication are highly cost-effective. For instance, providing information to parents on their children’s absenteeism and grades (through automated text messages, emails or phone calls) improves significantly student outcomes and reduces absenteeism.59 Lessons Learned 89. The Operation’s design is informed by lessons learned from other Bank projects , in particular the Colombia Access and Quality in Higher Education Project - PACES (Loans 8836-CO and 8701-CO, P160446), which provided lessons on strategies to increase the enrollment of students from disadvantaged socioeconomic backgrounds in quality tertiary education programs, as well as on processes and institutional agreements that promote quality assurance systems in tertiary institutions. Other Bank projects including components to support vulnerable students’ trajectories towards higher education helped the design process, including the Ecuador Transformation of the Tertiary Technical and Technological Institutes Project (Loan 8667-EC, P157425). 90. Previous experience and knowledge generated by projects that promoted access of vulnerable youth to higher education show financial incentives alone cannot support students’ trajectories, unless they are also supported by initiatives that promote resilience as a key socioemotional skill and various kinds of remedial education tools. The Operation builds on extensive previous experience from Bank analytical work building knowledge and containing initiatives to foster socioemotional skills, such as Brazil, Skills and Jobs (P133162), Peru New umbrella for Education, Skills & Employment (P147264), and the ongoing pilot initiative in the Province of Buenos Aires (PBA) within the framework of PROMER II. 91. The Operation would also benefit from lessons learned in Argentina in relation to the dissemination of student learning assessments and the process to build a culture of evaluation. The Analytical Services and Advisory task Supporting Evidence-Based Policy Interventions in Argentina (P164046), supported the dissemination of standardized test scorecards to school principals in schools in 56 Knowles (2015); Adelman et al (2016) 57 Faria et al (2017) 58 Cook et al., 2014, Heller el al (2016), Wang et al (2016). Other studies on effective coaching/tutoring interventions in K –12 education include Dobbie & Fryer (2013), Fryer (2014), Kraft (2015), Kosse et al. (2016), and Ander, Guryan, & Ludwig (2016); Banerjee et al. (2007); Carrell & Carrell (2006); Carrell & Carrell (2006); Carrell & Carrell (2006) 59 Bergman (2015), Rogers & Feller (2018), Berlinski et al (2016), Bennett & Bergman (2018) Page 34 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) vulnerable contexts in the provinces of Buenos Aires, Salta and La Rioja, as well as the previous collaboration with the SEE in the ongoing PROMER II, have also highlighted good practices in the empowerment of educational institutions’ self-evaluation capacity and provided lessons on political economy towards the construction of a solid national evaluation culture. 92. Preparation would also benefit from the support provided by the Bank to the AUH national program through the ongoing Children and Youth Protection Project (Loans 8903-AR and 8633-AR, P158791), which has helped close coverage gaps for vulnerable populations by supporting the ANSES in obtaining missing information on beneficiaries, thus allowing more children to enter the eligibility screening process. The Project aimed at exploiting new technologies to reduce paperwork to prove compliance, and reduced barriers through early alerts directed to beneficiaries and service providers, innovations aligned with the challenges faced by PROGRESAR and ASISTIRÉ. Assessment of the Program Expenditure Framework and Fiscal Sustainability 93. The Program expenditure framework is adequate to support the PforR. The Programs’ actual and expected expenditures are much greater than the amount of Bank financing, which represents around 25 percent of the total. The Program has a small aggregate impact on Argentina’s fiscal accounts and does not represent a sizeable risk to the fiscal consolidation program. The yearly budget of the Program only amounts to approximately 0.25 percent (US$250.6 million) of total government expenditures, and most of the Bank funding would represent a change in funding source to the existing budgetary lines of the MECCyT. Furthermore, even under a scenario where the new, more strict targeting criteria of the scholarship program—which represents the bulk of the Program’s expenditures—is eliminated, the total Program cost would still represent approximately 0.4 percent of the total government expenditures. The financial sustainability of the Program is also reinforced by the prioritization of social programs involving cash transfers by the Government in the current context (further details are presented in Annex 3). 94. In the context of the IMF Stand-By Arrangement, the federal government committed to achieving a primary fiscal balance in 2019 and a 1 percent of GDP surplus in 2020. This represents a fiscal effort as high as 3 percentage points of GDP, given the 2018 deficit of 2.7 percent of GDP and the impact of past measures on the 2019 balance (e.g. pensions indexation, tax cuts). Achieving fiscal consolidation targets is key for the GoA’s fiscal program to stabilize and reduce its debt-to-GDP ratio, which reached close to 88 percent in 2018. Fiscal room will be thus limited at the federal level. An additional reduction in public investment and subsidies and a transitory increase in the export taxes are expected to contribute the most to fiscal consolidation. These efforts will be accompanied by decentralization of spending responsibilities to provinces of part of electricity, transport, and water subsidies. A stable macroeconomic environment and a recovery in economic activity, with a positive impact on public revenues, will thus be key to achieving fiscal targets without further expenditure adjustments. 95. In this context, the proposed program would add up to about 0.25 percent (US$250.6 million) of total government expenditures, and most of the Bank funding would only represent a change in the funding source within existing budgetary lines of the MECCyT. In a context of strong fiscal consolidation, there is a risk that a number of programs including the proposed operation could be affected unless there is a commitment by the authorities to prioritize social programs involving cash transfers and ringfence these expenditures. This risk would be mitigated since social investments are at the heart of GoA’s Page 35 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) commitment to protect the vulnerable. B. Fiduciary 96. The Procurement and Financial Management (FM) systems’ capacity and performance, with the implementation of the proposed mitigating measures and agreed actions to strengthen the systems (which are reflected in the Program Action Plan, PAP), are adequate to provide reasonable assurance that the Program funds will be used for the intended purposes, with due attention to the principles of economy, efficiency, effectiveness, transparency, and accountability. 97. The overall integrated fiduciary risk rating is considered Moderate. The key fiduciary risks to the development outcomes of the Project that underpin the Moderate risk rating are as follows: (i) unfamiliarity of the Borrower with the PforR Instrument; (ii) special conditions included in the Standard bidding documents for goods, non-consulting and consulting services that do not indicate any thresholds in order to determine the weight of quality and cost bids for the total score; (iii) some inadequacies in contract management; and (iv) the risk of awarding a contract to a suspended or debarred firm or individual suspended by the Bank. 98. The proposed systems-and capacity-strengthening and/or mitigation measures, to address the above risks include the following: (i) close implementation support to be provided by the Bank’s fiduciary teams, coupled with the preparation of a POM; (ii) the implementing agency, with the assistance of the requesting areas, shall determine the maximum thresholds related to quality (technical) and cost (financial) weight in the special conditions included in the bidding documents; (iii) preparation of a manual, as a reference document for all staff involved in contract management and the training of relevant people accordingly; (iv) the Dirección de Contrataciones will indicate in the evaluation reports – in writing – the eligibility verification carried out and on June 30th of each year, the implementing agency will send to the Bank, the list of contracts awarded to firms and individuals; and (v) the external auditors’ Terms of Reference (TOR) will include a requirement to review Program expenditure for such ineligible contracts. 99. For the PforR, the MECCyT will follow the procedures of the national legislation. There are no potential high-value contracts identified under the Program at this moment. The proposed Program is not expected to finance any contract at or above prevailing Operations Procurement Review Committee (OPRC) thresholds, which at the time of the assessment were at minimum US$75 million for goods, information technology and non-consulting services and US$30 million for consulting services. 100. For the IPF, the MECCyT will conduct procurement using the Bank’s “Procurement Regulations for IPF Borrowers,” dated July 1, 2016, revised November 2017 and August 2018, and the provisions stipulated in the Legal Agreement, for the supply of goods, non-consulting and consulting services. At the national level, the MECCyT, through the DNB, would be responsible for carrying out procurement activities, as well as for monitoring and supervising the fiduciary arrangements at the provincial level, which also have structures in place to carry out procurement activities and are currently carrying out procurement activities under PROMER II. Page 36 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 101. A procurement capacity assessment of the PIU was carried out by the Bank. The assessment reviewed mainly the organizational structure for implementing the Project, and the experience of the existing procurement staff. The identified staff is integrated by two specialists with experience in managing the Bank’s and Inter-American Development Bank’s projects and they are familiar with handling procurement processes for low cost and complexity related to consulting services and non-consulting services. The identified staff has less experience in managing international open competitive processes and more complex processes. 102. Based on the specific needs identified to take the responsibility for the procurement function under this Project, the following actions are recommended to reduce risk and facilitate project implementation: (i) the POM will include a detailed description of the roles and responsibilities of the PIU and its interaction with the Technical Units that spells out technical, administrative and contract management tasks and responsibilities of each of them; (ii) the PIU procurement specialist and technical staff should attend procurement training within six month of Loan approval; and (iii) the Bank’s procurement team will closely assist the PIU during project implementation, in particular for more complex activities. . C. Legal Operational Policies Triggered? Projects on International Waterways OP 7.50 No Projects in Disputed Areas OP 7.60 No . D. Environmental and Social 103. The Bank’s Environmental and Social Systems Assessment (ESSA) evaluated that the capacity of the PforR Program systems is adequate to manage the Identified Risks. Argentina has a robust legal framework and an extended public and free education system that provides the opportunity for all students to access all levels of education. Implementation of these laws has been irregular and the effectiveness of the system relies on the capacity of the different provincial ministries of education. Nevertheless, this is a National Program that will be implemented at the federal level and MECCyT has assigned a team within the DNB to supervise the implementation of all environmental and social aspects of this Operation. In addition, it is important to highlight that organized civil society is very active in promoting inclusion in the edcuation system, thus providing an additional layer of monitoring to the system. 104. The ESSA identified groups that have systematically higher dropout rates and lower educational outcomes than the average population. These groups are indigenous peoples, afrodescendants, students with disabilities, and students that are part of the LGBTIQ60 community. The ESSA was prepared considering the core principles set in OP 9.00. It evaluated the potential range of environmental and social effects that may be relevant to the Operation and also assessed the institutional capacity of the 60 LGBTIQ is an acronym that includes lesbians, gay men, bisexual women and men, trans, intersex and queer individuals. Page 37 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) implementing agency and the environmental and social applicable regulatory framework to identify any gaps with the core principles and collectively design the measures to close these gaps. The ESSA did not identify any activity deemed likely to have significant adverse impacts on sensitive and valuable environmental components and processes. 105. In addition, the GoA Prepared an Environmental and Social Assessment (ESA) to identify enviromental and social risks associated with the Technical Assistance component.61 Both analyses were conducted in a participatory manner, including engagement processes with all relevant stakeholders62 from an early stage. Recommended actions resulting from this participatory process helped define a PAP to address issues related to the implementation of the activities under the PforR and an Environmental and Social Commitment Plan (ESCP) to address the risks associated with the activities that would be financed under the IPF component. Finally, the Government also prepared a Stakeholder Engagement Plan, containing the identification of stakeholders, the registry of early consultations, and the proposals for future engagement, which will ensure that the views of stakeholders are taken into account during project preparation and implementation. The ESA was published on the Borrower’s website and the Bank’s public website on April 23, 2019, the draft ESCP and Stakeholder Engagement Plan were published on April 30, 2019, and a final version of the ESCP was agreed during Negotiations.63 106. Both the ESSA and the ESA confirmed that Argentina is a multicultural country, with a large and diverse indigenous population, as well as a significant afro-descendant and migrant population, which is confronted with barriers that lead to higher dropout rates and underachievement due to structural and direct forms of discrimination, including bullying, sexual and other kinds of harassment.64 These documents also confirmed that school attendance and completion rates for people with disabilities are even lower, and that although there is no robust data available regarding LGBTIQ populations’ school attendance or completion rates at the regional level, there is evidence that high levels of discrimination and bullying suffered by these students also result in higher dropout rates.65 107. The Operation has a strong potential to reverse some of these trends and, for that reason, is expected to have only positive impacts on the targeted population. Nevertheless, it is uncertain if these programs are reaching all excluded minorities, as no disaggregated data is available or being produced. For that reason, one key aspect of the PAP and the ESCP will be to strengthen the Program’s outreach strategy and the MECCyT’s capacity to implement it to guarantee that the different programs are reaching 61 Given that there is no Guidance Note on Technical Assistance under the Bank’s ESF, the ESA incorporated the principles laid out in the Interim Guidelines on the Application of Safeguard Policies to Technical Assistance (TA) Activities in Bank-Financed Projects and Trust Funds Administered by the Bank (January 2014) to ensure its compliance with the principles and socioenvironmental standards of the Bank. 62 The Bank Team and the GoA held a first round of consultations with representatives of afro-descendants (February 12, 2019 and March 2, 2019), indigenous peoples (March 22-28, 2019) people with disabilities (February 22 and April 29, 2019) and people from the LGBTIQ community (March 28, 2019), both in the government and civil society organizations. 63 The final version of the ESCP was submitted for Bank disclosure on June 6, 2019. 64 Sexual harassment is higher towards afro-descendant female students as a product of oversexualization by other non- afrodescendant students. Migrant students usually face harassment from Argentinean students and some xenophobe groups that express that they do not have the rights to access public education. 65 For example, a 2007 study showed that 45 percent of transgender students dropped out of school, either due to transphobic bullying by their peers or to being excluded by school authorities. UNESCO, 2016, Out in the open, Education sector responses to violence based on sexual orientation and gender identity/expression. Page 38 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) targeted populations from excluded minorities, in culturally appropriate and timely manners. Experience from elsewhere shows that excluded minorities might not be captured under universal schemes. 108. Gender. Identified gender gaps and a plan of activities to address and measure them is available in Annex 10. In particular, the low participation of women in STEM careers will be targeted with specific evidence-based interventions to close the gender gap and engage more women in these careers. The gap related to gender-specific barriers to education progression will also be targeted through a study to identify the barriers and develop an action plan to tackle them. Both these gaps will be monitored in the results framework, as detailed in Annex 10. 109. The Operation would neither finance nor support any civil works activities. In this sense, the activities supported by the Operation are not expected to have any negative environmental or social impacts. No rehabilitation or construction of new infrastructure or other actions having an impact on the environment would be financed through the Operation. The Operation would be implemented at the national level, over a diversity of jurisdictions with different environmental, social and institutional settings. However, the overall risks and potential adverse environmental impacts are not considered significant. 110. Citizen Engagement. The Operation would have a two-layer Grievance Redress Mechanism (GRM). One layer would focus on PROGRESAR and would be implemented by ANSES in coordination with MECCyT. ANSES has a very active citizen engagement approach and a strong grievance redress mechanism. The team managing scholarship applications has shown flexibility and is open to revising the conditions of application to ensure that vulnerable groups do not face cconscious or unconscious barriers that might prevent them from benefiting from the program. A second layer GRM would be developed under MECCyT to cover the rest of the activities finaced by the Operation. Since MECCyT lacks a formal GRM, the ESCP includes the commitment to create one. The Results Framework also includes an indicator that would track the percentage of cases registered in PROGRESAR’s GRM and their timely resolution. 111. In addition, communities and individuals who believe that they are adversely affected as a result of a Bank supported Program-for-Results Financing operation, as defined by the applicable Bank Policy and Directive, may submit complaints to the existing program grievance redress mechanism or the WB’s Grievance Redress Service (GRS). The GRS ensures that complaints received are promptly reviewed in order to address pertinent concerns. Affected communities and individuals may submit their complaint to the WB’s independent Inspection Panel which determines whether harm occurred, or could occur, as a result of WB non-compliance with its policies and procedures. Complaints may be submitted at any time after concerns have been brought directly to the World Bank's attention, and Bank Management has been given an opportunity to respond. For information on how to submit complaints to the World Bank’s corporate Grievance Redress Service (GRS), please visit www.worldbank.org/grs. For information on how to submit complaints to the World Bank Inspection Panel, please visit www.inspectionpanel.org. . The activities that will be implemented under the Technical Assistance IPF are not expected to have negative environmental or social impacts. . Page 39 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) E. Risk Assessment 112. The overall risk for the operation is Substantial. The Political and Governance risk is Substantial, given that the country will undergo presidential and legislative elections in October 2019. This could potentially lead to a pause in the implementation of the programs and/or changes in the government’s priorities. This risk is mitigated, however, by the GoA’s commitment to protect the most vulnerable, in line with the targeted beneficiaries of this Operation. Furthermore, the largest programs began under previous administrations, which shows that they have a high level of buy-in from different political parties, and the country’s existing systems are expected to remain sound even if there is a governmental transition. 113. The Macroeconomic risk is High due to the current financial constraints in the public sector. If fiscal consolidation has a higher impact than anticipated in the short term on economic activity, broad political support and funding for the social programs under this Operation could be at risk. These risks are mitigated because social investment commitments are at the heart of the GoA’s priorities and the bulk of the operation is targeted to vulnerable populations. 114. Both Sector Strategies and Policies and Technical Design of Program risks are considered Moderate, given that the operation would support only existing programs that have been prioritized and are expected to continue for at least five more years. The implementation of these programs has also entailed collaboration and communication between the federal and provincial ministries, which is likely to mitigate any risks that may arise in the case of a political transition. 115. The integrated Fiduciary risk, which assesses the likelihood and impact of the Financial Management risk and procurement risk, considering governance and anti-corruption risks, that funds will not be used for intended purposes to achieve value for money with integrity in delivering sustainable development is assessed as Moderate. The main fiduciary risks and corresponding mitigation measures are discussed in detail in the Fiduciary Systems Assessment. 116. Environmental and Social Risk is Moderate. Environmental Risk is low given that the Operation would neither finance nor support any civil works, rehabilitation, construction, changes in the use of natural resources, pollution generation, or exposure to environmental risks. It is therefore not expected to have any negative environmental impacts. Social Risk is Moderate, as the Operation is expected to have only positive impacts on vulnerable and systematically excluded groups and no negative social effects. The MECCyT has experience working with World Bank Safeguards and received training on the Bank’s Environmental and Social Framework (ESF). Nevertheless, the support to these programs will require that the MECCyT carry out a participatory approach, which might require the strengthening of its social communication processes, citizen engagement, and beneficiary feedback mechanisms, to ensure inclusion and active participation of beneficiaries from vulnerable groups and avoid any kind of discrimination that might be present in the current system. In this context, the MECCyT is carrying out a series of consultations with all vulnerable groups identified, including afro-descendants, indigenous peoples, students with disabilities, and LGBTIQ students to avoid, minimize or mitigate the risks mentioned above. In addition, the PAP includes an action to strengthen the Program’s outreach strategy and the MECCyT’s capacity to implement it with a focus on vulnerable groups. Moreover, the social risk is also associated with the intended institutional strengthening of the National Evaluation Commission of Page 40 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Teacher Training Institutes, which could generate some level of conflict with teachers. The Operation would include teachers’ and students’ representatives in the stakeholder engagement process to avoid any potential tension associated to the latest activities. The Moderate rating is related more to the tensions intrinsic to the teacher evaluation component of the Operation, which should not, in principle, affect vulnerable minorities and/or students. MECCyT will develop the evaluation program in a participatory and transparent manner to avoid conflict within this component. Page 41 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . ANNEX 1. RESULTS FRAMEWORK MATRIX Results Framework COUNTRY: Argentina Improving Inclusion in Secondary and Higher Education Program Development Objective(s) The objectives of the Operation are to (i) reduce dropout rates in basic and higher education among the most vulnerable; and (ii) strengthen the Borrower’s education evaluation systems. Program Development Objective Indicators by Objectives/Outcomes RESULT_FRAME_T BL_ PD O Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 Support programs to reduce dropout in basic and higher education Average dropout rate of ASISTIRE schools in selected 11.70 11.30 10.80 10.30 9.90 9.40 provinces (Percentage) Persistence in the first two years of higher education for 15.60 15.80 16.00 16.40 16.60 16.70 the most vulnerable youth (Percentage) Strengthen student and teacher evaluation systems Percentage of students assessed through the 69.3% secondary (2017) 76.8% primary (2018) 72% secondary (2019) 79% primary (2020) 75% secondary (2021) 82% primary (2022) APRENDER evaluation nationwide (Text) Page 42 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) RESULT_FRAME_T BL_ PD O Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 Percentage of basic education students assessed through the 47.8% secondary (2017) 65% primary (2018) 50% secondary (2019) 67% Primary (2020) 56% secondary (2021) 70% primary (2022) APRENDER evaluation in the Participating Provinces with the Lowest Coverage (Text) Diagnostic evaluation system Action plan for Results report on the Instruments for Pilot of the evaluation for teacher training institutes DLI 8 No No evaluation and implementation of the evaluation developed carried out implemented (Text) dissemination of results pilot evaluation. . Page 43 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . Intermediate Results Indicator by Results Areas RESULT_FRAME_T BL_ IO Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 Improving the Management and Monitoring of PROGRESAR Scholarships Students benefiting from direct interventions to enhance 581,547.00 515,668.00 518,246.00 520,837.00 523,441.00 526,058.00 learning (CRI, Number) Students benefiting from direct interventions to 376,007.00 333,412.00 335,079.00 336,755.00 338,438.00 340,130.00 enhance learning - Female (CRI, Number) Percentage of applicants to PROGRESAR Higher Education 46.18 47.31 48.44 49.58 50.71 51.84 that meet the eligibility requirements (Percentage) Percentage of higher education scholarship candidates whose academic information has been DLI 1 75.00 80.00 85.00 90.00 95.00 95.00 provided by their academic institution (Percentage) Percentage of female higher education scholarship candidates whose academic 75.00 80.00 85.00 90.00 95.00 95.00 information has been provided by their academic institution (Percentage) Estimation of PROGRESAR Published document Estimation of coverage Estimation of coverage Estimation of coverage Scholarships' coverage by No with estimations and rates using study's rates using study's rates using study's education level (Text) methodology methodology methodology methodology Percentage of higher education DLI 2 28.70 30.70 32.70 35.00 34.70 36.70 Page 44 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) RESULT_FRAME_T BL_ IO Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 scholars that have renewed the scholarship (Percentage) Percentage of first-year higher education scholars enrolled in DLI 3 29.30 31.30 33.30 35.30 37.30 39.30 Strategic Careers (Percentage) Percentage of female higher education scholars that study STEM careers as a percentage 6.80 7.10 7.40 7.60 7.80 8.00 of female higher education scholars (Percentage) Tools to diagnose and overcome non-financial Website with diagnostic barriers to education Consultancies to carry Diagnostic study is DLI 4 No tools for scholars is Website is active Website is active progression and completion out studies hired completed active designed and implemented (Text) Pilot of the certification Academic certification system Certification system for Certification system for Consultancies to carry system for basic for basic education scholars DLI 5 No No basic education scholars basic education scholars is out studies hired education scholars is implemented (Text) is developed active implemented Supporting the National Student and Teacher Evaluation Systems Percentage of school principals accessing the APRENDER 59.70 65.00 70.00 78.00 80.00 85.00 Summary Report (Percentage) Percentage of school principals using the APRENDER Summary DLI 7 0.00 0.00 30.00 45.00 55.00 60.00 Report to enhance pedagogical practices (Percentage) IPF: Strengthen the Borrower's Capacity for the Design, Implementation and M&E of the Programs Page 45 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) RESULT_FRAME_T BL_ IO Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 Piloting and evaluation of a mentoring intervention Intervention and Pilot of the intervention Pilot of the intervention program to support scholars in No No Evaluation results report evaluation plan designed is implemented is implemented their academic progression (Text) Beneficiary Feedback. Percentage of cases registered in the GRM that are responded 0.00 0.00 0.00 20.00 50.00 75.00 and resolved in a timely manner (15 business days) (Percentage) Results of additional Develop additional modules for Instruments for parents Instruments for parents Instruments for parents No No modules included in APRENDER (Text) and SES produced and SES piloted and SES piloted school report Local adaptation of TEACH Adaptation plan Adapted manuals Pilot implemented in No No Training completed (Text) produced produced schools Number of Provinces with signed agreements for the implementation ASISTIRÉ in a 1.00 6.00 9.00 12.00 12.00 12.00 group of vulnerable schools (participating schools) (Number) Percentage of participating schools in selected provinces that meet the minimum 0.00 0.00 30.00 45.00 65.00 70.00 implementation requirements for ASISTIRÉ (Percentage) Number of provinces implementing the improved 0.00 0.00 3.00 5.00 7.00 9.00 early warning software Page 46 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) RESULT_FRAME_T BL_ IO Indicator Name DLI Baseline Intermediate Targets End Target 1 2 3 4 (including attendance data and additional variables) (Number) Pilot of ASISTIRÉ Family in a Intervention and Pilot of the intervention Pilot of the intervention No No Evaluation results report group of schools (Text) evaluation plan designed is implemented is implemented Impact evaluation of the Intervention and Pilot of the intervention Pilot of the intervention No No Evaluation results report revamped ASISTIRÉ (Text) evaluation plan designed is implemented is implemented Development of interventions Training for facilitators Intervention protocols International workshop Toolkit for schools is to improve the design of No No manual produced manual produced on dropout produced ASISTIRÉ (Text) . Page 47 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . Monitoring & Evaluation Plan: PDO Indicators Methodology for Data Responsibility for Data Indicator Name Definition/Description Frequency Datasource Collection Collection Selected Provinces are Buenos Aires, Chaco, Chubut, Corrientes, Jujuy and San Juan, which constitute the first cohort of provinces for the Administrative expansion of ASISTIRÉ. The data provided Average dropout rate of ASISTIRE schools indicator is the average by the The SPE uses RA data for Annual DNB - M&E Unit in selected provinces dropout rate of the Secretary of the estimations. selected schools in each Education Province. the identification Planning (SPE) code of the schools included in the estimation of the indicator is available in the POM. The baseline for this indicator is 2017. This indicator tracks access and persistence in higher education for the most vulnerable. The most Data is collected and vulnerable are defined as published by the INDEC, Persistence in the first two years of higher 20 to 30-year-olds whose Annual EPH and available online at DNB - M&E Unit education for the most vulnerable youth household income https://www.indec.gob. correspond to the first and ar/bases-de-datos.asp. second quintile. The indicator measures the percentage of vulnerable students that access higher Page 48 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) education programs and complete at least 2 years of studies at this level, as a percentage of the total vulnerable youth in this same age range. The indicator is estimated as an average of the data for the 4 trimesters of the calendar year. The baseline corresponds to the average of the first three trimesters of 2018 and a projection of the 2018 final trimester, given the avaiblable data. End target corresponds to average of the trimesters with available data from calendar year 2023, by the time of the project completion report. This indicator tracks the number of students that participate in APRENDER as a percentage of the total Data is published on the SEE national students enrolled in the MECCyT website. Percentage of students assessed through reports on assessed grades. The Annual Available online at DNB - M&E Unit the APRENDER evaluation nationwide APRENDER indicator alternates https://www.argentina.g coverage. between primary and ob.ar/educacion. secondary in line with the APRENDER assessment, which is applied yearly Page 49 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) alternating between primary and secondary. The indicator corresponds to the number of students that participate in APRENDER as a percentage of the total students enrolled in the assessed grades in the Provinces with Lowest Coverage. The Provinces with Lowest Coverage correspond to SEE reports annual Percentage of basic education SEE national the Provinces in the lowest student coverage of students assessed through the reports on quartile of student APRENDER by Province. APRENDER evaluation in the Annual. APRENDER DNB - M&E Unit participation rates Available online at Participating Provinces with the coverage by nationwide. When the https://www.argentina.g Lowest Coverage Province. quartile cutoff is not a ob.ar/educacion whole number, it would be rounded up to the next whole number to estimate the number of Provinces to include in the lowest quartile. The indicator is the simple average of the participation rate in these Provinces. This indicator follows the development and SEE Diagnostic evaluation system for teacher implementation of an SEE documents and Annual documents DNB - M&E Unit training institutes implemented evaluation system for reports and reports teacher training institutes, including: i) an evaluation Page 50 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) action plan that includes objectives of the system, scope, timeline for the development of instruments, a dissemination of results plan, and an implementation plan (including carrying out a pilot); ii) the development and validation of instruments for evaluation; iii) carrying out the pilot established in the action plan; and iv) a final report on the results of implementation of the . pilot. Page 51 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . Monitoring & Evaluation Plan: Intermediate Results Indicators Methodology for Data Responsibility for Data Indicator Name Definition/Description Frequency Datasource Collection Collection This indicator follows the number of direct beneficiaries of PROGRESAR Scholarships and is collected in the DNB Students benefiting from direct information system. The Annual DNB DNB - M&E Unit interventions to enhance learning indicator is expected to drop during the first year of implementation before stabilizing, as the monitoring and evaluation of eligibility improves. This indicator follows the number of direct female beneficiaries of Students benefiting from direct PROGRESAR interventions to enhance learning - Annual DNB DNB - M&E Unit Scholarships, and the Female data is collected in the DNB information system. This indicator is the Income percentage of PROGRESAR requirements DNB information system Percentage of applicants to PROGRESAR applicants to Tertiary non- are validated has the information on Higher Education that meet the eligibility Annual DNB - M&E Unit university and university by ANSES, eligibility requirements requirements that meet the eligibility while for applicants. requirements of the academic Page 52 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) scholarship. Eligibility requirements requirements include are certified income and academic by the requirements. academic institutions and reported to DNB. This indicator measures the share of higher education scholarship candidates that received an academic certification by their DNB receives Percentage of higher education institutions on time for the academic DNB requires academic scholarship candidates whose academic awarding of the scholarship certification institutions to issue the Annual DNB - M&E Unit information has been provided by their . It tracks the efficiency of from the academic certification academic institution the system to monitor academic from applicants. eligibility requirements and institutions. ensures that all applicants have their academic certification requirement assessed. This indicator measures the share of female Higher Education Scholarship Percentage of female higher candidates that received an education scholarship candidates academic certification by whose academic information has their institutions in time for Annual DNB DNB information system DNB - M&U been provided by their academic the scholarship institution adjudication. It tracks the efficiency of the system to monitor eligibility requirements and ensures Page 53 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) that all applicants have their academic certification requirement assessed. This indicator refers to the publication of a study that determines the coverage of PROGRESAR Scholarships The DNB will hire a firm by education level using for the elaboration of EPH and available data the study and the from ANSES. The report DNB - Report establishment of a Estimation of PROGRESAR Scholarships' includes coverage Annual with study methodology for its DNB coverage by education level estimations for vulnerable results. estimation, including populations, gender, information on coverage geographical location and for vulnerable education level, and populations. establishes a methodology to estimate the coverage in the future. This indicator follows scholars in their academic progression. It measures the share of higher education scholars that renew their scholarship, as Percentage of higher education scholars a percentage of the total DNB information Annual DNB DNB - M&E Unit that have renewed the scholarship scholars in the previous systems year. It excludes first and last-year scholars as they do not renew scholarships (first time scholars or graduation). The baseline's numerator is 2017 scholars Page 54 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) that renew in 2018 and the denominator is 2017 scholars that apply for the scholarship in 2018. This indicator follows the evolution of one of the incentives of PROGRESAR Scholarships related to the Percentage of first-year higher education enrollment in Strategic Annual DNB DNB information system DNB - M&E Unit scholars enrolled in Strategic Careers Careers. Strategic Careers for the indicator are detailed in the Project's Operations Manual. This indicator tracks a gender gap identified in the ESSA. It estimates the Percentage of female higher education share of STEM career scholars that study STEM careers as a DNB information scholars that are females. Annual DNB DNB - M&E Unit percentage of female higher education system. STEM careers include scholars careers in Science, Technology, Engineering and Mathematics. This indicator comprises the diagnosis and development of tools to Tools to diagnose and overcome non- support students in the DNB DNB documents and financial barriers to education progression completion of their studies. documents, Annual reports, and online DNB - M&E Unit and completion designed and The tools to diagnose non- reports and website active implemented financial barriers to website education progression and completion refer to a quantitative and qualitative Page 55 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) study that explores the causes for scholars’ dropout through surveys, structured interviews, focus groups, and analysis of administrative data. The minimum requirements of the study are detailed in the POM. The tools to overcome non-financial barriers to education progression and completion of students refer to tools available to students through PROGRESAR’s website on: guidance for the application process, information on returns to education by career, tools for academic readiness on specific subjects, short courses/videos on developing socio-emotional skills, and others defined in the POM. This indicator tracks the development and DNB Academic certification system for basic implementation of an Revision of DNB Annual documents DNB - M&E Unit education scholars implemented academic certification documents and reports and reports system for basic education scholars. The system will be Page 56 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) piloted in a group of adult schools. This indicator tracks the number of school principals (maximum of 1 per school) SEE that access the online administrative SEE reports on access to Percentage of school principals accessing APRENDER Report as a data from the Annual school report results by DNB - M&E Unit the APRENDER Summary Report percentage of school online system school principals principals of schools that of school participated in APRENDER reports and have a school report available online. This indicator tracks the percentage of schools in which principals and teachers report that they have used the data from the APRENDER Summary School Report to enhance SEE any pedagogical practices instruments in the school. This is developed to Percentage of school principals using the SEE report on results of measured by asking the capture the APRENDER Summary Report to enhance Annual APRENDER teacher and DNB - M&E Unit following question to the information pedagogical practices principal instruments school principals and included in teachers (for each school, APRENDER either the teacher or the assessments school principal should answer 1 for the indicator to be considered equal to 1. Otherwise, it is considered a zero): School principals: As a Page 57 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) result of the data from APRENDER, has any pedagogical practice been improved? If yes, can you provide any evidence or example? 1. Yes, and provides evidence or examples. For example, the school improvement plan, revision of lesson plans, sending teachers to professional development programs, etc. 2. Yes, but does not show any evidence or examples of this. 3. No Teachers: As a result of the data from APRENDER, have you made any revision toyour pedagogical practices? If yes, can you provide any evidence or example? 1. Yes, and provides evidence or example. For example, lesson plans, professional development programs, additional exercises for students, etc. 2. Yes, but does not provide any evidence or example. 3. No Page 58 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) This indicator follows the design, piloting and evaluation of an Piloting and evaluation of a mentoring DNB Revision of DNB intervention providing intervention program to support scholars Annual documents documents and DNB - M&E Unit mentoring to scholars in their academic progression and reports advancement reports (complementary interventions) in partnership with NEXOS. This indicator tracks the percentage of cases registered in PROGRESAR Scholarships' Grievance Redress Mechanism (GRM) that are responded and Beneficiary Feedback. Percentage of cases resolved within 15 business registered in the GRM that are responded DNB presents a report of days of their registration in Annual GRM reports DNB - M&E Unit and resolved in a timely manner (15 GRM the system. While a GRM business days) exists for submission of cases, the mechanism for response is being formulated and is expected to be implemented in 2020. This indicator follows the production, piloting and implementation of instruments for parents SEE reports Develop additional modules for Revision of SEE reports and on SES in the Annual and DNB - M&E Unit APRENDER and documents APRENDER assessment, documents and the report of the results included in the school report cards. Page 59 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) The indicator follows the adaptation of the TEACH classroom observation tool for Argentina, the SEE Revision of SEE advance Local adaptation of TEACH development of manuals to Annual documents DNB - M&E Unit reports implement it, the training and reports of observers, and the implementation of a pilot in a group of schools. The indicator measures the number of Provinces that sign an agreement with the National authorities to implement ASISTIRÉ in a group of vulnerable schools within each Province. The agreement must be signed by the relevant Provincial and National authorities Number of Provinces with signed and include the agreement NAT consolidates the agreements for the implementation NAT to implement ASISTIRÉ in a Annual agreements and DNB - M&E Unit ASISTIRÉ in a group of vulnerable schools documents group of vulnerable supporting documents (participating schools) schools. Each agreement will be accompanied by a supporting document with: i) the list of schools that are selected for the implementation of ASISTIRÉ with their identification number (CUE); and ii) a description of the procedure followed Page 60 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) to select the schools in the Province. In the baseline (2018), the Province of Buenos Aires had already signed an agreement. This indicator tracks the level of implementation of ASISTIRÉ in the selected schools and provinces, according to the implementation characteristics defined in the POM. Selected Provinces is the group of Provinces that signed the agreements by 2019. NAT Participating schools are Percentage of participating schools in administrative NAT report on the schools in selected selected provinces that meet the data on participating schools Provinces selected to scale Annual DNB - M&E Unit minimum implementation requirements schools implementation up ASISTIRÉ. Minimum for ASISTIRÉ participating requirements implementation in ASISTIRE requirements are defined in the POM, and include taking digital assistance, use of early warning, and follow-up interventions to at-risk students identified in the EWS. It is built by estimating the percentage of participating schools that are complying with the characteristics of ASISTIRÉ. Page 61 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) The denominator corresponds to the original list of participating schools in selected Provinces as found in the supporting documents of each agreement. The indicator corresponds to the number of Provinces that are implementing the improved early warning software. The improved NAT documents that the software: i) includes early warning system Number of provinces implementing the additional administrative that is being used in improved early warning software data to identify students at Annual NAT each Province, with DNB - M&E Unit (including attendance data and additional risk, such as academic details on the variables) performance; ii) has a information used as revised and improved input for the system. algorithm; and iii) has approved sensitivity and false alarms tests and the technical specifications described in the POM. This indicator follows the implementation and evaluation of a pilot NAT elaborates reports NAT Pilot of ASISTIRÉ Family in a group of program that provides documenting the Annual documents DNB - M&E Unit schools student attendance implementation of the and reports information to parents activity through an experimental evaluation Page 62 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) This indicator follows the impact and process Impact evaluation of the revamped NAT advance reports evaluation of ASISTIRÉ in a Annual NAT DNB - M&E Unit ASISTIRÉ and documentation Province through an experimental evaluation. This indicator comprises activities to improve the design of ASISTIRÉ. It includes: i) the systematization of the intervention protocols and training for facilitators NAT documentation and which will be compiled in a Development of interventions to improve reports on manual; ii) an international Annual NAT DNB - M&E Unit the design of ASISTIRÉ implementation on workshop on dropout activities. prevention in the first semester of 2021; and iii) the development of a toolkit for schools to tackle specific dropout risks identified by a diagnosis . study. Page 63 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . ANNEX 2. Disbursement Linked Indicators, Disbursement Arrangements and Verification Protocols . Disbursement Linked Indicators Matrix DLI_T BL_MATRI X Percentage of higher education scholarship candidates whose academic information has been provided by their academic DLI 1 institution Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Intermediate Outcome Yes Percentage 100.00 33.00 Period Value Allocated Amount (USD) Formula Baseline 75.00 Prior Results 80.00 35.00 min: 75% April 2020 0.00 October 2020 85.00 25.00 min: 80% April 2021 0.00 October 2021 90.00 25.00 min: 85% April 2022 0.00 October 2022 95.00 15.00 min: 90% Page 64 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) DLI_T BL_MATRI X DLI 2 Percentage of higher education scholars that have renewed the scholarship Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Outcome Yes Percentage 80.00 27.00 Period Value Allocated Amount (USD) Formula Baseline 28.70 Prior Results 30.70 25.00 min: 28.7% April 2020 0.00 October 2020 32.70 20.00 min: 30.7% April 2021 0.00 October 2021 34.70 20.00 min: 32.7% April 2022 0.00 October 2022 36.70 15.00 min: 34.7% DLI_T BL_MATRI X DLI 3 Percentage of first-year higher education scholars enrolled in Strategic Careers Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Intermediate Outcome Yes Percentage 40.00 13.00 Period Value Allocated Amount (USD) Formula Baseline 29.30 Prior Results 31.30 10.00 Page 65 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) April 2020 0.00 October 2020 33.30 10.00 April 2021 0.00 October 2021 35.30 10.00 April 2022 0.00 October 2022 37.30 10.00 DLI_T BL_MATRI X Tools to diagnose and overcome non-financial barriers to education progression and completion designed and DLI 4 implemented Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Output No Text 60.00 20.00 Period Value Allocated Amount (USD) Formula Baseline No Prior Results 0.00 April 2020 Diagnostic study and action plan published 20.00 October 2020 Website with information to guide the 40.00 application for scholarships April 2021 0.00 October 2021 0.00 Page 66 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) April 2022 0.00 October 2022 0.00 DLI_T BL_MATRI X DLI 5 Academic certification system for basic education scholars implemented Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Output No Text 10.00 3.00 Period Value Allocated Amount (USD) Formula Baseline No Prior Results 0.00 April 2020 0.00 October 2020 0.00 April 2021 Action plan for the certification system for basic 5.00 education October 2021 0.00 April 2022 Results report of the pilot of the certification 5.00 system for basic education October 2022 0.00 Page 67 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) DLI_T BL_MATRI X Percentage of basic education students assessed through the APRENDER evaluation in the Participating Provinces with the DLI 6 Lowest Coverage Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Outcome Yes Text 4.50 1.00 Period Value Allocated Amount (USD) Formula Baseline 65% primary (2018) Prior Results 0.00 April 2020 50% secondary 2.50 October 2020 0.00 April 2021 67% primary 1.00 October 2021 0.00 April 2022 56% secondary 1.00 October 2022 0.00 DLI_T BL_MATRI X DLI 7 Percentage of school principals using the APRENDER Summary Report to enhance pedagogical practices Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Outcome Yes Percentage 4.00 1.00 Period Value Allocated Amount (USD) Formula Baseline 0.00 Page 68 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Prior Results 0.00 April 2020 0.00 October 2020 0.00 April 2021 0.00 October 2021 45.00 2.00 April 2022 0.00 October 2022 55.00 2.00 DLI_T BL_MATRI X DLI 8 Diagnostic evaluation system for teacher training institutes implemented Type of DLI Scalability Unit of Measure Total Allocated Amount (USD) As % of Total Financing Amount Intermediate Outcome No Text 2.50 1.00 Period Value Allocated Amount (USD) Formula Baseline No Prior Results 0.00 April 2020 0.00 October 2020 Action plan for evaluation and dissemination of 1.50 results April 2021 0.00 October 2021 0.00 Page 69 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) April 2022 0.00 October 2022 Pilot of the evaluation carried out 1.00 . Page 70 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . Verification Protocol Table: Disbursement Linked Indicators DLI_T BL_VERIFICATI ON Percentage of higher education scholarship candidates whose academic information has been provided by their academic DLI 1 institution This indicator measures the share of higher education scholarship candidates that received an academic certification by their institutions in time for the scholarship adjudication (from the application inscription period start until September 30th of each calendar year). It tracks the efficiency of the system to monitor the academic eligibility by measuring the share of Description applicants that have their academic certification requirement assessed. Higher education scholarship candidates are those that applied to the scholarships for Tertiary non-university and University programs. Academic information refers to the document sent by each academic institution to the DNB containing a summary of the applicant's academic information required to assess eligibility. Data source/ Agency DNB Verification Entity Verification Agent (VA) The DLI will be accomplished when the MH sends the Bank a detailed document that reports that the target for the indicator is met, the estimations of the indicator, and the report of the VA. The document should include the total number of higher education candidates by scholarship, and the number of the candidates with the academic information formally shared by the education institution, and the estimation of the indicator. This information must be disaggregated by gender, Procedure geographical location, education level (tertiary non-university and university), and education institution. The VA report must include the procedure and results of the verification, which should cover a minimum number of randomly selected applicants’ folders to verify the compliance with their academic certification. The minimum number will be established in the POM. DLI_T BL_VERIFICATI ON DLI 2 Percentage of higher education scholars that have renewed the scholarship This indicator follows scholars in their academic progression. It measures the share of higher education scholars that renew their scholarship during the current academic year, as a percentage of the total scholars of the previous year that applied to renew the scholarship. It excludes first and last-year scholars as they do not renew scholarships (first time scholars or Description graduation). Current higher education scholars are those that received the scholarship for Tertiary non-university and University programs during the academic year. The baseline's numerator is 2017 scholars that renew in 2018 and the denominator is 2017 scholars that apply for the scholarship in 2018. Page 71 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Data source/ Agency DNB Verification Entity VA The DLI will be accomplished when the MH sends the Bank a detailed document that reports the target for the indicator is met, the estimations of the indicator, and the report of the VA. The document should include the total number of current higher education students that renewed the scholarship during the current academic year, and the total scholars of the previous year that applied to renew the scholarship *(excluding first and last-year scholars). This information must also be Procedure disaggregated by gender, geographical location, education level (tertiary non-university and university), education institution, area or career, and year of study. The VA report must include the procedure and results of the verification procedure, which must cover a minimum number of randomly selected scholars’ folders to verify that they received the scholarship the previous academic year. The minimum number will be established in the POM. DLI_T BL_VERIFICATI ON DLI 3 Percentage of first-year higher education scholars enrolled in Strategic Careers This indicator tracks the impact of PROGRESAR Scholarships incentives related to enrollment in Strategic Careers. Strategic Careers for the indicator are available in the POM. First-year higher education scholars are those that are enrolled for the Description first year of the program or career. In any given year, the indicator is computed as the ratio between first-year higher education scholars that enrolled in Strategic Careers (numerator), and the total number of first-year higher education scholars (denominator). Data source/ Agency DNB Verification Entity VA The DLI will be accomplished when the MH sends the Bank a detailed document that reports the target for the indicator is met, the estimations of the indicator, and the report of the VA. The document must include the total number of first-year higher education scholars that enrolled in strategic careers and the total number of first-year higher education scholars. Procedure This information must also be disaggregated by gender, geographical location, education level (tertiary non-university and university), education institution, area or career, and year of study. The VA report must include the procedure and results of the verification procedure which must cover a minimum number of randomly selected scholars’ folders to verify that they are enrolled in strategic careers. The minimum number and the list of strategic careers will be established in the POM. Page 72 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) DLI_T BL_VERIFICATI ON Tools to diagnose and overcome non-financial barriers to education progression and completion designed and DLI 4 implemented This indicator comprises the diagnosis and development of tools to support students in the completion of their studies. The tools to diagnose non-financial barriers to education progression and completion refer to a quantitative and qualitative study that explores the causes for scholars’ dropout through surveys, structured interviews, focus groups, and analysis of Description administrative data. The minimum requirements of the study are detailed in the POM. The tools to overcome non-financial barriers to education progression and completion of students refer to tools available to students through PROGRESAR’s website on: guidance for the application process, information on returns to education by career, tools for academic readiness on specific subjects, short courses/videos on developing socio-emotional skills, and others defined in the POM. Data source/ Agency MoE documents, online website Verification Entity VA DLR 4.1 will be accomplished when the MH sends the Bank the final version of the diagnosis of non-financial barriers and the action plan with specific actions to overcome them. The document must include a tentative deadline for each of the activities to be completed and specific outputs to achieve on each deadline. The document must include the VA report on the compliance of the document with the established criteria in the description of the indicator and the POM. Procedure DLR 4.2 will be accomplished when the MH sends the Bank a document with the link to the website with available tools to overcome non-financial barriers to education progression and completion for students and its compliance with the specifications established in the POM. The document must include the VA report on the compliance of the website with the established criteria in the description of the indicator and the POM. DLI_T BL_VERIFICATI ON DLI 5 Academic certification system for basic education scholars implemented This indicator tracks the development and implementation of an academic certification system for basic education scholars. Description The system will be piloted in a group of adult schools. Data source/ Agency DNB Verification Entity VA DLR 5.1 will be accomplished when the MH sends the Bank the document with the action plan for the development of the Procedure academic certification system for basic education scholars with a detailed schedule of activities and piloting plan. The document must include a tentative deadline for each of the activities to be completed as well as implementation Page 73 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) arrangements. The document includes the VA report, which must include the compliance of the action plan with the description of the indicator. DLR 5.2 will be accomplished when the MH sends the Bank a document with the report on the results of the piloting of the certification system in a group of adult schools, including the usage of the system, main challenges to implementation, lessons learned, and an escalation plan. The document includes the VA report, which must include the verification of participation of a minimum number of a random sample of adult schools in the pilot. The minimum number will be defined in the POM. DLI_T BL_VERIFICATI ON Percentage of basic education students assessed through the APRENDER evaluation in the Participating Provinces with the DLI 6 Lowest Coverage The indicator corresponds to the number of students that participate in APRENDER as a percentage of the total students enrolled in the assessed grades in the Provinces with Lowest Coverage. Students assessed refers to the number of students that participate in the census-based APRENDER as a percentage of the total number of students enrolled in the grades Description covered by APRENDER across public schools nationwide. The Provinces with Lowest Coverage correspond to the Provinces in the lowest quartile of student participation rates nationwide. When the quartile cutoff is not a whole number, it would be rounded up to the next whole number to estimate the number of Provinces to include in the lowest quartile. The indicator is the simple average of the participation rate in these Provinces. Data source/ Agency SEE Verification Entity VA The DLI will be accomplished when the MH sends the Bank a document that compiles the National APRENDER report Procedure published online on the SEE website, which includes student coverage of APRENDER nationwide and by Province, and the percentage of students assessed through APRENDER meets the established targets. DLI_T BL_VERIFICATI ON DLI 7 Percentage of school principals using the APRENDER Summary Report to enhance pedagogical practices This indicator tracks the percentage of schools where principals and teachers report that they have used the data from the APRENDER Summary School Report to improve/enhance any pedagogical practices in the school. This is measured by asking the following question to the school principals and teachers (For each school, either the teacher or the school principal Description should answer to 1 for the indicator to be considered equal to 1. Otherwise, it is considered a zero): School principals: As a result of the data from APRENDER, has any pedagogical practice been improved? If yes, can you provide any evidence or example? 1. Yes, and provides evidence or examples. For example, the school improvement plan, revision of lesson plans, Page 74 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) sending teachers to professional development programs, etc. 2. Yes, but does not show any evidence or examples of this. 3. No Teachers: As a result of the data from APRENDER, have you made any revision toyour pedagogical practices? If yes, can you provide any evidence or example? 1. Yes, and provides evidence or example. For example, lesson plans, professional development programs, additional exercises for students, etc. 2. Yes, but does not provide any evidence or example. 3. No Data source/ Agency SEE Verification Entity IVA The DLI will be accomplished when the MH sends the Bank a document reporting that the target for the indicator is met and the estimations of the indicator. The numerator is the number of school principals or teachers (maximum of 1 per school) that are using the improved APRENDER report to improve/enhance pedagogical practices at their school as determined in Procedure the indicator description. The denominator corresponds to the total number of schools surveyed for the education level APRENDER was implemented. The document includes the VA report, which must cover a minimum number of randomly selected schools to verify their answers to the questions in the school principal and teacher surveys. The minimum number will be established in the POM. DLI_T BL_VERIFICATI ON DLI 8 Diagnostic evaluation system for teacher training institutes implemented This indicator follows the development and implementation of an evaluation system for teacher training institutes, including: i) an evaluation action plan that includes objectives of the system, scope, timeline for the development of Description instruments, a dissemination of results plan, and an implementation plan (including carrying out a pilot); ii) the development and validation of instruments for evaluation; iii) carrying out the pilot established in the action plan; and iv) a final report on the results of implementation of the pilot. Data source/ Agency SEE Verification Entity VA DLR 8.1 will be accomplished when the MH sends the Bank a document with the action plan for the diagnostic evaluation system for teacher training institutes, with specific activities and timeline for implementation. The implementation plan includes a pilot to be carried out by October 2022, and comprises the stages to be achieved by that date. The document Procedure includes the VA report, which must verify compliance with the description of the indicator. DLR 8.2 will be accomplished when the MH sends the Bank a document that reports the number and list of teacher training institutes that participate in the pilot, a description of compliance with the goals established in the action plan, and the VA Page 75 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) report. The VA report must include the procedure and results of a verification of the implementation of the system in a . minimum number of randomly selected teacher training institutes. The minimum number will be established in the POM. Page 76 of 144 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 3. TECHNICAL ASSESSMENT COUNTRY : Argentina Improving Inclusion in Secondary and Higher Education Strategic Relevance 1. Rationale for Bank Engagement and Bank’s value added. There is a strategic rationale for the Bank to engage in the proposed Operation, as it is consistent with the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by supporting programs targeted towards students at risk of dropout and repetition, who overwhelmingly come from vulnerable backgrounds. The Operation is aligned with the commitment to ensuring sustainability and improved efficiency of priority education programs. Additionally, the Bank has a history of engagement with Argentina, through which it has acquired in-depth knowledge of both the education sector and the federal nature of the country. Given that this federal nature could present bottlenecks to implementation due to varying political will and implementation capacity among provinces, the Bank is well-positioned to rapidly respond to the Borrower’s request for support to programs that are carried out (or expected to be scaled up) nationwide by drawing on and incorporating lessons learned from previous and ongoing operations into their design. Furthermore, the Bank’s vast experience on improving information management systems, evaluation, and decision-making in the sector can be a great value added in the form of technical assistance, as it can draw from international experience and best practices to improve government systems. 2. No additional partners are expected. However, the Inter-American Development Bank (IADB) will partially support the activities under Results Area 1 until June 2019, through its project Programa de Apoyo a la Equidad y Efectividad del Sistema de Protección Social en Argentina (AR-L1302). The objectives of the IDB support are to fund PROGRESAR scholarships and contribute to the monitoring and information system of the program, to support: (i) the strengthening of academic trajectories of students supported by the PROGRESAR scholarships; (ii) a more efficient incorporation of former AUH beneficiaries to the PROGRESAR Scholarships system; and (iii) to contribute to early dropout alert systems. The total amount of support of the IDB component related to PROGRESAR Scholarships is US$320 million and the support is expected to end in June 2019, before the World Bank support for PROGRESAR Scholarships becomes effective. The objectives of the IDB operation are aligned with the development objectives of the World Bank support, and there would be dialogue and articulation between both institutions. Rationale for Public Sector Provision/Financing 3. Three factors provide a strong rationale for public sector financing of the interventions supported by the Program. First, individual acquisition of education generates positive externalities on society by making individuals more engaged and responsible citizens and increasing the overall level of productivity and growth in the economy. To the extent that individuals do not take these benefits into account when making education investment decisions, public sector investments can lead to efficiency gains. Second, there is imperfect information that makes individuals unaware of the importance of education in future labor opportunities and, even if they are aware, financial constraints might prevent them from financing these investments in the credit market. Third, the education system has the potential to promote equality of opportunities. Since dropout through secondary and higher education is particularly relevant for students from disadvantaged households, any policy aimed at reducing financial and non-financial barriers at these educational levels will benefit vulnerable groups and is therefore desirable from an equity perspective. Page 77 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Technical Soundness 4. Technical Soundness. This section presents a review of the available literature to provide further evidence that the Operation’s design is technically sound, based on international best practices, and in line with evidence-based interventions. 5. Financial support (CCTs, merit- and need-based scholarships) for school completion: a large body of evidence finds that scholarship programs almost always improve school participation outcomes (enrollment, attendance, completion and dropout) and can, in some cases, improve student learning. The 2018 improvements and modifications to PROGRESAR Scholarships, which promote more inclusion, merit-based benefits, and Strategic Careers in higher education, have also been identified in the literature as effective interventions for education outcomes. Evidence suggests that interventions that increase the benefits of attending school (such as scholarships) increase time in school,66 while those that increase the benefits of higher effort and better academic performance (such as merit-based scholarships) improve learning outcomes. The significantly positive effects on school participation are found in Brazil,67 Colombia,68 Cambodia,69 China,70 Honduras,71 Malawi,72 Mexico,73 Nicaragua,74 and Pakistan.75 In Malawi and Nicaragua, estimates calculated several years after the end of the program show that scholarship beneficiaries had higher cognitive skills, as measured by test scores. In contrast, in other countries (Cambodia, China and Colombia), there is no evidence of impacts on learning. Only one of the interventions to increase demand for education in developing countries has consistently worked to increase test scores across multiple contexts: merit-based scholarships.76 Overall, the findings indicate that merit-based scholarships can effectively increase test scores and attendance, while need-based scholarships tend to affect mainly enrollment and attendance, but not test scores. However, there can be an equity issue, as merit-based scholarships typically reach students who are already better off, which highlights the importance of combining this merit-based scholarship with strong targeting mechanisms. Evidence on the effect of merit-based scholarships for higher education students is also promising, although the results are more mixed.77 In particular, a few rigorous recent evaluations have shown large impacts. For instance, Angrist et al (2017) show that scholarships significantly boosted college enrollment and persistence. Four years after award receipt, randomly-selected scholarship winners were 13 percentage points more likely to be enrolled in college. 6. Non-financial support is also critical to strengthen the transition to higher education. This is particularly relevant for disadvantaged students who are the first generation in their family to have access to higher education and may lack support to make informed decisions and basic skills to succeed 66 Glewwe & Muralidharan (2015,), Damon et al (2018) 67 De Janvry et al (2012) 68 Barrera-Osorio et al (2011) 69 Barrera-Osorio & Filmer (2013) 70 Mo et al (2013) 71 Galiani and McEwan (2013) 72 Baird et al (2011) 73 Schultz (2004;); Behrman et al, 2009, 2011 74 Braham et al (2013) 75 Chaudhury and Parajuli (2010) 76 Kremer, Miguel and Thornton (2009), Friedman et al (2011), Blimpo (2014) 77 Dynarski (2008), Castelman (2014), Scott-Clayton (2011b), Scott-Clayton & Zafar (2016), Bettinger et al (2016); DesJardins & McCall (2014), DesJardin et al, (2010), Sjoquist & Winters (2012, 2015) Page 78 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) in higher education. Information-based programs that provide key data to students and parents have been shown to improve career choices in higher education. A recent RCT in Chile shows that providing information to future higher education students, through an online platform, on the potential earnings (and fee costs) corresponding to different enrollment choices, helped particularly low-socioeconomic students enroll in degrees with higher returns.78 The introduction of additional incentives for “strategic” careers (most high in demand) aimed at guiding the decision of new students could reinforce the effects of providing this information. Studies that analyze the role of financial aid in supporting strategic career (STEM) attainment in the U.S. find that eligibility for need-based financial aid increased STEM credit completion by 20 to 35 percent among academically-ready students in a large, public higher education system.79 Evidence from basic education also highlights the potential of information-based interventions. An RCT in the Dominican Republic finds that providing students with information on earnings differences by education reduced dropout by 7 percent in the subsequent year, and increased school completion by 0.2 years.80 Similarly, informing fourth grade students and their parents of earning differences by education levels in Madagascar increased average attendance by 3.5 percentage points.81 7. Socioemotional support and low-cost behavioral interventions are also essential to support students struggling with the more complex curriculum and environment of higher education. During the transition to a new education level, students can face frequent social setbacks and feelings of anxiety and isolation. Interventions that provide self-administered short programs on motivation and belonging have shown that college students benefit when they understand that challenges in the transition to college are common and improvable, and thus, that early struggles need not portend permanent lack of belonging or potential.82 In addition, a RCT showed that providing services that helped students set goals and acquire financial aid between their transition from high school to college increased college enrollment by 3 percentage points (even more for low-income students).83 Recent experimental evidence in the U.S. found that brief, one-time messages to students increased recurring tutoring attendance, leading to relevant changes in students’ study habits.84 8. Standardized student assessments are a key ingredient to shift the focus to learning and support students’ trajectories. The benefits of education—cultural, economic, and social—accrue to society only when learning occurs (OECD, 2010). For example, an increase of one standard deviation in scores on international assessments of reading and mathematics achievement levels has been linked to a 2 percent increase in annual growth rates of GDP per capita (Hanushek and Woessmann, 2007, 2009). For governments to learn what policies worked to improve learning and to avoid “flying blind,” they need credible data (World Bank, 2018). The Operation would support the strengthening of the student evaluation system. In a recent paper, Bergbauer, Hanushek and Woessmann (2018) find that the sole fact that countries adopt a standardized student assessment and make results of the evaluations public is associated with improvements in student achievement. As part of the Operation, the SEE would improve the report on the results of the assessment that each school receives to provide principals with information on what students mastered and what competencies they need to improve. A recent RCT in La Rioja, Argentina found that providing schools with such information (together with a short training) 78 Neilson et al (2018) 79 Castelman et al (2018) 80 Jensen (2010) 81 Nguyen (2008) 82 Yaeger et al (2016;); Walton & Cohen (2011) 83 Castleman & Page (2015) 84 Pugach & Wilson (2018.) Page 79 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) improved student achievement between 0.28 and 0.38 of a standard deviation, which represents around one additional academic year of learning (de Hoyos et al, 2018). 9. The assessment of teachers’ professional development is the second cornerstone of the education evaluation system. Improving teacher pre-service education is crucial to improve student learning. Teachers are the most important school-based determinant of student learning (Hanushek and Rivkin, 2010; World Bank, 2018). Research in Peru and Ecuador shows that teacher subject content knowledge (Metzler and Woessmann, 2012) and pedagogical practices (Araujo et al, 2016) in the classroom are among the most important factors to explain teachers’ value added. The Operation would strengthen the evaluation system of soon-to-be teachers in Argentina in all these dimensions, by (i) consolidating the current content and pedagogic knowledge assessment (ENSEÑAR); and (ii) complementing it with an assessment of what the teachers do in the classroom. This information would be used to provide feedback to teacher training institutes to improve their performance, to develop a suite of tools for self-evaluation, and to inform the design of tools and strategies for external evaluation initiatives in the future. Finally, as part of the technical assistance, the Operation would develop and pilot a strategy to integrate the use of student assessments and observations of what teachers do in the classroom with the systems of professional development principals and teachers. This is in line with current research on best practices in professional development (Popova, Evans, & Arancibia, 2016; Yoshikawa et al., 2015; Bambrick-Santoyo, 2012; Fryer, 2017). 10. The current design of ASISTIRÉ is aligned with best international practices, and the planned complementary interventions would reinforce its effectiveness. Extensive research has shown that it is more cost-effective to act early to prevent dropout than to try to reengage out-of-school youth in the educational system. EWS that rely on administrative data, even when they are based on basic algorithms, have proven to have high predictive power of dropout, allowing identification of children at risk and early action.85 A recent RCT in the U.S. showed that an EWS that relies on basic data (chiefly absences, grades, and behavioral problems), coupled with close follow-up of students at risk by the school team (case workers), reduced chronic absence and course failure by almost 20 percent after only one year of implementation.86 The improvements in the program’s design, which include strengthening the evidence- base of key interventions (such as closing skills gaps, teenage pregnancy prevention, and reinforcing the channels for family-school communications) would further boost the effectiveness of the program. At the secondary level, programs targeting key socioemotional skills87 with academic tutoring, as well as other supportive interventions (such as remedial education88 and teenage pregnancy prevention programs) are starting to show substantial effects on dropout for students in at-risk communities. For instance, a series of well-identified studies has shown that programs with a focus on student socioemotional skills (e.g. counseling programs,89 cognitive behavioral therapy, coaching and tutoring) improve mental health and, consequently, improve school engagement and student achievement. Interventions as diverse as life- planning skills, HIV prevention education,90 and vocational training, combined with reproductive health education, have shown promise in reducing adolescent pregnancy and pregnancy-related school 85 Knowles (2015); Adelman et al (2016) 86 Faria et al (2017) 87 Cook et al., 2014, Heller el al (2016), Wang et al (2016). Other studies on effective coaching/tutoring interventions in K –12 education include Dobbie & Fryer (2013), Fryer (2014), Kraft (2015), Kosse et al. (2016), and Ander, Guryan, & Ludwig (2016). 88 Banerjee et al. (2007) 89 Carrell & Carrell (2006) 90 Baird et al (2011), Duflo et al, 2011, 2015; Dupas (2011) Page 80 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) dropout.91 Similarly, evidence has also shown that providing information to parents on their children’s absenteeism and grades (through automated text messages, emails or phone calls) improves student outcomes and reduces absenteeism.92 Assessment of PROGRESAR’s Management and Information System93 11. Description. The PROGRESAR Scholarship Program involves the following steps related to: (i) scholarship enrollment; (ii) application assessment; (iii) scholarship approval and award; and (iv) payment and accountability. The Program’s enforcement authority is the MECCyT through the DNB, which interacts with other agencies, including ANSES, which was the enforcement authority until early 2018, and with which the MECCyT signed a cooperation agreement for participation at certain stages. Below is a detailed description of each process and sub-process. 12. Scholarship Enrollment. The enrollment period opens annually from February 1st to March 31st. The application process begins when a potential recipient enrolls as a program applicant. Two different enrollment mechanisms exist, depending on whether the application is submitted for the Basic Education scholarship, for the Technical and Professional Education scholarship, or for the Higher Education scholarship. For Basic Education Scholarships, most applications are submitted in person at the Comprehensive Service Units (UDAI) of ANSES, but they can also be filed before the Mobile Service Units (UDAM) of the same agency, or online. To complete the application at the UDAI, the applicant must first make an appointment either online or by phone (line 130). The application process requires filling in a form, which can be downloaded previously or be handed to the applicant in person. Together with the form, the applicant must submit a regular student certificate duly signed and stamped by the educational institution where he/she attends primary or elementary school. The data is manually entered into an ANSES platform. The ANSES staff checks the applicant’s personal information, identity card, identity information in the People’s Data Administration System (ADP), and that the stamp on the student certificate matches the authorized Single Institution Code (CUE). Once these checks are made, the process is approved. For more details on this process, see Figure A3.1. For enrollment in Higher Education scholarships, enrollment is through the MECCyT’s online platform, where the applicant must log in with a user name and password. This user name will be valid throughout the scholarship award process, since the Program will use this platform to communicate with the applicant (for instance, to notify the applicant of the need for additional information, the granting or denial of the scholarship, and the ways to file claims). By entering the user name and password, the applicant can fill in a form with his/her personal data and information on the area and institution where he/she is enrolled and/or studying. For more details on this process, see Figure A3.2. 13. Application Assessment. The application assessment process ensures the fulfillment of the terms and conditions set forth under the executive order that initiated the program (No. 90/2018). For this purpose, the MECCyT requests certain information from ANSES and Universities and Tertiary Institutions to determine the eligibility of the enrolled applicants. The socioeconomic assessment process aims to verify the applicant’s identity and validate personal and family economic status. At this point, the decision is made as to whether the applicant falls within the eligibility criteria established under the executive 91 Duflo et al (2015), Cristia et al. (2014) 92 Bergman (2015), Rogers & Feller (2018), Berlinski et al (2016), Bennett & Bergman (2018) 93 Based on an assessment carried out by the IDB in the context of the Programa de Apoyo a la Equidad y Efectividad del Sistema de Protección Social en Argentina (AR-L1302) and information provided by MECCyT. Page 81 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) order that created the program (No. 90/2018). The socioeconomic assessment is performed by ANSES, taking into account the income level reported by the applicant and his/her family. The academic requirement verification is as follows for the different scholarships. For Basic Education scholarships, the regular student certificate submitted by the applicant at the enrollment stage is deemed as sufficient. For this reason, no additional step is taken at this point. The information requested from Higher Education scholarships applicants through the enrollment form allows grouping of applicants by educational institution. In order to complete the applicants’ academic assessment for scholarships, MECCyT sends each university a TXT file for them to certify the applicant’s current career, enrollment year and progress. The tertiary institutions can access the information posted on the MECCyT’s web platform and certify the academic status of their applicants (year of entry, career and approved subjects). Finally, in the case of Teacher Training scholarships, the applicants must pass an exam administered by the SEE. Those who were granted this scholarship in the past must have passed 20 percent of the curricula subjects each year covered by the scholarship. 14. Scholarship approval and award. MECCyT receives information on the socioeconomic and academic assessments and, based on this information, initiates an information consolidation step to conclude the process. At this point, the decision is made to grant or deny the scholarship or to identify inconsistencies for later clarification. The scholarship amount is determined through a ministerial resolution and is determined based on the incentives scheme of the scholarships. 15. Payment and Accountability. The funds from the scholarships are paid throughout 10 months (from March to December) and are collected monthly in arrears. The recipients for Basic and for the first year of higher education receive 80 percent of the scholarship amount. The remaining 20 percent is withheld and paid the following year, once the requirements established under executive order No. 90/2018 are fulfilled. 16. Pre-settlement. MECCyT keeps the recipient database at the NB. Every month, it performs the pre-settlement process, which generates the list of scholars for the respective payment period, creating a TXT file that contains the personal data of the scholar, the payment period, and the items to settle. The pre-settlement process is sent to ANSES via an internal note (no amount indicated) and the TXT format file. Once this information is received, ANSES checks enrollment validity, existence of labor ID code (CUIL), and vital status of the individual according to the data contained in the People’s Data Administration system (ADP). ANSES updates its own file of PROGRESAR scholars (called Z2- PROGRESAR 2) based on the information submitted by MECCyT. Once these checks are completed, ANSES creates an issuance and payment order file, and assigns a payment terminal for scholars who have not yet been assigned one. Assignment is performed based on the individual’s address recorded in ANSES. The possible payment terminal options include: Banks (Nación or others), Correo Argentino, Rural offices or PIM (virtual wallet from Nación Servicios, via cell phone). Non-bank payment methods are available when scholars do not have easy access to a bank branch due to geographical location. ANSES sends the file to the scholar’s Payment area for a preliminary estimation of bank fees and notifies MECCyT of the amount to pay. This includes the amount to pay to the scholars and the applicable fees charged by the bank. 17. Payment. MECCyT then conducts the permanent updates and formalizes the payment order by sending a Note to the General Administration Directorate (DGA). The DGA generates an e-SIDIF file and submits it to the MH to allocate budget and send the funds to the Banco de la Nación Argentina account number 3793/42 (exclusive account for PROGRESAR Scholarships). With a second Note, it sends the Page 82 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) updated file to the Fund Payment Department, which is responsible for processing the payment to scholars. Once the funds have been transferred from MECCyT to ANSES, the latter uses an account in the Central Bank of the Republic of Argentina (BCRA), which receives the funds coming from the already mentioned Banco Nación account, and from where it makes the payments to the payment network (mainly banks and Correo Argentino). The monthly payment operation lasts approximately 40 days. In a large number of cases, the funds are credited to the scholars’’ savings accounts on day 1 of the payment operation; however, in some cases, the funds are not directly credited, so the payer institution keeps the funds until the scholar collects them at any time during that period. Likewise, Correo Argentino keeps the funds available for collection during that period of time. 18. Accountability of funds. Once that period is completed, the monthly payment operation is deemed as closed, and both the Banks and Correo Argentino must account for the received funds before the BCRA. For this purpose, they must submit a file stating the paid funds and the funds due for payment together with a note, which will be valid as a sworn statement for 100 percent of the funds sent. Likewise, the funds due for payment must be sent back to the BCRA account that originally made the transfer. At that point, the BCRA checks that the entities have reported the total number of cases they account for, the paid funds, the paid fees, the due amounts, and the unpaid fees. This accountability process before the BCRA can take up to 10 business days (or more in the case of Correo Argentino). The BCRA consolidates the information received by banks, posts the reported accounts into the system, and submits the information to ANSES. ANSES then consolidates the information on the results of the payment process, i.e., it makes a summary of the total funds transferred to the BCRA, how much was paid, how much was not paid and the amount of paid and unpaid fees. Based on this information, an electronic file is created, and a Final Report is sent to the MECCyT, including the summary data for the payment process with a table attached, specifying this information per payer institution and some additional data. Finally, ANSES takes the necessary steps to return the funds by instructing its Finance area to approve the return of funds. Thus, the Accounting area generates a Payment Order for the Banco Nación account 4982/45 (the account selected by the MECCyT for the return of funds). The Treasury area then executes the Payment Order and makes the payment to the MECCyT’s bank account. Figure A3.3 illustrates the scholarship payment process. Page 83 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Figure A3.1. Basic Education Scholarship Enrollment Process Performed in ANSES – MECCyT receives the final information directly from ANSES Page 84 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Figure A3.2. Higher Education Scholarships Enrollment Process Applicant ANSES MECCyT Universities Tertiary institutions Applicant enters Updates Progresar Web personal information Platform and sends TXT into the Progresar with applicant’s details Web Platform for validation Receives information and sends for SIU to Makes socio- update academic economic information assessment Updates academic path by CUIL number in Progresar Web Platform SIU updates data by CUIL number and reports the University Updates TXT and sends results to Updates TXT (only MECCyT those found in the system) and sends to MECCyT Consolidates data and issues Final Analysis Base Updates Web Platform to Notifies and, if communicate the approval, necessary, rejection or request for updates additional information. information or files claim Files claim or updates Yes data No End Page 85 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Figure A3.3. Payment Process Assessment of the Program Expenditure Framework 19. The scope of the proposed Operation would be to support a subset of Government Programs from 2019 to 2023, clustered around two results areas under a PforR Operation (US$300 million) and a Technical Assistance Operation (US$40.1 million) using the IPF instrument. The total cost of the activities Page 86 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) under the Operation is estimated at about US$1.133 billion, of which US$341 million would be financed through a Bank loan. 20. The PforR’s operating and investment budget totals US$1.093 billion, out of which US$300 million (27.4 percent) would be funded by the Bank. The scope of the PforR Program supports two Results Areas selected as priorities by MECCyT to foster access and completion at the secondary and higher education levels and to strengthen the student, teacher and institutional evaluation systems in the education sector. To reach these objectives, MECCyT is implementing two programs, which together conform the PforR Program: (i) the PROGRESAR Scholarship program for 18 to 24-year-olds; and (ii) the APRENDER student learning assessment system and the ENSEÑAR teacher training evaluation for recent graduates. 21. Results Area 1. PROGRESAR Scholarships (US$1.20 billion).94 This Results Area would finance the provision of education scholarships to beneficiaries, reimbursing approximately 23 percent of the total program spending over the proposed Operation duration. 22. Results Area 2. Supporting the implementation of the national standardized student learning assessment APRENDER,95 the ENSEÑAR assessment,96 and the evaluation of students applying to scholarships for teacher training (US$13 million). 94 PROGRESAR was established in 2014 as subsidy transfer to foster access and completion at all education levels for students aged 18-24 that belonged to the poorest households. The program was managed by ANSES and was conceived as a continuation of the AUH, which covers up to 17-year-olds. 80 percent of the monthly transfer was contingent on age and income, and 20 percent on regular attendance to a public education institution at any level. By 2017, the program reached 1 million beneficiaries, but monitoring of eligibility and attendance was carried out poorly. In 2018, the program was transferred to the MECCyT and was modified from a transfer to a scholarship program - PROGRESAR Scholarships- and significant improvements of the program have been introduced over the last two years. The main objectives of these changes were to align benefits with education outcomes, to promote more inclusion and merit-based benefits, and close the gap between education and labor markets by promoting Strategic Careers in higher education. 95 APRENDER was established in 2016 to obtain and to disseminate timely and quality information to better understand student learning, via a census-based standardized student learning assessment for primary (6th/7th grade) and secondary students (year 5/6) to : (i) measure trends in the quality of education over time; (ii) use the information to design strategies focused on improving quality; and (iii) share the results with provincial authorities, school principals, and the public to report on performance. The results are reported at the national, provincial and school levels and the SEE has actively adopted several strategies for the widespread dissemination of the results of APRENDER. They have done that both through a password-access online platform that provides each school with a tailor report analyzing the results in comparative perspective, and through an open online platform that allows for basic statistical analysis, disaggregated to the provincial level. 96 ENSEÑAR is an evaluation system for teacher training institutes. In 2017, the SEE launched ENSEÑAR to assess teacher training institutes’ quality through a student content knowledge assessment and a toolkit for self -institutional assessment. The SEE plans to improve the tools to assess the teacher training institutes and generate reports based on the findings from ENSEÑAR, including strategies for improvement to advise teacher training institutes and to advance towards a quality accreditation process based on relevant international experiences. Page 87 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Table A3.1. World Bank Financing – Expenditure Framework Component US$ Program (Bank funds) Jun 2019 2020 2021 2022 Jul 2023 Total Component 1: Support Government Programs 240 244 244 244 121 1093 (PforR) Results Area 1: 240 240 240 240 120 1080 PROGRESAR Scholarships Results Area 2: 0 4 4 4 1 13 National Student and Teacher Evaluation Systems 23. The Operation would be carried out through the MH and MECCyT. The MH, through the DPPEESA, would oversee overall implementation of the Operation and would be responsible for all actions related to disbursements and withdrawals. The MECCyT, through the DNB, would be responsible for the implementation of activities related to PROGRESAR Scholarships and for the coordination of the Operation, for fiduciary aspects, for technical and operational decision-making, and for monitoring and evaluation. The SEE would be responsible for student and teacher evaluations, including activities under APRENDER and ENSEÑAR. The SGE would be responsible for all activities related to ASISTIRÉ through its national coordination team. Further specifications for these roles and responsibilities are detailed in the POM. Financial sustainability 24. In the context of the IMF Stand-By Arrangement, the federal government committed to achieving a primary fiscal balance in 2019 and a 1 percent of GDP surplus in 2020. This represents a fiscal effort as high as 3 percentage points of GDP, given the 2018 deficit of 2.7 percent of GDP and the impact of past measures on the 2019 balance (e.g. pensions indexation, tax cuts). Achieving fiscal consolidation targets is key for the GoA’s fiscal program to stabilize and reduce its debt-to-GDP ratio, which reached close to 88 percent in 2018. Fiscal room will be thus limited at the federal level. An additional reduction in public investment and subsidies and a transitory increase in the export taxes are expected to contribute the most to fiscal consolidation. These efforts will be accompanied by decentralization of spending responsibilities to provinces of part of electricity, transport, and water subsidies. A stable macroeconomic environment and a recovery in economic activity, with a positive impact on public revenues, will thus be key to achieving fiscal targets without further expenditure adjustments. 25. In this context, the proposed program would add up to about 0.25 percent (US$250.6 million) of total government expenditures, and most of the Bank funding would only represent a change in the funding source within existing budgetary lines of the MECCyT. In a context of strong fiscal consolidation, there is a risk that a number of programs including the proposed operation could be affected unless there is a commitment by the authorities to prioritize social programs involving cash transfers and ringfence these expenditures. This risk would be mitigated since social investments are at the heart of GoA’s commitment to protect the vulnerable. Results Chain 26. Table A3.2 summarizes the Operation’s results chain, and how the DLIs plan to contribute to Page 88 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) achieve the expected results. Table A3.2. Results Chain Outcomes/intermediate Planned activities Products Objective outcomes PROGRESAR Scholarships Financial support - scholarships • Finance PROGRESAR scholarships • PROGRESAR Scholarships are • Basic financial needs of • Develop nationwide information system delivered to beneficiaries (by beneficiaries are met (*) to allow schools to certify enrollment and education level) • % of higher education attendance of students of PROGRESAR • Information system for scholarship candidates Basic Education academic certification is whose academic certification Increase implemented (DLI 5) is verified by their academic enrollment in Studies to inform improvements to the institution (DLI 1) basic program • % of applicants that comply education for • Study of the estimation of the PROGRESAR • Study of the program coverage with eligibility requirements vulnerable scholarships’ coverage and prioritization strategy by education level (Basic, youth • Diagnosis of the barriers to completion non-university higher faced by students • National helpdesk for scholars education and University) implemented • % of scholars’ re-enrolment Increase Tackle Information barriers (DLI 2) access to • Provide relevant information to potential • % of first-year scholars of higher beneficiaries in the scholarship higher education that study education for application process strategic careers (DLI 3) the most • National helpdesk to support scholars • Online toolkit for students to • Information system to certify vulnerable support their application, attendance of basic Tackle academic and motivational barriers academic progression and education is used by schools • Produce materials for academic and motivation is implemented (DLI (*) motivation support for students based on 4) • Students have access to the the diagnosis online toolkit • Pilot a mentorship program to provide academic, personal and motivational support to students Support the Student and Teacher Evaluation Systems Implementation and Dissemination of APRENDER • Develop the student assessments, including additional modules to widen its scope • Train jurisdictional personnel on the implementation of the assessment • Produce materials for the assessment, including developing new modules to add • APRENDER is implemented in new measures (e.g. on SES, household schools (DLI 6) education levels, etc.) • % of jurisdictional units that • Analyze assessment outcomes to produce received the training and Strengthening school reports awareness process for the • School principals use the student, • Coordinate strategies and logistics to assessment report to improve teacher and disseminate assessment outcomes pedagogical practices (DLI 7) institutional • Adapt and pilot “Teach” classroom • APRENDER results are evaluation observation tool to Argentina disseminated system • School principals’ access and Improve APRENDER reports use enhanced reports and • Collect international experiences on supplementary pedagogical reporting results from standardized materials assessments • APRENDER coverage increase • Focus groups with school directors on how to improve the APRENDER Summary • New APRENDER Summary report. Report uploaded in the platform Page 89 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) • Conduct new analysis on APRENDER for school Directors to access it assessments, optimizing the use of • Studies produced cumulative databases. • Teach Argentina is developed • Analyze additional APRENDER data in and piloted technical schools to characterize the educational dynamics of its students. • Analyze the human capital, infrastructure and other supplies of primary and secondary schools, with the purpose of building clusters. Evaluation of teacher institutes • International workshop on experiences on evaluation and self-evaluation of teacher • Self-evaluation of Institutes is training institutes. • Diagnostic evaluation system for implemented • Set up a committee of experts on teacher training plan (DLR 8.1) • Proposal for an external evaluation and self-evaluation of teacher • ENSEÑAR evaluation is evaluation system agreed training institutes implemented with stakeholders • Develop a suite of tools for self-evaluation • % institutes that receive • Diagnostic evaluation system including the adaptation of ¨Teach¨ customized reports with for teacher training is • Disseminate the suite of tools diagnostic of teacher practices implemented (DLR 8.2) • Implement ENSEÑAR (*) • Develop a proposal for an external evaluation system Discuss the proposal with relevant stakeholders Support the scale-up of ASISTIRÉ Scale-up of ASISTIRÉ in selected vulnerable schools nationwide • Development of key documents to scale- up the program (facilitators profile and • Number of Provinces with screening, training plan, training manual signed agreements for the for schools, diagnostic tools, intervention implementation of ASISTIRÉ in a protocols comprising key evidence-based group of vulnerable schools interventions on socio-emotional skills— • Documents are finalized and • Schools implementing e.g. motivation, study skills, etc.— printed ASISTIRÉ (and meeting teenage pregnancy prevention, and • ASISTIRÉ facilitators are trained minimum requirements) remedial education) in the revamped training • Hire and train ASISTIRÉ facilitators • Schools receive the program • Acquisition of program inputs (including inputs ICT technology, internet connectivity, • School teams are trained in the etc.) new online attendance system • Follow-up interventions Reduce • Train schools in use of the new online targeted to “at risk” students dropout rates attendance system in schools implementing in secondary ASISTIRÉ education in Revamp of ASISTIRÉ ASISTIRÉ • Improve accuracy of EWS schools in the • Analysis of main dropout causes selected • Improve evidence-based content of Provinces facilitators training (focus on key barriers • Implementing the improved • Improvement in students’ such as key socio-emotional skills, version of the EWS attendance, motivation, and teenage pregnancy prevention and • New training modules are performance (*) remedial education) finalized and printed • Develop dropout prevention toolkit • Dropout prevention toolkit (learning materials to support the distributed to schools evidence-based interventions covered in • ASISTIRÉ Familia implemented in the training) a number of provinces • Develop cell phone app ASISTIRÉ Familia to track real-time attendance and other key data in a sample of schools) (*) Expected outcome not measured by the Operation Page 90 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Lessons Learned 27. The Operation’s design is informed by lessons learned from other Bank projects , in particular the Colombia Access and Quality in Higher Education Project - PACES, which provided lessons on strategies to increase the enrollment of students from disadvantaged socioeconomic backgrounds in quality tertiary education programs, as well as on processes and institutional agreements that promote quality assurance systems in tertiary institutions. Other Bank projects including components to support vulnerable students’ trajectories towards higher education helped the design process, including the Ecuador Transformation of the Tertiary Technical and Technological Institutes Project. 28. Regarding improvements towards a quality assurance system, previous experiences indicate that governance issues and the capacity of training institutions to carry out the reforms are crucial elements for success. As sustained in PACES, if institutions do not have autonomy for decision-making or do not have the managerial or financial capacity to carry out the necessary changes for accreditation, quality is unlikely to improve. Building consensus and capacity within tertiary institutions and designing a sensible and organized transition towards a national unified strategy, as learned in the task Design of a Monitoring and Evaluation Strategy for the Higher Education Reform in Chile Project (P157268), is therefore a provision included in the technical assistance for a quality assurance system. 29. The support for M&E systems conducted in the framework of the Analytical Services and Advisory Supporting Evidence-Based Policy Interventions in Argentina, which supported dissemination and knowledge-sharing to school principals in schools in vulnerable contexts in the provinces of Buenos Aires, Salta and La Rioja, and previous collaboration with the SEE in the ongoing PROMER II. It also highlighted good practices in the empowerment of educational institutions’ self-evaluation capacity and provided lessons on political economy towards the construction of a solid national evaluation culture. 30. Previous experience and knowledge generated by projects that promoted access of vulnerable youth to higher education show financial incentives alone cannot support students’ trajectories, unless they are also supported by initiatives that promote resilience as a key socioemotional skill, as well as various kinds of remedial education tools. The Operation builds on extensive previous experience from Bank analytical and advisory tasks building knowledge and containing initiatives to foster socioemotional skills, such as Brazil, Skills and Jobs (P133162), Peru New umbrella for Education, Skills & Employment (P147264), and the ongoing pilot initiative in PBA within the framework of PROMER II. 31. Preparation will also benefit from the support provided by the Bank to the AUH national program through the ongoing Children and Youth Protection, which has helped close coverage gaps for vulnerable populations by supporting the ANSES in obtaining missing information on beneficiaries, thus allowing more children to enter the eligibility screening process. The Project aimed at exploiting new technologies to reduce paperwork to prove compliance, and reduced barriers through early alerts directed to beneficiaries and service providers, innovations aligned with the objectives of Components 1 and 2. Economic Justification 32. This section presents the economic and financial analysis of the Project . The analysis will focus Page 91 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) on: (i) financing and strengthening PROGRESAR Scholarships; and (ii) promoting the expansion of ASISTIRÉ – the most important program for dropout prevention in lower secondary education. These activities concentrate most of the Project resources (92 percent).97 33. The Operation is expected to yield substantial economic and social benefits through its contribution to increased educational attainment, but also through an improvement in educational quality. The Operation is expected to reduce dropout in Secondary and Higher Education –especially among the poorest households – and therefore to increase the number of years of education in the population. From the individual point of view, the Operation would have a direct positive impact on lifetime earnings of the beneficiaries through an increase in their labor productivity. Other individual labor outcomes are also expected to be affected, since higher educational attainment facilitates labor market entry for youth and increases the probability of being employed and having a better job. From a social perspective, the Operation would raise human capital endowment, having a positive impact on growth98 and therefore reducing poverty rates. Moreover, since it is targeted to disadvantaged populations, it would potentially influence the reduction of income inequality. Its impact on the society goes beyond the individual’s outcomes, since a more educated population has better health, less crime, higher democratic participation, more environmental consciousness, etc.99 Since the programs involved also aim to improve educational quality, all these effects are largely augmented.100 34. The analysis will focus on the monetary private benefits deriving from an increase in the number of years of education of the beneficiaries. Despite all the individual and social gains resulting from the Operation, many of them cannot be captured in the economic analysis since the available data do not allow it. Both PROGRESAR and ASISTIRÉ should lead to an increase in the number of years of education. This approach would only capture the most readily quantifiable benefits, as better educated individuals also benefit from improved health and greater life satisfaction, and society enjoys the multiple positive externalities provided by a more educated population. Moreover, the positive effect of the Operation on education quality translates into an increase in the returns to schooling that is more difficult to predict and quantify and is therefore not included in the economic analysis. Bearing this in mind, the economic benefits that are computed are only a lower bound of the Operation’s potential returns. 35. Efficiency. The Operation’s efficiency is evaluated with a cost-benefit analysis that compares the estimated costs of each of the Components of the Program with the expected benefits associated to the PDOs. The net effect of the Project on beneficiaries is estimated using a present discounted value (PDV) approach. This approach entails estimating the stream of benefits and costs of schooling over the lifetime of a representative student with and without the Operation. Given the structure of benefits and costs presented, an Internal Rate of Return (IRR) is calculated, which is defined as the discount rate that equates the net present value of the investment per student with the net present value of all earnings differential over a student’s life cycle. Both the PDV and the IRR of the Project allow us to assess whether it is a good investment from an economic point of view. 97 The remaining portion correspond to the student and teacher evaluation activities and are not included in the estimations. 98 As in an augmented Solow model. 99 See for instance Acemoglu and Angrist (2000) and Moretti (2004) for more on social returns to education. 100 See for instance Patrinos and Saellariou (2008) and Bertschy et al. (2009) for more on the impact of quality on earnings and Hanushek and Woessmann (2008) for more on its effect on growth. Page 92 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 36. Costs. The total estimated cost of ASISTIRÉ is US$14.4 million, US$4.4 million of which are expected to be devoted to an improvement in the design of the Program,101 while the other 10 are expected to finance its expansion. In effect, while the program reached 220 schools and around 50,000 students in 2018, it is expected to substantially expand in the following years. ASISTIRÉ is expected to be implemented in 800 new schools over the Program’s timeframe. If the average ASISTIRÉ beneficiaries per school are preserved in this expansion, 181,818 new students will be added to the 50,000 that benefited in 2018. If the number of students served by this Program expands linearly over the years until reaching 231,818 and distributing the cost of the improvement and the expansion accordingly among the projected beneficiaries gives as an average annual cost per student of US$6 for the improvement in the design and US$55 for the implementation of the expansion, totalizing an average cost per student of US$61. This value will be used as the estimated average cost per year and beneficiary of ASISTIRÉ. 37. The cost per scholar of the PROGRESAR Scholarships in the five-year period requires several assumptions. While there is a fixed budget of US$240 million each year, the students are distributed among 3 modalities: youth aged 18-24 in Basic education and Technical professional training, students aged 18-30 in Tertiary non-university studies, and individuals aged 18-30 pursuing a University degree. Assuming that the PROGRESAR students remain constant during the five years of the Operation in the level achieved in 2018, as well as their distribution across the different modalities of the program, 259,380 students would receive the scholarship in Basic education and Technical professional training, 151,000 in higher non-university education, and 195,000 in university level each year.102 Under this scenario, the yearly cost per scholar of the PROGRESAR Scholarships remain fixed at US$350 per scholar in Basic Education and Technical professional training, while the cost per scholar ranges between US$420 and US$620 with an even higher cost for teacher training in higher non-university institutes, and between US$420 and US$1300 in university. Due to a lack of data on the latter two modalities, we estimate the average cost per scholar for the latter two modalities by dividing all the program resources not devoted to Basic Education and Technical professional training (US$149.2 millions) by the number of students in these two modalities (346,000) and the resulting cost is US$431 per scholar each year. 38. Benefits. The benefits from the Operation are summarized in Figure A3.4. Another expected benefit in the framework includes improved management capacity through the Strengthening of the MECCyT (Subcomponent 4.4), which is not incorporated in the economic analysis. If it were, even higher returns would be expected, as future reforms would also be affected by this increase in efficiency. As previously explained, the social benefits and improvements in quality affecting the returns to schooling are also not considered. Thus, the numbers presented in this economic analysis should be considered as a lower-bound estimate of the Operation’s benefits. 101This amount includes the cost of the impact evaluation, which is added to the cost to be conservative. 102These assumptions imply a political decision of maintaining the amount of the scholarship constant in US dollars across the 4- year period instead of increasing coverage at the expense of a reduction in the amount of money that is transferred. Page 93 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Figure A3.4. Framework of Project Benefits 39. Estimating the impact of PROGRESAR Scholarships on short-term outcomes. We rely on rigorous evaluations of the impact of education scholarships to get a sense of the range of the effect that PROGRESAR can have on education attainment and other short-term outcomes.103 A review of the literature that evaluates the impact of scholarships is summarized in Table A3.3. Typically, the studies assess the effects of either merit-based or poverty-based scholarships on educational attainment (via enrollment and grade progression) and/or learning outcomes (captured by test scores).104 Poverty-based programs tend to deliver less-promising results in terms of learning outcomes but are still effective to increase the educational attainment of beneficiaries. Table A3.3. Summary of Evaluations of the Impact of Scholarships on short-term outcomes Effect on Effect on Study Type of scholarship Level of intervention educational test scores attainment Filmer, D. & Shady, N. (2009). "School Enrollment, Basic education 0.24 years of Selection and Test Scores". Development Research Poverty-based not significant (secondary) schooling Group, The World Bank. Blimpo, M. P. (2014) “Team Incentives for 0.66 years of Education in Developing Countries: A Randomized Basic education Merit-based - schooling (0.27 Field Experiment in Benin.” American Economic (secondary) sd) Journal: Applied Economics 6(4): 90-109 Behrman, J., Parker, S., Todd, p., and Wolpin, K. (2015). “Aligning Learning Incentives of Students 0.41 years of Basic education and Teachers: Results from a Social Experiment in Merit-based - schooling (0.17 (secondary) Mexican High Schools.” Journal of Political sd) Economy 123 (2): 325-364. Angrist, J., Autor, D., Hudson, S., and Pallais, A. "Evaluating Post-Secondary Aid: Enrollment, Poverty and Merit- Higher education 0.52 years of - Persistence, and Projected Completion Effects." based (college) schooling NBER working paper 23015. 103There is not a rigorous impact evaluation of PROGRESAR Scholarships. 104In the Table, we express the estimated effects on learning outcomes (always reported as standard deviations of the exam score) in terms of years of schooling. To that end, we use the PISA equivalence between learning outcomes and school attainment (0.41 standard deviations equals the effect of 1 year of schooling). Page 94 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Duflo, E., Dupas, P. and Kremer, M. (2017). The 0.32 years of 0.09 years of Poverty and Merit- Basic education Impact of Free Secondary Education: Experimental schooling schooling (0.15 based (secondary) Evidence from Ghana. (annualized) sd annualized) 40. There are limitations to using this literature review to estimate the impact of PROGRESAR on short-term outcomes. First, the experiments evaluated in the literature have limited external validity. Second, PROGRESAR scholarships are different to the programs evaluated in the literature in several dimensions. For instance, the age group that is covered is different, and more importantly, the majority of PROGRESAR beneficiaries attend either tertiary university education or tertiary non-university. To the best of our knowledge, however, no rigorous evidence exists in developing countries on the impact of scholarships at the Tertiary level, while the existing evidence for developed countries is not conclusive.105 In an effort to avoid possible upward biases in the estimation of the benefits, we chose to use the lowest effect found in the literature. Under this conservative assumption, the impact of the financial support of PROGRESAR scholarships is estimated as 0.24 years of schooling per year of exposure to the program. 41. This effect does not include the expected impact of the complementary activities included in the Operation. The effect of these complementary activities comes from a literature review of the effect of selected interventions on students that is averaged and multiplied by a correction factor of 0.1 trying to capture the relatively lower intensity of these activities in PROGRESAR Scholarships with respect to those found in the literature. This additional effect is included in the sensitivity analysis. 42. To estimate the impact of ASISTIRÉ on short term outcomes, we rely on the limited empirical evidence on EWS along with targeted interventions. A recent RCT in the U.S. showed that an early warning system that relies on basic data (chiefly absences, grades, and behavioral problems), coupled with close follow-up of students at-risk by the school team (case workers), reduced chronic absenteeism and course failure by almost 20 percent after only one year of implementation (Faria et al, 2017). The statistically significant difference of 20 percent (or 5 percentage points) that the paper finds in course failure between students in treatment and control schools would translate into 0.05 years of education for the average treated student if course failure would be equivalent to grade failure. Since only a fraction of students who fail a course end up failing the grade, we assume half of the effect find in this paper for ASISTIRÉ (0.025).106 However, ASISTIRÉ has distinctive features that differ from the evaluated program. Because of this, the alternative that is explored in the sensitivity analysis section is the use of an average effect among the impacts that are found in the literature on interventions on at-risk students that affect the same channels: support to the students (mainly remedial education and socioemotional skills) and communication with parents. A correction factor of 0.1 is applied to the estimated effects since ASISTIRÉ has a significant lower intensity intervention with respect to the reviewed programs. 105 Evidence from the U.S. on merit-based scholarships suggest either a positive effect on academic achievement (Dynarski 2008, Castelman 2014, Scott-Clayton 2011, Scott-Clayton and Zafar 2016, Bettinger at al. 2016) or no effect at all (DesJardins and McCall 2014, DesJardin at al. 2010, Sjoquist and Winters 2012, Sjoquist and Winters 2015). Quasi-experimental identification strategies on poverty-based scholarships find either positive or null effects (see Deming and Dynarski 2009 (US), Nielsen at al. 2010 (Denmark), Dearden et al. 2014 (UK) for positive effects and Baumgartner and Steiner 2005 (Germany) for null effects). 106 We would obtain a similar estimate if we use instead the only rigorous evaluation of an EWS in a developing country (the USAID-funded School Dropout Prevention Pilot Program (SDPP) in Cambodia, India, Tajikistan, and Timor-Leste). This program had a statistically significant favorable impact on dropout rates and grade progression in Cambodia, where 63.3 percent of students in EWS + Engagement schools enrolled in the ensuing grade, while only 61.5 percent did so in the control group. This statistically significant difference of 1.8 percent points can be translated into an increase in 0.018 years of education for the average treated student, which is similar to the 0.025 chose to use in our baseline estimation. Page 95 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 43. To translate the expected increase in years of schooling for a student into wage improvements, we will use the standard labor economics model to estimate the returns of education, namely the Mincer (1974) equation: = ̂1 + ′ ̂ + ̂ + ̂ Where Wi is the logarithm of the hourly wage of individual i; Si is the years of schooling of i; X is a vector of controls; ̂ , and ̂ , ̂ are the estimated parameters; and ̂ is the error term. 44. The yearly monetary benefit of the Project can be captured by the increase in the years of schooling times the estimated returns to education, as described in the following equation: ̂ ∗ ∆ = ∗ where are the monetary benefits of the Project at time t, ∆ captures the impact in years of schooling; ̂ corresponds to the returns to education, and stands for the expected labor income in year t for a student without the Project. 45. The Mincer equation for Argentina is estimated using microdata from EPH corresponding to the second semester of 2017. For the estimation, the set of controls we include comprises a sex dummy, potential experience, potential experience squared, regional dummies, a wage-earner dummy, and six sectorial variables. The parameters in the equation are estimated by Heckman full maximum likelihood using individuals aged 25–55. The selection equation includes the same covariates in the wage equation plus number of children, number of children interacted with the gender dummy, a marriage indicator and a school attendance binary variable. However, alternative estimations of the returns to schooling in Argentina are used to check the robustness of the results. We use the yearly labor income per capita (US$) as the reference income Yi.107 Multiplying this Yi by the estimated returns in the Mincer equation ̂ and the estimated impact on short-term outcomes ∆ we obtain the yearly benefits of the Project per student 108, which are summarized in the Table below. The yearly benefits of ASISTIRÉ amount to US$15 per year, while the benefits of PROGRESAR total US$186 per year. Table A3.4. Benefits per student each year by Program Component and Results Area Expected Return to Impact in Yearly labor income education years of benefits without the schooling of the project project Yt β Δ St Bt ASISTIRÉ + PROGRESAR Scholarships $9,915 0.0814 0.265 $ 214 ASISTIRÉ $9,915 0.0814 0.025 $ 20 PROGRESAR Scholarships $9,915 0.0814 0.240 $194 Sources: (1) Expected labor income without the project computed as the average monthly labor income in nominal LCU (computed from SEDLAC microdata - second semester 2017) * 12 * average nominal exchange rate in second semester of 2017: $14386.84*12*(1/17.413). (2) Return to education estimated from a Mincer equation estimated by Heckman full maximum likelihood. (3) Impact in years of schooling of ASISTIRÉ comes from the only rigorous evaluation of an Early Warning System with impact in dropout rates, while the impact of PROGRESAR comes from the minimum effect found in the literature of scholarships. 107 Other alternatives are explored in the sensitivity analysis section. 108 An implicit assumption in this estimation is that the hours of work are not affected by the Operation. Page 96 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 46. Cost-benefit analysis. The last step of the analysis is to compare costs and benefits. Using a discount rate, the yearly cost per student is brought to the present, as well as the whole stream of yearly future benefits . We assume that students benefited from the Operation enter the workforce at age 21 and retire at age 61 (after 40 years of work) earning US$ per year more than their peers. For the PROGRESAR scholarships, we maintain the 40-year timeframe but delay the entrance to the workforce until the age of 25.109 With a conservative discount rate (5 percent), the Net Present Value of the income benefits of the average Operation beneficiary are compared with its cost, and the IRR is computed. The Table below summarizes the results. Results from the economic analysis suggest that benefits from the expansion of ASISTIRÉ are 4 times its cost, while benefits from PROGRESAR scholarships in Basic Education and Technical Professional Training exceed 7 times its cost. The Benefit/Cost ratio for improving PROGRESAR scholarships in higher education is a bit lower than in the other level (5.8), which is explained by the relatively higher cost of these types of scholarships. However, the robustness of this result to the possible presence of heterogeneous impacts on short-term outcomes (i.e. a non-linear return to education) is explored in the sensitivity analysis. Table A3.5. Results of the economic analysis Benefit/Cost Internal Rate of Ratio Return (IRR) PROGRESAR scholarships Basic Education 7.1 19 % PROGRESAR scholarships Higher education 5.8 17 % Scale-up of ASISTIRÉ 4.0 13 % 47. Sensitivity analysis. The analysis is based on several parameters and assumptions that can be relaxed or modified to check the robustness of results. We test the consistence of the results to: (i) different discount rates; (ii) different estimations of the impact of the program on schooling; (iii) different estimations of the returns to schooling (in particular, allowing for non-linearity in the returns to additional years of education at basic, tertiary non-university, and university education); (iv) different reference incomes; (v) different profiles of insertion in the labor market (varying the entrance and retirement age, or the timeframe). The results are presented in the following tables. Table A3.6. Benefit/Cost ratios with different discount rates 5% 7% 10% PROGRESAR scholarships Basic Education 7.1 4.9 3.1 PROGRESAR scholarships Higher education 5.8 4.0 2.5 Scale-up of ASISTIRÉ 4.0 2.7 1.7 109More specifically, we assume that the benefits of the project start after 7 years for ASISTIRÉ and after 8 years for PROGRESAR. Moreover, all the computations are expressed per year of investment to avoid having to make additional assumptions regarding the number of years of exposure to the program. Page 97 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Table A3.7. Benefit/Cost ratios with alternative estimations of the impact of the program on schooling110 With impact estimations from programs Baseline affecting the same channels (revamped/improved programs) PROGRESAR scholarships Basic Education 7.1 7.4 PROGRESAR scholarships Higher education 5.8 6.0 Scale-up of ASISTIRÉ 4.0 6.3 Table A3.8. Benefit/Cost ratios with non-linear returns to schooling111 With non-linear Baseline returns to schooling PROGRESAR scholarships Basic Education 7.1 3.9 PROGRESAR scholarships higher non-University education 5.8 6.1 PROGRESAR scholarships University 5.8 8.0 Scale-up of ASISTIRÉ 4.0 2.2 Table A3.9. Benefit/Cost ratios with the same return to education (in %) but a lower reference income112 With lower Baseline reference income PROGRESAR scholarships Basic Education 7.1 3.2 PROGRESAR scholarships Higher education 5.8 2.6 Scale-up of ASISTIRÉ 4.0 1.8 Table A3.10. Benefit/Cost ratios postponing 5-years the stream of benefits With delayed insertion Baseline in the labor market PROGRESAR scholarships Basic Education 7.1 5.6 PROGRESAR scholarships higher education 5.8 4.5 Scale-up of ASISTIRÉ 4.0 3.1 110 The assumed impacts of ASISTIRÉ and PROGRESAR on schooling in these estimations are 0.044 and 0.255, respectively. The impact of ASISTIRÉ results from the 0.15 years of schooling from interventions on students coming from Heller et al. (2017) and Oreopoulos et al. (2017) scaled-down by a factor of 0.1, plus the intervention on parents in Berlinski et al. (2016) equivalent to 0.029 years of schooling. The impact of PROGRESAR results from adding to the baseline estimation the impact of complementary activities, assumed to have the same impact as the intervention on students in ASISTIRÉ (0.15*0.1=0.015). Since we assume students served in the first year of the Operation would not perceive the benefit from the improved programs, a correction factor of 3/4 was also applied to the additional yearly benefits. 111 Returns to an additional year of schooling at basic, tertiary and university education are separately estimated by including degree completion dummies in the Mincer equation and dividing the estimated effect of an additional degree by the theoretical number of years necessary to complete each level. Estimated returns per year of education are therefore 0.0450 for basic education, 0.0857 for tertiary and 0.1136 for university education. 112 The reference income in these estimations is the per capita labor income of the bottom 50 percent of the income distribution. Page 98 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 4. FIDUCIARY SYSTEMS ASSESSMENT COUNTRY : Argentina Improving Inclusion in Secondary and Higher Education 1. Conclusion 1.1 Reasonable Assurance 1. The Procurement and Financial Management (FM) systems’ capacity and performance, with the implementation of the proposed mitigating measures and agreed actions to strengthen the systems (which are reflected in the Program Action Plan, PAP), are adequate to provide reasonable assurance that the Program funds will be used for the intended purposes, with due attention to the principles of economy, efficiency, effectiveness, transparency, and accountability. 1.2 Risk Assessment 2. The overall integrated fiduciary risk rating is considered Moderate. The key fiduciary risks to the development outcomes of the Project that underpin the Moderate risk rating are as follows: (i) unfamiliarity of the Borrower with the Program for Results (PforR) Instrument; (ii) special conditions included in the Standard bidding documents for goods, non-consulting and consulting services that do not indicate any thresholds in order to determine the weight of quality and cost bids for the total score; (iii) some inadequacies in contract management; and (iv) the risk of awarding a contract to a suspended or debarred firm or individual suspended by the Bank. 3. The proposed systems-and capacity-strengthening and/or mitigation measures, to address the above risks include the following: (i) close implementation support to be provided by the Bank’s fiduciary teams, coupled with the preparation of a Project Operations Manual (POM); (ii) the implementing agency, with the assistance of the requesting areas, shall determine the maximum thresholds related to quality (technical) and cost (financial) weight in the special conditions included in the bidding documents; (iii) preparation of a manual, as a reference document for all staff involved in contract management and the training of relevant people accordingly; (iv) the Dirección de Contrataciones will indicate in the evaluation reports - in writing - the eligibility verification carried out and on June 30th of each year, the implementing agency will send to the Bank, the list of contracts awarded to firms and individuals; and (v) the external auditors’ Terms of Reference (TOR) will include a requirement to review Program expenditure for such ineligible contracts. 1.3 Procurement Exclusions 4. There are no potential high-value contracts identified under the Program at this moment. The proposed Program is not expected to finance any contract at or above prevailing Operations Procurement Review Committee (OPRC) thresholds, which at the time of the assessment were at minimum US$75 million for goods, information technology and non-consulting services and US$30 million for consulting services. 2. Scope 5. The scope of the proposed Project will be to support a subset of Government Programs from 2019 to 2023, clustered around two results areas under a PforR Operation (US$300 million) and a Technical Assistance Operation (US$40.1 million), that would use the IPF instrument. The total cost of the activities Page 99 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) under the Project is estimated at about US$1.133 billion, of which US$341 million would be financed through a Bank loan. 6. The PforR’s operating and investment budget totals US$1.093 billion, out of which US$300 million (27.44%) will be funded by the Bank. The scope of the PforR Program supports two Results Areas selected as priorities by the Ministry of Education, Culture, Science and Technology’s (Ministerio de Educación, Cultura, Ciencia y Tecnología, MECCyT) to foster access and completion at the secondary and higher education levels and to strengthen the student, teacher and institutional evaluation systems in the education sector. To reach these objectives, MECCyT is implementing two programs, which together conform the PforR Program: (i) the PROGRESAR Scholarship program for 18 to 24-year-olds; and (ii) the APRENDER student learning assessment system and the ENSEÑAR teacher training evaluation for recent graduates. 7. Results Area 1. PROGRESAR113 Scholarships (US$1,080 million). This Results Area would finance the provision of education scholarships to beneficiaries, reimbursing approximately 23 percent of the total program spending over the proposed Project duration. 8. Results Area 2. Supporting the implementation of the national standardized student learning assessment APRENDER,114 the teacher evaluation ENSENAR115 and the evaluation required to students applying to scholarships for teacher training (US$13.0 million). The Operation would support the annual census-based evaluations for the period 2020-2023. This Results Area also includes the evaluation of teachers-to-be in their last year, before graduation (through the ENSEÑAR evaluation; optative for teachers, and that has already been conducted once in 2017). 113 PROGRESAR was established in 2014 as subsidy transfer to foster access and completion at all education levels for students aged 18-24 that belonged to the poorest households. The program was managed by the National Administration of Social Security (Administración Nacional de la Seguridad Social, ANSES) and was conceived as a continuation of the AUH, which covers up to 17- year-olds. 80 percent of the monthly transfer was contingent on age and income, and 20 percent on regular attendance to a public education institution at any level. By 2017, the program reached 1 million beneficiaries, but monitoring of eligibility and attendance was carried out poorly. In 2018, the program was transferred to the MECCyT and was modified from a transfer to a scholarship program - PROGRESAR Scholarships- and significant improvements of the program have been introduced over the last two years. The main objectives of these changes were to align benefits with education outcomes, to promote more inclusion and merit-based benefits, and to close the gap between education and labor markets by promoting Strategic Careers in higher education. The program offers four types of scholarships that focus on i) overaged students completing primary and/or secondary education; and ii) students accessing and completing higher education. 114 APRENDER was established in 2016, together with the SEE, to obtain and to disseminate timely and quality information to better understand student learning, via a census-based standardized student learning assessment for primary (6th/7th grade) and secondary students (grade 5th/6th) to : (i) measure trends in the quality of education over time; (ii) use the information to design strategies focused on improving quality; and (iii) share the results with provincial authorities, school principals, and the public to report on performance. The results are reported at the national, provincial and school levels and the SEE has actively adopted several strategies for the widespread dissemination of the results of APRENDER. They have done that both through a password-access online platform that provides each school with a tailored report analyzing the results in comparative perspective, and through an open online platform that allows for basic statistical analysis, disaggregated to the provincial level. Since 2016 APRENDER has increased coverage of schools (from 88 to 95 percent) and of students within those schools (71 to 78.7 percent). 115 ENSEÑAR is an evaluation system for teacher training institutes. In 2017, the SEE launched ENSEÑAR to assess teacher training institutes’ quality through a student content knowledge assessment and a toolkit for self -institutional assessment. The SEE plans to improve the tools to assess the teacher training institutes and generate reports based on the findings from ENSEÑAR, including strategies for improvement, to advise teacher training institutes and to advance towards a quality accreditation process based on relevant international experiences. Page 100 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 9. The main implementing agency of the Project would be the MECCyT, through its National Directorate for Educational Scholarships (Dirección Nacional de Becas Educativas, DNB), under the MECCyT. This Directorate would be responsible for coordination of the Project, for fiduciary aspects (including operational and procurement plans), technical and operational decision-making, and for M&E. Within the MECCyT, Project activities would be carried out by three separate technical units, as follows: (i) the Secretariat for Educational Management (Secretaría de Gestión Educativa, SGE) would be responsible for all activities related to ASISTIRÉ; (ii) the DNB would be responsible for implementation of activities related to the PROGRESAR scholarships; and (iii) the Secretariat of Education Evaluation (Secretaría de Evaluación Educativa, SEE) would carry out the APRENDER assessment and all other evaluations and/or assessments under the Project. The DNB would coordinate and oversee Project activities among these technical units to ensure proper implementation and would be the Bank’s main technical counterpart. The Ministry of Finance (Ministerio de Hacienda, MH) would be the other Project executor, with clear roles and responsibilities, differentiated between the MECCyT and MH. 10. Pursuant to the World Bank’s (Bank) Policy and Directive for Program for Results Financing, the Bank’s fiduciary team, comprising FM and procurement specialists, conducted an integrated Fiduciary Systems Assessment (FSA) of the Federal fiduciary systems and entities involved in the Program. The overall objective of the FSA was to determine whether the fiduciary systems of the Program provide reasonable assurance that the Program financing proceeds will be used with due attention to the principles of economy, efficiency, effectiveness, transparency and accountability. 3. Review of the Public Financial Management Cycle 3.1 Planning and Budgeting 3.1.1 Adequacy of Budgets 11. The Project will follow the Federal budget cycle and process. The formal budget process in Argentina consists of four stages. The first stage entails the formulation of the Executive’s budget proposal, and the second involves the congressional discussion of the proposal and approval of the budget law. In the third stage, the proposal is implemented, and in the fourth, the execution of the budget is evaluated and controlled. 12. In a nutshell, during the formulation stage, medium-term ceilings on budget aggregates and ministry/agency expenditures will be proposed by the National Budget Office (Oficina Nacional de Presupuesto, ONP) and approved by the Head of the Cabinet, and the ministries/agencies will then prepare their preliminary draft budgets on that basis. The MECCyT’s, budget unit acts as the focal point for preparation of the budget documentation to be sent for ONP review. Consequently, there is due process between the implementing agency and the ONP for negotiating the annual budget. The MECCyT’s detailed planning and budgeting activities encompass the following: (a) planning and providing assistance to the various budget sector units; (b) analysis and adjustment of proposed programs with priorities and budget availabilities, and (c) consolidation and preparation of the final budget documentation. Staff are familiar with the budget cycle and respective rules and regulations. The ONP, the ruling authority of the budget process, then evaluates the anteproyectos (preliminary projects) prepared by the ministries/agencies and puts together the aggregate proposal. The budget is prepared using institutional, program, functional, source of financing, and economic classifications, amongst others. Page 101 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 13. The Executive must submit the annual budget116 to the National Congress for its approval before September 15, when the approval stage begins. The National Congress then evaluates the proposal through the Comisión de Presupuesto y Hacienda (Budget Committee) of both chambers and can summon members of the Executive to discuss the issues it finds relevant. The National Congress is empowered to approve or reject the budget proposal, to introduce modifications, and to remove articles. After the National Congress approves the budget, the Executive can introduce line item vetoes. 14. During the next stage, the implementation stage, the Executive distributes the expenditure authorizations among the line ministries and introduces modifications. The ONP prepares quarterly Physical and Financial Monitoring Reports of the National Administration, as required by the Financial Administration Law (LAF) of 1992, revised in 2006. The legislature provides adequate oversight during the planning stage of the budget cycle, but only weak oversight during the implementation stage of the budget cycle, as the legislature is not consulted prior to the government shifting funds between administrative units117 specified in the Enacted Budget, spending unanticipated revenue, or reducing spending due to revenue shortfalls. 15. Budget execution is recorded in the Federal Government's Integrated Financial Management Information System (Sistema Integrado de Información Financiera, or e-SIDIF) which began operations in 1993. It has been improved in phases, and now provides capabilities for modern treasury management, including features to record and control commitments, process payments, account for cash and accrual transactions, and prepare financial statements. e-SIDIF is used by all Central government budget entities. e-SIDIF is also linked to the Debt Management and Information System (SIGADE). The MH which manages e-SIDIF, controls and verifies budget execution. e-SIDIF uses comprehensive expenditure classifications where expenditures can be classified and reported by institution, category, financing source, economic use, programs, and function. e-SIDIF also has modern Business Intelligence tools for the manipulation of data and generation of custom reports. Greater flexibility/disaggregation in the Chart of Accounts has also resulted in an improvement in the automatic registration of transactions and facilitated comparisons of accounting to budget data, with the system capable of recording multi-currency accounting entries. e- SIDIF also has an automated bank reconciliation, that facilitates the task of reconciling information between bank statements and the accounting records, to guarantee the reliability, accuracy and completeness of the accounting records. Integrations between the e-SIDIF system and the transversal systems ComprAR, ContrarAR, LOYS, GAT has also contributed to greater transparency. 16. During the final stage, the evaluation and control stage, the National Congress—through the Government Supreme Audit Institution (Auditoria General de la Nación, AGN)—performs ex post controls regarding budget implementation and must evaluate and approve the Cuenta de Inversión (the annual budget execution report and financial statements elaborated by the Executive). The AGN provides adequate budget oversight. Under the law, it has full discretion to undertake audits as it sees fit. Moreover, the head of the institution is appointed by the legislature and cannot be removed without 116 Argentina’s score of 50 on the 2017 Open Budget Index is lower than its score in 2015 as Argentina has decreased the availability of budget information by reducing the information provided in the Executive’s Budget Proposal and has failed to make progress in producing and publishing a Mid-Year Review or Citizens Budget online in a timely manner and publishing a Pre-Budget Statement that only contains minimal budget information. Argentina’s score of 13 out of 100, indicates that it also provides few opportunities for the public to engage in the budget process. This is higher than the global average score of 12. 117 This risk will be monitored during supervision missions, to ensure there is adequate budget allocated to implement the Program. Page 102 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) legislative approval, which bolsters its independence. Finally, the AGN is provided with sufficient resources to fulfill its mandate. 17. In addition, Argentina did not have a functioning Independent Fiscal Institution (IFI) before December 21, 2016, when the Oficina de Presupuesto del Congreso (Congressional Budget Office, CBO) was created by Law 27.343 as an important source of independent, nonpartisan budget information. As part of the IMF Stand-by-Arrangement (SBA), the Argentine government has undertaken to adequately resource and staff the CBO, so that it can effectively: (i) evaluate budgetary and macroeconomic forecasts (including those contained in the annual budget and mid-year budget report); (ii) provide independent costing to the National Congress of new policy initiatives; and (iii) assess the government’s fiscal plans, including the annual budget. There would also be a comprehensive examination of the CBO’s design features to ensure that it is fully able to achieve the government’s desired objectives. 18. All the Program (and Project) budgeting transactions will be processed through e-SIDIF (and UEPEX) by the respective executing agencies. 19. Review of the budget structure and classification-Program Budget Composition by Program/Activity and Expenditure Categories: Page 103 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Component 1: Support Key Government Programs (PforR) FY 2019 FY2020-2023 FY2019-2023 FY2019-2023 Results Area 2: National Student and Results Area 1: PROGRESAR Scholarships Teacher Evaluation Systems Detai Category of Expenditure based on Results Area Results Area #1 Results Area #1 Results Area #2 Results Area #2 Total Year 1 Total Year 1 Total Years Total Total led Economic Classification #1 "Otorgamiento "Otorgamiento APRENDER Y APRENDER Y (FY19 (FY19 2,3,4,5 Duration of Duration of Budg "Otorgamien de Becas de Becas ENSEÑAR ENSEÑAR Budget) - Budget) - Program Program et to de Becas Compromiso Socioeducativas "Evaluación "Acciones Million ARS Million USD *(Estimated *(Estimated *(Estimated Line Educativas: Docente" " Educativa" Complementarias Million ARS) Million ARS) Million USD) PROGRESAR" Programa 49 - Programa 49 - Programa 32 - para el Programa 49 Actividad 43 Actividad 02 Actividad 3 Fortalecimiento - Actividad 40 de la Evaluación Educativa - PROMER II " 49.0.0.4 Transfers Grants / Transferencias al 0.3.4.11 Sector Privado para Financiar Gastos $ 9,480.62 $- $- $ - $ - $ 9,480.62 USD 232.65 $ 33,203.10 $ 42,683.72 USD 1,047.45 .5.1.3 Corrientes - Becas 49.0.0.4 Transfers Grants / Transferencias al 3.3.4.11 Sector Privado para Financiar Gastos $ - $300.00 $- $ - $ - $ 300.00 USD 7.35 $ 1,026.90 $ 1,326.90 USD 32.55 .5.1.3 Corrientes - Becas 32.0.0.3 Gastos en Personal - Personal .3.4.11. $- $- $ - $ - $ - $ - USD 0.00 $ 13.83 $ 13.83 USD 0.34 1.1.1 Permanente - Retribución del cargo 32.0.0.3 Gastos en Personal -Personal .3.4.11. Permanente - Retribuciones que no $- $ - $ - $ - $- $ - USD 0.00 $7.86 $ 7.86 USD 0.19 1.1.3 hacen al cargo 32.0.0.3 Gastos en Personal -Personal .3.4.11. Permanente - Sueldo Anual $- $ - $ - $ - $- $ - USD 0.00 $ 1.81 $ 1.81 USD 0.04 1.1.4 Complementario 32.0.0.3 Gastos en Personal -Personal .3.4.11. Permanente - Contribuciones $- $ - $ - $ - $- $ - USD 0.00 $ 5.87 $ 5.87 USD 0.14 1.1.6 Patronales 32.0.0.3 Gastos en Personal - Servicios .3.4.11. Extraordinarios - Retribiciones $- $- $- $- $- $- USD 0.00 $ 0.53 $ 0.53 USD 0.01 1.3.1 extraordinarias 32.0.0.3 Gastos en Personal - Servicios .3.4.11. Extraordinarios - Sueldo Anual $- $- $- $- $- $- USD 0.00 $ 0.04 $ 0.04 USD 0.00 1.3.2 Complementario 32.0.0.3 Gastos en Personal - Servicios .3.4.11. Extraordinarios - Contribuciones $- $- $- $- $- $- USD 0.00 $ 0.14 $ 0.14 USD 0.00 1.3.3 Patronales 32.0.0 Gastos en Personal - Personal .3.3.4. Contratado - Retribuciones por $- $- $- $- $- $- USD 0.00 $ 5.14 $ 5.14 USD 0.13 11.1.8 .1 Contrato 32.0.0 .3.3.4. Gastos en Personal - Personal 11.1.8 $- $- $- $- $- $- USD 0.00 $ 1.60 $ 1.60 USD 0.04 Contratado - Adicionales al Contrato .2 32.0.0 Gastos en Personal - Personal .3.3.4. 11.1.8 Contratado - Sueldo Anual $- $- $- $- $- $- $- $1 $1 $0 .3 Complementario Page 104 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 32.0.0 Gastos en Personal - Personal .3.3.4. 11.1.8 Contratado - Contribuciones $- $- $- $- $- $- $- $2 $2 $0 .5 Patronales 32.0.0 .3.3.4. Bienes de Consumo $- $- $- $- $- $- $- $2 $2 $0 11.2 32.0.0 .3.3.4. Servicios Básicos $- $- $- $- $- $- $- $ 20 $ 20 $0 11.3.1 32.0.0 .3.3.4. Servicios Técnicos y Profesionales $- $- $- $- $- $- $- $ 339 $ 339 $8 11.3.4 32.0.0 .3.3.4. Servicios Comerciales y Financieros $- $- $- $- $- $- $- $ 61 $ 61 $1 11.3.5 32.0.0 .3.3.4. Pasajes y viáticos $- $- $- $- $- $- $- $6 $6 $0 11.3.7 32.0.0 .3.3.4. Otros Servicios $- $- $- $- $- $- $- $ 50 $ 50 $1 11.3.9 32.0.0 .3.3.4. Equipos para computación $- $- $- $- $- $- $- $ 14 $14 $0 11.4.3 .6 Total $ 9,481 $ 300 $- $- $- $ 9,781 $ 240 $ 34,760 $ 44,540 $ 1,093 Page 105 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Adherence of the budgeted Program expenditure and its execution to the Government’s priorities Budget vis-à-vis Actual Expenditures, 2016-18-Results Area #1-PROGRESAR FY 16 FY 17 FY 18 Actual Expenditure (million AR$) 7,780 8,156 9,978 Original Budget (million AR$) 8,196 8,758 9,998 Budget Outturn 94.9% 93.1% 99.7% Budget vis-à-vis Actual Expenditures, 2016-18-Results Area #2-APRENDER and ENSENAR FY 16 FY 17 FY 18 Actual Expenditure (million AR$) 232 294 258 Original Budget (million AR$) 292 367 388 Budget Outturn 79.4% 80.1% 66.4% 20. Assessment of the Program’s financial sustainability and funding predictability-The Program’s budget is realistic, prepared with due regard to the Argentina’s Government’s policy, and implemented in an orderly and predictable manner. There is a reasonable expectation that the required resources will be appropriated in the financial years when required. 3.1.2 Procurement Planning 21. Project Implementation Unit should gather procurement requirements from the different programs - according to annual budget allocation - and provide all necessary information to MECCyT’s Dirección de Contrataciones (Procurement Unit) - depending on the Dirección General de Administración y Gestión Financiera - to elaborate Annual Procurement Plans. Office of National Procurement (“Oficina Nacional de Contrataciones”) centralize Annual’s Procurement Plans and post it in its web site or in the Electronic Procurement System named “Comprar.ar”. At the moment, there are advances in procurement planning with estimated costs and procurement methods for the activities financed by the Program. However, there are no advances in planned dates. The detail of planning is a list of demand for goods, non-consulting and consulting services prepared by the implementing unit. This list is prepared based on the five-year plan, but the level of detail does not get into planned dates. 22. The national framework of regulations is solid and transparent and is familiar to both public officials and the private sector. Decrees N° 1.023/2001 and N° 130/2016 are the main legal texts regulating procurement in the country, they contain the set of rules of the National Procurement System (“Régimen de Contrataciones de la Administración Nacional”), and regulates procurement of goods, works and services. 23. The procurement framework does not distinguish between goods, non-consulting services and consulting services. The same rules apply to all. However, it is important to note that according to the specific characteristics, the nature and complexity of the specific goods or services, the selection criteria may mainly consider non-economic factors, such as technical, scientific, artistic or other capacities. 24. Open competitive bidding is the default procurement method, as defined by Article 10 of Decree N° 130/2016. All procurement opportunities are advertised regardless of estimated cost are published at Page 106 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) “Compr.ar” or at Office of National Procurement free-access web site and official gazette. 3.1.3 Procurement Profile of the Program 25. Procurement under the Program involves goods, consulting services, and non-consulting services: i) Information technology equipment for trainings; ii) Individual consultants; iii) Consulting services to improve the school reports to increase salience and usage of the reports by school principals (‘Operativo APRENDER’ audiovisuals design; database and software development); iv) Non-consulting services: data processing; design, printing and distribution of publications and materials 26. These goods, consulting services and non-consulting services are not expected to have a significant adverse impact on the environment and/or affected people, as defined in the Policy and Directive on Program for Results Financing, and the resulting contracts are below the OPRC thresholds (substantial risk activities). During the implementation, the Bank will screen Program execution to ensure compliance with the PforR policy requirements. 27. Procurement is the responsibility of the MECCyT, specifically the Dirección de Contrataciones (Procurement Unit) - depending on the Dirección General de Administración y Gestión Financiera - which implements the competitive bidding processes for the Program. Its activities include preparing and issuing bidding documents, responding to bidders’ questions, and reviewing the bid evaluation reports. 28. At MECCyT, the DNB would coordinate and oversee Project activities among technical units to ensure proper implementation and would be the Bank’s main technical counterpart. The SGE would be responsible for all activities related to ASISTIRÉ; the DNB would be responsible for implementation of activities related to the PROGRESAR scholarships; and (iii) the SEE would carry out activities related to APRENDER assessment and all other evaluations and/or assessments under the Project. 3.1.4 Procurement Performance 29. In 2015 there were a total of 6 processes with a total amount of approximately US$ 45 million (5 Licitaciones Públicas and 1 Licitación Privada) for procurement of Goods - with an average of 6.3 bids received; in the case of Non-consulting services there were 4 processes (Licitaciones Públicas and Privadas) –total amount US$400,000, with an average of 2.75 bids received. The average evaluation period for all processes was of 46.8 days considering all processes. 30. In 2016 there were a total of 5 processes for procurement of Goods: 2 Licitaciones Públicas and 2 Licitaciones Privadas, the average number of bids received was 4.75; and 1 Contratación directa. The total amount was US$556,000. For Non-consulting services there were only 2 processes (1 Licitación Privada and 1 Contratación Directa) - approximately US$55,500-, the number of bids received in the competitive bidding was 3. The average evaluation period for all processes was of 35.5 days. 31. In 2017 there were a total of 8 processes (3 Licitaciones Públicas and 1 Licitación Privada / 4 Contrataciones Directas) for procurement of Goods – US$9.6 million. The average number of bids received Page 107 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) in competitive biddings was 8.5. For Consulting Services were 2 Contrataciones Directas for a total amount of US$115,660. For Non-consulting services in a total 8 processes (6 Licitaciones Privadas and 2 Contrataciones Directas) – US$12.5 million- the average number of bids received in competitive biddings was 3.6. The evaluation of bids for all processes took an average of 28 days. 32. In 2018 there were a total of 8 processes for procurement of Goods: 4 Licitaciones Públicas and 1 Licitación Privada, and 3 Contrataciones Directas, the average number of bids received in competitive biddings was 8. The total amount for goods was US$ 564.300. In procurement of Consulting Services for the same period (3 Concursos Públicos and 2 Contrataciones Directas; total amount US$1,925,600) the average number of bids received in competitive biddings was low, 1.75 bids. For Non-consulting services were 12 processes (7 Licitaciones Públicas, 1 Concurso Público and 4 Contrataciones Directas), total amount US$10,753,900. In competitive biddings, the average number of bids received was 2.6. The average evaluation period for all processes was of 51 days. 33. The technical sectors within the MECCyT are responsible for steps in the process, including securing budget allocation, preparing TOR or technical specifications and cost estimates, providing technical opinions during bid evaluations, managing contracts, receiving goods and services, and paying suppliers and consultants. 3.2 Budget Execution 3.2.1 Treasury Management and Funds Flow 34. There are adequate arrangements for transferring Program funding from the MH to the MECCyT, to ensure funds are made available in line with implementation plans and that funds are made available in an orderly and predictable manner as further described below. 35. The flow of funds for the separate Programs, will be as follows: PROGRESAR PROGESAR Scholarships benefits are paid for 10 months (between March and December). The beneficiaries of the compulsory level and the first year of higher education, receive 80% of the amount of the scholarship. The remaining 20% is retained and will be paid the following year, once the compliance with the requirements established in Decree No. 90/2018 is certified. Pre-payment MECCyT maintains the database of eligible beneficiaries. Every month, it carries out the pre-payment process, by which the list of beneficiaries is generated for the respective payment period and a file is prepared in TXT format, with the personal data of the beneficiaries, the payment period and the amounts to be paid. The said information is then sent to the National Social Security Agency (Administración Nacional de la Seguridad Social, ANSES) through an internal note, accompanied by the file in TXT format. Once this information is received, ANSES carries out certain validations: if it is a valid registration, if the individual Page 108 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) tax identification number (CUIL) exists or does not exist and if the person is deceased according to data from the People Data Administration (ADP). ANSES then updates its own PROGRESAR eligible beneficiaries’ database (called Z2-PROGRESAR 2) based on the information sent by MECCyT. Once these validations have been carried out, ANSES generates a payment file, and assigns a payment method for any first-time beneficiary. The assignment is made according to the address registered in ANSES. The possible payment options (i.e. payment agents) are: Banks (Banco de la Nación or others), Argentine Mail (Correo Argentino) Parajes Rurales o PIM (virtual transfer via cellphone). The non-bank payment mechanisms apply, in those cases in which, due to their geographical location, the beneficiary does not have a bank branch to which he/she can easily access. ANSES then sends the completed payment file to its Payments Department, to perform the preliminary calculation of bank commissions/charges and informs MECCyT of the amount to be settled. This includes the amounts to be paid to the beneficiaries and the corresponding bank commissions. Payment MECCyT then makes any permanent updates to its database and formalizes the payment instruction with the sending of a note to the General Directorate of Administration (DGA). This generates an entry in e- SIDIF, for the MH to make the budget allocation and to release the funds to a specific account held at the Banco de la Nación (Account No. 3793/42 used for the PROGRESAR Scholarship Program). From this account, funds are then transferred to ANSES’s account at the Central Bank of Argentina (BCRA). ANSES uses this account at the BCRA to make the payments to the beneficiaries via the payment agents’ network. With a second note, MECCyT then sends the updated payment file to the Benefit Payment Department, in charge of recording the payment to the beneficiaries. The monthly payment operation lasts approximately 40 days. In many cases, the funds are credited to the beneficiaries’ specific bank accounts on day 1 of the payment operation. However, given that there are situations in which there is no direct crediting of the funds, the paying institution keeps the funds to be paid to the beneficiary, once the beneficiary formally identifies himself/herself. Post-Payment Once this 40-day period has expired, the monthly payment operation is considered closed. Payment agents must account for the funds received from ANSES, via the BCRA. To this end, they must present a file of payments made to beneficiaries and payments not claimed by the beneficiaries, together with a note, which has the character of a sworn statement. They must return/refund the unpaid amounts to the BCRA account from which they received the initial transfer. At that time, the BCRA checks that the required payment agents, have submitted their renditions, including information relating to the benefits paid, the commissions paid, the unpaid amounts and the commissions not paid. This process of rendition to the BCRA can take up to 10 business days, with the term being slightly longer in the case of Correo Argentino. The BCRA then consolidates the information received by the payment agents and sends the information to ANSES. Page 109 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANSES then performs a review of the information on the results of the payment process. Based on this information, an electronic file is generated, and a final report is sent to the MECCyT, where the summary data of the payment process is highlighted, accompanied by an annexed detailed table. Finally, ANSES instructs its Finance Area to approve the refund/return of unclaimed amounts, from which the Accounting Area then generates a Payment Order, to transfer the funds to an account at the Banco de la Nación (Account No 4982/45 designated for the effective return/refund of funds). APRENDER and ENSEÑAR The flow of funds will follow the normal/standard process. Payments are made in line with approved budget authorizations and limits. Payments under the two Programs can be made either, from an account controlled by the MH on the MECCyT’s behalf, for payments that exceed the amount of AR$500,000.00 each; or in cases where the payments are below this threshold, the MECCyT makes the payments from an account under its’ own control. In this latter case, the MH transfers funds to the MECCyT’s account every quarter, based on forecasts provided by the MECCyT. The Federal Government uses a Single Treasury Account (STA). Treasury balances are calculated and consolidated on every business day. In 2017 the list of Financial Institutions that operate as Payment Agents of the System was expanded, incorporating 3 new banks, currently operating with a total of 38 banking entities. The deployment of the modules linked to Treasury management reached 105 organisms, representing 87.50 percent of the total. 36. Disbursement of Bank loan proceeds will be made at the request of the Borrower upon achievement of Disbursement-Linked Indicators (DLIs). Some DLIs (see DLI matrix) are scalable, thus allowing for disbursements to be proportional to the progress towards achieving the targeted DLI value. Other DLIs are not scalable, as the indicators relate to actions that are either achieved or not. 37. Verification protocols- Verification of DLR 1 for DLIs 1, 2, and 3 will be undertaken by individual consultants. The Borrower will then either, contract a consulting firm (to be paid from the proceeds of the IPF Operation) or a government agency (AGN or SIGEN for example), that has the capacity to provide independent verification for the remaining DLIs, to ensure credible verification of the achievement of the DLIs. The Verification Agents selected, will need to be acceptable to the Bank, and will thus be assessed by the Bank, to ensure it/they, has/have the necessary independence, experience and capacity of ensuring credible verification. 38. The MECCyT will prepare technical reports to document the status of achievement of DLIs. The technical reports will be verified by the IVA, appointed by the MECCyT, as per TOR agreed with the Bank. On validation of DLIs by an IVA, the MECCyT will communicate the achievement of DLIs and corresponding DLI values to the MH. 39. Results achievement notification-MH will notify the Bank of DLI achievement, supported by the relevant evidence and documentation. Following the Bank's review of the complete documentation, including any additional information considered necessary to confirm the achievement of the DLI results, the Bank will confirm the achievement of the DLI(s) and the level of Program financing proceeds available for disbursement against each DLI. Page 110 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 40. Disbursement requests (Withdrawal Applications) will be submitted to the Bank by the MH using the Bank's e-disbursement (Client Connection) system. A copy of the Bank's official communication, confirming the DLI achievement, should be attached to the disbursement requests. 41. Under the PforR Operation, funds will be disbursed from the Bank to the Borrower, in U.S. dollars into an account indicated by the Borrower and acceptable to the Bank, based on the achieved DLIs. An amount of US$70 million of the PforR operation will be disbursed on account of the DLIs met by the Borrower between October 1, 2019 and the date of the Legal Agreement (Prior Results). Should the Legal Agreement be signed on or after October 1, 2019, these would no longer be Prior Results. The Operation does not intend to use the functionality of PforR Advances. 42. The Bank may decide, without formally extending the Closing Date, to disburse or approve the use of proceeds of the loan for Withdrawal Applications received within six months after the Closing Date for DLIs achieved by the Borrower before the Closing Date. Upon the Borrower’s request, the Bank may decide to extend the period for receipt of such Withdrawal Applications. 43. The General Conditions state, that if after the Closing Date the Borrower fails to provide the Bank evidence satisfactory to the Bank that the withdrawn loan balance does not exceed the total amount of Program Expenditures (payments made on or after the Signing Date but before the Closing Date), the Borrower shall, upon notice from the Bank, promptly refund to the Bank such excess amount of withdrawn balances.118 3.2.2 Accounting and Financial Reporting 44. The Cuenta de Inversión, essentially the Federal Government’s annual financial statements, are prepared by the National Accounting Office (Contaduría General de la Nación, CGN) following national financial reporting and accounting standards (cash transitioning to accrual119) in accordance with the requirements of Law No. 24.156 of 1992, and complementary regulations. The annual financial statements cover the central government, including social insurance funds. The AGN, which reports to the National Congress, audits the financial statements. The last accounts audited by AGN were those of 2016.The last accounts approved by the National Congress were those of 2013 (approved in October 2015). The Law No. 24.156 of 1992, also empowers the CGN to set public sector accounting standards. IPSAS have not been adopted, but CGN has begun to develop public sector accounting standards that are harmonized with accrual basis IPSAS – although there is no clear timeframe for doing so. There are written policies and procedures covering all routine accounting and related administrative activities and only authorized entities and persons, may change or establish new accounting principles, policies or procedures. 45. In addition to recording the Project’s transactions in e-SIDIF, the Federal Government system specially designed for the execution of multilateral finance operations, the System of Financial Administration and Control for Agencies Executing Foreign Loans (Sistema de Administracion y Control 118 Alternatively, in the last year of the Program, the MH in coordination with the Bank, may reconcile the audited Program Expenditure with the DLI amounts disbursed by the Bank to date, and any shortfall in the Program expenditure in relation to DLI disbursement, could be adjusted from the final DLI claim. 119 As per IFAC’s International Public Sector Financial Accountability Index. Page 111 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Financiero para las Unidades Ejecutoras de Prestamos Externos, UEPEX) will be utilized to maintain the Program’s accounts, using information extracted from e-SIDIF. UEPEX provides a good ex-ante internal control framework and it is considered adequate for accounting purposes. The cash basis of accounting will be used to record the Program’s transactions. UEPEX can produce financial reports and accounting and financial staff are adequately trained to use and maintain the system. 46. The e-SIDIF/UEPEX systems will be used, to generate the PforR Operation and IPF Operation annual financial statements using the cash basis, both in the currency of Argentina (Argentinian Peso-AR$) and in United States Dollars (US$). The MH will prepare the semesterly IFRs relating to the PforR Operation with the MECCyT responsible for the preparation of the IFR’s relating to the IPF Operation. The MH will then submit these two different formatted IFR’s, no later than 45 days after the end of each reporting period. At the end of each fiscal year, the MH will prepare the annual financial statements for the PforR Operation that will be audited. The final semester PforR Operation IFRs with accompanying notes, will serve as the PforR Operations annual financial statements. At the end of each fiscal year, the MECCyT will prepare the annual financial statements for the IPF Operation that will be audited. The final semester IPF Operation IFRs with accompanying notes, will serve as the IPF Operations annual financial statements. The MECCyT will be responsible for liaising/coordinating (and, if required, contracting) with the financial statement auditors, the audit of the PforR Operation’s annual financial statements and the IPF Operation’s annual financial statements, but the submission of a single combined audit report (covering both Operations) to the Bank within 6 months after the end of each fiscal year will fall under the responsibility of the MH. 3.2.3 Procurement Processes and Procedures 47. Decree N° 1.023/2001 is the main legal text regulating procurement in the country, contains the set of rules of the National Procurement System (“Régimen de Contrataciones de la Administración Nacional”), and regulates procurement of goods, works and services. It is important to note that according to the specific characteristics, the nature and complexity of the specific goods or services, the selection criteria may mainly consider non-economic factors, such as technical, scientific, artistic or other capacities. In these cases, procurement shall be carried out in a multiple stage process, where the quality of the bid and the cost of the goods or services in the selection of the successful bidder is taken into account. Equally important to note is the fact that there are several legal texts regulating different aspects of procurement. 48. Through Decree N° 1030/2016, Decree’s N° 1.023/2001 regulation was approved (“ Reglamento del Régimen de Contrataciones de la Administración Nacional”), in order to promote technological development, incorporate information and communication technologies, simplify procedures with the objective of facilitate and speed up interaction between National Administration and citizens, improve efficiency, quality and sustainability, fight against corruption and promote ethics and transparency. The regulation applies to procurement of goods, consulting services and non-consulting services, excluding works. 49. Disposición 63 - E/2016 set up the standard biding document (“Pliego Único de Bases y Condiciones Generales”) and the special condition’s bidding document (“Pliego de Condiciones y Bases Particulares”) minimum requirements, inter alia, technical specifications, special clauses, estimated contract duration, bid’s validity period and currency and evaluation criteria. Page 112 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 50. The provisions in the legal framework and procedures are based on clear and mandatory rules, and as the rest of procurement’s legal framework it is available and freely accessible by the public at Office of National Procurement web site www.argentinacompra.gov.ar. 51. National procurement is executed through the Electronic Procurement System named “Compr.ar” www.comprar.gob.ar. The system was launched in 2016, it allows to manage national public procurement via internet and enables to monitor procedure’s progress in real time. Through the system, central’s administration business needs are made publicly available, firms or individuals who wish to participate are able to submit their bids, and anyone who wishes to be informed is able to access to detailed procurement’s processes information (bidding opportunities and documents, amendments, bid opening records, evaluation reports, complaints, contracts award or other outcomes of the procurement process, contracts and purchase order, contract’s amendments and termination). 52. Procurement Methods. Open competitive bidding. It’s the default procurement method and provides all bidders an equal opportunity to bid. There are two types of open competitive bidding: a) “licitación pública”, in case the selection criteria mainly consider economic factors; and “concurso público”, the selection criteria mainly consider non-economic factors, such as technical, scientific, artistic or other capacities. Limited competitive bidding. bidders are required to be registered in the Supplier’s Information System (“Sistema de Información de Proveedores”) as a condition for submitting bids, and contract estimated cost is not above AR$ 8 million (USD 2 million equivalent). As in the case of “open competitive bidding” there are also two types of limited competitive bidding: a) “licitación privada”, in case selection criterion mainly considers economic factors; and “concurso privado”, in case selection criterion mainly considers non- economic factors, such as technical, scientific, artistic or other capacities. National / International approach. Considering market approach options, in national biddings (“licitaciones” and “concursos nacionales”) only those bidders with legal address in the country, or their headquarters is located in the country, or have a local office properly registered are eligible for submitting bids; and in international biddings (“licitaciones” and “concursos internacionales”) bidders whose headquarters is located abroad and do not have a local office properly registered in the country, and those eligible for national biddings are eligible for submitting bids. Single stage. Licitaciones or concursos públicos or privados shall be a single stage process when evaluation of bids and bidder’s capacities are carried out in the same stage. Multiple Stage process. The procurement framework does not distinguish between goods, non-consulting services and consulting services. The same rules apply to all, but according to the specific characteristics, the nature and complexity of the specific goods or services, a Licitación or Concurso (open or limited) shall be held in a multiple stage process, where the quality of the bid and the cost of the goods or services in the selection of the successful bidder is taken into account. In these cases, procurement shall be carried out in a multiple stage process. Technical and financial bids shall be submitted in separate sealed envelopes at the same time. The procurement unit shall publicly open all technical bids received by the deadline for the submission of bids at the designated place stipulated in the bidding document. Technical Page 113 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) evaluation committee shall evaluate the bids in full conformity with the provisions and the criteria indicated in the bidding document and elaborate a pre-selection report. After the pre-selection report (technical evaluation report) is completed, pre-selected bids financial envelopes shall be opened in public. At the same time, bidder’s financial envelopes whose bids did not meet the minimum qualifying technical score shall be returned unopened. The bidding document shall set the selection criteria to determine the most advantageous bid, considering two options: i) bid obtaining at least the minimum qualifying score and the lowest cost shall be awarded; or ii) bid obtaining the highest total score - obtained by weighting the quality and cost scores and adding them - shall be awarded. Direct Selection (Contratación Directa) may proceed, inter alia, when according to regulations it is not possible to apply other selection method and the contract estimated cost is not above AR$1.6 million (US$0.4 million equivalent); to perform or purchase scientific, technical or artistic works which can be executed only by certain firm, artist or specialist; the required good or service is proprietary and obtainable only from one source and convenient substitutes are not available. 53. Procurement opportunities are advertised regardless of estimated cost at “Compr.ar” or at Office of National Procurement free-access web site. “Licitación pública” and “concurso público” are advertised in the official gazette and on “Compr.ar” website. “Licitación privada” and “concurso privado” are also advertised on the same website and in addition, invitations are sent to at least five suppliers previously registered in the Supplier’s Information System (“Sistema de Información de Proveedores”). Open international competitive procurement (“licitación and concurso público internacional”) is advertised on the Office of National Procurement’s website, and in UNDB on line or in the Bank’s DG Market website. 54. Standard bidding documents do not indicate minimum and/or maximum thresholds in order to determine the weight of quality and cost bids for the total score. The implementing agency shall determine the technical and financial weight in the in the special conditions bidding document in a case by case basis, taking into account the complexity of the assignment and the relative importance of quality. 3.2.4 Contract Administration 55. There are controls to ensure that goods, non-consulting services and consulting services are delivered and comply with bid. Standard bidding documents and special conditions provide specific clauses regarding contract management and supplier performance. For procurement of goods samples may be requested, which are sent to the different sectors as a benchmark for verification upon delivery. For consulting services, the TOR used as technical inputs for the bidding documents provide a means of comparison with what is delivered. Quality control is performed before payment by: i) requesting area, monitors the goods or services are delivered within the stipulated time for execution of the contract; and ii) Comisión de Recepción Definitiva - integrated by three members who were not involved in the selection process for the contract - verifies if the goods and services comply with the technical specifications and TOR as set in the bidding document and contract. 56. When goods and services do not comply with bid technical specifications, or the consulting services do not comply with the TOR, suppliers are subject to monetary penalties and can be debarred. Contract management is performed in two stages, based on timely execution and quality control. Most of the mechanisms are related to correct supplier performance via penalties and sanctions. However, there Page 114 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) is no formal contract management procedure so that the requesting area could prevent disputes, and if they occur, seek to resolve them amicably by mutual consultation, before following regular administrative procedures. Contract disputes are settled following administrative procedures described in Law 19.459 (Ley de Procedimiento Administrativo). The mechanisms are well-known to government and officials and to the private sector. For contracts that are already under implementation, a firm’s first step is to file an administrative petition. Once the administrative process is finished, a lawsuit could be filed to a judicial authority. 57. The Borrower, through DNB, will collect baseline data on the performance of contract management during 2019 to monitor performance during implementation and to measure how the contract management has improved as a result of implementing the remedial actions identified in the PAP. 3.2.5 Complaint Mechanisms 58. Before the public opening of bids, firms or individual members of the public are able to submit questions and/or complaints regarding the bidding documents through Comprar.ar system. Any bidder may inform to the implementing unit any fault, error or defect detected, in order to be considered during bids evaluation. After the public opening of bids, bidders are allowed to review or make copies of other bidder’ bids for a two-day period. However, information provided by bidders which they have marked as confidential shall not be disclosed. During bids evaluation, information relating to the examination, clarification, evaluation, and recommendations concerning award, shall not be disclosed to bidders or other persons until the publication of the evaluation report. After the publication of the evaluation report, anyone may complain about decisions adopted directly to the agency implementing the bidding. Any submitted complaint is solved at the contract award act. After the contract award, anyone may appeal the decisions adopted, according to the provision in Law N. 19.549 (Ley de Procedimiento Administrativo). Once the administrative process is finished, a lawsuit could be filed to a judicial authority. Comments, claims or complaints do not stop a procurement process. However, the implementing agency could decide to stop the process for reasons of public interest or to avoid serious damage to the individual or firm who filed de complaint. 3.2.6 Procurement Risks 59. The procurement system provides reasonable assurance that the fiduciary principles of transparency, economy, efficiency, effectiveness, and accountability will be met. It also provides an adequate mechanism to ensure fairness and guarantees the right to appeal decisions on individual bidding processes. 60. Transparency. Decree N° 1030/2016 (“Reglamento del Régimen de Contrataciones de la Administración Nacional”) was issued in order to promote technological development, incorporate information and communication technologies and simplify procedures and facilitate and speed up interaction between National Government and citizens. Accordingly, the Electronic Procurement System named “Compr.ar” (www.comprar.gob.ar) was launched in 2016. All competitive procurement processes are published online at Compr.ar. The system allows to manage national public procurement via internet and enables to monitor procedure’s progress in real time. National’s administration business needs are made publicly available, firms or individuals who wish to participate are able to submit their bids, and Page 115 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) anyone who wishes to be informed is able to access to detailed procurement’s processes information. Compr.ar uses technology that guarantees neutrality, security, confidentiality and user’s identity, based on public standards that allows information backup and processes records, besides, it allows integration with other information systems. 61. Accountability. The processes and procedures supporting the implementation of procurement in Argentina assign very clear roles and responsibilities for all steps of the process. Officials (Minister, Secretary, Director, etc.) are responsible for authorizing expenditures and he or she will perform this function with the advice of specialized teams in budget and planning, procurement, bid evaluation, legal departments, financial management, etc. Each of these specialized teams is individually accountable for their specific contribution. Oversight of this operation is performed by control agencies, such as the Auditoría Interna and Sindicatura General de la Nación, which ensure compliance with legislation. For instance, the procurement system will only allow a bidding process to move forward if the required budget appropriation had been previously made in the budgetary information system. Any payment can only be made if a contract or purchase order had been duly awarded and recorded in the contract information system. 62. Fairness. The procurement legal framework in the country is solid and well known to all stakeholders. Also, there are four ways through which bidders and any citizen can voice complaints during a bidding process: before the public opening of bids, after the public opening, after the publication of the evaluation report, and after the contract award. 3.3 Internal Controls 3.3.1 Internal Controls 63. The internal control environment to be used for the Program is anchored in Argentina’s legal and institutional framework. The internal controls relevant to the Program include arrangements to provide reasonable assurance that: (i) operations are conducted effectively, efficiently, and in accordance with relevant financing agreements; (ii) financial and operational reporting is reliable; (iii) applicable laws and regulations are complied with; and (iv) assets and records are safeguarded. Transactions processing will use the Federal Government’s internal approval processes and systems, that provide for reasonable segregation of duties, supervision, quality control reviews and reconciliation. The processes flows appear to be well understood. 64. DNB will be responsible for overall Program coordination and implementation with the assistance of the SEE for Results Area 2 of the Program. There are specific units responsible for carrying out, managing and monitoring each of the different program variations with specific monitoring systems used for each, following detailed norms and regulations. Thus, there is adequate control over and stewardship of Program activities and funds. 65. The Program’s Results Area 1 (the bulk of the Operation) is essentially a conditional cash transfer (CCT) program. CCT program’s implementation experience has shown the importance of information systems that allow for efficient management of the operating cycle. Without question, CCT programs demand the intensive use of information systems to identify potential beneficiaries, determine eligibility, Page 116 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) register new beneficiaries in the program, verify compliance with conditionalities, deliver payments, and update the list of beneficiaries. The systems for granting, monitoring compliance with eligibility criteria, and co-responsibilities for and making payment of benefits under the PROGRESAR Program were detailed, in a recently completed assessment conducted by the Inter-American Development Bank (IDB). The assessment conducted by the IDB was reviewed by the Bank and the conclusion that: (i) the systems are sufficiently robust and secure to base the rendering of accounts for the transfers; and (ii) that there are opportunities to reinforce the integrated nature of these systems, reducing the possibility of errors in the transfer of information between their respective subsystems, is concurred with. ANSES also plays a critical role in this process as a wholesale payment agent. An in-depth review of ANSES’ internal controls framework was made during preparation of other Bank financed operations.120 It was determined that the ANSES’ controls are in place, are reliable and continue to operate as planned. 66. All the Project’s transactions will be processed within e-SIDIF and UEPEX, that enforces strict segregation of duties, controls the preparation and approval of transactions to ensure that these transactions are properly executed and recorded (i.e. different units or persons authorize the transaction and record the transaction), and guarantees the confidentiality, integrity and availability data. The four stages in the budget execution cycle: authorization (gasto o crédito autorizado), commitment (compromiso), verification (devengado), and payment (pagado) are reflected as separate stages/control points, and thereby ensure adequate segregation of duties. e-SIDIF and UEPEX contains a series of controls, which effectively limit expenditure commitments and payments to cash availability and approved budget appropriations. 67. All accounting and support documents are retained on a permanent basis, using a system121 that allows for easy retrieval for the authorized user. 68. The Project’s internal control system will be documented in a Project Operational Manual (POM) that will encompass both the PforR Operation and IPF Operation. The POM will comprise descriptions, flow charts, policies, templates and forms, user-friendly tools, tips and techniques to ensure that the approval and authorization controls continue to be adequate and are properly documented and followed with adequate safeguarding of the Project’s assets. The POM should be prepared by the DNB and be approved by the Bank and be maintained/updated throughout the Projects’ life. 69. The Access to Public Information Law (Ley 27.275) regulates the right of access to public information. The Law, which considers “all government-held information” to be public, was approved by the National Congress in September 2016 and entered into force in September 2017. The Law provides procedures for processing information requests, covering obligations concerning disclosure, and the duty to provide data in an open format. The law also envisages sanctions – not yet determined - for those who deny access to information and determines exceptions that normally comply with international standards of freedom of information, inter alia, national security, defense and international relations; public safety; the prevention, investigation and prosecution of criminal activities; privacy and other legitimate private interests; commercial and other economic interests, be they private or public. A new Agency for Access to Public Information was created, which will operate within the Executive branch, although “with 120 Children and Youth Protection Project (P158791) Original Loan and Additional Financing. 121 The Federal Government has also recently implemented an Electronic Document Management System (EDM). Page 117 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) operational autonomy,” according to the law. The text is broad in terms of the list of institutions legally bound to respond to requests for access to information: besides the various branches of the state, it includes companies, political parties, trade unions, universities and any private entity to which public funds have been allocated, including public service concessionaires. The Agency was created to ensure compliance with the law. Its functions include advising people who seek public information and assisting them with their request. 70. The Code of Ethics for Public Servants of the National Executive Power was approved by Law No. 25.188 of October 26, 1999 and contains the set of rules that pertain to the conduct of public servants and includes penalties to be applied for non-compliance with these standards. The code informs the principles and duties of public servants, as well as qualities of the public servants, his/her obligations toward the well-being of the population, and the prohibitions and punishments derived from the irregular service of his/her functions that recall the fundamental principles of public administration. 71. Decrees No. 201/2017 and 202/17 are additional control and transparency and integrity mechanisms in judiciary and procurement processes to avoid officials’ feasible conflicts of interest. These rules incorporate additional transparency and monitoring tools in judiciary and extra judiciary processes and in any procedure in which de government awards a contract. A special and more transparent procedure is created, in cases where firms and suppliers state they have a judiciary or extra judiciary conflict with the National Government or any relationship with the President, Vice-President, Chief of Cabinet, Ministers or any official involved in procurement processes. In both cases, every file should be advertised to allow to be checked and monitored by any citizen. 3.3.2 Internal Audit 72. The Project will be subject to internal audit by the Internal Audit Units (IAUs) of all the executing Federal agencies, as well as the General Syndicate of the Nation (Sindicatura General de la Nación, SIGEN), which is the Federal Government's Internal Audit Agency, under the jurisdiction of the executive branch. SIGEN is an integral part of the Federal Government’s internal control system providing the following core services: (i) assessing the adequacy and effectiveness of the internal control system (internal audit of the executive branch); (ii) supervising and coordinating the actions of the IAUs and approving their audit plans; and (iii) compliance auditing of procurement processes and contracts. 73. SIGEN can carry out its audits, after prior notification to the organizational unit being audited; it documents its findings and reports directly and continuously to the President (executive). SIGEN has unlimited right to information within the scope of their audit mandates. They have access to all premises and facilities and are authorized to conduct talks with any member of the organizational unit being audited to clarify questions. 74. SIGEN relies heavily on the use of data and the analysis of information from the sectoral and transversal systems to perform its duties; the e-SIDIF system, e-procurement portal, electronic filing system, and the open data portal are key sources of information. The Bank is actively engaging with SIGEN to: (i) provide access to the Business Intelligence Tools that the Modernization Ministry is deploying in the Administration to better analyze existing information in the several digital platforms available to allow SIGEN to effectively produce high-quality reports; and (ii) design and implement an open government Page 118 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) plan, to publicly disseminate SIGEN’s own reports, as the internal audit reports are not easily available for public consultation on SIGEN’s website. 75. SIGEN has conducted an internal audit review relating to PROGRESAR122 in 2016. There were no internal audit reviews conducted for APRENDER and ENSENAR to date. To the extent that internal (or external) audit reports become available in the future relating to any of the Programs, the status of the auditors’ findings and recommendations will be followed up, during supervision missions. 3.3.3 Program Governance and Anticorruption Arrangements 76. The Borrower will use the existing country systems to take all appropriate measures to prevent fraud and corruption relating to the Program. In 1999 the Oficina Anticorrupción (Anti-corruption Office, OA) was created (Law No. 25.233) to promote transparency in public procurement procedures, in order to seek to ensure the correct end efficient utilization of public funds. This OA provides technical assistance, elaborates and coordinates programs to fight against corruption and intervenes in procurement processes. In addition, OA elaborates and proposes preventative policies and legal reforms regarding anti- corruption matters, to improve efficiency in corruption practices prevention, investigation and sanction. The OA shall be involved in case any irregularity in firm’s documentation, member’s societies, etc. is found. 77. The current administration has publicly stressed that fighting corruption is at the top of the political agenda, to re-establish trust in leaders and public institutions. Since taking office in December 2015, the current administration has moved quickly, pushing through the National Congress several bills and proposing amendments to enhance current legislation to promote transparency, fight corruption and strengthen public integrity. New or overhauled laws have been passed or are being discussed in the areas of corporate criminal liability, access to information, ethics and integrity, plea bargain and asset recovery; and accountability mechanisms have been strengthened significantly, such as those of the OA. Notably, at the time of taking office Argentina ranked 107 out of the 175 countries in Transparency International's Corruption Perceptions Index but rose 22 places to 85 in 2018. The current corruption scandals123 represent both a challenge and an opportunity for further institutional reforms. 78. The Criminal Code (Law No. 11.179) establishes responsibilities, accountabilities, and penalties for officials and citizens in relation to acts of corruption. Furthermore, Law No. 27.401 establishes criminal liabilities related to bribery, traffic of influences and illicit enrichment, inter alia, for both domestic and foreign companies, state-owned or not. The Argentine Criminal Code applies to any public officer and to any individual who engages in corrupt practices with public officers. The Argentine Criminal Code defines 122Relevamiento y análisis de los procedimientos implementados y los mecanismos de control interno imperantes en la Administración Nacional de la Seguridad Social (ANSES), relativos a la operatoria de cumplimiento de las condiciones y otorgamiento de la prestación del PROGRAMA DE RESPALDO A ESTUDIANTES ARGENTINOS (PROGRESAR) - The main findings were a lack of coordination between ANSES and the committee’s setup to oversee PROGRESAR; the lack of a dedicated unit within ANSES to administer the Program; deficiencies in verifying the eligibility and performance criteria of beneficiaries under the Program; and errors in the payment process. An objective of the IPF TA operation, is to improve the functioning of the PROGRESAR Program. These deficiencies will thus be addressed as part of the Bank’s support to the Program under the IPF Operation, that will be monitored during implementation. 123 In part, the revelations surrounding the ‘’ Cuadernos’’ scandal— which started to unfold at the beginning of August 2018, involving fraud and corruption charges of public officials and a large number of private entrepreneurs during previous administrations - are fruits of such strengthened commitment, measures and institutions. Page 119 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) the offenses of domestic bribery (active, passive and improper lobbying), corruption and transnational bribery. Other national sources of legislation dealing with anti-corruption are the Ethics in Public Service Act, the National Public Employment Regulation Act, the Code of Ethics in Public Service for Public Officials of the National Executive Branch, Decree No. 1162/00 (obligations of public officials or employees of reporting crimes known in the exercise of their duties) and Decree No. 1023/01 (contracting system of the public administration).The Argentine Criminal Code prohibits government officials from receiving or accepting money or other benefits/gifts in order to do or not to do something related to their duties (active bribery); and further prohibits any individual from personally, or through an intermediary, giving or offering any gift to a public officer (passive bribery). 79. In Argentina, bribery offences, whether domestic or foreign, are Federal crimes. Investigations may be initiated by a complaint made by any person to the police, public prosecutor, or judge. Allegations of corruption could be filed at the OA either by e-mail, anonymously or with protection of identity, at the Procuraduría de Investigaciones Administrativas or at the government agency where the offence took place. Mecanismo de Reporte de Alto Nivel (MRAN) is another available channel to report irregularities and acts of corruption for bidders and officials involved in infrastructure procurement processes. MRAN is a tool promoted by the Organization for Economic Cooperation and Development (OECD) and Basel Institute on Governance for preventing corruption and ethic irregularities. The aim is to provide, from the early beginning, a trustable, specific and well-known channel to companies and other interesting parties involved in procurement processes to report red flags, in terms of ethic misbehavior and briberies. 80. Public officials who become aware of the commission of a crime while performing their duties, must report it to the authorities. Enforcement authorities are legally required to commence the investigation once becoming aware of a bribery allegation. The investigation is conducted by the investigating judge or can be delegated to a public prosecutor. The Federal procedural system grants the judge the authority to direct the proceedings, and it is this judge who may take any measures appropriate for the discovery of the truth (seizures, requests for information, raids, phone tapping, etc.). Once the investigation is completed, the judge may decide whether the matter should move to the oral trial stage, in which case a separate judge or judicial panel conducts the trial until its completion. There is thus adequate institutional capacity to monitor and address governance and corruption issues. Issues of major concern are the delay in the investigation and prosecution of complex economic crime cases, the lack of resources to expedite and pursue the investigations, and the lack of judicial and prosecutorial independence. Similarly, when corruption practices involved private companies, authorities are unable to establish legal liabilities or impose sanctions to such entities, as responsibilities were easily transferred to individual employees, thus resulting in impunity situations and the prevalence of corruption systems. Until 2017, Argentina was one of the few countries, not to comply with several provisions of the United Nations Anti‐Bribery Convention, including those related to corporate liabilities. 81. Incidences of fraud and corruption in the program identified by public officials will immediately be reported to the DNB and to the OA, who initiates a formal investigation procedure. In the case the incidence is reported by any person directly to the OA, the OA informs the DNB and could require additional information. Page 120 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 82. In accordance with the Bank’s Anti-Corruption guidelines for PforR operations, the Borrower needs to ensure that “any person or entity debarred or suspended by the Bank is not awarded a contract under or otherwise allowed to participate in the Program during the period of such debarment or suspension.” The Borrower will implement its commitment to comply with this abovementioned requirement by: (i) enforcing national rules - Article 68 and Decree No. 1030/2016 – which states that entities and individuals debarred or suspended by the Bank and/or IADB because they have been sanctioned under the Bank's fraud and corruption policy, are ineligible to be awarded a contract; (ii) inclusion of clauses in the contracts stating activities within the Program boundary are carried out in accordance with the ‘Guidelines on Preventing and Combating Fraud and Corruption in Program-for- Results Financing’, dated February 1, 2012 and revised July 10, 2015; and (iii) indicate in the evaluation reports - in writing – that the eligibility verification has been carried out. In addition, the external auditors’ TOR will include a requirement to review Program expenditure to confirm that contracts are not awarded to debarred or suspended firms. The DNB will immediately report any allegation of fraud and corruption to the Bank, through exchange of letter, and every semester (together with the IFRs) a report will be prepared containing all alleged cases, with an updated status of the respective actions taken. 3.4 Auditing 3.4.1 Program Audit 83. For purposes of the Program, the external audit will be conducted by the AGN124 due to its mandate. The AGN is the current auditor for Federal-level projects being financed by the Bank and has the capacity to deliver a quality audit on time. 84. The AGN will follow agreed TOR acceptable to the Bank and will conduct the audit in accordance with the International Standards for Supreme Audit Institutions (ISSAIs), a framework of standards formulated by The International Organization of Supreme Audit Institutions (INTOSAI) or national auditing standards if, as determined by the Bank, these do not significantly depart from international standards. 85. The audited Program financial statements (that is, the fiscal years’ last semester IFR) will also be prepared in accordance with accounting standards acceptable to the Bank (that is, IPSAS issued by the International Public Sector Accounting Standards Board of the International Federation of Accountants or national accounting standards where, as determined by the Bank, they do not significantly depart from international standards). 86. The auditors will be required to issue an opinion on the Program’s annual financial statements and produce a management letter in which any internal control weaknesses are identified, with a view to contributing to the strengthening of the control environment. The auditor’s report will be submitted to the Bank no later than six months after the end of the fiscal year. The Bank will review the audit report and will periodically determine whether the audit recommendations are satisfactorily implemented. 87. The Bank also requires that the Borrower disclose the audited Program financial statements in a manner acceptable to the Bank and following the Bank’s formal receipt of these statements from the 124In the event that the AGN is not able to conduct the audit, a private auditor acceptable to the Bank will be hired to conduct the audit, to be paid for under the IPF Operation. Page 121 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Borrower, the Bank will also make them available to the public in accordance with The World Bank Policy on Access to Information. 3.5 Procurement and FM Capacity 3.5.1 Staffing Should be Adequate in Both Numbers and Experience 88. The DNB and SEE has a team of fiduciary staff (FM and procurement) and is responsible for monitoring the implementation of the Program’s integrated fiduciary aspects. Staff are professional, experienced, and knowledgeable on governmental policies and procedures, and although they have prior experience in implementing Bank IPF Operations, this will be the first PforR Operation in Argentina. 4. Program Systems and Capacity Improvements Summary of Program Systems and Capacity Improvements Type of Risk Mitigation Action Timing Action Prepare Operational Manual By Negotiations Special conditions included in the standard bidding The implementing agency (DNB), with the documents do not assistance of the requesting areas, shall indicate any determine the maximum thresholds related During PAP thresholds in order to to quality (technical) and cost (financial) implementation determine the weight weight in the special conditions included in of quality and cost the bidding documents. bids for the total score. Contract Prepare a manual as a reference document management: lack of for all staff involved in contract management During PAP contract and train relevant people accordingly. implementation management manual Risk that contracts Dirección de Contrataciones will indicate in will be awarded to the evaluation reports - in writing - the firms and/or eligibility verification carried out; On June During PAP individuals debarred 30th of each year, the implementing agency implementation or suspended by the will send to the Bank the list of contracts Bank awarded to firms and individuals Risk that contracts will be awarded to TOR for the external auditors will include a firms and/or External requirement to review Program expenditures Annual individuals debarred Supervision for ineligible contracts. or suspended by the Bank Page 122 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Fraud and Corruption Promptly inform the Bank of any credible and Semiannual PAP material allegations of fraud and/or corruption regarding the Program as part of the overall Program reporting requirements. 5. IPF Technical Assistance (TA) Operation-FM Arrangements 89. The purpose of the IPF Operation is to strengthen the Programs’ design, M&E, and the institutional capacity of the MECCyT, and thereby secure the long-term implementation sustainability of the Program. 90. The IPF Operation would consist of the following components: i) Subcomponent 2.1: Strengthen PROGRESAR Scholarship Program (US$11 million): This component would include activities to: a) improve the information and management systems to monitor eligibility requirements; b) strengthen the support to scholarship beneficiaries in their education trajectories, with tailored interventions for the most vulnerable; and c) evaluate the results of the program. ii) Subcomponent 2.2: Support the Secretariat of Education Evaluation (SEE) (US$ 13 million). This component would support the implementation of APRENDER. iii) Subcomponent 2.3: Strengthen the expansion and implementation of ASISTIRÉ (US$14.4 million): This subcomponent will support the revamp of the design and the impact evaluation of the program, and its scale-up to several provinces in the country. iv) Subcomponent 2.4: Strengthen Project Management (US$1.7 million). This component will provide support to DNB in MECCyT, to strengthen its M&E capacity and the development of a beneficiary feedback mechanism for all the components of the project. 91. The TA component’s FM arrangements will rely to a large extent on the same procedures and systems of planning and budgeting, accounting, internal controls, financial reporting and auditing highlighted earlier for the PforR Operation. 92. The implementation of Subcomponent 2.2 of the IPF, would also involve the transfer of funds to Provincial authorities and decentralized implementation of certain activities. Provincial Executing Units (UEP’s) already established under the Bank’s previous Rural Education Improvement Project–PROMER I (P070963) and current Rural Education Improvement Project-PROMER II (P133195), will be responsible for the provincial implementation, under the coordination/supervision of DNB. 93. As the assessments previously conducted in the context of the previous Bank projects may be stale, we have opted to conduct a full assessment of a sample of UEPs125 (based on their relative economic size and importance) involved in the component. The key existing controls that can be highlighted include: (i) the use of UEPEX at national and provincial levels to maintain the project accounts and keep control of the advances to the Provinces. UEPEX features allow, inter alia: (a) on-line recording and monitoring of transactions; and (b) reconciliation of advances to Provinces; (ii) creation of specific budget lines in the 125 The Provinces of Buenos Aires, Corrientes and Santa Fe. Page 123 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Provincial budget; (iii) submission of monthly bank account reconciliations to DNB; and (iv) the advances to Provinces will continue to be made, solely to pay for eligible expenditures as incurred (or to be incurred126), with the advances required to be documented by Provinces to the DNB within a maximum period of 90 days before receiving any further advances or transfer of funds. 94. The disbursement of funds from the Bank to DNB, will be processed in accordance with Bank procedures as stipulated in the Legal Agreement and Disbursement and Financial Information Letter (DFIL). The primary disbursement method will be Advances. The Project will be also able to process Reimbursements and Direct Payments under the Project, if required. All payments will be made using the e-SIDIF and UEPEX systems, once payment obligations have been committed and verified. 95. The following diagram indicates de flow of funds for the Advance disbursement method: i) The funds will be transferred to a specific (to be opened specifically for the Project) segregated bank account (DA) administered by DNB. This account will be opened at a commercial bank acceptable to the Bank (Banco de la Nación) as is usual for Bank operations with the Federal Government. This account will be denominated in United States Dollars (USD). Funds will be transferred from the DA to an operational bank account in local currency (AR$) from which payments for eligible expenditures in local currency would be made by the DNB. ii) In the case of Subcomponent 2.2, funds will be further advanced to the UEP’s in AR$, from the operating account, to pay for eligible expenditures as incurred (or to be incurred), with the advances required to be documented by Provinces to the DNB within a maximum period of 90 days before receiving any further advances or transfer of funds. iii) Payment processes will be registered in e-SIDIF and UEPEX by the DNB and UEPs and the records will be reconciled at the end of each month. iv) The Statement of Expenditures (SOEs) will be prepared by the DNB, with information available in e-SIDIF and UEPEX and they will be supported by the accounting records. 126 Certain types of expenditures are approved ex-ante by the DNB Page 124 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 96. The proposed Fixed Ceiling for the DA will be US$2,500,000. The Minimum Application Size (MAS) for Direct Payment and Reimbursement Withdrawal Applications will be USD 500,000 equivalent. The frequency for the presentation of eligible expenditures paid from the DA, is at least once every six months. 97. The DNB will report on the use of Advances and process Reimbursement requests, through Withdrawal Applications supported by Statement of Expenditures (SOEs) specified in the DFIL.127 Direct payments will be documented by Records. The DNB will sign off on the Withdrawal Applications documenting expenditures, based only on actual expenditures, ensuring that the Loan proceeds were exclusively used for eligible expenditures. The Project will also have a six-month Grace Period to document expenditures incurred before the Closing Date. 98. The General Conditions require the Borrower to retain all records (contracts, orders, invoices, bills, receipts, and other documents) evidencing eligible expenditures and to enable the Bank’s representative to examine such records. They also require the records to be retained for at least one year following receipt by the Bank of the final audited financial statement required in accordance with the Legal Agreement or two years after the Closing Date, whichever is later. Borrowers are responsible for ensuring that document retention beyond the period required by the Legal Agreement complies with their government’s regulations. 99. Category of expenditures and financing percentages: Category Amount of the Loan Percentage of Expenditures to be financed (including Disbursement Linked Allocated to DLRs (inclusive of Taxes, except for any taxes Indicator, as applicable) (expressed in USD) levied for financial transactions) (9) Goods, non-consulting services, 40,147,500 100% consulting services, Training and Operating Costs for the Project (10) Front-end Fee 852,500 Amount payable pursuant to Section 2.03 of this Agreement in accordance with: (i) Section 2.05 (b) of the Program General Conditions; and (ii) Section 2.07 (b) of the Project General Conditions (11) Interest Rate Cap or Interest Amount due pursuant to: (i) Section 4.05 (c) Rate Collar premium of the Program General Conditions; and (ii) Section 4.05 (c) of the Project General Conditions TOTAL 341,000,000 100. The e-SIDIF/UEPEX systems will be used, to generate the PforR Operation and IPF Operation annual financial statements using the cash basis, both in the currency of Argentina (Argentinian Peso-AR$) and in United States Dollars (US$). The MH will prepare the semesterly IFRs relating to the PforR Operation with the MECCyT responsible for the preparation of the IFR’s relating to the IPF Operation. The MH will then submit these two different formatted IFR’s, no later than 45 days after the end of each reporting 127 In case of advances, a Designated Account Activity statement will also be required. Page 125 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) period. At the end of each fiscal year, the MH will prepare the annual financial statements for the PforR Operation that will be audited. The final semester PforR Operation IFRs with accompanying notes, will serve as the PforR Operations annual financial statements. At the end of each fiscal year, the MECCyT will prepare the annual financial statements for the IPF Operation that will be audited. The final semester IPF Operation IFRs with accompanying notes, will serve as the IPF Operations annual financial statements. The MECCyT will be responsible for liaising/coordinating (and if required contracting) with the financial statement auditors, the audit of the PforR Operation’s annual financial statements and the IPF Operation’s annual financial statements, but the submission of a single combined audit report (covering both Operations) to the Bank within 6 months after the end of each fiscal year will fall under the responsibility of the MH. 101. The following reports for the IPF Operation will be required to be included in the semesterly IFRs: i) IFR 1 –Sources and Uses of Funds by Disbursement Category (period to date, year-to-date, Project-to-date) showing budgeted amounts versus actual expenditures, (i.e., documented expenditures), including a variance analysis; ii) IFR 2 –Uses of Funds by Project Component (period to date, year-to-date, Project-to-date) showing budgeted amounts versus actual expenditures, (i.e., documented expenditures), including a variance analysis; and iii) IFR 3 – DA bank reconciliation and accompanying bank statements. 102. Borrower Actions to Prevent and Combat Fraud and Corruption in connection with the Use of Loan Proceeds. In furtherance of the above-stated purpose and general principles, the Borrower will: i) take all appropriate measures to prevent Fraud and Corruption in connection with the use of Loan proceeds, including (but not limited to) (i) adopting appropriate fiduciary and administrative practices and institutional arrangements to ensure that the proceeds of the Loan are used only for the purposes for which the Loan was granted, and (ii) ensuring that all of its representatives involved with the project, and all recipients of Loan proceeds with which it enters into an agreement related to the Project, receive a copy of the Bank’s IPF Anti-Corruption Guidelines and are made aware of its contents; ii) immediately report to the Bank any allegations of Fraud and Corruption in connection with the use of Loan proceeds that come to its attention; iii) if the Bank determines that any person or entity referred to in (a) above has engaged in Fraud and Corruption in connection with the use of Loan proceeds, take timely and appropriate action, satisfactory to the Bank, to address such practices when they occur; iv) include such provisions in its agreements with each recipient of Loan proceeds as the Bank may require giving full effect to the Bank’s IPF Anti-Corruption Guidelines; v) cooperate fully with representatives of the Bank in any investigation into allegations of Fraud and Corruption in connection with the use of Loan proceeds; and vi) if the Bank declares any recipient of Loan proceeds ineligible take all necessary and appropriate action to give full effect to such declaration. 6. IPF Technical Assistance (TA) Operation-Procurement Arrangements 103. The MECCyT will conduct procurement using the Bank’s ‘Procurement Regulations for IPF Page 126 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) Borrowers’, dated July 1, 2016, revised November 2017 and August 2018, and the provisions stipulated in the Legal Agreement, for the supply of goods, non-consulting and consulting services. At the national level, the MECCyT, through the National Directorate for Educational Scholarships (Dirección Nacional de Becas Educativas, DNB (Project Implementation Unit), would be responsible for carrying out procurement activities, as well as for monitoring and supervising the fiduciary arrangements at the provincial level, which also have structures in place to carry out procurement activities and are currently carrying out procurement activities under 8452-AR Second Rural Education Improvement Project (P133195). 104. A procurement capacity assessment over the PIU was carried by the Bank. The assessment reviewed mainly the organizational structure for implementing the Project, and the experience of the existing procurement staff. The identified staff is integrated by two specialists with experience in managing the Banks’ and Inter-American Development Banks’ projects and they are familiar with handling procurement processes for low cost and complexity related to consulting services and non-consulting services, the identified staff has less experience on managing international open competitive processes and more complex processes. 105. Based on the specific needs identified to take the responsibility for the procurement function under this Project, the following actions are recommended to reduce risk and facilitate project implementation: (i) the Operation Manual will include a detailed description of the roles and responsibilities of the PIU and its interaction with the Technical Units that spells out technical, administrative and contract management tasks and responsibilities of each of them; (ii) the PIU procurement specialist and technical staff should attend procurement training within six month of Loan approval; (iii) the Bank’s procurement team will closely assist the PIU during project implementation, in particular for more complex activities. 106. Goods to be procured under the IPF will include, among others, information technology equipment to improve support to digital systems, tablets with connectivity, and materials for trainings. 107. Consulting services to be procured under the IPF will include: (a) study to improve the information and management systems to monitor eligibility requirements through the strengthening of the information and evaluation system; (b) study of the estimation of the PROGRESAR scholarships’ coverage, and develop a prioritization strategy for selecting beneficiaries; (c) diagnosis of the main non-financial barriers students face to progress in their studies and graduate; (d) develop tools for students to close critical cognitive and non-cognitive skills gaps; (e) institutional strengthening for the provincial evaluation units; f) technical assistance for: (i) analysis of school performance in technical schools; (ii) analysis of the supply of primary and secondary schools; (iii) analysis of the social characteristics of the students and their distribution in the clusters in order to characterize homogenous / heterogeneous schools; (iv) analysis of the pedagogical conditions that affect learning; (g) improve the precision of the EWS, this includes enriching the data used to identify students at risk and developing a more accurate algorithm; (h) strengthen the delivery and evidence-based content of ASISTIRÉ interventions through the support the development of specific modules tackling barriers in the dropout prevention toolkit; (i) development and piloting of a cell phone app (ASISTIRÉ Familia) for parents; (j) impact and process evaluation of the revamped version of the program. Page 127 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 108. Non-consulting services to be procured under the IPF will include, among others, data processing; design, printing and distribution of publications and materials. 109. Individual Consultants to be procured under the IPF will include several consultants’ profiles for Mesa de Ayuda and the technical team, several consultants’ profiles for SEE and provinces, project managers, coordinators, trainers, data analysts, local project coordinators and facilitators. 110. A Project Procurement Strategy for Development (PPSD) and a draft Procurement Plan for the first 18 months of project implementation which provides the basis for the procurement processes is been developed. The PPSD and the plan is expected to be agreed upon between the Borrower and the Bank before negotiations. The Procurement Plan will be updated in agreement with the Bank on an annual basis or as required to reflect the current Project implementation needs. 6. Implementation Support 111. The proposed fiduciary implementation support includes the following: i) Reviewing the implementation progress, focusing on the achievement of the Program results and implementation of the Action Plan; ii) Monitoring the performance of fiduciary systems and audit reports, including implementation of the PAP and application of the PforR ACG; iii) Monitoring the PforR financial statement reporting process and assisting the client as necessary; iv) Monitoring changes in fiduciary risks of the Program and, as relevant, compliance with the fiduciary provisions of legal covenants; v) Reviewing the Program implementation with the sector team to assess the timeliness and adequacy of the Program funds appropriation; and vi) Continually assessing and monitoring the performance of the FM and procurement systems under the Program and providing suggestions for improvement. Page 128 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 5. SUMMARY ENVIRONMENTAL AND SOCIAL SYSTEMS ASSESSMENT COUNTRY : Argentina Improving Inclusion in Secondary and Higher Education 1. The Bank’s Environmental and Social Systems Assessment (ESSA), which was prepared considering the core principles set in OP 9.00, evaluated the potential range of environmental and social effects that may be relevant to the Operation and assessed the institutional capacity of the implementing agency and the applicable environmental and social regulatory framework to identify any gaps with the core principles and collectively design the measures to close these gaps. The ESSA did not identify any activity deemed likely to have significant adverse impacts on sensitive and valuable environmental components and processes. 2. Argentina has a robust legal framework and an extended public and free education system that provides the opportunity for all students to have access to all levels of education. There are some challenges related to enforcement, but civil society is active in promoting inclusion. The National Law of Education (Law #26206/2006) establishes an obligatory education scheme from kindergarden to secondary school. Nevertheless, only 60 percent of students graduate from secondary school. The Country also has a national andiscrimination law (Law 23592/1988) and has created a National Institute against Discrimination, Xenophobia and Racism (INADI) through Law 24515/1995. In addition, Argentina has approved an Antibullying Law (Law 26892/2013) to eradicate conflict from all educational institutions. Implementation of these laws has been irregular and the effectiveness of the system relies on the capacity of the different provincial ministries of education. Nevertheless, it is important to highlight that organized civil society is very active in promoting inclusion in the edcuation system. 3. The ESSA identified groups that have systematically higher dropout rates and lower educational outcomes than the average population. These groups are indigenous peoples, afro-descendants, students with disabilities, and students that are part of the LGBTIQ128 community (Trans students in particular). In addition, the GoA Prepared an Environmental and Social Assessment (ESA) to identify environmental and social risks associated with the Technical Assistance (TA) component.129 Both analyses were conducted in a participatory manner, including engagement processes with all relevant stakeholders130 from an early stage. Recommended actions resulting from this participatory process helped define a PAP to address issues related to the implementation of the activities under the PforR and an Environmental and Social Commitment Plan (ESCP) to address the risks associated with the activities that would be financed under the IPF component. Finally, the Government will also prepare a Stakeholder Engagement Plan, containing the identification of stakeholders, the registry of early consultations, and the proposals for future 128 LGBTIQ is an acronym that includes lesbians, gay men, bisexual women and men, trans. intersex and queer individuals. 129 Given that there is no Guidance Note on Technical Assistance under the Bank’s ESF, the ESA incorporated the principles laid out in the Interim Guidelines on the Application of Safeguard Policies to Technical Assistance (TA) Activities in Bank-Financed Projects and Trust Funds Administered by the Bank (January 2014) to ensure its compliance with the principles and socioenvironmental standards of the Bank. 130 The Bank Team and the GoA held a first round of consultations with representatives of afro-descendants (February 12, 2019 and March 2, 2019), indigenous peoples (March 22-28, 2019) people with disabilities (February 22 and April 29, 2019) and people from the LGBTIQ community (March 28, 2019), both in the government and civil society organizations. Page 129 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) engagement, which will ensure that the views of stakeholders are considered during project preparation and implementation. The ESA was published on the Borrower’s website and the Bank’s public website on April 23, 2019. The draft ESCP and Stakeholder Engagement Plan were published on April 30, 2019, and a final version of the ESCP was agreed during Negotiations131. 4. Both the ESSA and the ESA confirmed that Argentina is a multicultural country, with a large and diverse indigenous population, as well as a significant afro-descendant and migrant population, which are confronted with barriers that lead to higher dropout rates and underachievement due to structural and direct forms of discrimination, including bullying and sexual harassment.132 These documents also confirmed that school attendance and completion rates for people with disabilities are even lower, and that although there is no robust data available regarding LGBTIQ populations' school attendance or completion rates at the regional level, there is evidence that high levels of discrimination and bullying suffered by these students also result in higher dropout rates. 133 5. The Operation has a strong potential to reverse some of these trends and, for that reason, is expected to have only possitive impacts on the targeted population. Nevertheless, it is uncertain if these programs are reaching all excluded minorities, as no disaggregated data is available or being produced. For that reason, one key aspect of the PAP and the ESCP will be to generate an outreach strategy to guarantee that the different programs are reaching targeted populations from excluded minorities, in culturally appropriate and timely manners. Experience from elsewhere shows that excluded minorities might not be captured under universal schemes. 6. This Operation would neither finance nor support any civil works activities. In this sense, the activities supported by the Project are not expected to have any negative environmental impacts. No rehabilitation or construction of new infrastructure or other actions having an impact on the environment would be financed through the Operation. The Operation would be implemented at the national level, over a diversity of jurisdictions with different environmental, social and institutional settings. However, the overall risks and potential adverse environmental impacts are not considered significant. 7. Social risks are linked to potential exclusion of vulnerable students either by inefficiencies in the outreach strategies aimed at these groups, or by conscious or unconscious discrimination processes under the current system that is reflected in higher levels of bullying suffered by afro-descendants and migrant students, LGBTIQ students and students with disabilities, which has a direct impact on higher school dropouts for these groups. 8. During preparation, an environmental and social assessment was prepared by the GoA to analyze risks associated to the TA and determine the need to define an outreach strategy that may contribute to avoid, minimize or mitigate those potential risks. 131 The final version of the ESCP was submitted for Bank disclosure on June 6, 2019. 132 Sexual harassment is higher towards afro-descendant female students as a product of oversexualization by other non-afro- descendant students. Migrant students usually face harassment from Argentinean students and some xenophobe groups that express that they don’t have rights to access to public education. 133 For example, a 2007 study showed that 45 percent of transgender students dropped out of school, either due to transphobic bullying by their peers or to being excluded by school authorities. UNESCO, 2016, Out in the open, Education sector responses to violence based on sexual orientation and gender identity/expression. Page 130 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 9. The Borrower prepared a Stakeholder engagement plan that includes consultations with other government institutions and organizations from the civil society. A first series of these consultations took place as part of the Project preparation, from February to April 2019. These consultations included the National Institute of Indigenous Peoples Affairs, the Human Rights Secretary of the Ministry of Justice (including its referents for afro-descendant people and for LGBTIQ population, as well as the National Agency for Disability) and other areas of the MECCyT such as the National Directorate for Socio-educative Policies that do not have a role in project implementation but offered valuable information to the social assessment, including the identification of barriers to access for students from vulnerable groups. The plan also includes the continuation of consultations with organizations from the civil society that represent vulnerable groups such as indigenous peoples’ representatives, members of the afro- descendant community, members of the LGBTIQ community and organizations related with people with disabilities during Project implementation. In addition, the ESCP includes actions to enhance the identification of potential beneficiaries of the Program and the dissemination of information about the benefits of the scholarships. It also includes a study to define the Program's coverage and prioritization scenarios with an analysis on these vulnerable groups. 10. The IPF Component will neither finance nor support any civil works activities. In this sense, the activities supported by the TA are not expected to have any negative environmental impacts. Social risks are linked to potential exclusion of vulnerable students either by inefficiencies in the outreach strategies aimed at these groups, or by conscious or unconscious discrimination processes under the current system. The IPF will finance TA to strengthen the institutional capacity of the DNB, the National Team of ASISTIRÉ (e.g. improving of information and management systems; impact evaluation) and the Secretariat of Education Evaluation (support to the process of evaluation and accreditation of teacher training technical institutes development, including the development of standards, training of external peer reviewers, development of data base, pilot implementation of external evaluation, and the institutional strengthening of the National Evaluation Commission of Teacher Training Institutes). The Operation will include teachers’ unions into the stakeholder engagement process to avoid any potential tension associated to the latest activities. These consultation process will be incorporated into the Stakeholder Engagement Plan. 11. The MECCyT has broad experience working with World Bank Safeguards through the implementation of two investment operations focused on rural education and three operations to enhance secondary education besides other social inclusion and social protection programs that included educational outcomes. For this Operation, the DNB will have two professionals focused on the implementation of the environmental and social aspects of the Project, in addition to the support of other institutional teams like the Under secretariat of Inclusive Education, which includes the team working with intercultural bilingual education (EIB) and others working on generating better opportunities for students with disabilities. No additional hiring will be required to implement activities and actions included in the Project design. Nevertheless, its record on citizen engagement and beneficiary feedback has been moderately satisfactory since, although it has several informal channels to address questions, complaints and/or proposals from different stakeholders, it lacks a formal grievance redress mechanism (GRM). The activities under this Operation require that the MECCyT engage in a participatory approach, which might entail strengthening its citizen engagement and beneficiary feedback mechanisms. The creation of a GRM for the entire Program has been included as an action in the Project’s ESCP. Page 131 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 12. In addition, the PROGRESAR Scholarship will continue to have its own GRM managed by ANSES, the National Social Security Agency. ANSES has a very active citizen engagement approach and a strong GRM that includes a webpage,134 a toll-free number (130), more than 450 offices countrywide, and official accounts in the main social networks (Facebook, Twitter, Instagram). In addition, the MECCyT created a helpdesk with its own communication channels including a webpage,135 a toll-free telephone number (0800-999-1066), and other telephone lines136 to complement ANSES’ own channels. 13. The DNB is already engaging stakeholders to inform these decisions. Preliminary consultations with vulnerable groups have already provided relevant data (e.g. indigenous communities required to review the due dates for inscription, afro-descendant youth requested to improve the outreach strategy within their communities, and trans community members required to change the age threshold). 14. To address risks identified with the Program-for-Results financing and with the IPF component, both the PAP and the ESCP have specific actions that will eliminate, minimize, mitigate and/or compensate the associated risks respectively. 15. The Program Action Plan includes: (i) the creation of baseline on PROGRESAR students belonging to different vulnerable groups; (ii) the preparation of a study of the estimation of the PROGRESAR Scholarships’ coverage and development of a prioritization strategy to select beneficiaries; (iii) actions to strengthen the Program's outreach strategy and the MECCyT’s capacity to implement it with a focus on vulnerable groups; and (iv) the enhancement of access to PROGRESAR Scholarships for students belonging to vulnerable groups. 16. The ESCP includes: (i) the creation and/or strengthening of participatory sectoral and intersectoral working groups at the local level to identify and assess key barriers and drivers for teens between 12 and 14 years old to engage in significative educational trajectories; (ii) the generation of codes of conduct for ASISTIRE facilitators; (iii) the provision of training to ASISTIRE facilitators to identify students from indigenous communities and other vulnerable groups and to help them prepare a reengagement strategy that is culturally appropriated; (iv) the identification of schools that will implement ASISTIRÉ with high level of indigenous people to define a culturally appropriate strategy for preventing dropouts; (v) the development of a revised communication strategy including graphic design, social networking, and workshops, with a focus on vulnerable groups, to ensure their access to the Program’s benefits if they are eligible; and (vi) the implementation of specific consultations with teachers and students on the instruments to evaluate education quality. 134 https://www.anses.gob.ar/becas-progresar 135 https://becasprogresar.educacion.gob.ar 136 (+5411) 4129-2038 / 2046 / 2048 / 2075 Page 132 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) . ANNEX 6. PROGRAM ACTION PLAN Action Description Source DLI# Responsibility Timing Completion Measurement Study of the Technical Dirección Nacional de Due Date 31-Jul-2020 Study completed and delivered. estimation of the Becas (MECCyT) PROGRESAR Scholarships' coverage and development of a prioritization strategy to select beneficiaries Strengthening the Environmental Dirección Nacional de Recurrent Quarterly Report on stakeholder Program's outreach and Social Becas (MECCyT) engagement actions, including strategy with a Systems meetings with institutional and focus on vulnerable civil society representatives of groups. vulnerable groups, communication channels to reach different targets, and outreach resources and campaigns implemented. Enhancement of Environmental Dirección Nacional de Recurrent Semi- Report on enhancement of access to and Social Becas (MECCyT) Annually accessibility in the registration PROGRESAR Systems process for students with scholarship for disabilities, establishment of students belonging student help desk, review to vulnerable requirements for different groups vulnerable groups to ensure their inclusion, establish prioritization criteria. Creation of baseline Environmental Dirección Nacional de Due Date 28-Jun-2019 Report with socioeconomic on PROGRESAR and Social Becas (MECCyT) profile of students, with a focus scholars belonging Systems on identifying students to different belonging to vulnerable groups vulnerable groups (i.e. indigenous peoples, afrodescendants, strudents with disabilities, LGBTIQ students). Creation of a Technical PIU Due Date 30-Sep-2019 Document confirming the Technical Table to conformation of the Technical coordinate Table. technical assistance activities under Component 2 The implementing Fiduciary PIU Recurrent Yearly Report identifying the agency (DNB), with Systems maximum thresholds/weights Page 133 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) the assistance of determined, including a brief the requesting description and reasonableness areas, shall of the decision adopted by determine the procurement process. maximum thresholds related to quality (technical) and cost (financial) weight in the special conditions included in the bidding documents. Prepare a manual Fiduciary PIU Other During TBD as a reference Systems implementa document for all tion staff involved in contract management and train relevant people accordingly. Dirección de Fiduciary Direccion de Recurrent Yearly Report certifying that Contrataciones will Systems Contrataciones evaluation reports include in indicate in the writing the eligibility verification evaluation reports - carried out. The Report will in writing the include the relevant eligibility documentation (copies of verification carried evaluation reports). out; On June 30th of each year, the implementing agency will send to the Bank the list of contracts awarded to firms and individuals Promptly inform Fiduciary PIU Recurrent Semi- Report identifying any credible the Bank of any Systems Annually and material allegations of credible and fraud/and/or corruption. The material allegations Report will include the relevant of fraud and/or documentation. corruption regarding the Program as part of the overall Program reporting requirements. . . Page 134 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 7. IMPLEMENTATION SUPPORT PLAN COUNTRY : Argentina Improving Inclusion in Secondary and Higher Education 1. The proposed implementation support plan (ISP) is consistent with the Government Program that this Operation would support and considers the Program-specific challenges and risks, as defined in the SORT. It also draws on lessons learned from other results-based operations and previous operations in the education sector in Argentina. 2. The ISP is based on mechanisms that would enable enhanced implementation support to the GoA, timely and effective monitoring of the Program, and guidance on technical, fiduciary, environmental, and social aspects. The ISP would entail review meetings and missions with the GoA; regular technical meetings; field visits, as needed; and monitoring and evaluation. The Bank’s implementation support would broadly consist of: i) Capacity-building activities to strengthen the ability to implement the Program, covering the technical, fiduciary, and environmental and social dimensions ii) Provision of technical advice and implementation support geared towards the attainment of the PDOs, DLIs, and intermediate results indicator. iii) Ongoing monitoring of implementation progress, including regular review of key outcome and intermediate indicators, and identification of bottlenecks iv) Monitoring changes in risks to the Program, identification of appropriate mitigation measures, and compliance with the Loan Agreement Main Focus of Implementation Support Time Focus Skills Needed First 12 months • Monitor Program budgeting and • Project management, monitoring, allocation and evaluation • Monitor DLIs and Results Framework • Financial management • Monitor Program Action Plan • Procurement management • Evaluate technical performance • ESF management • Evaluate fiduciary systems’ performance • Fine-tune DLI verification mechanisms and their effectiveness • Review staffing of implementing agency • Assist client in formulation of terms of reference and assessment of resources required 12-24 months • Monitor Program budgeting and • Project management, monitoring, allocation and evaluation • Monitor DLIs and Results Framework • Financial management Page 135 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) • Monitor Program Action Plan • Procurement management • Evaluate technical performance • ESF management • Evaluate fiduciary systems’ performance • Carry out Mid-Term review to assess the Operation’s performance 24-54 months • Monitor Program budgeting and • Project management, monitoring, allocation and evaluation • Monitor DLIs and Results Framework • Financial management • Monitor Program Action Plan • Procurement management • Evaluate technical performance • ESF management • Evaluate fiduciary systems’ performance • Begin implementation completion and results report in the final 6 months of implementation Task Team Skills Mix Required for Implementation Support 3. The table below presents the estimated number of staff weeks and trips (for Bank staff and specialized consultants) that would be required during the first two years of implementation. The first two years would require more staff weeks given that the bulk of disbursements are planned for this timeframe. Staff Weeks No. of Trips Skills Needed per Year per Year Task Management, Monitoring and Evaluation Task Team Leader (HQ), Senior Economist 12 3 Task Team Leader (Country Office), Senior Education Specialist 12 3 Analyst 12 3 Program Assistant 6 0 Specialized consultants To be determined Procurement Specialist 4-6 2 Financial Management Specialist 4-6 2 Financial Management Analyst 4 2 Social Specialist 4 2 Environmental Specialist 4 2 Page 136 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 4. The table below shows the resources required after the first two years. Staff Weeks No. of Trips Skills Needed per Year per Year Task Management, Monitoring and Evaluation Task Team Leader (HQ), Senior Economist 10 2 Task Team Leader (Country Office), Senior Education Specialist 10 2 Analyst 10 2 Program Assistant 6 0 Specialized consultants To be determined Procurement Specialist 4-6 2 Financial Management Specialist 4-6 2 Financial Management Analyst 4 2 Social Specialist 4 2 Environmental Specialist 4 2 Page 137 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 8. DETAILED DESCRIPTION OF IPF COMPONENT COUNTRY : Argentina Improving Inclusion in Secondary and Higher Education 1. Component 2 would be financed using an IPF component for an amount of US$40.1 million. The purpose of this component would be to strengthen the programs’ design, M&E, and the institutional capacity of the MECCyT, and thereby secure the long-term implementation sustainability of the Government program, and to support the scale-up and evaluation of a dropout prevention program. This component would comprise additional financing, with the possibility of crowding in resources during implementation. The components would improve the quality of implementation and support improvements in the programs’ design to increase their effectiveness. 2. Subcomponent 2.1: Strengthen PROGRESAR Scholarship Program (US$11.0 million). This Subcomponent would focus on tackling non-financial barriers to effectively support students in their learning trajectories, which are especially relevant for the most vulnerable students. Besides financial barriers, low income students face access, academic, professional and personal/motivational barriers to complete their education. This Subcomponent would include activities to: (i) improve the information and management systems to monitor eligibility requirements and targeting; and (ii) strengthen the support to scholarship beneficiaries in their education trajectories with tailored interventions for the most vulnerable. The Subcomponent would be financed through technical assistance and would finance firm consultancies and the purchase of goods. 3. The bundling of financial and non-financial support to students is expected to increase the effect of the program on student outcomes. Several randomized studies have compared financial support-only interventions against wraparound services, showing higher impacts of the latter on students’ progression and outcomes.137 The PROGRESAR Scholarship Program is the platform to provide financial support to vulnerable students, and has the potential to expand and provide non-financial support to their scholars. 4. The Subcomponent would support the strengthening of the information and evaluation system to monitor the certification of regular attendance of PROGRESAR students in basic education. Applicants to PROGRESAR Basic Education constitute the most vulnerable population the program serves, presenting the lowest income and the lowest education level, as they are completing their basic education studies (primary or secondary). The current application process to access this scholarship has barriers that the Operation would help minimize to ensure these applicants have full access to the program. One of these barriers is the certification of regular attendance, which is provided by the education institution they attend (adult education schools). To be eligible, applicants need to present their formularies in person to local ANSES offices, where the review of eligibility requirements is carried out. The certification of regular student status must be provided by the student, who must request it to the institution and then present it with the application formulary. In practice, this implies that the applicant devotes more time and effort to apply to the scholarship relative to the other level applicants, who apply online and whose certification is provided directly from the institution to the MECCyT. The Operation would support the design and 137 See Angrist et al., 2009, Carrell and Sacerdote, 2017, Clotfelter et al., 2017, Evans et al., 2017 Page 138 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) implementation of a national information and evaluation system to monitor the certification of regular attendance of students receiving PROGRESAR Basic Education. This would reduce the application barriers to basic level students and improve the monitoring and evaluation system of the program. By 2018, students from this level attend more than 13,000 institutions. The establishment of an information system will reduce the bureaucracy students have to face to receive their certifications. 5. The Subcomponent would use evidence-based international experience and behavioral interventions to support potential and current students in the application process by providing relevant information. The Operation would support the development and piloting of information interventions to potential and current scholars, with the objective of orienting their career decisions and minimizing potential biases that can affect their performance. These interventions include, but are not limited to, providing students with information on: i) returns to education by career ii) additional support programs (e.g. tutoring) iii) reminders on the application process deadlines and steps to complete enrolment and scholarship application iv) benefits for strategic careers 6. The information would be provided through the PROGRESAR Scholarship website, videos, text messages, and visits to schools, among others. Some of these activities would be carried out in partnership with other National programs. 7. The Subcomponent would support a series of studies that will provide a diagnosis of the current coverage of the program, and the main barriers students face to complete their studies. The Operation would carry out a study on the coverage of PROGRESAR Scholarship, which would provide estimations on the fiscal cost of the expansion of the program, and scenarios of financial sustainability in the presence of increases in the demand of the program. In line with this, the study would include potential eligibility alternatives, in view of possible fiscal constraints, by ensuring the targeting to the most vulnerable students (i.e. low-income students, indigenous populations). Additionally, the Operation would finance a study of the characterization of students to map the main barriers they face to progress in their studies and graduate. The results would be used to generate tailored strategies to assist specific profiles of students in their academic progression. These include: i) online modules to close specific academic gaps in specific subjects or areas as identified by the diagnosis ii) online modules to develop specific socioemotional skills, and increase their motivation and belonging iii) encouraging peer-tutoring attendance with low-cost interventions iv) others as a revealed by the diagnosis. 8. Finally, the subcomponent would support the piloting and evaluation of a scholars’ mentoring program in line with evidence-based interventions. The program, to be carried out jointly with the National program NEXOS, would design and pilot a mentoring intervention for first-year scholars. The pilot would be rigorously evaluated to assess its results on academic progression and other relevant outcomes Page 139 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) and assess its potential scalability given the results of the evaluation. The design would explore the possibility of using peer-student mentoring. 9. Subcomponent 2.2 would include activities to implement APRENDER and ENSEÑAR at the provincial level and support the SEE to strengthen and expand the national evaluation system (US$13.0 million). This subcomponent would include activities to support the institutional strengthening of the provincial evaluation units that are responsible for the local implementation of activities, and to provide technical assistance to expand APRENDER contents. The activities include, but are not limited to: 10. Supporting SEE in the development and use of new instruments for APRENDER. This includes extending APRENDER to incorporate the perspective of parents through the design and application of a complementary questionnaire that will be applied to a sample to be defined, aimed at investigating socioeconomic characteristics of the household, participation of parents in the student's trajectory, perceptions of the dimensions to be agreed linked to education in general, and to intra- and extra-school dimensions associated with schooling and learning. It would also include the design of a new module of APRENDER to evaluate socioemotional skills. 11. The subcomponent would provide technical assistance to increase knowledge on students’ learning, through further analysis of APRENDER outcomes and data. This includes: i. An analysis of the social characteristics of the students and their distribution in the clusters to characterize homogenous/heterogeneous schools according to their enrollment; ii. An analysis of the pedagogical conditions that affect learning, including gender differences in human capital endowment, opinions and attitudes in principals and teachers and their link with school climate, self-concept of students, and learning outcomes; iii. An analysis of school performance in technical schools, carrying out a characterization of national or regional scope using quantitative information using the results of APRENDER information on student learning, together with the opinions systematized in complementary questionnaires, characterize the educational dynamics of the students; iv. The analysis of the supply of primary and secondary schools, with the purpose of building clusters in terms of: i) assessment of infrastructure according to principals and students; ii) quartile of vulnerability Education Social Context Index (ICSE); iii) endowment of human capital in principals and teachers; iv) school climate; and v) student self-concept. 12. The subcomponent would provide technical assistance to deepen knowledge of teaching practices, focusing on tools for self-evaluation and improvement. This includes the adaptation of the TEACH instrument for classroom observation in primary and secondary schools in Argentina and piloting the tool in a sample of 200 schools, with the correspondent school reports on the results from its application, including the definition of a sub-sample of 30 schools to provide tailored feedback and follow- up coaching to principals and teachers on how to improve management and teaching practices. It would also include support for the technical improvement of the design of self-evaluation toolkits for teaching training institutes, focusing on tools for improvement of teaching practices. Page 140 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) 13. Finally, the subcomponent would provide technical assistance to support knowledge and initiatives on self- and external evaluation of teaching institutes. This includes the design of the instruments for external evaluation of teaching training institutes, including the design of a system to distribute online results report for each institution. It would also provide support for knowledge activities to incorporate best practices in the strategy for evaluation and self-evaluation of teacher training institutes, such as (i) one international workshop on experiences on evaluation of teacher training institutes; (ii) support for the conformation of a committee of experts on evaluation and self-evaluation of teacher training institutes; and (iii) the design and development of a national strategy for an external evaluation system of teacher training institutes. 14. Subcomponent 2.3: Strengthen the expansion and implementation of ASISTIRÉ (US$14.4 million). 15. The improvement in the design of the program would include improving the accuracy of the EWS; strengthening the delivery and evidence-base content of ASISTIRÉ interventions; and reinforcing the channels for family-school communication. To improve the precision of the EWS, the Operation would enrich the data used to identify students at risk (e.g., the academic history of students) and would develop a more accurate algorithm (e.g., based on machine-learning methods). This data will not be published, it will only be made available to the personnel authorized by the jurisdictions in line with Art. 97 of the 2006 National Education Law. ASISTIRÉ has already moved in this direction via a short-term technical assistance with the CAF Development Bank of Latin America and the University of Buenos Aires, and the project will take this assistance as the step zero toward the new EWS. Since the data capacity of each province is quite heterogeneous, the Operation would take this into consideration when designing the “enhanced” EWS. 16. To strengthen the delivery and evidence-based content of ASISTIRÉ interventions, the Operation would be guided by rigorously-evaluated international experiences by lessons learned from other projects. The Operation would strengthen the recruitment process of ASISTIRÉ facilitators, systematize their training, and align the content, protocols, and delivery mechanisms of the interventions with global evidence. Both the available data and the international evidence highlight the importance of strengthening interventions specifically targeting remedial education, socioemotional skills, and teenage pregnancy. The Operation would support the development of specific modules tackling these barriers in the dropout prevention toolkit.138 This toolkit would also be shared with other schools with high dropout rates, such as FARO schools. An international workshop to discuss global evidence and the strategy to strengthen the evidence base of ASISTIRÉ’s interventions would be carried out. All these activities would consolidate the role of ASISTIRÉ as the innovation hub in the national strategy to prevent dropout. 17. As discussed in the sector context, the data shows that while dropout rates are higher for boys, a significant share of girls do not complete secondary education (approximately 1 out of 4). The reasons are quite different by gender. Economic reasons and lack of motivation/learning deficits stand out for boys, but teenage pregnancy is by far the most important barrier. The technical assistance would particularly 138The Operation will benefit from the dropout prevention interventions being piloted in PBA (a joint-collaboration project between the Province, the Bank, and CAF), which is being financed by PROMER II. Page 141 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) take this diagnostic into account when designing the set of interventions comprising the dropout prevention toolkit. 18. The subcomponent would also enhance the monitoring of students’ attendance by their parents and their school progress in general. Recent experiences from the United States, Chile, and Brazil suggest that using new communication channels such as text messages or emails to share data on absences, grades, and behavioral problems (the variables usually collected by early warning systems) is a highly cost- effective strategy to improve school outcomes and reduce dropout. Indeed, experimental evidence shows that this type of intervention has had large impacts on a variety of outcomes, often requiring less than a year of implementation. Some countries, like Uruguay, have moved a step forward, and parents have access to detailed data through cell phone apps. The intuition is that the students’ families often underestimate the absence rates and the overall performance of their children, and that providing such information would allow them to better monitor their children’s progress and effort. In other words, the early warnings trigged by systems like ASISTIRÉ could also target another key actor: the students’ families. Taking this potential into account, the Operation would support a pilot app aimed at parents (“ASISTIRÉ Familia”) in a few provinces implementing the program. The app would allow, among other things, tracking their children’s attendance and grades (if available) in real time, and could be used to send important reminders to parents, such as the risk of losing the regular student status if children accumulate too many absences. 19. The subcomponent would support both a process and an impact evaluation of the revamped ASISTIRÉ. The gradual rolling out of the program, as well as the likely need of rationing the program in most provinces (i.e., the number of schools willing to participate will probably exceed the project ’s capacity), would allow for the carrying out of a rigorous impact evaluation of the revamped program. In particular, a randomized control trial (RCT) may be conducted in one or two provinces. The analysis would be complemented by a process evaluation to understand the challenges in implementing the program in the field. Both studies would help identify strengths and weaknesses of the program and would be used to further improve the design and implementation of the different components. 20. The subcomponent would also support the implementation at scale of the program throughout the country. The plan for the next five years is to gradually roll out the program to 12 provinces, reaching up to approximately 1,200 schools. The schools would be selected in each province considering chiefly their vulnerability index and the implementation logistics.139 As a first step, each province participating in the program will have to sign an agreement with the national government. The Operation would finance a number of activities to support the scaling-up of the program, including: (i) strengthening the capacity of the national team, including hiring project managers, coordinators, trainers, and data analysts; (ii) building the provincial teams, including recruiting, hiring, and training the local project coordinators and a team of facilitators (two for every five schools); and (iii) supporting the acquisition of key inputs for program implementation, such as tablets with connectivity. 21. Subcomponent 2.4: Strengthen Project Management (US$1.7 million). This subcomponent would provide support to the PIU in MECCyT to strengthen its M&E capacity. This subcomponent would 139 This includes mostly the suitability of their infrastructure to implement the online attendance system, and the distance among participating schools to reduce the costs of sharing the ASISTIRE facilitators. Page 142 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) also finance the annual financial audits throughout the Operation’s lifespan and the VA for the verification of the DLIs. Page 143 of 145 The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 9. RATIONALE AND THEORY OF CHANGE OF THE GoA’S PROGRAM Program/Activities Intermediate Outcomes Evidence PDO/ Outcomes Edo and Marchioni, 2018 Reduce financial barriers Evans et al (2018), AUH (reduce opportunity cost) to Ganimian & Murnane, 2016 access secondary Improve returns to education Secundaria 2030 through higher relevance Improve returns to education Ages 12-17 Escuelas FARO Reduce through lower skill gaps dropout rates in basic and Dropout prevention - ASISTIRÉ higher Knowles, 2015; -Real-time attendance data Adelman et al, 2016 education - Early Warning System Early prevention for the - ASISTIRE Facilitators most - Articulation of resources (Local vulnerable roundtables) Targeted comprehensive Oreopoulos, Brown & - Strengthen dropout prevention Lavecchia, 2015 toolkit (e.g design and pilot teenage support to at-risk students pregnancy interventions) - Improve accuracy of EWS Ganimian & Murnane, 2016 Reduce financial barriers PROGRESAR Scholarship for (reduce opportunity cost) to basic education access secondary Baird, et al 2011; Barrera- - Financial support targeted to the poor Osorio, 2018; Barrera- youth Reduce non-financial barriers Osorio & Filmer, 2016 - Strict conditionalities (progress) to complete basic education Angrist, Lang, & Oreopoulos, Reduce financial barriers 2009; Oreopoulos et al., 2012 PROGRESAR Scholarship for (reduce opportunity cost) to higher education access and complete higher Ages 18-24 - Financial support education - Strict monitoring of conditionalities Baird, et al 2011; Barrera-Osorio, - Merit-based component 2018; Barrera-Osorio & Filmer, - Additional incentive for strategic careers 2016 Improve attendance, (in high demand by labor market) progression and performance Technical assistance - Support programs that offer Reduce non-financial barriers Angrist, Lang, & Oreopoulos complementary support to adult education 2009; students to complete basic and higher - Support programs that offer education complementary support to transition from secondary – higher education - IE to fine-tune design World Development Report 2018; De Hoyos et al, 2017; Bergbauer, APRENDER Student assessment Hanushek and Woessman, 2018 Monitor and track learning gaps - Census-based, periodical and comparable - Dissemination and use of results Improve student Bruns, B., & Luque, J. (2014) and Monitor and track teacher quality teacher Teacher evaluation over time evaluation - Data on teacher quality and skills gaps systems Quality assurance Evaluate quality of teacher training Page 144 of 145 teacher training institutes institutes - Support design and implementation of a quality assurance system for teacher training institutes The World Bank Improving Inclusion in Secondary and Higher Education (P168911) ANNEX 10. GENDER GAPS AND ACTIONS Program Gap Actions Monitoring PROGRESAR Women in STEM Implementation of gender- Results Framework includes Scholarships careers specific evidence-based indicator to follow this gap: interventions to engage women Female higher education in STEM careers. scholars as a percentage of total higher education scholars studying STEM careers Gender-specific Study to identify gender-specific Results Framework includes DLI barriers for barriers to education progression 4: Development and progression and and development and implementation of tools to graduation implementation of an action plan address barriers to education to tackle these barriers using progression, which includes the successful national and diagnostic for females and the international development and programs/interventions. implementation of an action plan to address these gaps. This indicator is linked to disbursements. Teacher and Lack of An analysis of the pedagogical Advances will be monitored in student information of conditions that affect learning, quarterly reports. evaluation gender including gender differences in differences on human capital endowment, external opinions and attitudes in conditions principals and teachers and their affecting link with school climate, self- learning concept of students, and learning outcomes. ASISTIRÉ While secondary Teenage pregnancy is by far the Advances will be monitored in school dropout most important barrier for girls. quarterly reports. rates are higher The technical assistance would for boys, a design a set of interventions in significant share the dropout prevention toolkit to of girls do not target this specific barrier. complete secondary education (approximately 1 out of 4). The reasons are quite different by gender. Page 145 of 145