THE WORLD BANK 37265 From Schooling Access to Learning Outcomes An Unfinished Agenda An Evaluation of World Bank Support to Primary Education THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. WORLD BANK INDEPENDENT EVALUATION GROUP From Schooling Access to Learning Outcomes: An Unfinished Agenda An Evaluation of World Bank Support to Primary Education 2006 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2006 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 09 08 07 06 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750- 4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW , Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Photo credit: Classroom in town of Kampala, Uganda, by Jon Jones/Sygma/Corbis. ISBN-10: 0-8213-6792-7 ISBN-13: 978-0-8213-6792-6 eISBN: 0-8213-6793-5 DOI: 10.1596/978-0-8213-6792-6 Library of Congress Cataloging-in-Publication Data has been applied for. Ainsworth, Martha, 1955– Committing to results : improving the effectiveness of HIV/AIDS assistance: an OED evaluation of the World Bank’s assistance for HIV/AIDS control / Martha Ainsworth, Denise A. Vaillancourt, Judith Hahn Gaubatz. p. cm. — (Operations evaluation studies) Includes bibliographical references. ISBN-13: 978-0-8213-6388-1 ISBN-10: 0-8213-6388-3 1. Economic assistance—Developing countries—Evaluation. 2. AIDS (Disease)—Economic aspects—Developing countries. 3. HIV infections—Economic aspects—Developing countries. 4. AIDS (Diseasse)—Developing countries—Prevention. 5. HIV infections—Developing countries—Prevention. 6. World Bank. I. Vaillancourt, Denise. II. Hahn Gaubatz, Judith. III. Title. IV. World Bank operations evaluation study. HC60.A4575 World Bank InfoShop 2005 Independent Evaluation Group E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (IEGKE) Facsimile: 202-522-1500 E-mail: eline@worldbank.org 362.196'9792'0091726—dc22 Telephone: 202-458-4497 2005052329 Facsimile: 202-522-3125 Printed on Recycled Paper Contents vii Acknowledgments ix Foreword xiii Executive Summary xix Acronyms and Abbreviations 3 1 Introduction 3 Advancing Primary Education: A Worldwide Goal 4 The Evolution of World Bank Policy on Primary Education 7 Evaluation Objectives and Design 11 2 Trends in World Bank Support to Primary Education 11 Evolution in Lending for Primary Education 16 Evolution of Objectives, from Expansion to Learning Outcomes 18 Bank-Supported Analytic Work 19 Performance Ratings of Primary Education Projects 23 3 Improving Access and Learning Outcomes for the Disadvantaged 23 Primary School Access 32 Improved Student Learning Outcomes 41 4 Better Management for Better Outcomes 41 Improving Management Performance 43 Decentralization 44 Community Control and Accountability 45 Teacher Incentives 46 Monitoring, Evaluation, and Research 49 5 Conclusions and Recommendations 49 Conclusions 52 Recommendations 55 Appendixes 57 A: World Bank Policy Objectives and Strategies for Primary Education 59 B: Study Methods FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA 63 C: Projects in the Portfolio Review Sample 67 D: Education Projects and Lending Amounts,1963–2005, by Country 71 E: What Are Fast-Track Initiative Countries Targeting? 73 F: Primary Education Evaluation Terms of Reference for Country-Level Analysis 77 G: Case Study Summaries 99 H: External Advisory Panel Comments 105 I: Management Response 111 J: Chairman’s Summary: Committee on Development Effectiveness (CODE) 115 Endnotes 123 Bibliography Boxes 4 1.1 How Much Has the World Bank Committed to Primary Education? 6 1.2 Primary, Basic, and General Education 8 1.3 Evaluation Building Blocks 15 2.1 Sectorwide Lending Support in Uganda 19 2.2 Analytic Work in Case Study Countries: Where Are the Learning Outcomes? 24 3.1 Measuring Primary School Access 28 3.2 Conditional Cash Transfers: A Panacea for Reaching the Poor? 33 3.3 Measuring Learning Outcomes 34 3.4 Improved Student Learning Outcomes in Three Countries 35 3.5 Low Priority for Early Reading Skills: Mali and Peru 38 3.6 Trade-Off between Improved Access and Student Learning Gains: Is It Inevitable? 43 4.1 Toward More Coherent Capacity Building in Africa Figures 5 1.1 Trends in Gross Primary Enrollment Ratios by Region, 1970–2000 12 2.1 Increase in World Bank Commitments to Primary Education, 1963–2004 13 2.2 Increase in New Commitments for Primary Education, by Region 14 2.3 New Commitments to Primary Education by Managing Sector 15 2.4 Education Commitments by Subsector and Time Period 25 3.1 Increases in Gross Primary Enrollment Ratios in Countries Receiving Bank Support 29 3.2 Reducing Enrollment Gaps in Vietnam 30 3.3 Yemen: Trends in Primary Enrollments (grades 1–6) by Gender 35 3.4 Percent of Sixth Graders Performing Sufficiently on Language Examinations in Uruguay, by Income Level Tables 13 2.1 Cumulative Projects and Commitments for Primary Education, 1963–2005, by Region 17 2.2 Objectives of Education Investment Projects that Allocate at Least 50 Percent of Expenditure to Primary Education 18 2.3 Analytic Work on Primary Education Managed by the Education Sector, Fiscal 2000–05 iv CONTENTS 20 2.4 IEG Ratings of Completed Primary Education Projects, by Year of Approval 24 3.1 Outcomes by Enrollment Objective for Completed Primary Education Projects 32 3.2 Outcomes by Objective for Completed Primary Education Projects 42 4.1 Performance on Education Sector Management Activities for Completed Primary Education Projects v Acknowledgments This evaluation was conducted by the Indepen- White was task manager of the impact evaluation dent Evaluation Group (IEG)–World Bank at the of the Bank’s primary education support in request of the Bank’s Board of Executive Ghana. The four case study teams were as Directors. The report was written by H. Dean follows: Mali: Penelope Bender (leader), Abou Nielsen (task manager), with inputs from Martha Diarra, Koffi Edoh, and Marc Ziegler; Pakistan: Ainsworth, William Hurlbut (editor), and Maurice Boissiere (leader), Safiullah Baig, Manisha Modi. Maria Pilar Barquero provided Manisha Modi, and Fareeha Zafar; Peru: Martin administrative support. Heather Dittbrenner Carnoy (leader), Martin Benevides, Santiago edited the document for publication. The Cueto, and Amber Gove; Romania: Sue Ellen evaluation report is based on inputs from many Berryman (leader), Amber Gove, Dana Sapatoru, IEG staff and consultants who contributed and Anca Tirca. IEG would like to express background papers, desk reviews, and field- appreciation to Livia Benavides, Ana Marie Sandi, based studies. Tahseen Sayed, and Atou Seck for facilitating the Maurice Boissiere authored background work of the case study teams in the field. studies on the rationale for public investment in The evaluation also greatly benefited from primary education and the determinants of the constructive advice and perspectives of a primary education outcomes. The World Bank distinguished External Advisory Panel, consist- portfolio of lending and nonlending activities was ing of David Archer (ActionAid International), reviewed by a team consisting of William Merilee Grindle (Harvard University), Beatrice Cummings (analysis report and drafting); Manisha Okyere (University of Cape Coast, Ghana), and Modi (database construction and analysis); Gillian Paulo Renata Souza (former Minister of Perkins and Howard White (analysis design and Education, Brazil). The evaluation team is methods); and Akiko Sawamoto and Maki enormously grateful for the panel’s feedback on Tsumagari (coding and tabulating). the intermediate outputs and final report. The Fieldwork conducted for the evaluation responsibility for the interpretation of the included seven Project Performance Assessment evidence and the report’s conclusions rests with Reports (PPARs), one impact evaluation, and four the evaluation team, however. country case studies. The PPARs on Honduras, The evaluation team also extends its appreci- Niger, Uruguay, and the Republic of Yemen were ation to the following individuals for their led and prepared by Helen Abadzi; on Vietnam comments and insights on various drafts and by Martha Ainsworth; on Uganda by Gillian intermediate products: Soniya Carvalho, Perkins; and on India by Dean Nielsen. Howard Victoria Elliot, Nils Fostvedt, Patrick Grasso, vii FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Amina Ibrahim, Nalini Kumar, Keith Lewin, Maria up the evaluation Web site and electronic Cristina Mejia, Jennifer Nielsen, Kyle Peters, bulletin board. Ernesto Schiefelbein, Klaus Tilmes, and Howard Finally, we wish to acknowledge financial White. The team also thanks Julius Gwyer, Maria support from the government of Japan through Mar, and Alex McKenzie for their help in setting a Consultant Trust Fund grant. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group–World Bank: Ajay Chhibber Manager, Sector, Thematic, and Global Evaluation: Alain Barbu Task Manager: H. Dean Nielsen viii Foreword he Education for All (EFA) movement, launched in 1990, has resulted T in an extraordinary mobilization of World Bank and country resources in support of basic education over the past 15 years. World Bank EFA financing, mostly focused on primary education, has become increasingly progressive, targeting the most disadvantaged countries and often the dis- advantaged within countries. In most parts of the world, Bank and country remarkably stable: universal primary school investments have led to significantly improved completion, equality of access for girls and access to primary education through the other disadvantaged groups, and improved construction of new schools and the reduction student learning outcomes. This Independent of other physical, financial, and social barriers. Evaluation Group (IEG) evaluation was Nevertheless, tens of millions of children in mounted to assess the extent to which these the developing world—mostly girls, the poor, objectives have been met in countries and other disadvantaged—remain out of school, supported by the Bank. The main objectives of hundreds of millions drop out before completing the evaluation were to assess World Bank primary school, and of those who do complete, a assistance to countries in their efforts to large proportion fail to acquire desired levels of improve their basic knowledge and skills base knowledge and skills, especially in the poorest through the provision of quality primary countries of South Asia and Sub-Saharan Africa. education, and to provide lessons for countries Beyond achieving universal completion of in their development strategies and for the Bank primary education, a Millennium Development in its support to those strategies. Goal (MDG), the remaining EFA challenge is to Evaluation findings show clearly that World ensure that all children, particularly the disadvan- Bank financial support for primary education has taged, acquire the basic knowledge and skills that increased since 1990. Nearly 90 percent of the are crucial for poverty reduction. Bank’s $14 million primary education portfolio Over the years of Bank support for EFA and has been committed since that date. The share of its world conferences in 1990 and 2000, the primary education lending allocated to the Bank’s policy objectives for increased support to poorest countries has more than doubled over primary education have been simple and this same period, from 26 to 54 percent. Commit- ix FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA ments rose in all geographic regions, but most children are reaching the country’s mastery level notably in Latin America and the Caribbean, in English, and in India half of 7-year olds are South Asia, and Sub-Saharan Africa. unable to read a short paragraph fluently. Poor Expanding enrollments was one of two delivery of educational services was at the root of subgoals in reaching universal completion, and low student performance, and much of that can in this the evaluation showed widespread be traced back to weak subsector management, success in Bank-supported primary education including weak incentives for improving learning projects. About 69 percent of projects in the outcomes. study sample reached their enrollment This evaluation presents the following main expansion goals. In the 12 countries where IEG recommendations: made field visits to Bank-funded projects, gross enrollment ratios have increased an average of • Primary education efforts need to focus on im- 19 percentage points over the past 10–12 years. proving learning outcomes, particularly among In countries such as Mali and Uganda, increases the poor and other disadvantaged children. were explosive. Projects from outside the The MDG push for universal primary enroll- Education Sector often contributed heavily to ment and completion, although a valuable in- this goal, through their emphasis on building termediate goal, will not suffice to ensure that schools and community support; national policy children achieve the basic literacy and nu- contributions included reducing or dropping meracy that are essential to poverty reduc- school fees. tion. To reduce poverty, countries in The other subgoal in universal completion partnership with the Bank need to make im- was improving internal efficiency (reducing proved learning outcomes a core objective in dropout and repetition). This goal was their primary education plans and focus on the underemphasized in Bank-supported projects, factors—shown by country-level analysis— even in countries with very poor efficiency most likely to influence such outcomes in the records (for example, Niger). Where it was an local context, recognizing that improving explicit objective, only about a quarter of Bank- learning outcomes for all will require higher supported projects were successful. Equity of unit costs than universal completion. access for girls was often pursued by projects in • Efforts are urgently needed to improve the countries having gender disparities and performance of sector management in sup- generally reached their access targets even port of learning outcomes. This implies the though boy-girl gaps were often not closed. need for sound political and institutional analy- Equity for the poor was somewhat less often ses, taking into account the incentives faced by pursued, but still taken up with a high level of officials and teachers to improve the quality of success. The focus of equity efforts was on teaching and learning; for strengthened ac- access, not on learning outcomes. countability and supervision systems that cover Although it is a key Education Sector concern, learning outcomes in disadvantaged commu- improvement in learning outcomes was not as nities’ schools; and for improved monitoring often an objective in primary education projects: and evaluation systems that track learning out- about one in three included them in explicit comes over time among different income and objectives or in performance indicators. Of the social groups, cover staff and system per- 12 field-visit countries, only 5 even had formal formance (not just inputs and outputs), and in- systems for tracking student learning. Of the clude incentives to ensure that findings are Bank-supported projects that included improv- used in decision making. The Bank should re- ing learning outcomes, most of them did so quire all new Country Assistance Strategies to successfully. However, even where learning include learning outcomes indicators. improved, absolute levels of student achieve- • Finally, the Bank needs to work with its devel- ment were very low, particularly among the opment partners to reorient the Fast-Track Ini- disadvantaged: in Ghana only 5 percent of tiative (FTI) toward supporting improved x FOREWORD learning outcomes, in parallel with the MDG tries in setting up suitable learning assessment emphasis on primary school completion. This systems; and revisions of cost and financing gap will require some reframing of FTI goals and estimates to include the higher unit costs of objectives; the addition of relevant items in reaching the most disadvantaged and sup- the “indicative framework”; assistance to coun- porting improved learning outcomes for all. Vinod Thomas Director-General Evaluation xi Executive Summary asic knowledge and skills—not educational attainment—are key to re- B ducing poverty. Raising enrollments and completing primary school- ing are necessary—but not sufficient—to ensure basic literacy and numeracy. Developing countries and partner agencies such in the poorest countries of South Asia and Sub- as the World Bank need to focus on raising Saharan Africa. Beyond achieving universal learning outcomes, particularly among disadvan- completion of primary education, which is one taged children, to realize the poverty reduction of the Millennium Development Goals (MDGs), benefits of investing in primary education. the remaining EFA challenge is to ensure that all The Education for All (EFA) movement, children, particularly the disadvantaged, acquire launched in 1990, has resulted in an extraordi- the basic knowledge and skills that are crucial nary mobilization of World Bank and country for poverty reduction. resources in support of basic education over the During the 1990s and into the current past 15 years. World Bank EFA financing, mostly decade, World Bank policy on primary focused on primary education, has become education was conveyed in a series of policy and increasingly progressive—targeting the most strategy papers and updates. These were also disadvantaged countries, and often the the basis of its support of EFA conferences in disadvantaged within countries. In most parts of 1990 and 2000. Over these years, the Bank’s the world, Bank and country investments have policy objectives for primary education have led to significantly improved access to primary been simple and stable: universal primary education through the construction of new school completion, equality of access for girls schools and the reduction of other physical, and other disadvantaged groups, and improved financial, and social barriers. student learning outcomes. Nevertheless, tens of millions of children in The World Bank has promoted a variety of the developing world—primarily girls, the poor, strategies for achieving these objectives. Strate- and other disadvantaged groups—remain out of gies have ranged from improving internal school; hundreds of millions drop out before efficiency and building institutional capacity in completing primary school; and of those who the 1980s, to aggressively supporting girls’ do complete it, a large proportion fail to acquire education, improving teacher education, and desired levels of knowledge and skills, especially creating achievement assessment systems in the xiii FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA early 1990s, to increasing community involve- Together, these 50 projects comprise the ment, school autonomy, decentralization, and evaluation’s “portfolio sample.” The evaluation early childhood education in the late 1990s. also drew on recent, in-depth Independent The Bank also endorsed the MDG calling for Evaluation Group (IEG) field assessments of universal completion of primary education by primary education projects in seven countries; 2015 and subsequently cosponsored the Fast- an impact evaluation of Bank support for Track Initiative as a means of accelerating primary education in Ghana; and country case progress toward that goal. The Bank’s 2005 studies in Mali, Pakistan, Peru, and Romania. Education Sector Strategy Update commits the Bank to maintaining momentum on EFA and the Bank Support to Primary Education Has MDGs, while at the same time strengthening Grown Rapidly “education for the knowledge economy” From 1963, the first year of Bank lending to (secondary, higher, and lifelong education). Its education, to 2005, the total amount of Bank strategy emphasizes increased focus on results, lending to primary education was an estimated systemwide approaches, and closer collabora- $14 billion. Nearly 90 percent of Bank lending tion with other donors. for primary education has occurred since the This evaluation has two objectives. The first is beginning of the EFA movement in 1990. to assess World Bank assistance to countries in About two-thirds of this lending has been in their efforts to improve their basic knowledge the form of International Development Associa- and skills base through the provision of quality tion credits. The share of primary lending to primary education, particularly since the countries accounting for the poorest 40 percent beginning of the EFA movement in 1990. of the global population has more than doubled The second objective is to provide lessons for over the past 15 years, from 26 to 54 percent. countries in their development strategies, and Commitments rose in all geographic Regions, for the Bank in its support of those strategies. but most notably in Latin America and the Early findings of the evaluation have been Caribbean, South Asia, and Sub-Saharan Africa. incorporated in the 2005 Education Sector Absolute increases in Bank financing for Strategy Update. This evaluation is intended to primary education were accompanied by help the Bank work more effectively with substantial decreases in support for vocational partner countries in converting these strategies education; funding for tertiary and secondary into results-oriented programs. education remained steady. The amount of Bank A review of the Bank’s lending portfolio for analytic work on primary education from 2000 primary education examined documents from to 2005 has remained stable at about 17 more than 700 projects that allocate funds to products per year. Relatively few of these primary education; about 440 of these projects products have focused primarily on learning originated in the Education Sector. They were outcomes. reviewed to assess the volume, substance, and A growing share of lending for primary geographic reach of Bank lending for primary education has been through projects managed education. outside the Education Sector and through A smaller group of 198 projects allocated at development policy (adjustment) lending. For least half of their funding to primary education. the most recent five-year period, 31 percent of From this pool a random sample of 35 all commitments to primary education were completed and ongoing projects was drawn to through components of projects managed by examine in-depth policy implementation, other sectors. Increases in Bank support for effectiveness, sustainability, and institutional primary education have often been matched by development. In addition, a purposive sample increases in the country partner’s financial of 15 projects with the highest allocations to commitment to primary education, sometimes primary education from other sectors was influenced by development policy agreements examined. between the Bank and the country. xiv EXECUTIVE SUMMARY Meeting Policy Objectives through projects that do not originate in the The main policy objectives of the sector since Education Sector. Also, in recent years, demand- 1990 have been to expand primary school side policies have been successfully imple- enrollments and completion, improve equity of mented by governments, often with the support access, and bolster learning outcomes. About of the Bank, such as eliminating school fees (as two-thirds of primary school investment in Uganda and Malawi) and providing scholar- projects included an expansion objective. About ships (Pakistan) or conditional cash transfers the same proportion covered equity of access (Mexico). (mostly for girls and the poor). National equity objectives were also generally Regrettably, relatively few projects (less than reached, at least in terms of increasing the 60 percent) had objectives to reduce school enrollment of girls and children from poor dropout and repetition rates (improving families (the Reublic of Yemen, Mali). But equity internal efficiency). This is key for raising gaps between poor children and more primary completion rates. advantaged children did not always close. Only about one in five projects had an explicit Improving completion rates through reducing objective to improve student learning out- dropout and repetition (improving internal comes. This does not mean that projects were efficiency) was often underemphasized, even in unconcerned about quality: almost all aimed for countries with very poor efficiency records improvements in educational quality, but until (Niger). Where it was an explicit objective, recently this was mostly seen in terms of delivery countries succeeded in only about a quarter of of inputs and services. Most projects also aimed Bank-supported projects. to strengthen education sector management or governance. Key lessons: The objectives of development policy lending for primary education were similar to those for • A trade-off between improved access and stu- investment projects, except that all of these dent learning gains can be avoided with explicit projects aimed to expand enrollment, and even planning for improved learning outcomes and fewer focused on learning outcomes. Invest- strong political commitment to that goal. ment projects that were managed by other • If primary school completion rates are raised sectors but had considerable support for by automatically promoting children to the primary education generally focused on next grade or without heeding student learn- increased enrollments and equity. ing outcomes, then higher completion rates will not reflect improvements in knowledge and Expanding Access skills—which is the ultimate policy objective— Access expansion was the most successfully met especially among the disadvantaged. objective in Bank-supported primary education • Many of the strategies used to rapidly increase projects: 69 percent reached their expansion access, such as “big bang” fee reductions, use of goals. In the 12 IEG field study countries where contract teachers, double-shifting, and auto- the Bank supported enrollment gains, gross matic promotion, have had negative effects on enrollment ratios increased an average of 19 learning outcomes, at least in the short run, and percentage points over the past 10–12 years. In some of these strategies are difficult to sustain. countries such as Mali and Uganda, enrollment more than doubled. Improved Learning Outcomes Enrollment expansion has generally come Little of the Bank’s recent analytical work through supply-side interventions: creating new covering primary education has focused mainly schools within easy walking distance of home, on learning outcomes and their determinants. hiring more teachers, or activating community This suggests that an adequate evidence base to support. An increasing amount of Bank support inform efforts to raise learning outcomes is for supply-side expansion programs is coming often lacking. Many countries still do not xv FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA generate the information they need to design comes later; expansion and quality improve- solutions to improve low learning outcomes ment can be successfully undertaken together among the disadvantaged, and there has not and can have mutually reinforcing effects. been adequate experimentation with local Moreover, competing pressures may make it solutions and their evaluation with respect to difficult to undertake quality retrofitting at a their impact on learning outcomes. later date. Few of the sample investment projects in the • Failure to provide reading skills in the early pri- portfolio aimed to improve learning outcomes mary school years—among both the advan- (less than one in three). Among those that did, taged and disadvantaged—is often at the root however, the majority were successful. Sample of weak learning outcomes. projects showing the most improvement in • Although the Fast-Track Initiative has been a learning outcomes were in Latin America (Chile, strong force in encouraging rapid increases in Mexico, and Uruguay) and India, and all are enrollment and completion, as the main chan- cases where national commitment to learning nel of coordinated donor support to primary outcomes and their measurement is high. education it could have a much sharper focus Among 12 countries where the evaluation on improving learning outcomes. undertook field studies, only 5 had repeated measures of learning outcomes. In three of Better Management for Better Outcomes these—Ghana, India, and Uruguay—learning Improved sector management has been a goal improved over time, at least in part due to of virtually all Bank-supported primary project interventions. There were even fewer education projects, but performance has been countries where improving learning outcomes below expectations. Only one project in four among the disadvantaged was an objective, but achieved this objective. Only 25 percent of where this was a goal (such as in Uruguay and primary education projects received an IEG India), results were positive, and gaps between rating of substantial or better on institutional the more and less advantaged narrowed. development impact. Particularly weak were Even in countries where learning outcomes activities aimed at improving central manage- have improved, absolute levels of student ment, such as planning, policy making, and achievement are still low. For example, in Ghana budgeting. only 10 percent of children reached the The Bank supported decentralization efforts country’s mastery levels in math and 5 percent in most study countries, often with good results in English. In India, half of 7- to 10-year-olds (Honduras and India), but in some cases there were unable to read fluently a short paragraph was ambiguity in what the different levels of grade 1 difficulty. covered, nonalignment of administrative and Social fund and other community-driven financial features of decentralization, and projects, which have typically emphasized undertraining of local government staff for their school construction, are often loosely linked to new tasks. The extent to which some forms of sector policies. They have frequently overlooked decentralization might be contributing to the need for complementary investments in increased school system inequities has not been school quality and do not always have adequate adequately assessed. technical input from education experts. School-level management activities were relatively more effective, and so were efforts to Key lessons: empower communities in school improvement efforts. This was particularly true for physical • More, better, and more contextualized analyt- improvements but not for improved teaching or ical work is needed on learning outcomes and learning. Various approaches to more equitable their determinants at the primary level. teacher distribution have been tried with mixed • Countries need to resist the temptation to in- results, the most promising being the recruit- crease access first and improve learning out- ment of local (often untrained) youth, as long as xvi EXECUTIVE SUMMARY provisions can be made for their professional factors most likely to affect learning out- development, career paths, and job security. comes in a given country’s context. This Project monitoring and evaluation has will require more analysis of student learn- typically tracked outputs, rather than outcomes ing and its local constraints and facilitators. or impacts, but this appears to be changing. Bank ■ The Bank and governments need to rec- support has helped governments establish ognize that reaching children not yet en- management information, student assessment rolled and improving low achievement levels systems, and research capacity, but their quality will raise the unit costs of primary education. and the degree to which they have been used for improving policy and practice have been limited. • Efforts are urgently needed to improve the performance of sector management Key lessons: in support of learning outcomes. This im- plies that: • Sector management and governance might ■ Programs to improve sector management and have been better dealt with had there been bet- governance need to be based on sound po- ter institutional and political assessments at litical and institutional analyses that take into the outset. account the incentives faced by officials and • Weak management incentives at all levels can teachers to improve the quality of instruction be a constraint, especially to the improvement and learning outcomes. Accountability and of education quality. There are often more re- supervision systems need to be adapted to wards for increasing the number of schools support improved learning outcomes. than for the difficult tasks of redistributing ■ Primary education managers need to: (a) teachers, implementing a new curriculum, or track learning outcomes over time—not just doing effective monitoring and evaluation. the average, but among different income and • Few Bank-supported country programs di- social groups; (b) monitor individual staff rectly addressed teacher recruitment and per- and system performance indicators, for both formance incentives; particularly lacking are centralized and decentralized activities; and performance incentives related to student (c) create and use incentives to encourage learning outcomes. staff to improve and use technical skills. All new Country Assistance Strategies should in- Recommendations clude learning outcome indicators. ■ Analytic, assessment, and research activi- • Primary education efforts need to focus ties need to be oriented to informing key on improving learning outcomes, par- management and policy issues, with incen- ticularly among the poor and other dis- tives to ensure that the findings are used in advantaged children. The MDG push for decision making. One such research prior- universal primary completion, while a valu- ity would be to assess the impact of decen- able intermediate goal, will not ensure that tralized management on inequalities across children achieve the basic literacy and nu- income and social groups and to identify meracy that are essential to poverty reduc- mitigation measures of any adverse effects. tion. This means that: ■ Improving learning outcomes needs to be a • The Bank needs to work with its devel- core objective of all support for primary ed- opment partners to reorient the Fast- ucation, with a particular focus on achieving Track Initiative to support improved equity in learning outcomes by gender and learning outcomes, in parallel with the among the poor or otherwise disadvantaged. MDG emphasis on primary completion. ■ The Bank’s primary education assistance— This will require the following: whether sponsored by the Education Sec- ■ Reframe the goals and objectives of the tor or other sectors—needs to focus on the Fast-Track Initiative to include improved xvii FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA learning outcomes for all, in addition to peated learning assessments capable of school completion for all. tracking outcomes separately for disadvan- ■ Require learning achievement indicators and taged groups, including the poor. targets in country Fast-Track Initiative pro- ■ Revise cost and funding gap estimates to (a) posals and add items to the indicative frame- reflect the costs of achieving basic learning work that are directly related to learning outcomes (not simply primary completion) outcomes, such as instructional time, teacher and (b) take into account the increased unit attendance, or availability of textbooks. costs of expanding access to and improving ■ Assist countries, financially and technically, learning outcomes among children from dis- to set up suitable systems to conduct re- advantaged backgrounds. xviii ACRONYMS AND ABBREVIATIONS APL Adaptable Program Loan CAS Country Assistance Strategy CCT Conditional cash transfer CSR Country Status Report DPEP District Primary Education Project (India) DPL Development policy lending EdSIP Education Sector Investment Program EDUCO Community-Managed School Program (El Salvador) EFA Education for All EMIS Education management information system ESAC Education Sector Adjustment Credit (Uganda) ESW Economic and sector work EU European Union FTI Fast-Track Initiative GDP Gross domestic product GER Gross enrollment rate HIPC Heavily indebted poor countries IBRD International Bank for Reconstruction and Development (World Bank) ICR Implementation Completion Report IDA International Development Association IEG Independent Evaluation Group (formerly OED) LLECE Latin American Laboratory for Assessment of Quality of Education M&E Monitoring and evaluation MDG Millennium Development Goal MECEP Primary Education Quality Project (Peru) MER Ministry of Education and Research (Romania) NER Net enrollment ratio NGO Nongovernmental organization OECD Organisation for Economic Co-operation and Development OED Operations Evaluation Department (now IEG) PCR Primary completion rate PEP Primary Education Project (Vietnam) PETDP Primary Education and Teacher Development Project (Uganda) PISA Programme for International Student Assessment PPAR Project Performance Assessment Report PROHECO Community Education Program (Honduras) PRSC Poverty Reduction Support Credit PRSP Poverty Reduction Strategy Paper PSRL Programmatic Social Reform Loan QAE Quality at entry xix FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA QAG Quality Assurance Group SAP Social Action Program SAPP Social Action Program Project SAR Staff Appraisal Report SWAp Sectorwide approach TIMSS Third International Mathematics and Science Study UNESCO United Nations Educational, Scientific, and Cultural Organization UPE Universal primary education USAID United States Agency for International Development OED changed its official name to the Independent Evaluation Group (IEG) in December 2005. The new designation “IEG” will be inserted in all IEG’s publicatons, review forms, databases, and Web sites. xx Chapter 1: Evaluation Highlights • The World Bank has committed approximately $14 billion for primary education since 1963. • Primary education contributes to poverty reduction primarily by improving basic knowledge and skills. • Basic knowledge and skill levels, even among school graduates, have often been very low in developing countries. • Bank policy objectives for primary education have emphasized universal access and improved learning outcomes. • The education Millennium Development Goals and the Fast-Track Initiative emphasize primary school completion, not learning outcomes. 1 Introduction rimary education is a powerful lever for poverty alleviation and social and P economic growth (World Bank 2002b). Its results can be empowering, enabling graduates to take charge of their lives and make more informed choices, contribute to the building of a democratic polity, increase earning po- tential and social mobility, improve personal and family health and nutrition (particularly for females), and enable women to control their fertility.1 Advancing Primary Education: alone has committed about $14 billion for A Worldwide Goal support to primary education in more than 100 World Bank studies in the early 1980s showed low- and middle-income countries (box 1.1).3 relatively high rates of return to investments in Primary enrollments grew rapidly in the 1960s primary education (Psacharopoulos and and 1970s, but stagnation and setbacks in the Woodhall 1985). More recent research shows 1980s were brought on by economic that it is the knowledge and skills acquired downturns, especially in Sub-Saharan Africa during primary education rather than the (figure 1.1). number of years of schooling completed that An estimated 103 million 6- to-11-year-olds in make a difference in personal economic developing countries—or about one-fifth of the mobility (Glewwe 2002) and national economic total—were still not in growth (Coulombe, Tremblay, and Marchand school in 2001 (UNESCO About a fifth of 2004; Hanushek and Kimko 2000).2 Thus, to the 2004). United Nations developing country extent that public investments in primary global monitors now education are effective in conveying these predict that, at current children still lack access learning outcomes, support for primary trends, nearly 47 million to primary education. education is central to the World Bank’s children will still be out mandate of poverty reduction. of school in 2015 (UNDP 2005). Developing countries, the World Bank, and About 80 percent of out-of-school children the international community have invested were in low-income countries in South Asia and heavily in primary education over the past few Sub-Saharan Africa, and 15 percent were in the decades. Since 1963, when it began lending for Middle East and North Africa (World Bank education, through mid-2005, the World Bank 2002b). Within countries, access to primary 3 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 1.1: How Much Has the World Bank Committed to Primary Education? If all primary education projects were devoted completely to pri- the “general education” code, is to allocate half of general edu- mary education, then calculating the World Bank’s commit- cation expenditures to primary education. Many social fund and ments would be straightforward. However, primary education is community-based or -driven projects also allocate funds for “other often part of a larger investment or sector adjustment activity that social services,” and it is left to the communities to decide what includes other education subsectors and improvements in man- will be financed. agement and administration. This makes it difficult to attribute It was not possible to calculate what share of these types of them to any one level of schooling. Further, primary education funds was ultimately used by communities to finance primary ed- can also be found as part of projects in other sectors, such as ucation inputs, but given the proliferation of this type of project (IEG agriculture, community development, or HIV/AIDS, or incorpo- 2005c) and the fact that basic education is often among commu- rated in development policy lending (DPL) with objectives related nities’ top priorities, primary education financing from this source to primary education, even though the budgetary support is not could be substantial. earmarked for specific sectors. The Independent Evaluation Group (IEG) estimate of $14 billion The evaluation used two internal databases to estimate primary in Bank commitments to primary education since 1963 is based on education expenditure—one maintained by the Bank’s Education data for projects originating in the Education Sector, on data for Sector exclusively for projects originating in that sector, and a Bank- projects originating in other sectors, and the Education Sector’s wide database that covers projects in all sectors, including edu- convention of allocating half of general education commitments cation. Both databases attribute percentages of project spending to primary education. to specific subsector codes, including primary education.a For DPL, including Poverty Reduction Support Credits (PRSC), In recent years a “general education” code, which can in- the database with all projects attributes a notional share of the total clude all types of education expenditures, has come into wide- commitment for budgetary support to as many as five sectors, based spread use. The convention followed by Education Sector on an assessment by the task team leader of the frequency of the management, based on analysis of a subsample of projects with sector’s occurence in the policy matrix. education is unevenly distributed by gender, performance standards, with results in many low- income, ethnicity, and disability and between income, rural areas being “only marginally better rural and urban areas. than for children who have not completed school” Two-thirds of out-of-school children were (Watkins 2001, p. 105).4 Such results are echoed in girls, a share almost unchanged from a decade the United Nations Educational, Scientific, and before (Watkins 2001). Moreover, children from Cultural Organization’s EFA Global Monitoring the richest 20 percent of households in develop- Report, aptly subtitled “The Quality Imperative” ing countries are three times more likely to be in (UNESCO 2004). school than those from the poorest 20 percent (UNDP 2005). The Evolution of World Bank Policy on Among children already enrolled in primary Primary Education school, learning outcomes have often been low— The World Bank’s commitment to universal in some cases disastrously low—reflecting primary education dates back to its 1980 widespread ineffectiveness in teaching and Education Sector Policy Paper, which learning processes. National test data from emphasized for the first time the relatively high Bangladesh, Brazil, Ghana, Pakistan, the Philip- rates of return to primary education (World Bank pines, and Zambia all 1980). 5 The Bank’s 1990 policy paper, Primary Education, portrayed primary education as the For those enrolled, low show a majority of those who leave primary school foundation of a country’s human capital develop- learning outcomes are to be achieving well below ment (World Bank 1990). It concluded with a widespread. their countries’ minimum challenge to developing countries and to itself: 4 INTRODUCTION Figure 1.1: Trends in Gross Primary Enrollment Ratios by Region, 1970–2000 140 120 Gross primary enrollment ratio 100 80 60 40 20 0 1970 1975 1980 1985 1990 1995 2000 Sub-Saharan Africa East Asia and Pacific Middle East and North Africa Latin America and the Caribbean South Asia Eastern Europe and Central Asia Sources: UNESCO Statistical Yearbook (1999) for 1970–95 for all Regions except Europe and Central Asia and World Development Indicators (World Bank 2004h) for that Region and the year 2000. Note: The gross primary enrollment ratio (GER) is defined as the number of children enrolled in primary school as a percent of the primary school-age population. It can exceed 100 be- cause of the enrollment of over-age children, due to late enrollment or repetition. The net primary enrollment ratio (NER), which is the number of children enrolled of primary school age as a percent of the primary school-age population, is always lower and cannot exceed 100. None of the Regions has achieved an NER of 100. “adequate funding of a good-quality primary whether people ac- Equitable access and education system that is widely and equitably tually learn as a result available is … a critical priority for both national of those opportuni- improved learning were budgets and external aid.” The twin policy ties, i.e., whether they early Bank policy objectives of more equitable access and incorporate useful concerns. improved student learning set a pattern for all knowledge, reasoning subsequent policy papers. It was this focus that ability, skills, and values. The focus of basic the Bank took to the first Education for All (EFA) education must, therefore, be on actual conference, held in Jomtien, Thailand, in 1990, learning acquisition and outcome, rather where nations and development agencies than exclusively upon enrollment, contin- committed to “meeting basic learning needs” of ued participation in organized programmes, children and adults.6 While the resulting World and completion of certification require- Declaration on Education for All committed to ments.7 achieving universal primary education by the year 2000, it underscored that the ultimate In a 1995 review of Priorities and Strategies objective of these efforts is learning: for Education (World Bank 1995), the Bank gave top priority to “basic” education, which included Whether or not expanded educational but was not limited to primary education, opportunities will translate into meaningful emphasizing sectorwide policy reform; equity of development … depends ultimately on access for the disadvantaged (girls, the poor, 5 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA The second Education for ethnic minorities, the cess to and complete free and compulsory ed- disabled, and those in ucation of good quality All conference prioritized remote or hardship • Eliminating gender disparities in primary and improved learning areas); and institutional secondary education by 2005 and achieving outcomes. development, including gender equality in education by 2015, with a the capacity to measure focus on ensuring girls’ full and equal access learning outcomes (see box 1.2). A new to and achievement in basic education of good Education Sector Strategy Paper in 1999 quality reaffirmed the commitment to basic education— • Improving all aspects of the quality of education especially for the poorest and for girls—and to so that recognized and measurable learning out- systemic reform (World Bank 1999). comes are achieved by all, especially in literacy, Subsequently, the Bank supported the Dakar numeracy, and essential life skills” (UNESCO Framework for Action 2000, p. 8) The education MDGs that was the result of a emphasize primary second EFA conference, In 2000 the Bank also endorsed the Millen- school completion, not the World Education nium Development Goals (MDGs). The goal for Forum, held in Dakar, education overall was to ensure “that by 2015 learning outcomes. Senegal, in April 2000 children everywhere, boys and girls alike, will be (UNESCO 2000). The able to complete a full course of primary Dakar Framework resulted in a renewed global education,” plus a push for gender equity in commitment to primary education, not simply access, and literacy for youth age 15–24. to improved access and quality of instruction, However, unlike both the 1990 World Declara- but to equitable achievement of learning tion on Education for All and the 2000 Dakar outcomes. Specifically, it advocated: Framework for Action, the MDGs primarily address the issue of access to primary education • “Ensuring that by 2015 all children, particularly and do not include an explicit goal with respect girls, children in difficult circumstances, and to either the quality of instruction or to learning those belonging to ethnic minorities have ac- outcomes, such as literacy or numeracy. In 2002, the World Bank and other regional, bilateral, and international development agencies established the Education for All–Fast- Box 1.2: Primary, Basic, and General Education Track Initiative (FTI) as a means of accelerating progress toward the MDG of universal primary school completion by 2015 in low-income For the purposes of this evaluation, primary education is defined as the countries (World Bank 2004d).8 The FTI is a “general school education at the first level [plus non-formal education partnership between national governments and at this level], programs designed to give skills in numeracy and liter- donors. Countries can qualify for FTI support by acy and to build the foundations for further learning.” Depending on the submitting a Poverty Reduction Strategy Paper conventions in a country, this would include the first five to eight years (PRSP) or the equivalent and a national of formal education. The term basic education includes primary in- education plan, both of which prioritize univer- struction but can also cover a broader set of educational programs, in- sal primary education.9 Countries are also cluding lower secondary education, early childhood education, adult expected to commit to monitorable policy, literacy, and life-skills or nonacademic nonformal education programs. service delivery, and financing targets using More recently, project designers inside and outside the Bank’s Education “benchmarks” specified in the Indicative Sector have begun using the term general education to define the Framework (see Appendix E).10 In return, content of education projects, a term that is sometimes used inter- donors are expected to scale up technical and changeably with primary education or to describe projects covering financial resource mobilization to support these more than four subsectors of education. country-driven programs and to harmonize their support.11 6 INTRODUCTION The Bank’s 2005 Education Sector Strategy project support for adult The Fast-Track Initiative Update, which incorporated some early findings literacy programs has was created to accelerate from this evaluation, committed the Bank to been previously reviewed maintaining momentum on EFA and the MDGs, (Abadzi 2003), as has its attainment of the MDGs. while at the same time strengthening “education support to secondary for the knowledge economy” (secondary, higher, education (Perkins 2004); and a review of support and lifelong education; World Bank 2005b). The for early childhood development is being sector plans to work through the FTI to maintain planned. An evaluation of support to primary momentum on EFA in the low-income countries. education—by applying findings and lessons to It strongly supports strengthening the results the key assumptions and strategies of current orientation of the sector (greater attention to programs—has the potential to substantially education outcomes), suggesting that key influence the strategic agenda and effectiveness of education outcome indicators be included in all future policies aimed at the EFA goals, especially new Bank country-level planning documents basic knowledge and skills acquisition for all. (Country Assistance Strategies). The key questions addressed by this evalua- Over the 15-year period in which primary tion are the following: education has been a priority for the Bank, policy objectives have been remarkably stable • To what extent have the Bank’s policies for and can be summed up as follows: universal primary education been implemented? primary school enrollment (and, more recently, • How effective and sustainable have Bank-sup- completion); equality of access for girls (gender ported programs in primary education been in parity) and other underserved groups; and helping countries increase access to schooling improved learning outcomes. Because universal and improve learning outcomes, especially for enrollment and completion assume equity of the most disadvantaged among and within access, there are really two policy objectives: countries? universal enrollment and completion, and • To what extent has support to primary education improved learning outcomes.12 promoted institutional development? • What are the lessons from experience, in terms Evaluation Objectives and Design of key factors or de- The overall objective of this evaluation is to terminants of effec- The Bank’s policy assess the development effectiveness of World tiveness of the Bank’s objectives over 15 years Bank assistance to countries in their efforts to assistance for primary improve their basic knowledge and skills base education? consistently emphasized through the provision of quality primary improved access and education to all children, particularly since the The evaluation traced learning. beginning of the EFA movement in 1990. the World Bank’s support While the global EFA strategy advocates many to countries through the results chain, from Bank channels for pursuing its learning goals, including inputs (such as finance, policy dialogue, and analytic schooling at the primary and lower secondary work) to government inputs (policies and plans, levels, nonformal education, early childhood public spending, and institutional capacity), to development, adult literacy, and life skills educational system inputs/service delivery (for programs, this evaluation focuses on Bank example, classrooms, textbooks, trained teachers, support for publicly provided primary education. and supervision/com- This is not to deny the importance of the other munity involvement), to The Bank’s main channels, but reflects the fact that primary outputs (primary school contribution to EFA has education has been the main vehicle of the Bank’s enrollments and com- come through its support assistance to EFA and that IEG has yet to conduct pletion), to outcomes a comprehensive evaluation of the effectiveness (basic knowledge and to universal primary of Bank support to this subsector. The Bank’s skills acquisition, and education. 7 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 1.3: Evaluation Building Blocks • Literature reviews of (a) the rationale for investing in primary 5 Education Sector adjustment projects, and 15 adjustment education and (b) the determinants of primary education and investment projects managed by other sectors that al- outcomes in developing countries (Boissiere 2004a, b). located the most to primary education (IEG 2004d). Together • Review of World Bank documents on primary education pol- these 50 projects comprise the portfolio sample. icy, project design and completion reports, education sec- • Field-based evaluations of completed primary education proj- tor retrospectives (annual reports), research and policy ects: Project Performance Assessment Reports (PPARs) in Hon- dialogue reports, plus IEG evaluations of related sectors and duras, India, Niger, Uganda, Uruguay, Vietnam, and the Republic subsectors. of Yemen, and an impact study on basic education in Ghana. • An inventory and review of the portfolio of primary educa- • Field-based country case studies for an in-depth, contextu- tion projects sponsored by the Education Sector of the Bank alized view of the impact of the entirety of the Bank’s lend- and by other sectors, covering more than 700 Bank-financed ing, analytical work, and policy dialogue on primary education projects in more than 100 countries in a general way, and for in Mali, Pakistan, Peru, and Romania. (See Appendix F for more in-depth analysis, a random sample of 30 primary ed- more details on case study selection and methods and Ap- ucation investment projects (20 completed and 10 ongoing), pendix G for summaries of the case study reports.) welfare/employment outcomes).13 A full work, with respect to their magnitude and description of the analytic framework, the geographic distribution, objectives, the activities evaluation design, and instruments is in supported, and overall performance in meeting Appendix B. Note that learning outcomes (basic their objectives. knowledge and skills) are the ultimate results in Chapter 3 examines in depth the experience the “results chain” but that access to and of Bank support in helping countries meet the completion of primary education of good quality central objectives of improving both access to are among the major inputs to achieving them. primary education and learning outcomes. This This approach resulted in a number of discrete rich experience not only points to key activities or intermediate outputs that served as accomplishments and shortcomings but also to building blocks for the evaluation (box 1.3), most key lessons for countries and the Bank in of which can be accessed in the evaluation Web enhancing the performance of primary site (www.worldbank.org/ieg/education). The education systems. evaluation also builds on the findings of a joint Chapter 4 examines Bank support to evaluation of donor support to basic education, countries in improving sector management and conducted in 2003 with 12 other donors and led governance in their pursuits of better by the Netherlands (Joint Evaluation 2003).14 educational outcomes. The next chapter provides an overview of the Finally, Chapter 5 presents the study’s main evolution of the portfolio of lending and analytic conclusions and recommendations. 8 Chapter 2: Evaluation Highlights • Primary education commitments increased dramatically with the EFA movement in 1990, especially in low-income countries. • A growing share of lending for primary education has been managed outside the Education Sector. • Projects managed by other sectors focus mainly on increased enrollment. • Only one in five primary education projects has learning outcomes as an explicit objective. • A large share of analytic work in education covers primary education, but little of it focuses mainly on learning outcomes. 2 Trends in World Bank Support to Primary Education ew financial commitments to primary education jumped dramatically N in the early 1990s as the World Bank embraced the EFA movement. Total lending for primary education for the 27 years leading up to 1990 was about $1.7 billion.1 Evolution in Lending for provide any support to Education for All drove Primary Education primary education have primary education In the five years after the 1990 EFA conference been in the form of and the Primary Education Policy Paper (World International Develop- commitments to new Bank 1990), the number of projects supporting ment Association (IDA) highs in the 1990s. primary education roughly doubled and credits, rising from 59 commitments more than tripled (figure 2.1). percent before 1990 to 74 percent in 2000–04. During the five-year period beginning in 2000, the year of the second EFA conference, the Primary education commitments rose in number of projects continued to climb, but new all Regions, most noticeably in Latin America financial commitments leveled off. In 2005, the and the Caribbean, South Asia, and Sub-Saharan first year in the current five-year period (not Africa (figure 2.2). Lending for primary shown in the figure), the number of projects education did not increase significantly in continued to increase, but the commitment Europe and Central Asia until 1995–99, follow- levels remained flat.2 In all, between 1990 and ing the break-up of the former Soviet Union, 2005, lending for primary education increased while it initially declined in the Middle East and sevenfold above previous years, to about $12.3 North Africa before recovering in 2000–04. Latin billion. Thus, around 88 percent of all Bank American and South Asian countries have commitments for primary education have been borrowed the most for primary education ($4.4 approved since 1990.3 billion and $3.6 billion, respectively), followed The share of primary education commit- by Sub-Saharan Africa ($2.6 billion), but Sub- ments going to the countries accounting for the Saharan Africa had the largest number of poorest 40 percent of the global population has projects financing primary education (table 2.1). also more than doubled, from 26 to 54 percent, consistent with the Bank’s strategy (IEG A growing share of lending for primary 2004d).4 About two-thirds of projects that education has been through projects 11 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Figure 2.1: Increase in World Bank Commitments to Primary Education, 1963–2004 250 4,500 4,000 200 Second EFA Conference 3,500 Primary education (Dakar 2000) Number of projects policy paper and 3,000 US$ millions first EFA Conference 150 (Jomtien, Thailand, 1990) 2,500 2,000 100 1,500 50 1,000 500 0 0 1963–69 1970–74 1975–79 1980–84 1985–89 1990–94 1995–99 2000–04 Fiscal year of approval Projects Commitments Source: World Bank database of projects managed by the Education Sector. Notes: a. The World Bank fiscal year runs from July 1 to June 30; for example, fiscal year 2000 covered July 1, 1999, to June 30, 2000. b. This figure includes projects in all sectors with any primary education expenditure—half of commitments coded as general education were assumed to be for primary education. c. The entire commitment for a project is allocated to the year the project was approved. d. In fiscal 2005 a total of 70 projects with an estimated US$818.4 million in commitments to primary education was approved. Much of the growth in managed by other projects, 4 percent are for other investment sectors and for projects, and 1 percent for emergency lending.8 primary education development policy The dramatic increase in primary education lending has been in lending (DPL).5 For the commitments managed by other sectors is due projects managed by most recent five-year to a proliferation of projects with relatively small period, 31 percent of all primary education components. As can be seen sectoral units other than commitments to primary in figure 2.3, the number of newly approved the Education Sector. education were from projects with primary expenditure managed by com- ponents of projects the Education Sector has stabilized at about 70 managed by other sectoral units (see figure 2.3). In per five-year period, while the number managed fiscal 2005 the share reached 53.5 percent. by other sectors has increased every period Projects managed by the Education Sector that since 1990. That accounts for more than two- were approved in fiscal 2000–04 remain predomi- thirds (69 percent) of projects with any primary nantly traditional investment projects, as in previous education spending in 2000–04. periods.6 However, among the 31 percent of The projects managed by other sectors commitments managed by other sectors, nearly half committed, on average, $8 million to primary (15 percent) are develop- education, while those managed by the Many of the projects from ment policy lending Education Sector had primary education commitments of about $40–$55 million each.9 other sectors have small (including7 PRSCs [3 percent]), 11 percent are Only 2 percent of the 343 projects with any primary education for social funds or primary education expenditure managed by components. community- driven–type other sectors since 1963 allocated half or more 12 T R E N D S I N W O R L D B A N K S U P P O R T T O P R I M A R Y E D U C AT I O N Figure 2.2: Increase in New Commitments for Primary Education, by Region 1,600 1,400 1,200 1,000 $US millions 800 600 400 200 0 1963–69 1970–74 1975–79 1980–84 1985–89 1990–94 1995–99 2000–04 Fiscal year of approval Sub-Saharan Africa East Asia and Pacific Middle East and North Africa Latin America and the Caribbean South Asia Europe and Central Asia Sources: World Bank databases, one for projects managed by the Education Sector and a second for projects managed by other sectors. Notes: Includes projects in all sectors with any primary education expenditure; half of commitments coded as general education were assumed to be for primary education. The entire commitment for a project is allocated to the year the project was approved. of project commitments to primary education. primary education components managed by other Among projects with any primary expenditure sectors and to policy-type lending has potential managed by the Education Sector, however, the implications in several areas: the relevance of figure was 49 percent. primary education lending to sector policies and The shift in the composition of the portfolio strategies; the adequacy of supervision, monitor- from primary education investment projects ing, and evaluation; and the effectiveness and managed by the Education Sector to smaller impact of primary education lending.10 Table 2.1: Cumulative Projects and Commitments for Primary Education, 1963–2005, by Region Sub-Saharan Latin America South East Asia and Middle East Europe and Africa and the Caribbean Asia Pacific and North Africa Central Asia Total Number of projects approved 280 176 65 84 57 68 730 New commitments (millions of nominal US$) 2,619 4,356 3,649 1,886 760 814 14,084 Source: World Bank database. 13 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Figure 2.3: New Commitments to Primary Education by Managing Sector 4,500 180 4,000 Number of 160 800 projects, other sectors 3,500 140 Commitments: $US million 284 1,250 Number of projects 3,000 120 2,500 100 2,000 80 48 3,144 1,500 3,375 60 2,730 Number of 1,000 projects, 40 1,642 education 500 sector 20 0 0 1963–89 1990–94 1995–99 2000–04 Time period Education sector Other sectors Sources: World Bank databases, one for projects managed by the Education Sector and a second for projects managed by other sectors. Note: Includes projects with any primary education expenditure; half of commitments coded as general education were assumed to be for primary education. However, many of the projects approved in approach. In some places the approach consists the most recent period are still active, and few of of a mix of adjustment and investment instru- the completed policy-type lending operations ments. See box 2.1 for a description of the have been independently assessed.11 Thus, this Bank’s sectorwide approach in Uganda. evaluation has not been able to assess fully and systematically the relative advantages and Since 1990 the share of Bank education disadvantages of these different approaches, the lending allocated to primary education management of primary education lending by has increased, while that to some other other sectors, or the differential impact of subsectors, particularly vocational edu- various kinds of projects on learning outcomes. cation, has diminished. Figure 2.4 shows the Also, in recent years an increasing number of funding commitments over five consecutive five- countries have begun to integrate their year periods. Most striking is the strong growth education reform efforts through sectorwide of funding for the sector as a whole, until the planning and program support (as opposed to downturn in 2000–04. project support) from donor agencies. A sector- Concerning subsector support, until 1990 wide approach (SWAp) to financial assistance education lending was predominantly for has become a growing feature in the Bank tertiary and vocational education, a reflection of education portfolio—for the Bank’s focus on manpower planning and Lending for primary example, in its support technical skills development. The early 1990s education grew, while to primary education in saw a massive shift to primary education that of other education Uganda and in India (not (already under way in the late 1980s), reflecting yet evaluated). However, the growth of the poverty-alleviation agenda and subsectors fluctuated there is no particular the Bank’s support for the EFA movement. or fell. Bank instrument for this This continued until 2000–04, when its 14 T R E N D S I N W O R L D B A N K S U P P O R T T O P R I M A R Y E D U C AT I O N Box 2.1: Sectorwide Lending Support in Uganda In 1996 Uganda’s President Museveni made a pre-election com- evidence available, the rapid expansion of enrollments led to a de- mitment to eliminate primary school fees for up to four children terioration in both education quality and learning outcomes. in each family as of the new school year, ushering in what is now A more conventional Bank investment project, the Primary Ed- called the big bang approach to universal primary education. The ucation and Teacher Development Project (PETDP), had already been policy almost doubled primary school enrollments in 1997. under implementation for five of its seven years when ESAC was To help the Ministry of Education and Sports cope with this en- launched. With the new sectorwide project, PETDP was re-ener- rollment explosion, the government and the Bank quickly prepared gized and reoriented to the new universal primary education effort. a sectorwide program in 1988, called the Education Sector Ad- In the end, ESAC and PETDP became mutually supportive. justment Credit (ESAC). The project aimed to improve the efficient ESAC is highly regarded for its role in improving sector planning use of public resources and the availability of quality inputs, and and budgeting functions, but PETDP was considered essential as to strengthen sector management. ESAC funds, combining an IDA a source of innovative ideas and training. credit of $80 million and a Highly-Indebted Poor Country (HIPC) debt- reduction grant of $75 million, were disbursed annually in the form Rapid expansion of inputs, financed through ESAC, de- of budget support. The ESAC acted as a catalyst for a group of fund- pended on development over time through PETDP of sys- ing agencies working together in budget support to the sector. tems for teacher development, textbook procurement, An IEG evaluation of ESAC found that the project was successful classroom construction, and some capacity to further in partially mitigating the effects of the explosive expansion on ed- develop and manage these systems. It has not yet been ucation quality. Ultimately, however, ESAC targets for reducing demonstrated in Uganda that capacity and institution pupil:teacher and pupil:textbook ratios were not met, largely be- building needs in the sector can be sufficiently ad- cause of unanticipated, continuing growth in enrollments. From the dressed through budget support alone (IEG 2004c, p. 28). Figure 2.4: Education Commitments by Subsector and Time Period 10,000 8,000 US$ millions 6,000 4,000 2,000 0 1980–84 1985–89 1990–94 1995–99 2000–04 Time period General/other Adult lit/NFE Vocational Tertiary Secondary Primary Pre-primary Sources: World Bank databases, one for projects managed by the Education Sector and one for projects managed by other sectors. Note: This covers education lending from the education and other sectors; half of commitments coded as general education were assumed to be for primary education. 15 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA commitments stabilized, compared with all frequently cited objectives of 30 randomly selected other subsectors, except general education, in projects that allocate at least half of commitments which they fell. Commitments to vocational to primary education, among those that had education fell steadily from 1990 onward; those closed since fiscal 1995 or that were still active as of to secondary and tertiary have waxed and the end of fiscal 2004. waned. Commitments to general education For the purpose of this evaluation, the term increased, as a reflection of the above- primary education project refers to investment mentioned increase in lending from outside the projects managed by the Education Sector that education sector. allocate at least half of all commitments to primary education. The 20 closed projects in table 2.1 were Government financial commitment to approved during the period 1988–96, and the 10 primary education has also increased in active projects were approved roughly a decade many countries, often influenced by Bank later, from 1998 to 2004. policy development support. Among the 12 countries where IEG conducted field studies, The objectives most frequently cited— half increased primary education’s share of total found in virtually all primary education public education expenditure between 1995 and projects—were to improve sector manage- 2003. Three of them—India, Mali, and Niger— ment or governance and to improve the did so by 15–35 percent- quality of education. Whereas in completed Government financial age points. In three projects, quality of education was mostly commitment to primary countries the propor- indicated by increases in inputs (books and tions stayed the same, materials) and outputs (trained teachers), education has also and in two others the ongoing projects have also included learning grown. proportions declined. outcomes as indicators of quality improve- During the late 1980s ment.13 In addition, roughly two-thirds of all and 1990s, government financial commitment to projects aimed to expand enrollments and primary education became one of the focal points improve equity with respect to gender, urban- of World Bank adjustment (development policy) rural residence, the poor, the disabled, or lending in many countries. In all four African otherwise disadvantaged children.14 A little countries studied—Ghana, Mali, Niger, and more than half attempted to improve the Uganda—adjustment projects were launched in “internal efficiency” of primary education which lending conditions included moving or systems by reducing repetition and dropout holding expenditures in primary (or basic) rates.15 These objectives were remarkably stable education to a relatively high level (40–60 percent across both completed and active projects. of total). All four countries met or exceeded their adjustment targets, despite political and Only one in five primary education economic challenges, which shows how seriously projects had an explicit objective to these conditions were taken.12 improve learning outcomes or basic skills. This was equally true for both completed and Evolution of Objectives, from Expansion ongoing projects. A separate review of appraisal to Learning Outcomes documents covering the 23 primary education projects managed by the Education Sector and Investment projects managed by the approved in fiscal 2005 and most of fiscal 200616 Education Sector that found, again, that only about one in five projects Almost all projects were mainly con- (22 percent of the total) had an explicit develop- address sector cerned with primary ment objective to improve learning outcomes. education had mul- management and tiple objectives. Table Adjustment and development policy education quality. 2.2 presents the most lending projects that support primary 16 T R E N D S I N W O R L D B A N K S U P P O R T T O P R I M A R Y E D U C AT I O N Table 2.2: Objectives of Education Investment Projects that Allocate at Least 50 Percent of Expenditure to Primary Education (percent of projects with objective) Projects completed since fiscal 1995 Active projects All projects Stated objective (n = 20) (n = 10) (n = 30) Improve sector management or governancea 95 100 97 Improve educational qualityb 90 100 93 Increase enrollment 65 70 67 Improve equityc 60 65 62 d Increase internal efficiency 60 50 57 Improve learning outcomes 20 20 20 Sources: IEG 2004d, table 4, and project appraisal documents. a. Includes sector governance, management capacity, monitoring, and evaluation. b. Usually expressed in terms of inputs and outputs. c. Equity with respect to gender, the poor, rural, ethnic minorities, disabled, and otherwise disadvantaged. d. Reduced dropout and repetition. education pursued a somewhat different Only 30 percent had an Only one in five projects mix of objectives.17 Unlike investment objective of raising aims to improve learning projects, these projects all focused on increased educational quality; and enrollment, and only about 60 percent covered just one in five aimed to outcomes. sectoral management. However, they were improve sector manage- similar to investment projects in that almost all ment. None had learning outcomes objectives. covered improved quality (again, mostly inputs and outputs), about 60 percent equity improve- Primary education projects since 1990 ment, and about half improved educational allocate less to “hardware” and more to efficiency. “software.” In the Also, as with investment projects, few (20 1960s, 1970s, and into Projects managed by percent) had learning outcomes objectives. A the 1980s, the Bank other sectors aimed to growing number of DPLs are multisectoral emphasized “hardware” increase enrollments and PRSCs, of which 28 in 18 countries (some having (civil works and goods, equity, but not learning multiple PRSCs) had a basic education focus including distribution of approved by the Bank during fiscal 2001–05. textbooks). outcomes. Among these projects, about 61 percent In response to re- covered quality improvement or service search showing the influence of curriculum delivery. About 45 percent covered improving reform, better teaching, good management, and access and increasing or maintaining funding for community involvement (Lockheed and education (or primary education). In only two Verspoor 1991; Fuller 1987),18 emphasis in the countries, Nicaragua and Uganda, were learning 1990s shifted to software outcomes emphasized. (services and manage- Education infrastructure ment) and, within hard- Investment projects containing primary ware, from civil works to now gets only slightly education managed by other sectors were textbooks. Civil works more than curriculum, almost entirely focused on improved and textbooks were teaching, management, enrollment and equity objectives. Sixty financed in 93 percent of percent of these had equity improvement as an the projects, but the and community objective, and half cited increased enrollment. share of hardware in involvement. 17 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA overall project expenditures has declined Bank-sponsored analytic work on primary from 82 percent before 1990 to 53 percent in education has rarely put its main focus on fiscal 2000–04. The share for textbooks learning outcomes. Among the 14 studies increased from about 2 percent before 1990 to delivered in fiscal 2000–05 that focused about 11–12 percent in the early 1990s and exclusively on primary education, only three has been maintained, while the share of civil contained in-depth assessments of learning works in primary education commitments has outcomes.20 The others focused on primary declined by half, from 45 percent to 22 education strategy, finance, curriculum, and percent. enrollments. The 89 studies touching on primary education as part of the overall education agenda Bank-Supported Analytic Work covered topics such as national education strate- gies and reforms, finance and cost-effectiveness, Analytic work related to primary teacher training and incentives, private education and financed by the Bank’s education, decentralization, textbook quality, Education Sector stabilized during fiscal community involvement, education manage- 2000–05 at about 17 products per year (see ment, and girls’ education. table 2.3).19 This covers roughly two-thirds to Only three of these sectorwide papers, all three-quarters of all analytic work managed by managed by the Latin America and Caribbean the Education Sector Region, had as their major focus learning A large share of during those years. outcomes and achievement.21 In the Africa education analysis is on Much of this work was Region, the Human Development Sector has conducted in the con- sponsored more than a dozen Country Status primary education. text of sectorwide Reports (CSRs) in education and health as reviews. A few studies inputs into debt-reduction decisions, sector (14 of 103 in this 6-year period) focused plans, PRSPs, and PRSCs. exclusively on problems arising in primary Initially (1999–2000) the CSRs did not cover education—either in a learning outcomes, but they subsequently did The analytic work is specific country or in the showcase some student achievement rarely focused primarily context of regional or measures in relation to funding levels and, in global primary edu- some cases, school and socioeconomic status. on learning outcomes. cation papers. However, using these measures in planning Table 2.3: Analytic Work on Primary Education Managed by the Education Sector, Fiscal 2000–05 Analytic work Analytic work exclusively on on education, Total Percent of Percent of primary education including primary analytic Total total total Fiscal year Country Regional Country Regional work with analytic exclusively w/any delivered level or global level or global any primary work primary primary 2000 3 0 9 7 19 24 13 79 2001 0 0 9 0 9 13 0 69 2002 1 2 13 4 20 25 11 74 2003 1 2 12 0 15 23 13 65 2004 1 0 15 4 20 33 5 61 2005 4 0 11 5 20 26 15 77 Total 10 4 69 20 103 146 10 71 Sources: World Bank 2002a, 2003a, 2004a, 2005a, and the Education Sector Web site (http://education.worldbank.org). 18 T R E N D S I N W O R L D B A N K S U P P O R T T O P R I M A R Y E D U C AT I O N still appears to be problematic: in Madagascar, Performance Ratings of Mozambique, and Rwanda, which have Primary Education Projects completed CSRs that addressed learning All Bank-financed projects are subject to self- outcomes and which subsequently launched evaluation shortly after they are completed. new PRSCs, there is no mention of learning These evaluations are then validated by IEG. outcomes in the PRSCs. Projects are rated on their outcomes in relation Much analytic work sponsored by other to their objectives, sustainability, and institu- sectors of the Bank also is relevant to primary tional development impact.23 Table 2.4 education—for example, in public expenditure summarizes the ratings for primary education reviews, country economic memoranda, and projects.24 poverty assessments, generally sponsored by the Poverty Reduction and Economic Manage- Overall, the outcome of 82 percent of ment Network. When this work is added to the primary education projects was rated Education Sector work, the volume of analytic moderately satisfactory or better. These work relevant to primary education during ratings are above the average for the rest of the 2000–05 more than doubles.22 However, here education sector (78 too there is little work with direct relevance to percent) and substan- Eighty-two percent of learning outcomes. Likewise, this evaluation‘s tially higher than the primary education case studies in Mali, Pakistan, Peru, and Romania average for all sectors (72 projects have been rated (see box 2.2) showed little explicit coverage of percent). Ratings for learning outcomes in the Bank-supported sustainability were lower, moderately satisfactory analytic work related to primary education. with 62 percent rated or better on outcomes. Box 2.2: Analytic Work in Case Study Countries: Where Are the Learning Outcomes? The case studies for Mali, Pakistan, Peru, and Romania all found 1999 that helped to build consensus on sector improvements. The that the Bank had supported useful analytical work relevant to improvements subsequently appeared in Bank-supported invest- primary education. They were, however, generally light in their ment and adjustment projects, emphasizing better infrastructure, treatment of learning outcomes. bilingual education, school autonomy, teacher policy, equity, and Mali. Several studies have been undertaken emphasizing ac- accountability. While Bank support also built strong research and cess (especially gender equity), but a comprehensive sectorwide assessment capacity in the government, it did not press for a lon- review has yet to be done. The use of the results of analytic work gitudinal analysis of student learning outcomes or for impact as- has added to the credibility of the Bank team and has helped the sessments of project interventions (both within the competence government adopt a pattern of planning based on data. However, of local researchers). the Bank sector work has not focused on learning outcomes (ex- Romania. The Bank supported sector work related to the coun- cept in a bilingual education pilot) or on the constraints to the de- try’s economic transition, which was fed into its Education Reform livery of goods to its resource-starved schools. Project. In 2000 a World Bank Institute case study was conducted Pakistan. The Bank supported a sectorwide review in 1988 that on education decentralization, and in 2002 an Education Policy Note set the stage for subsequent policy dialogue and lending, but this was released. Capacity has been built for solid student assess- is out of date. It also supported some influential studies on specific ments, but little attention has been given to mining the outcomes themes, such as demand for girls’ schooling and devolution of ed- data for findings related to improved policy and practice. Also, ed- ucational management. Overlooked have been studies of institutional ucation has not been included in public expenditure reviews (ex- capacity and institutional incentives. Also, the quality and accuracy cept for the most recent) and is almost absent from country of ministry data, and how to improve them, need to be studied. economic memoranda, indicating incipient but still underdeveloped Peru. Two large diagnostic studies were undertaken in 1993 and intersectoral linkages and planning. 19 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Table 2.4: IEG Ratings of Completed Primary Education Projects, by Year of Approval Sustainability Institutional Outcome (% moderately (% likely or development impact satisfactory or better) highly likely) (% substantial or high) Fiscal year approved Before 1990 76 50 20 1990–94 89 66 19 1995–99 85 76 38 All primary education projects 82 62 25 (Number of projects) (117) (104) (106) All education projects excluding primary 78 66 46 All Bank-supported projects 72 50 36 Source: World Bank database, as reported in IEG 2004d. Note: Primary education projects are defined as those managed by the Education Sector and that allocated at least half of total commitments to primary education. IEG introduced sus- tainability and institutional development impact ratings more recently than the outcome rating, so early projects were not rated in these dimensions. The comparison ratings for all ed- ucation projects and all Bank-supported projects are for those projects that closed in fiscal 1990–2001. likely or highly likely to be sustainable over the tional objectives figured in virtually all projects whole period, somewhat below the average for managed by the Education Sector. other education projects, but above the Bank- These IEG ratings convey the extent to which wide average for 1990–2001. projects achieved their overall objectives. In Over time, the sustainability ratings for primary virtually all cases, however, there was more than education projects have steadily improved— one objective, and many of the projects included about three-quarters of the most recent projects objectives for other subsectors. are rated likely or highly likely on sustainability. The next two chapters take a closer look at This is in contrast with the ratings for institutional the extent to which the Bank’s primary development. The overall average of 25 percent of education support has successfully met individ- projects with substantial or high institutional ual objectives having to do with better development impact is well below that for the outcomes, such as expanded enrollments and Education Sector (excluding primary) and the learning achievement (chapter 3) and improved Bank-wide average for recent years. governance and institutions (chapter 4). They The considerable improvement in institu- draw on findings from both the portfolio review tional development ratings (to 38 percent and field-based project assessments, case substantial or high) in the late 1990s is encour- studies, and an impact evaluation to point to aging, but the ratings are still lower than for successful and unsuccessful strategies, lessons other education projects. They are also low in an learned, and the value added by the Bank’s absolute sense, particularly given that institu- involvement. 20 Chapter 3: Evaluation Highlights • Primary education projects have been effective at expanding access. • Although projects have met their equity of access objectives in many cases, gaps between advantaged and disadvantaged children often are not closing. • Reducing high dropout and repetition has been underemphasized. • Projects were relatively ineffective in improving educational quality. • Learning outcomes are generally not measured, but they have improved in some countries, even among the poor. • Though reading is the foundation of learning, few projects support improved early reading skills. • The optimal strategy for improving learning outcomes depends on country conditions and institutions. In the best cases, access and learning are pursued together. 3 Improving Access and Learning Outcomes for the Disadvantaged his chapter assesses the effectiveness of Bank support for improved learn- T ing outcomes (basic knowledge and skills) among the disadvantaged as the main educational driver of poverty reduction. It also identifies lessons learned from that experience. It begins with findings on improved access to and the poor), and internal efficiency to assess primary education, especially for the disadvan- primary school access. taged, given that this is a necessary (but not sufficient) prerequisite for learning, and then Enrollment expansion is one of the addresses the effectiveness of attempts to raise objectives supported by the Bank where learning outcomes for those in school.1 By the efforts have been most effective. Among disadvantaged this evaluation refers to those completed projects with increased enrollment who are generally underserved by public as an objective, 69 percent fulfilled it (table 3.1). education: primarily girls and the poor, but also Data from the IEG field-based studies (PPARs ethnic minorities, the disabled, and those who and country case studies) show how gross live in remote or hardship areas. enrollment rate (GER) in this group of Bank- supported countries increased an average of 19 Primary School Access percentage points over the past 10–12 years. Most Bank-supported projects since 1990 In some countries the rates have been measure “access” in terms of expanded enroll- nothing short of explosive (figure 3.1). In ments (or enrollment ratios) and improved countries experiencing rapid growth of the equity for the disadvantaged (see box 3.1). school-age population (Mali, Pakistan, the Access could also refer to primary school Republic of Yemen), the increases in enrollment completion, which is what the MDGs highlight. ratios are even more remarkable. Not all of this Before 2000, this view was not frequently taken, expansion can be attributed to Bank financial and few countries had reliable school comple- support—in Uganda, for example, the elimina- tion data. More often, countries and projects tion of school fees was focused on internal efficiency measures — the driving force. Projects have generally dropout and repetition—related to completion rates. This review therefore uses enrollment Impact studies in met their enrollment ratios, measures of equity (especially for girls Ghana and India show objectives. 23 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 3.1: Measuring Primary School Access The most widespread indicators of access are the enrollment ratios, gross and net. • Gross primary enrollment ratio (GER) = No. of children of any age enrolled in primary school x 100 No. of children of primary school age (can exceed 100, due to enrollment of over- or underage children) • Net enrollment ratio (NER) = No. of children of primary school age enrolled in primary school x 100 No. of children of primary school age (cannot exceed 100) • Primary school completion rate (PCR)a No. of students completing last year of primary school x 100 No. of children of official graduation age • Gender equity or parity is measured as the ratio between boys’ and girls’ enrollment ratios. • Internal efficiency generally refers to— • Dropout: A child’s leaving school after having been enrolled (low persistence) • Repetition: The requirement that a child repeat one or more grades. a. There are several ways to define primary school completion. This is the definition used by the World Bank. convincingly how Bank-supported projects impact was found (IEG 2004a; Jalan and Glinskaya influenced enrollment gains in those 2003).2 The Bank supported not only the financ- countries, largely through the provision of ing of construction costs but also the develop- new or renovated facilities (in India, for ment of innovative and cost-effective building hundreds of thousands of new students). Large designs and construction or contracting infrastructural expansion was the pattern in most procedures (India, Niger, Peru, Romania, Uganda, countries where IEG did fieldwork and project and Vietnam). Table 3.1: Outcomes by Enrollment Objective for Completed Primary Education Projects Fulfillment of objective (percent; n = 20) Number Partially Objective covering objective Fulfilled fulfilled Unfulfilled Undetermined Increased enrollment 13 69 0 23 8 Improved equity 12 75 25 0 0 Improved access for girls 9 55 22 22 0 Improved internal efficiency 12 25 42 25 8 Sources: IEG 2004d, table 13, and World Bank project appraisal and Implementation Completion Reports. 24 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D Figure 3.1: Increases in Gross Primary Enrollment Ratios in Countries Receiving Bank Support 160 140 120 Enrollment rate 100 80 60 40 20 0 Mali Pakistan Yemen Uganda India 1990 2002 Sources: Case studies and PPARs. Note: Data for India are for the 42 districts covered by the Bank-supported District Primary Education Project, using dates 1991 and 2001. One model was used extensively in India, posting local teachers in A frequent consequence of Indonesia, and other countries with good remote and hardship rapid enrollment results, especially in increasing community areas, it also lowers the interest in the school. It transferred funds to financial incentives for expansion has been a local school committees or councils, which then entering teaching and decline in teacher directly managed the construction or rehabilita- reduces the job security qualifications. tion activities. of those so hired, es- pecially among teachers paid by the community. Much more evaluative research is needed In India critics worry about how this is to show whether and how contract eroding professional standards for teachers and teaching is cost-effective, equitable, and creating a second tier of teachers relegated sustainable in specific settings. In many low- mostly to the poorer communities (Govinda and income countries (India, Mali, Niger, and Josephine 2005). Supporters of the program Pakistan) the rapid provision of new schools and point to research showing the relatively high classrooms has been accompanied by the hiring dedication of contract teachers as measured by of contract (or, in India, para) teachers. These daily attendance (SIEMAT 2005).3 In low-income teachers generally have minimal teacher countries, expansion of access does not training, receive a fraction of the regular teacher necessarily involve the use of contract teachers: salary (around one-sixth to one-half), can be Ghana, Uganda, and the Republic of Yemen—all hired locally (generally on a year-to-year basis), with the help of World Bank and other develop- and are often paid from community funds. ment agency financing—have been able to While this route provides governments a more expand enrollments while at the same time affordable and flexible option for staffing their improving the proportion of teachers who are expanding number of classrooms and a way of fully trained.4 25 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Much expansion of access Much expansion of Social Funds 2000, a set of impact studies access has come conducted by the Bank’s Poverty Reduction and has come through Bank through initiatives Economic Management Network (World Bank units outside the managed by Bank 2000a), showed that social funds projects have Education Sector. units outside the had uneven effects on “welfare” (including Education Sector— educational) outcomes in Bolivia, Honduras, for example, through social funds, public Peru, Nicaragua, and Zambia. In three of these works projects, and PRSCs. These initiatives five countries, social funds programs resulted in bring both benefits and risks. An IEG evaluation no better improvement in student enrollments (IEG 2004d) found social funds projects to be than control programs; in two of three, there “remarkably successful” in supporting was no relative improvement in student infrastructural expansion, especially in the absenteeism. In Bolivia, the only country where building of schools. For example, in the Republic student achievement was a focus, there was no of Yemen, two completed social funds projects better learning in social funds schools than in created places for more than 1 million students control schools. over a nine-year period. But the country’s Basic This lack of attention to learning outcomes Education Project, mounted in the Ministry of also shows up in the PRSCs. Only 4 of the 28 Education during an earlier but overlapping have included them in their objectives or nine-year period, provided for fewer than 50,000 performance indicators, and three of the four (IEG 2005e). During the same period, public were from Uganda. works projects in the Republic of Yemen created Reasons for this low emphasis and uneven space for another 283,000. performance on educational outcomes are often Likewise, Bank-supported social funds rooted in the limited scope of such projects. projects during the mid 1990s established 3,000 Social Funds 2000 suggests that discrete subpro- new classrooms in the Arab Republic of Egypt, jects arise in response to a community’s enough for about 120,000 students, and 4,400 perceived need for infrastructure improvement new classrooms in Cambodia, enough for about “rather than being driven by the objective of 175,000 students (IEG 2002). More recently, achieving a specific development impact,” such PRSCs have frequently emphasized expanded as an increase in basic knowledge and skills. This access: of the 28 such projects mounted during means that crucial complementary investments fiscal 2001–05, almost half had explicit access or “software inputs” (such as staff training and expansion objectives. capacity building) related to development An important potential benefit of social funds impact are often overlooked.5 is community ownership of the program, which The Bank’s Education Sector Unit has means, at least in the case of the Republic of commented on this. In its 2005 retrospective Yemen, that schools built under these programs (World Bank 2005a), it reports that documents are maintained by the community. A benefit of for projects managed by other sectors rarely PRSC expansion is that it is done in the context include significant details about the education of a broad poverty-reduction strategy, including component and its relationship to the country’s improvements in governance and financial educational policies or goals. Frequently the lists management. of professionals preparing the projects do not include education specialists (World Bank Such programs do not A significant risk of 2005a).6 PRSC documents are an example of such programs is that underreporting significant education details: emphasize learning their focus on quanti- only 5 of 28 mentioned anything about low outcomes. tative growth can learning levels in the countries, even though overshadow improve- improved knowledge and skills are the most ments in educational quality and outcomes, important educational factors in poverty including student learning outcomes. reduction.7 26 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D Another approach to enrollment expansion readiness of children The lack of attention to promoted by the Bank in Mali, Niger, and who attended these quality arises from the Uganda has been double-shifting, which in early childhood pro- Africa generally means holding morning and grams (World Bank community’s focus on afternoon shifts in the same school, taught by 2003f). infrastructure. the same teacher. Opposed by teacher unions In the Republic of and many parents, this approach has helped to Yemen and Pakistan, parental reticence to enroll increase enrollments in Mali and Niger. their daughters was successfully addressed in Bank- However, it has also reduced scheduled instruc- supported projects by building schools for girls, tional time by as much as 40 percent, a common recruiting female teachers, and providing scholar- explanation for poor academic performance in ships (in Pakistan). These solutions were not these countries.8 without problems. In the Republic of Yemen, The Bank also supported double-shifting in budget constraints created by structural adjust- the Uganda Sector Adjustment Credit of the late ment led to the recruitment of just over half of the 1990s, but its implementation was resisted until targeted number of female teachers. In Pakistan, 2004, when it began to be piloted in some the scholarship program districts. When it was introduced in Mali and was discontinued when Double-shifting may Niger, it was not piloted, and the trade-offs project funding ended increase enrollment but between this form of expansion and the loss of (and therefore was not usually cuts into instructional time were not mentioned in sustainable). In Niger and instruction time. project risk statements. Uganda, demand con- straints were not high- The Bank has also supported interventions lighted in project planning, because there was so to increase the demand for primary much pent-up demand. education, where this has been a However, pockets of low demand are now constraint to increased enrollments. In appearing, even in places where new schools some locations expansion is constrained by have been built, which suggests the appearance demand features (high opportunity costs associ- of new demand constraints and the need for ated with loss of children’s contribution to new solutions as countries begin to reach out to family income, low perceived benefits of the most disadvantaged. One such solution, education, and constraining cultural patterns for considered promising by many, is the use of girls), and in some cases World Bank support has conditional cash transfers (see box 3.2). addressed these features.9 Of the many efforts in India to increase In some countries parents are increasingly demand for primary education among girls, low- expressing a demand for improved caste children, and tribal populations, the Bank- educational quality by enrolling their supported DPEPs adopted two: public awareness children in private schools, which they campaigns emphasizing the value of primary perceive to have higher standards than education to individuals and communities and public schools. School surveys in Ghana showed lengthening the hours of early childhood an increase in private primary school enrollments education centers so that older girls—generally from about 5 percent of the total in 1988 to more called on to mind younger siblings—could than 20 percent in 2003. In Mali, private school attend a full school day.10 The former was associ- enrollments plus those in community schools— ated with large increases in primary school those sponsored by enrollments during the first two years of project nongovernmental organi- Demand for quality in implementation (and then diminishing returns), zations (NGOs) and education is increasingly the latter with marginal improvements in older community groups— girls’ attendance, plus some improvement—of grew to about 25 percent expressed in private undetermined magnitude—in the school of the total in 2003. school enrollments. 27 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 3.2: Conditional Cash Transfers: A Panacea for Reaching the Poor? Several middle-income countries, most of them in Latin America, countries CCTs have often replaced other, less-efficient social have raised school enrollments and health outcomes among the safety nets, in many low-income countries CCTs would present an poor through conditional cash transfer (CCT) programs that make entirely new safety net program. payments to the poorest households, provided they enroll their chil- To be affordable in the face of much larger need, CCTs in low- dren in school and take them to health care providers for check- income countries likely would have to precisely target a relatively ups. In Mexico, an upper-middle-income country where the primary small group of the “poorest poor.” The targeting mechanisms, enrollment rate was already over 90 percent, the impact of the Pro- monitoring requirements, and administrative structure of these gresaa program on primary enrollment was statistically signifi- programs are complex and generally very demanding in data and cant but small: 0.74–1.07 percentage points for boys and 0.96–1.45 implementation capacity. Particularly in countries where the avail- points for girls, controlling for household and school characteris- ability or quality of schooling may constrain raising enrollments or tics.b CCT programs supported by World Bank projects and linked achievement, cash transfers (a demand-side intervention) may not to primary education outcomes are under way in Brazil, Colombia, be the least-cost way of achieving a particular outcome. Jamaica, and (most recently) Turkey, among others, and will be sub- Thus, the impact of pilot CCT programs in low-income countries jected to impact evaluations (Rawlings and Rubio 2003). needs to be carefully evaluated against alternative strategies for The impact of CCTs on primary enrollments in low-income achieving educational outcomes—be they enrollment, attendance, countries is potentially much greater. In Nicaragua, for example, or learning—to assess the cost effectiveness and sustainability an impact evaluation of the Red de Proteción Social Pilot Project of alternatives. Over the next three years, the Bank’s Human De- found that primary enrollment in the treatment areas rose 22 per- velopment Network will sponsor impact evaluations of CCT pro- centage points higher than in the control areas, starting from a base- grams in six low-income countries (Bangladesh, Burkina Faso, line enrollment rate of 68.5 percent. However, in low-income Cambodia, Lesotho, Nicaragua, and Pakistan), drawing on a grant countries, cost-effectiveness, affordability, implementation ca- from the Bank Netherlands Partnership Program, and cofinanced pacity, and sustainability also loom large. While in middle-income by ongoing lending operations. a. Programa de Educación, Salud y Alimentación. In 2002, this program was renamed Oportunidades and its objectives were broadened. b. Impacts on secondary enrollment and attendance were greater. However, there were no significant positive impacts of Progresa on achievement test scores com- pared to the control groups (Behrman, Sengupta, and Todd 2000). Much of the recent growth in private of such schools and how the governments could education has been in low-cost secular schools. best deal with their proliferation. Such schools have also increased rapidly in number in recent years in countries such as Equity concerns were a prominent feature India and Pakistan, although there are no hard of most primary education projects. Of the data to show by how much (in India such investment projects in the portfolio sample (30), schools are “unrecognized” and are therefore about two-thirds of the completed projects and not included in government statistics). about 80 percent of ongoing project had equity Families, even those in the lower-income features. Likewise, about 80 percent of adjustment brackets, are increasingly turning to such and non-Education Sector projects have equity schools under the assumption, sometimes improvement features. The main target groups for founded and sometimes not, that they lead to these projects have been girls and the poor, but better learning outcomes. In the countries rural and indigenous children also are target visited by IEG for this evaluation there has been groups. The disabled were singled out in only 10 little information or percent of completed investment projects but in Equity efforts focused policy discussion, either 30 percent of the ongoing investment projects. within the Bank or in the more on access than on countries, about the Most investment and adjustment projects learning outcomes. growth and effectiveness with equity features were concerned with 28 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D equity in enrollments. Only about half were Vietnam, in parallel with Equity concerns featured concerned with equity of treatment—eliminating the Bank-financed Pri- prominently in most bias against disadvantaged children at school. mary Education Project Improved equity in learning outcomes was a (PEP) that emphasized projects. concern in only a third of completed investment improved access in projects, one-half of ongoing projects, and fewer underserved areas, gaps in enrollment across than 10 percent of the adjustment and non- consumption quintiles and ethnic groups have Education Sector projects. Fulfillment of the been substantially reduced (figure 3.2). equity objectives in these projects was quite Equity for girls. A more detailed look at the high—around 75 percent, which is not too completed sector investment projects shows surprising, given that they were mostly concerned that only about a third focused on gender issues. with enrollment gains, which have generally been This figure is misleading, however, because in promoted effectively. many countries gender equity had already essentially been reached. In countries with Nevertheless, there were wide differences gender disparities, about two-thirds of projects in the extent to which gaps between the had objectives addressing this. All focused on disadvantaged and more advantaged were girls’ access to primary education; only one closing. In Mali, where project expansion goals (DPEP in India) also focused on closing the have been reached, huge differentials remain gender learning achievement gap. between the capital city and outlying areas. In Most of the projects with gender equity India, enrollment gaps for girls and scheduled objectives (five of seven) satisfactorily met them. castes were largely closed, but not for scheduled However, this does not necessarily mean that tribes (indigenous people). More positively, in gender gaps are closing.11 Figure 3.2: Reducing Enrollment Gaps in Vietnam 100 96.4 95.9 94.2 94.8 92 90.5 87.7 89.2 90.7 90 82.1 80 Percent of children enrolled 70.5 70 61.8 60 50 40 30 20 10 0 Ethnic minorities Kinh and Chinese Poorest quintile Wealthiest quintile 1992–93 1997–98 2003–04 Sources: IEG calculations, using data from the 1992–93, 1997–98, and 2004 Vietnam Living Standards Surveys. Note: Kinh are ethnic Vietnamese. 29 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA In Niger, where Bank policy dialogue and While the relative gap in enrollments financial support during the 1990s emphasized between boys and girls has narrowed somewhat, rapid increase in access to primary education, about half a million fewer girls than boys are particularly for girls, the GER for girls between enrolled; thus a large gender gap persists.12 A 1990 and 2003 increased from 24 to 36 percent. more serious concern is the finding on how few However, boys’ enrollments increased by projects are focused on closing the learning gap similar amounts, leaving the gender gap differential between boys and girls, which would unchanged. involve addressing the biases girls often experi- In Mali, where Bank-supported enrollment ence once they are enrolled. gains were dramatic, the gap between girls’ and Equity for the poor. Among completed invest- boys’ enrollments actually widened. In the ment projects in the portfolio sample, 10 were Republic of Yemen, over the course of two basic found to have objectives targeting improved education projects and a series of public works equity for the poor. Because there are poor in and social funds projects, both male and female both lower- and middle-income countries, all primary enrollments have increased (figure 3.3). projects were eligible for this objective (perhaps The government built new schools and with the exclusion of the two emergency classrooms, deployed female teachers, provided projects). Thus, just over half of the sample materials, and encour- projects focused on improving outcomes for the Few projects focused on aged community involve- poor, a worrisome finding given the Bank’s ment, all to encourage poverty-alleviation mission. Nine of ten of these improved learning higher female enroll- projects aimed to improve access for the poor outcomes for the poor. ments, especially in rural (the 10th was in Mexico, where access was not areas. an issue).13 Four projects included improved Figure 3.3: Yemen: Trends in Primary Enrollments (grades 1–6) by Gender 2,000 1,800 1,600 Thousands of enrollments 1,400 1,200 1,000 800 600 Basic education project Basic education expansion project (effective in 1993) (closing in 2007) 400 200 Public Works and Social Funds Projects (effective 1996; closing in 2009) 0 1995–96 1996–97 1997–98 1998–99 1999–2000 2000–01 2001–02 2002–03 2003–04 2004–05 School year Male Female Sources: Republic of Yemen, Ministry of Education, School Census 1998–2004, and Central Statistical Organization’s Statistical Yearbook, various years. 30 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D learning outcomes for the poor, a higher improvement of com- Recent efforts to improve incidence than with projects aimed at equity for pletion rates among the internal efficiency have girls, but still low.14 In seven of the ten projects, poor, rural residents, objective outcomes were considered as having and girls. not been effective. been fulfilled. Of the four covering learning outcomes, all were considered to have fulfilled Efforts to improve internal efficiency have their objectives, an indication that learning not been very effective. Only 25 percent of outcomes can be improved for the poor, if given sample projects having internal efficiency as an government commitment of resources and objective fulfilled that objective (although 42 effort. percent did so partially; see table 3.2). Uruguay, for example, aspired to reduce its first grade Internal efficiency has been underempha- repetition rate from 21 to 10 percent through sized even in countries with very poor Bank-supported improvements in preschool records, and it is not effectively done education, in-service teacher training, and where implemented. In relatively few of the supervision; however, it only succeeded in countries where IEG did fieldwork for this lowering it to about 17 percent. evaluation did Bank-supported projects attempt India aspired to reduce its primary school to improve internal efficiency, even those dropout rate below 10 percent in 42 districts in countries with very low completion rates. For the DPEP but only did so in the state of Kerala, example, in Niger, where the Project Comple- where the dropout rate was already low. There tion Report (PCR) reported a very low 28 was little discussion in DPEP documents of percent in 2003 (despite a policy of automatic reasons for high dropout rates and no specific promotion to the next grade), there were no strategies for dealing with them. Recent analyses explicit internal efficiency objectives or in India (Azim Premji Foundation 2004) have components in completed projects supported suggested that it is, at least in part, a by the Bank. consequence of low student learning gains, a This was also true for Mali, which reports a finding echoed in Ghana, Mali, Niger, Peru, and PCR of 40 percent (2003) and Uganda, which Uganda. Thus, staying in school improves reported a PCR of 58 percent in 2000. In Mali, learning outcomes, but good learning outcomes teaching children initially in their first language along the way also influence staying in school. contributed significantly to improved internal Policies to automatically promote children to efficiency in a Bank-supported pilot, but the the next grade have also been enacted in countries government has had difficulty bringing its such as India, Niger, and Vietnam as another way bilingual education program to scale. to improve internal efficiency. However, the Data on dropout and repetition rates are efficiencies gained by automatic promotion might rarely reported across family income levels or be undermined by other social groupings (ethnic group and so increasing the numbers The MDGs and the Fast- forth). Where they are reported, substantial of children who complete Track Initiative are disparities are found: in Peru assessments primary school without covering the year 2002 show primary school having learned much. driving attention to completion rates for the extremely poor to be Niger’s automatic-pro- enrollment increases. 54 percent compared with 87 percent for the motion policies lowered non-poor (IEG 2005c). In Vietnam, nationwide repetition rates, but at the end of the cycle some primary school drop-out rates were lower than 3 30–35 percent of students had to be held back percent in 1999, but in the 189 districts where because they could not pass the leaving exam, and 70 percent of disadvantaged people reside, they some dropped out at that point. were 12 percent (World Bank 2005g). Encourag- ingly, in both of these countries current Bank- The joint donor agency EFA FTI has been a supported projects have prioritized the strong force in encouraging rapid 31 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA increases in enrollment and completion Improved Student Learning Outcomes in some low-income countries, but it has not been a force in learning outcomes. The Basic knowledge and skills acquisition Bank’s analytical work at the time the FTI was (learning outcomes)—particularly among launched in 2002 developed a standard for the least advantaged students—is what computing primary school completion (see box enrollments and perseverance in school 3.3) and drew up lists of countries “on-track“ must be about if primary education is to and “off-track” for universal primary completion contribute to poverty reduction. Surpris- by 2015 (Bruns, Mingat, and Rakotamalala ingly, few projects in the Bank’s portfolio had 2003). The 10 best-performing low-income specific objectives to improve learning countries on primary school completion were outcomes, and until recently, few even had selected for creating FTI benchmarks such as learning outcomes among their performance pupil:teacher ratio and primary education indicators. recurrent spending as a percent of total education recurrent spending (see Appendix Among the 6 of 20 completed sector E). Countries that were off track for reaching investment projects that did aim to MDGs were invited to apply for FTI assistance improve learning outcomes, however, 4 and urged to consider the quantitative did so satisfactorily and one partly so (see benchmarks based on the average of the best- table 3.2). Among the four, three were from performing countries in setting their own Latin America (Chile, Mexico, and Uruguay), and indicative framework goals. one was from India. 16 Instead of learning The 2000 EFA goal of ensuring “recognized outcomes, the more general objective of and measurable learning outcomes” for all is not “improving educational quality” appears in a focal point of the FTI. Thus, there are no almost all project designs.17 On that objective, learning achievement goals in FTI and no indica- the sample projects were found to be relatively tors of whether countries are on or off track in ineffective, with 39 percent fulfilling their providing basic learning and skills for all. Except objectives, compared with around 70 percent for the benchmark on for expansion objectives (table 3.2). Basic knowledge and “spending on inputs skills must improve if other than teachers,” Primary education projects approved there are no specific during the past two years also rarely had primary education is to benchmarks related to learning outcomes in their objectives; few contribute to poverty learning. The 2004 FTI emphasized learning outcomes for the reduction. framework document poor. As an extension to the portfolio review, encourages countries to the evaluation team examined objectives in the track student learning measures and the quality 23 projects that were approved during the past of teaching, but so far these have not become two fiscal years (2005–06). As in the portfolio part of the indicative framework.15 sample, about one in five of these projects had Table 3.2: Outcomes by Objective for Completed Primary Education Projects Fulfillment of objective (percent; n = 20) Number Partially Objective covering objective Fulfilled fulfilled Unfulfilled Undetermined Improved learning outcomes 6 67 17 0 17 Improved educational quality 18 39 27 33 0 Sources: IEG 2004d, table 13, and World Bank Project Appraisal Documents and Implementation Completion Reports. 32 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D Box 3.3: Measuring Learning Outcomes What to measure. The 1990 EFA declaration advocated measuring How are results used? Sometimes standard assessments are both “learning tools (such as literacy, oral expression, numer- used in a high- stakes manner to determine the academic stand- acy, and problem solving) and basic learning content (such as ing (and often future educational opportunities) of all students knowledge, skills, values, and attitudes).” The 2000 EFA update and their schools. Increasingly, countries are choosing to conduct prioritizes measurable learning outcomes, in terms of literacy, lower-stakes assessments, which are used to assess national/re- numeracy, and essential life skills. There is no international gional progress and/or to diagnose teaching effectiveness over standard or agreement on what knowledge and skills to meas- time. Such assessments are often given to samples of schools and ure; most countries with testing programs at the primary school students episodically (for example, every second or third year). level cover at least knowledge and skills in language and math- What has the Bank supported? Since its 1990 policy paper, the ematics. This evaluation has not adopted a strict definition of de- Bank has had a strategy of supporting countries in collecting and sired learning outcomes, preferring to use the broad and inclusive reporting student achievement data. The portfolio review for the formulation of basic knowledge and skills. study shows the share of lending projects having assessment How to measure learning outcomes. Countries are increasingly components increasing from less than 20 percent before 1990 to turning to standardized tests, which cover the same items and use near 70 percent during 1990–94 and between 55 and 68 percent the same format across the country. There are two ways of rep- since then. Recent global support for assessment capacity build- resenting the results: norm-referenced (how well test takers per- ing has been provided by the World Bank Institute (the training arm formed relative to others) and criterion-referenced (how well the of the Bank) and through the ongoing Global Student Learning As- test takers performed compared with a standard of excellence, sessment Initiative (funded through the development grant facil- sometimes put as percent “mastery”). In addition to national as- ity managed by the Education Network). The latter aims to help sessments, some developing countries participate in internation- countries develop the capacity to participate in one of the global ally coordinated assessments, such as the TIMSS, the Progress or regional assessments mentioned above. Recent early reading in International Reading Literacy Study, OECD’s Programme for In- assessments in India (Pratham 2006) and Peru (Abadzi 2005) show ternational Student Assessment (PISA), or regional exams, such that meaningful student assessments can be done quite rapidly and as those coordinated by the LLECE and the Southern and Eastern without the extensive institution building required for participation Africa Consortium for Monitoring Educational Quality. in a regional or international assessment. an explicit learning outcomes objective; about and Uruguay (see box 3.4), showing that it is not two-thirds had learning outcomes within their just in middle-income countries where learning performance indicators. Only 5 of the 15 aiming improvement is possible. However, in Honduras to track learning outcomes specifically and Romania, little improvement was noted. In mentioned an aim to track outcomes among the Pakistan, Peru, and Vietnam, assessment systems poorest, which will make it difficult to show have been established with the help of Bank- whether the projects are contributing to poverty financed projects, but they have not produced reduction. Only a quarter of them had baseline results that can be compared over time. learning outcomes data prior to project approval.18 Absolute levels of achievement, even in countries where positive change has occurred, Fieldwork conducted by IEG found that are generally far from satisfactory, despite Bank support can contribute to improved investment in quality improvement. Countries learning outcomes in both low- and middle- often set their own criteria income countries. In only 5 of the 12 countries or levels of test perform- Absolute levels of visited by IEG (Ghana, Honduras, India, Romania, ance that demonstrate achievement are and Uruguay) had there been repeated outcome subject matter mastery measurements using standardized tests. Student and then measure the generally far from achievement improved over time in Ghana, India, proportion of students satisfactory. 33 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 3.4: Improved Student Learning Outcomes in Three Countries In 2003 Ghanaian children completing nine years of basic edu- duced below the targeted 5 percent; some reduction was also ob- cation scored higher on tests of math and English than those com- served for lower castes but relatively little for scheduled tribes. pleting 10 years of basic education 15 years before (IEG 2004a). On Uruguay’s grade 6 assessment, students at all income lev- Improvement was observed for children from households of all els made gains from 1996 to 2002 (project period). However, those income levels, although greater improvement was found in bet- from disadvantaged backgrounds improved significantly more ter-off households. Also, there has been a 23 percent improve- than those from more advantaged backgrounds—18 and 19 per- ment on a standard criterion-referenced test between 1992 and centage point improvements in language and math for the poor 2000 in both English and math. compared with 2 and 6 percent, respectively, among the non- Math and language scores in India improved significantly over poor. National assessment results found a growing share of chil- a six-year period in the 42 districts participating in the DPEP, both dren receiving 60 percent satisfactory or better over the period at the grade 1 level and the penultimate lower primary education year 1996–2002. On the PISA exam, Uruguay’s 15-year-olds scored (grade 3 or 4). Gender disparities in achievement were generally re- above those of other participating Latin American countries. Source: IEG 2004a, 2006a; World Bank 2003g. reaching those levels. In Ghana, where average There are few instances of improved test scores increased over 15 years, fewer than 10 learning outcomes among the disadvan- percent of students have reached the mastery taged, but those that have been demon- level in math and fewer than 5 percent in English. strated show that it is possible to close In India, in 16 of 42 districts’ grade 3 and 4 gaps. For primary education to be a significant students were not performing at the minimum factor in poverty reduction, it is especially level (40 percent correct) in language, and a important that the poor and other underserved recent independent assessment of literacy levels groups acquire basic knowledge and skills. revealed that almost 50 percent of 7- to 10-year- Generally, where average levels of cognitive olds could not read fluently at the first-grade achievement are low, the levels for the level.19 underserved are even lower. India, Peru, Even in Uruguay, where scores on interna- Romania, Vietnam, and other countries have tional standard tests are above regional norms, acknowledged large differences in achievement fewer than half of grade 6 students reached across gender, urban/rural, and social status mastery levels in mathematics. In countries lines. Some countries, however, have been using without trend data, absolute learning levels are Bank assistance to address this problem. Box 3.4 also very low. Mastery in French and math among shows how, in the three countries where Bank- grade 6 students in 1999 in Niger was 13 and 11 financed projects have supported improvement percent, respectively; in the Republic of Yemen, in achievement, learning for the disadvantaged grade 6 students’ mastery of Arabic and math was has improved also, dramatically so in Uruguay 19 and 9 percent, and in Peru it was 8 percent for (see figure 3.4). Also, ongoing projects in India, Spanish and 7 percent for math. In Vietnam, only Peru, Romania, and Vietnam have targeted the 51 percent of grade 5 students were found to rural poor and otherwise disadvantaged and perform as “independent readers.” Mali and have goals to improve learning outcomes Pakistan have no among those groups. Improving learning standard test data, but Reading is considered the foundation of all outcomes among the were observed during IEG fieldwork to have school learning (Alexander, Entwistle, and disadvantaged is very low student learning Olson 2001), yet Bank-supported projects possible. levels.20 have rarely contained specific support for 34 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D Figure 3.4: Percent of Sixth Graders Performing Sufficiently on Language Examinations in Uruguay, by Income Level 100 90 87.7 80 85.4 88.0 75.5 70 70.2 73.9 65.4 60 61.0 61.5 Percent 54.8 58.4 50 51.4 46.7 48.5 40 37.1 30 20 10 0 1996 1999 2002 1996 1999 2002 1996 1999 2002 1996 1999 2002 1996 1999 2002 Very Disadvantaged Average Advantaged Very disadvantaged advantaged Source: IEG 2006a. improved reading skills in early grades. Strategies for improving learning Common problems identified in the case studies outcomes and their effectiveness vary, and PPARs have been the lack of data on early depending on country conditions and primary school reading skills and failure of govern- constraints. The three Bank-supported ments to set and track reading targets for grades 1 countries that have registered solid learning and 2. Box 3.5 reveals major problems with early gains have common explicit national policies acquisition of reading skills in two countries. and strong national commitment to improve Box 3.5: Low Priority for Early Reading Skills: Mali and Peru Poor reading skills in early grades (slow speed and low fluency) wide range of schools, case study teams could find no lower- is hypothesized to be behind much of the poor performance that grade children who could read sentences from their textbooks and appears in achievement tests later on, as well as early dropout only a few in the sixth grade who could. and repetition, particularly among the poor (see Abadzi 2005). In Peru, case study investigators cite a study conducted by the In Mali, poor reading levels were highlighted in the prepara- World Bank and local researchers in 22 low-income area schools tion of an ongoing project (World Bank 2000b) that supported a pro- across various geographic and cultural groupings. First and sec- gram called Reading First. This program was to prioritize reading ond graders were asked to read a simple 60-word paragraph in the first and second grades and was to be assessed against a taken from a grade 1 language textbook and were timed. Only goal of 80 percent of second graders’ reading satisfactorily. In the about 25 percent of first graders and 54 percent of second graders course of project implementation, this program has been de-em- were able to read one word or more, and those who could only read phasized, and four years into the project no appropriate reading at rates of 9 and 29 words a minute on the average, respectively, assessment instrument has been developed. During visits to a well below a modest Latin American standard of 30–60 words. Source: IEG case studies. 35 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA student learning outcomes. However, the affecting a child’s readiness for school (better approaches that they have taken to improve access to preschool education) and extra instruc- learning outcomes have been quite diverse. tional time (the provision of “full-time” schools). Bank support in Ghana was mainly channeled Evaluation results show that with targeted to the construction and rehabilitation of interventions, learning gaps between social buildings and the provision of textbooks; these groups can be reduced. had positive impact on learning outcomes (IEG 2004a). Such basic inputs are still not universal These three countries reveal a kind of in Ghana—they are often lacking in the most sequencing of learning outcomes support: disadvantaged communities, suggesting room from the provision of basic inputs to for improvement. Still more improvement will teacher support and pedagogical renewal be needed, however, if the country is to substan- to targeted interventions for the most tially increase student mastery levels in language disadvantaged.22 Where the needs at one level and math. Future steps will require changes in are not met, it would be unlikely that efforts at what happens inside the classroom—better use the next would bear fruit. For example, Mali has of instructional materials, more time on task, attempted to undertake pedagogical improve- and more effective pedagogy. ment through a bilingual education program, India’s DPEP programs exemplify a Bank but such interventions appear to be support package that covers both basic inputs overwhelmed by the fact that few books and and systems of teacher support and “pedagogi- materials are getting to schools (the cal renewal.”21 Although there has been no student:book ratio is between 2 and 12), and evaluation of the impact of these projects on buildings are severely overcrowded (pupil: student learning, it is likely that some of the teacher ratio is, on average, 67:1) in part because country’s positive test results are attributable to only about half as many buildings as planned better buildings and widespread distribution of were constructed under Bank-financed projects. much-improved textbooks. In Uganda, enrollment expansion was Yet India is starting to reach full provision of dramatic—mostly propelled by the removal of such inputs; further increments in learning are school fees—but was not accompanied by likely to rely more on “what children learn and sufficient expansion of physical facilities and how they learn”(Ayyar and Bashir 2004). In at books, despite efforts by the Bank to ensure that least one state—Kerala—where pedagogical school quality would not drop. By 2005, the renewal has been almost revolutionary, research average number of students per classroom was has found that project schools perform better 94, and 3 students were sharing a single than those in nonproject areas. In most DPEP textbook. Learning outcomes plummeted (IEG locations, however, pedagogical renewal has not 2004c) but recovered partially in 2003 (World yet overcome traditional methods and student Bank 2005i).23, 24 mastery remains limited. In Peru and Romania, fundamental educational inputs were in place, but learning Evaluation results show that with targeted outcomes were still low or flat. In Romania, Bank interventions, learning gaps between social support helped upgrade buildings and provide groups can be reduced. Improved learning better textbooks, but last-minute agreements outcomes in Uruguay had little to do with primary removed in-service teacher training from the school buildings and learning materials (which flagship Educational Reform Project; conse- have been fully available for many years). The quently, little effort was made to improve biggest improvements— teacher performance, and little improvement Targeted intervention can those for children from was observed. This was a major impediment to low-income families— improved learning outcomes. reduce learning gaps came as a result of In Peru, buildings were upgraded, improved between social groups. targeted interventions materials distributed, and substantial in-service 36 I M P R O V I N G A C C E S S A N D L E A R N I N G O U T C O M E S F O R T H E D I S A D VA N T A G E D training provided. Nevertheless, the school ment grants, created special education programs system showed no signs of improved teaching for those with special needs, expanded and learning, allegedly a result of low incentives: preschool education, and screened children for falling teacher salaries (low morale) and almost health difficulties.25 Likewise, Jordan created no supervision or teacher accountability. targeted interventions (with good learning In middle-income countries such as Uruguay, outcomes), based on detailed diagnostic there was little to be gained from improving information gained from achievement test basic inputs or even teachers’ general instruments (World Bank 2005e). knowledge and skills. Closing the learning gap This chapter began with an assertion that required an intervention that went to the roots improved access is a necessary prerequisite for of the learning problems in the target group, learning among the disadvantaged. That does which were poor readiness for school and the not mean, however, that access should be need for extra instructional time. attended to first and learning outcomes later— Other middle-income countries have shown once most children are in school. In the best the value of Bank support in reaching out to cases presented above, increased access and pockets of poverty and underachievement and improved learning were closing performance gaps. In Mexico, a compen- pursued at the same A trade-off between access satory education program supported by the Bank time and were found to and learning can be provided special support to disadvantaged be mutually supportive avoided. schools serving non-Spanish-speaking families (when children enter and made inroads into closing their performance and stay in school, their learning usually gaps (with the mainstream) through a combina- improves, and where good learning outcomes tion of materials development (didactic materials are observed, children and their parents are and textbooks in indigenous languages), updated more attracted to the school). A better example audiovisual technology, professional development is Indonesia, where primary school access and of teachers, and grants to parents and community learning improvement got seriously out of leaders for school improvement programs balance. Before quality retrofitting of its primary selected by the group (World Bank 2004b). schools could be undertaken, increased In Chile, a Bank-supported primary education attention and financing had moved to lower improvement project focused on improved secondary education (World Bank 2003b). learning outcomes (cognitive and affective) At times, however, access can compete with among students in marginalized and rural areas. learning outcomes, especially when expansion It showed cognitive improvements during uses resources intended for quality improve- 1992–96 that were far greater than the national ment or when it proceeds too quickly. Is the average. In addition to training teachers and trade-off between access and learning outcomes providing free textbooks, the project intensified inevitable? Box 3.6 outlines conditions under classroom supervision, offered school improve- which the trade-off can be avoided. 37 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Box 3.6: Trade-Off between Improved Access and Student Learning Gains: Is It Inevitable? Educational planners and managers often face hard choices, tives to support groups for whom expenses represent se- given the reality of severely constrained budgets and compet- rious constraints. ing interests. Expanding access generally enjoys a kind of po- • Adequate provision of essential resources. Expansion with litical mandate, buttressed by national and international quality requires an increase in marginal costs, largely because declarations, and having a win-win quality—both providers and of the extra challenge in reaching the formerly unreached consumers enjoy its tangible and timely benefits. Improving (Roberts 2005). Adequate resources include safe buildings, learning outcomes is also a popular concept but requires a learning materials (in the mother tongue where relevant), much more complex chain of events, leading to hard-won, often trained and motivated teachers, and instructional time (time barely visible and deferred benefits (Grindle 2004). Pursuing the on task). India increased its spending per elementary student popular expansion agenda can often be done in ways that un- from $25 to $44 from 1993 to 2002 (in 2002 prices); Ghana in- dermine the learning improvement one (as when classrooms be- creased the school day from four hours to five. come overcrowded or trained teachers are not provided); • Improvement in education system efficiency. This increases effectively pursuing both at the same time is rare. the availability of resources within any given budget enve- However, the experience of Ghana and India has shown that lope. Ghana reduced the number of pre-university schooling the trade-off is not inevitable. Here are some of the factors that years from 17 to 12; India aspired to reduce dropout below support expanding access and improving learning outcomes at 10 percent (still in progress). the same time: • Accountability for results. Both countries have established school committees with the potential for (if not yet full ef- • Explicit planning for and high political commitment to im- fectiveness in) overseeing use of funds and quality im- proved learning outcomes. This requires strong leadership provement. In Ghana, committees plan improvements based and political skill in building the consensus needed to over- on district achievement data from the previous year, and the come resistance and inertia. frequency of supervised visits to schools has increased. In • Realistic pace of expansion. Big bang approaches to ex- India, village education committees are expected to monitor pansion can overwhelm education systems. Ghana and teacher attendance; in one state annual student test scores India both have expanded gradually, using targeted incen- are reported to the state legislature. 38 Chapter 4: Evaluation Highlights • Efforts to improve management have not been sufficiently founded in institutional-political analysis. • Support for local government and school management of primary education has been more effective than support for central gov- ernment management has been. • Community management increased parental involvement and improved facilities and staffing, but not quality of instruction. • Few country programs directly address teacher recruitment and performance incentives. • Recent projects have given more attention to outcomes evaluation than earlier projects did. • Systems for monitoring, student assessment, and research have rarely been used in decision making. 4 Better Management for Better Outcomes mproved sector management has been considered an essential ingredient I in a government’s attempt to effectively and efficiently turn educational resources into learning gains. In sector strategy papers, improved sector management—governance reform, systemic reform, and decentralization— has been singled out as a priority (World Bank 1990, 1995, 1999). Virtually all primary education projects reviewed planning, policy making, and budgeting for this evaluation aimed to improve sector for primary education, were able to only management or governance, which includes partially fulfill those plans (table 4.1). None objectives such as strengthening management of the projects fully met its targets for such systems, decentralizing planning and decision activities, and nearly half failed to provide making, increasing community control and enough information to evaluate management accountability, and strengthening monitoring performance in these areas, which in itself and evaluation (M&E) systems. indicates poor management. Yet performance has been below expecta- tions in this area: only one in four projects with Management objectives often have been objectives to improve sector management overambitious and not sufficiently ground- fulfilled them, although 58 percent did so ed in institutional-political analysis. For partially. This puts performance almost on par example, in Peru’s Primary Education Quality with the worst performing objective in the Project, school autonomy and accountability and portfolio (improving internal efficiency, see regional decentralization were not implemented chapter 3). It is also consistent with the overall largely because of a lack of consensus and politi- IEG institutional development ratings (see cal will. That could have been anticipated with chapter 2) for primary education projects, which better political analysis during preparation. showed only 25 percent are rated substantial or Moreover, while project better on institutional development impact. planners foresaw a pos- Management sible change in govern- performance objectives Improving Management Performance ment (as revealed in have often been overly appraisal documents), Most projects with designs to improve they did not anticipate ambitious and only central management, in areas such as that 15 ministers of partially achieved. 41 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Table 4.1: Performance on Education Sector Management Activities for Completed Primary Education Projects (n = 20) Performance Number of projects Partially Activities within objective Fulfilled fulfilled Unfulfilled Undetermined Total Central management (planning, policy making, budgeting) 13 0 54 0 46 100 Decentralized management By local government 13 31 38 8 23 100 By schools 12 50 25 8 17 100 Source: IEG 2004d. education would be appointed in 16 years, greatly guidelines and learning materials have been weakening the ability and willingness to widely distributed, but there is no means implement strategic planning, policy making, and of ensuring or even determining compliance other system management efforts. or use. Similarly, the government in Pakistan and the Finally, the frequent use of project implemen- Bank rushed into two large Bank-supported tation units in Bank-supported projects, such as social action projects, each allocating more than those in Mali, Niger, and Peru until 2001, has $100 million to basic education, without a clear created animosity between relatively well-paid understanding of their complex management— unit staff and government education officials especially financial management—require- and little, if any, transfer of skills. ments. A good capacity and political assessment might have revealed not only low technical skills Management improvement efforts are at the provincial and district levels but also a more often evaluated on inputs and culture of patronage, which frequently led to the outputs than on outcomes. Only about half misallocation of funds. (47 percent) of the randomly selected projects having management improvement objectives Weak incentive systems Ineffective manage- evaluated were with respect to management ment performance outcomes (improved planning and reporting, constrain management can often be traced to better budget oversight, and so forth). Most performance. weak incentive sys- tracked only inputs and outputs, like the tems. In Mali, the provision of office equipment and staff training. reward systems of donors have created an The failure to monitor and evaluate manage- environment in which central government ment outcomes weakens the incentive for managers only undertake tasks when the per improved performance. A recent IEG evaluation diem is considered high enough. Routine tasks of capacity building in Africa found that “projects not covered by such incentives are often left almost always achieved their target numbers of undone or are done by consultants, further individuals to be trained, but on the critical undermining management capacity building. question of whether new skills were acquired The case study for Pakistan cites weak and translated into organizational performance, management incentives the record seems weak” (IEG 2005a, p. 32). Management as a major obstacle to improvement is more improving educational Management capacity building in operations drawing on World Bank financ- often evaluated on inputs quality. In Peru, the government has been ing has been fragmented and largely and outputs than on unable to create account- ineffective (Busto, Smith, and Skoelv 2006; IEG outcomes. ability at the school level: 2005a, d, e). Fragmentation comes largely from 42 BETTER MANAGEMENT FOR BETTER OUTCOMES the lack of a coherent strategy and an overem- government staff for Management capacity phasis on individual technical skills. It also their new tasks. building in Africa has results from the uncoordinated and confusing mix of capacity-building efforts across the Decentralization can been fragmented and various development agencies. Making capacity have adverse effects largely ineffective. building into a core activity of donor support on education system within a sectorwide framework is seen as a clear equity. With respect to decentralization and path for bringing more coherence and clout to educational outcomes, the IEG impact study in the undertaking (box 4.1). Ghana revealed how decentralization in that country (involving a greater share of financing by Decentralization districts and communi- ties) has led to disparities The effects of Decentralization of educational manage- in resource availability decentralization on ment is supported by an increasing share between poor and non- of Bank primary education projects, but poor areas. Similarly, case education management the effects of this on educational access study managers for Peru have not been and quality, especially for the disadvan- perceived vast inequali- established. taged, have not been established. Consis- ties in district capacity to tent with its 1999 sector strategy, Bank support effectively manage education under the country’s for decentralized management has increased.1 proposed new decentralization laws. About 80 percent of ongoing projects in the In contrast, Uruguay, perhaps Latin America’s portfolio finance decentralization to local most centralized country, has been very success- government, compared with 60 percent of ful in improving equity of outcomes in its completed projects. An even higher share of education system through targeted interventions ongoing projects—90 percent—finances to poor communities. Examples like this call for a school-level management, compared with 60 more nuanced and more evidence-based percent of completed projects. education sector position on decentralization, The Bank supported decentralization efforts especially with respect to the disadvantaged. in most field-based study countries, often with good results (Honduras and India), but in some Bank support for local government and cases (Romania and Pakistan) there was ambigu- school management of primary education ity in what the different levels covered, nonalign- was more effective ment of administrative and financial features of than for central Decentralization can decentralization, and undertraining of local government manage- have adverse effects. Box 4.1: Toward More Coherent Capacity Building in Africa The IEG evaluation of capacity building in Africa identified four sets outcomes objectives, and coordinates efforts across pro- key elements to ensure coherence of management capacity gram components and related public sector reforms building interventions, as follows: • An implementation process that arranges in the right se- quence measures to strengthen relevant institutional, orga- • Capacity needs assessments conducted with stakeholder nizational, and human resource capacity participation • M&E processes that assess progress and suggest necessary • A management structure that aligns the public sector im- course corrections. provements being sought with country development goals, Source: IEG 2005a, p. 33. 43 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Support for local and ment, but leaves which is designed to put accountability into the room for improve- hands of the clients of education, is to empower community management ment. Planned activities communities to manage their own schools (see of primary education has for strengthening local World Bank 2004h).3 Increased community control been more effective than government manage- was a feature of Bank support in several field-study ment of primary educa- countries: Honduras, India, Mali, Pakistan, support for central tion were fulfilled in Romania, and the Republic of Yemen. Honduras’ management. about a third of Community Education Program (PROHECO), completed projects; for piloted in a completed project and brought to school-level management activities that number scale in the ongoing Community-Based Education was about one-half (table 4.1). The field-based Project, fully exemplifies this approach in that it studies also showed mixed results for decentral- empowers community education associations to ization. open schools, hire and pay teachers, support In India, the large network of the DPEPs put school improvements (for example, through school improvement planning and implementa- parent and teacher training programs), and tion in the hands of village committees and maintain school buildings. district officials. Many communities and districts Other countries, with Bank support, have seized this opportunity to mobilize the energy also created school committees that have and creativity of their members for improved exercised many of these functions. Evaluations access and learning outcomes. However, have revealed that school committees do get set because DPEP was in a minority of the districts in up and often effectively manage school most of the states where it was implemented and construction, rehabilitation, maintenance, and because it is managed through a special network the hiring of teachers (World Bank 2003g; IEG outside the bureaucracy, there is some doubt as 2004a; Durston 1999). An evaluation of El to whether the innovations will take root in the Salvador’s Bank-supported Community Man- mainstream (Ayyar and Bashir 2004). aged School Program (EDUCO), in which In both Pakistan and Romania, the Bank schools are managed autonomously by supported a policy of decentralization, community education associations, found that underpinned by analytical work, but the countries student absences were lower in community- scaled up the reforms before their models were managed schools compared with traditional fully developed. Consequently, there is much public schools, after controlling for student, ambiguity over what level of government is school, and participation characteristics responsible for different functions and how (Jimenez and Sawada 1999).4 functions transferred to local governments will be The evidence to date about the effectiveness of financed.2 Partly because of such ambiguities, community management in improving the quality technical training of local government officials in of instruction and student learning outcomes is need of management skills was not effective thin (Roberts-Schweitzer, Markov, and Tretyakov (Pakistan) or even covered (Romania). 2002). Student test scores have improved under PROHECO in Honduras, but this was more a Community Control and Accountability reflection of improved teacher attendance, smaller classes, and fewer school closings than of Empoweringcommunities to manage improved instruction—for example, use of educational funds has multigrade teaching (Vegas 2005). In the case of Community management increased parental in- EDUCO, learning achievement on standardized has brought volvement in schools tests of children in community-managed schools improvements, though and brought improve- was no different from those of children in ments in facilities and traditional public schools, when child, school, and not for quality of staffing. The ultimate participation characteristics were controlled for instruction. form of decentralization, (Jimenez and Sawada 1999).5 44 BETTER MANAGEMENT FOR BETTER OUTCOMES In Ghana and India, school committees are have, there were Operating through primarily engaged in annual planning meetings sometimes perverse and infrastructural improvements (IEG 2004a; consequences or un- traditional World Bank 2003g). Because so many countries sustainable results. accountability structures have bought into this model of community The Bank supported has been associated with empowerment, it is important that it be fully and teacher selection and carefully evaluated with respect to its impact on performance incentives improved learning learning. in two of four case study outcomes. countries. In Mali, the Strengthening traditional accountability Bank’s policy dialogue advocated an increase in mechanisms operating through head standards for primary school teacher candidates teachers and school supervisors has also (secondary school completion) and the use of been a focus of Bank support and has been contract teachers. However, these measures associated with improved learning reduced the availability and use of trained outcomes. In Ghana the IEG impact study teachers, because few high school graduates were showed instructional leadership by head interested in teaching primary school. teachers and supervisors to affect improved In Peru, Bank-funded analytic work put heavy teaching methods. That in turn affected learning emphasis on meritocratic methods for hiring gains. Support to Ghana from other develop- and rewarding teachers. The government ment partners helped to increase supervision adopted teacher tests to select new teachers and visits from about five to between six and nine to reconfirm the appointments of acting ones, times a year over the 1988–2003 period. but the system was only used once and has not Nevertheless, support was uneven, given that become routine. Also, it has successfully piloted only about 44 percent of teachers reported the use of locally applied incentives to encour- direct contact with a supervisor in 2003 (IEG age improved teacher attendance. A clear gap is 2004a). the fact that none of the meritocratic methods In Uruguay and Chile, support for intensified included any rewards for improving student supervision (at least one visit per month) has learning outcomes.7 been associated with relatively high achieve- ment gains in project schools. In Chile, supervi- A number of strategies have been sion was reoriented from a focus on inspection implemented to attract teachers to rural to pedagogical improvement. Principals were and underserved areas, with varying also retrained in developing school-based degrees of success. In most countries, serious improvement projects that engaged teachers in teacher shortages in marginalized areas have efforts to improve teaching and learning (World undermined efforts at improving learning Bank 2001a). among the disadvan- In Peru, where learning outcomes are below taged. For example, Few country programs expectations, feedback and accountability Uganda has an average directly address teacher mechanisms are weak or nonexistent. In India, pupil:teacher ratio of enrollment growth has outpaced the expansion 55:1, but in one selection and of supervision systems, which, since they were relatively poor district it performance incentives. integrated with the DPEP, have fallen into is 90:1. In Ghana, the relative neglect in some locations.6 proportion of schools with pupil:teacher ratios above 50:1 was far above the national average in Teacher Incentives the disadvantaged northern region (54 percent). To attract teachers to hardship areas, the Few Bank-supported country programs have Bank has supported the construction of teacher directly addressed teacher selection and houses, cash incentives for teaching in rural performance incentives, and where they areas, and local recruitment of teachers, 45 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA especially females. In Uganda, teacher housing an M&E system during project preparation; (ii) was added to a Bank-financed project to attract the absence of M&E specialists on the Bank’s and retain teachers in rural schools, raising project appraisal teams; (iii) the selection of teacher morale (IEG 2004c). Financial in- inappropriate indicators to evaluate impact and centives were offered to attract teachers to rural measure outcomes; and (iv) a general failure to areas in a successful pilot project in Peru. In the convince the borrower of the usefulness of these Republic of Yemen, teachers rejected housing systems as a management tool, and of their signif- in favor of a financial incentive for rural school icance for governance and the development of a service that they could use to cover expenses learning culture. wherever they chose to live. However, new professional requirements for Primary education projects, especially teachers will make it harder to attract teachers those initiated during the late 1980s to to rural areas. Now teacher candidates must be mid-1990s, were even less focused on secondary school graduates, preferably female, outcomes and impact evaluation; more with tertiary-level teacher training. Because few recent projects have been giving results rural women can meet these standards, most more attention. Among the 20 completed new teachers are urban women for whom a Education Sector investment projects examined modest rural service bonus will be insufficient to in depth for this study, only 44 percent had plans induce them to move or even commute to a to evaluate improved quality of schooling using poor rural village. outcome measures, whereas nearly all (94 For such reasons, countries like Honduras percent) planned to monitor outputs (IEG have supported community management of 2004d, table 8). schools and hiring of local (but often less- Only one in 10 proposed to compare findings qualified) teachers. The program has enlisted with a control or comparison group, or had teachers for remote schools, but those teachers some provision for assessing the counterfactual. have expressed apprehensions about job Comparable figures for outcome/impact evalua- security and access to retirement and profes- tion of sector management objectives were 26 sional development benefits (World Bank and 11 percent.11 More recent projects are 2001b). Such threats to keeping teachers in better on both counts: 80 percent of active difficult assignments have not been effectively projects have plans to measure quality improve- addressed in many locations.8 ment features through outcome indicators and about 30 percent are designed to do impact Monitoring, Evaluation, and Research evaluation; the comparable figures for improved sector management are 80 and 20 percent.12 The evaluation designs of many Bank- While the design features of the new projects do supported education projects have not point to more borrower acceptance of results been outcome or results oriented.9 A recent evaluation as a management tool, optimism assessment by the Bank’s Quality Assurance should be tempered by the fact that, in the past, Group (QAG) found that only 68 percent of all only two-thirds of projects with planned education projects designed in fiscal 1999–2002 outcome evaluations carried them out (IEG had satisfactory designs 2004c, box 4).13 Evaluations have for outcome or impact typically not focused on evaluations, despite the The Bank has supported capacity building outcomes or results, fact that overall “quality in M&E in most countries with primary at entry” ratings were education projects, many including the although this is changing. satisfactory for more than development of national learning assess- 85 percent of them.10 ments. In India, Pakistan, and Uganda, it The QAG assessment attributed continuing supported the establishment of Educational weaknesses in M&E to: (i) the failure to establish Management Information Systems (EMIS). In 46 BETTER MANAGEMENT FOR BETTER OUTCOMES Pakistan, Peru, Romania, and Vietnam, it In Romania, capacity Bank-supported M&E supported systems of assessing student was not built with regard systems have rarely been learning. In India it supported the conduct of to systems for using data educational research. from any source in policy used in decision making. Assessment systems are up and running in analysis or strategic plan- Peru and Romania and are considered among the ning, and little Bank support had been provided most enduring and effective features of Bank- for cross-sectoral analysis and planning. In Peru, assisted projects. The large increase in research the failure of the Quality Measurement Unit to output in India suggests that a “culture of influence educational decision making resulted research” is taking root, spurred by the DPEPs. in part from lack of staff training. In India, the blossoming research productivity has often been Where functioning M&E systems have been irrelevant to policy and system improvement, established, however, rarely have results research topics having been chosen in an been used in decision making. The useful- unsystematic manner without policy develop- ness of M&E products has been undermined by ment goals in mind (World Bank 2003g). poor quality and lack of attention by policy In a more positive vein, Vietnam’s first (and, makers. In the countries where EMIS have been to date, only) national reading and mathematics developed there are widespread and lingering assessment, financially and technically problems with data quality: in Uganda school supported by the Bank, led to a significant shift data frequently cannot be verified; in Pakistan in the ministry’s focus toward learning there are discrepancies in simple enrollment outcomes. The results stimulated a public data between EMIS and household surveys. debate on the quality of schooling and Neither Romania nor Peru has effectively used prompted new government initiatives to student test results for planning purposes and in improve learning outcomes among ethnic policy making. minorities and the poor. 47 5 Conclusions and Recommendations Conclusions here has been an extraordinary mobilization of lending and non-lend- T ing resources in support of the Bank’s primary education policies over the past 15 years. Lending for primary education has become increas- ingly progressive with respect to the poorest countries and has often been directed toward the most disadvantaged within countries. In most parts of the world, these investments have • In the few places where knowledge and skills significantly improved access to primary acquisition have been measured over time, education through the construction of new positive changes have been observed. However, schools and the reduction of other physical, often absolute or mastery levels in basic sub- financial, and social barriers. Beyond achieving jects are still low, particularly among the dis- universal completion of primary education, which advantaged. is one of the MDGs, the remaining EFA challenge • Failure to provide reading skills in the early pri- is to ensure that all children, particularly the mary school years—among the disadvantaged disadvantaged, acquire the basic knowledge and and advantaged alike—is often at the root of skills that are crucial for poverty reduction. weak learning outcomes. • Raising primary school completion rates—the While the Bank has effectively helped main MDG and FTI objective—is an important countries improve enrollments even among part of the story. However, where this agenda has underserved groups, it has been less effective been promoted at the expense of good learning in helping them reduce school dropout rates opportunities, large numbers of children, es- and increase learning outcomes, especially pecially those from disadvantaged backgrounds, among the disadvantaged. have been completing school without having gained the knowledge and skills they need. • Few Bank programs, although they are in- • A trade-off between improved access and stu- creasing in number, have been designed ex- dent learning gains can be avoided, but only plicitly to improve student learning outcomes with careful strategizing. This should begin for the disadvantaged (even among those fo- with explicit planning for and strong political cusing on improving the quality of schooling). commitment to improved learning outcomes. 49 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA The optimal strategy to improve learning • The Bank’s Country Assistance Strategies have outcomes for the disadvantaged in any not consistently included educational out- given country will depend on local comes (including learning outcomes) in their conditions and institutions. Those countries performance indicators. The 2005 Education where Bank support has led to improved Sector Strategy Update has called for this to be learning for the disadvantaged have taken quite institutionalized in all regions. different routes, consistent with their conditions • Sector management capacity, a common weak and constraints. What has had the largest impact point in Bank-supported primary education in Ghana may not have the same impact in projects, might have been better dealt with Uganda; what is effective in Honduras may not had there been better-organizational capacity be so in Peru. assessments at the outset (some countries were The Bank has acknowledged the importance clearly overstretched by their Bank-supported of countries’ creating their own solutions, based project agendas) and better capacity-building on their unique mix of problems, financial, programs. In addition, many country programs cultural and political resources, and constraints. supported by the Bank failed to pre-assess the Some of the lessons learned in this evaluation strength of political forces acting both in sup- related to this approach are as follows: port of and against the change agenda. Instead, there was a widespread presumption that de- • Many countries still do not generate the kind cisions are made on the basis of rational plan- of information they need to design solutions ning and technical merits. Politically motivated to low learning outcomes among the disad- threats to project implementation and success vantaged. They need data on participation were rarely even mentioned in risk assessments rates, delivery of services and quality inputs, and of the many projects reviewed here; thus, no learning outcomes by gender and by socio- mitigation strategies were formulated. economic status that can be tracked over time. • There has not been adequate experimentation Of the relatively few countries that aim to im- with local solutions through the evaluation of prove learning outcomes, many begin with- pilot projects, with respect to their impact on out any baseline data. learning outcomes. Adopting international good • Where the distribution of learning outcomes practice norms without sufficient experimen- across gender and family income levels has tation can lead to unanticipated results that been measured, specific interventions for the can undermine program effectiveness. disadvantaged can be designed. Unfortunately, many countries have not used student learn- Relatively ineffective government efforts ing data—even when available—in a diagnos- to improve sector management and tic manner or for program planning and governance have been a weak link in the improvement. results chain from Bank support to • Economic and sector analysis often is out of learning outcomes. date or not sufficiently comprehensive to un- derstand the factors behind low participation, • Many primary education projects had objectives persistence, and learning among the disad- to improve sector management, but few of vantaged. These reasons will change the closer them fulfilled their objectives. The most seri- a country comes to universal enrollment: the ous lapses were on improvements at the cen- reasons for nonenrollment and/or low learn- tral level, but results at the local government ing outcomes among the last 20 percent will level were also quite weak. Most projects ex- be very different from those for the other four amined evaluated their results using input or quintiles. Bank-supported projects and pro- output criteria (number of staff members grams have often been unengaged or unsuc- trained) as opposed to outcomes (changes in cessful in building local capacity for the conduct management behavior). This echoes the IEG of such sector assessments. evaluation of capacity building in Africa. 50 C O N C L U S I O N S A N D R E C O M M E N D AT I O N S • The Bank supported decentralization efforts in based projects, mostly covering equity im- most of 12 IEG field study countries, often provement or enrollment increases (learning with good results (Honduras and India). Many outcomes and sector management were not cases, however, saw several points for im- present). Their project activities were pre- provement: ambiguity in what the different dominantly school construction and rehabili- levels covered, nonalignment of administra- tation. It was not clear the extent to which tive and financial features of decentralization, governments were committed to covering the and undertraining of local government staff qualitative inputs and recurrent costs (teach- for their new tasks. Also, there are serious ers, books) for this expansion in infrastruc- questions about the effects of decentralization ture. In the Republic of Yemen major Bank on the educational opportunities and out- support for infrastructure coming through comes for the disadvantaged. public works and social funds projects was ac- • In most of the field study countries, weak man- companied by the adequate provision of agement incentives at all levels were con- trained teachers and textbooks. In other coun- straints, especially to the improvement of tries, however, there was often weak interac- educational quality. There were more rewards tion between the non-Education Sector for increasing the number of schools than for programs and Education Sector programs and redistributing teachers, implementing a new recurrent resources. curriculum, doing effective M&E, or even im- • The direct involvement or influence of the Ed- proving test score results. ucation Sector on these education compo- • Few Bank-supported country programs di- nents does not appear to be strong (see chapter rectly addressed teacher recruitment and per- 3). The Education Sector Strategy Update of formance incentives. Performance incentives 2005 shows a determination to increase the related to student learning outcomes are par- focus on learning outcomes, but it is not clear ticularly lacking. Among the many strategies to how well this shift in priorities will be com- attract teachers to disadvantaged areas, the municated to supporters of education in the hiring of locals shows the most promise, as long other sectors. as those teachers have access to professional • The strong presence of basic education in growth opportunities and job security. growing numbers of PRSCs is a good sign that basic education is viewed as a poverty-reduc- Increasingly, Bank support for primary tion instrument, but an examination of recent education is coming through components PRSCs shows few that are focused on the main in projects originating in other sectors. education driver of poverty reduction, namely, This phenomenon has boosted the resources knowledge and skills acquisition. Equally few available to primary education and exemplifies a mention low learning outcomes as a contrib- growing Bank preference for cross-sectoral (or utor to poverty. A careful evaluation of this comprehensive) development approaches. new instrument is needed to determine the ex- Moreover, in social funds and community-driven tent to which the basic education features of development, support for primary education is such credits are based on a good analysis of driven by community demand. However, as so what is needed to improve educational im- many of these efforts are recent, few have been pact and whether such features help set the evaluated, so their effects on educational stage for better learning outcomes. Also need- outcomes are still not well understood. Some ing assessment are the adequacy of supervision preliminary findings suggest the need for of quality improvement features of the PRSC caution. and whether educational outcomes are specif- ically and effectively evaluated. • The portfolio review of 10 projects managed outside the Education Sector showed those The Bank’s 1990 primary education paper projects to have fewer objectives than sector- called for adequate funding for “good- 51 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA quality primary education to become ■ The Bank’s primary education assistance— widely and equitably available.” Although whether sponsored by the education sector Bank and country funding for primary or other sectors—needs to focus on the education has increased sharply since factors most likely to affect learning out- then, the question still remains whether it comes in a given country’s context. This has been adequate and sustainable. will require more analysis of student learn- ing and its local constraints and facilitators. • Reaching the previously underserved will re- ■ The Bank and governments need to rec- quire marginal increases in unit costs; also ognize that reaching children not yet en- reaching beyond school attendance to improve rolled and improving low achievement levels learning outcomes will mean increased costs. will raise the unit costs of primary education. • So far, estimates of funding gaps for universal primary education (see Bruns, Mingat, and • Efforts are urgently needed to improve Rakotamalala 2003) are based on steps needed the performance of sector management to ensure universal completion. A more ap- in support of learning outcomes. This im- propriate approach, given the learning out- plies that: comes “imperative” [UNESCO’s term], would ■ Programs to improve sector management factor in the costs of providing acceptable learn- and governance need to be based on sound ing levels for all. This will clearly result in in- political and institutional analyses that take creased funding gap estimates, but the message into account the incentives officials and is a timely one, given renewed interest in EFA teachers face to improve the quality of in- by G-8 countries (G-8 Group 2006). struction and learning outcomes. Account- • The Bank’s 2005 Education Sector Strategy ability and supervision systems need to be Update signals an intention to maintain mo- adapted to support improved learning out- mentum for EFA, including a greater focus on comes. learning outcomes, while at the same time ■ Primary education managers need to: (a) broadening its mandate to support “educa- track learning outcomes over time—not tion for knowledge economies,” without any just the average, but among different in- net additions to total Bank education sector come and social groups; (b) monitor indi- staffing. It is difficult to understand how this vidual staff and system performance more demanding EFA agenda can be pursued indicators, for both centralized and decen- with diminished staff resources. tralized activities; and (c) create and use incentives to encourage staff to improve Recommendations and use their technical skills. All new Coun- try Assistance Strategies should include • Primary education efforts need to focus learning outcomes indicators. on improving learning outcomes, par- ■ Analytic, assessment, and research activi- ticularly among the poor and other dis- ties need to be oriented to informing key advantaged children. The MDG push for management and policy issues, with incen- universal primary completion, while a valu- tives to ensure that the findings are used in able intermediate goal, will not ensure that decision making. One such research prior- children achieve the basic literacy and nu- ity would be to assess the impact of decen- meracy that are essential to poverty reduc- tralized management on inequalities across tion. This means that: income and social groups and to identify ■ Improving learning outcomes needs to be a mitigation measures of any adverse effects. core objective of all support for primary ed- ucation, with a particular focus on achieving • The Bank needs to work with its devel- equity in learning outcomes by gender and opment partners to reorient the Fast- among the poor or otherwise disadvantaged. Track Initiative to support improved 52 C O N C L U S I O N S A N D R E C O M M E N D AT I O N S learning outcomes, in parallel with the ■ Help countries, financially and technically, MDG’s emphasis on primary completion. to set up suitable systems for conducting This will require that the Bank and its partners repeated learning assessments that are do the following: capable of tracking outcomes separately ■ Reframe the goals and objectives of the FTI for disadvantaged groups, including the to include improved learning outcomes for poor. all, in addition to school completion for all. ■ Revise cost and funding gap estimates to (a) ■ Require learning achievement indicators reflect the costs of achieving basic learning and targets in country FTI proposals and add outcomes (not simply primary completion) items to the indicative framework that are and (b) take into account the increased unit directly related to learning outcomes, such costs of expanding access to and improving as instructional time, teacher attendance, or learning outcomes among children from availability of textbooks. disadvantaged backgrounds. 53 APPENDIXES APPENDIX A: WORLD BANK POLICY OBJECTIVES AND STRATEGIES FOR PRIMARY EDUCATION Document Policy objectives Sector strategies Priority recipients Education: Sector Provide basic education to Maximize internal efficiencies in education. Strategy Paper all youth and adults Improve institutional capacity. (1980) (irrespective of sex, ethnicity, and socioeconomic status). Relate education to work and environment. Primary Education: Expand primary school Organize Bank funding support for long-term institutional A World Bank completion. development. Policy Paper (1990) Expand girls’ access to Support countries in: Countries having education. • Efficiently allocating additional resources reform agendas Improve children’s • Enacting aggressive measures for girls’ enrollment (nearby learning. schools, toilet facilities, female teachers, cash incentives, and so forth) • Regularly collecting/reporting student achievement data • Reforming initial/in-service teacher training and support • Strengthening educational management • Strengthening preschool education • Integrating nutrition and health into primary/preschool programs. Priorities and Improve educational Allocate greater share of lending to basic education. The poor and Strategies outcomes. Encourage education reform, based on economic analysis. underserved for Education (1995) Increase equity for poor, Encourage household involvement in education. within countries females, other Promote school autonomy. disadvantaged. Increase both supply of and demand for basic education. Encourage free basic education; some cost recovery for other levels. Education Sector Support EFA goals, Support countries in: The poorest coun- Strategy (1999) especially for girls. • Increasing early interventions (early child development and tries with lowest Improve quality of school health) enrollment rates teaching and • Increasing innovative delivery (distance education, open learning. learning, information technology) • Strengthening systemic reform (standards, curriculum, and assessment of governance/decentralization) • Apply Bank operating principles (focus on client, compre- hensive analysis and selective action, focus on develop- ment impact, good use of knowledge, productive partnerships). Education Sector Maintain momentum on Support countries through: Low-income Strategy Update EFA and MDGs. • Collaborations with other donors, including through the countries with (2005) Strengthen education for Fast-Track Initiative PRSPs and educa- the knowledge economy. • Increased focus on results tion plans • Systemwide approaches, including analysis of education sector as well as macroeconomic context. Note: EFA = Education for All; MDGs = Millennium Development Goals; PRSPs = Poverty Reduction Strategy Papers. 57 APPENDIX B: STUDY METHODS Conceptual Framework Thematic reviews covered the following topics: The framework depicts the main explicit channels (that is, mentioned in project • Rationale for public investment in primary ed- documents) through which Bank assistance ucation in developing countries (see Boissiere (analytic work, policy dialogue, and lending) 2004b) can influence educational outcomes—for • Determinants of learning outcomes in devel- example, through government policy and oping countries, including both supply- and de- institutional capacity, specific features of the mand-side factors (see Boissiere 2004a) educational system (facilities, books, teacher • External assistance to primary education in status and quality, teaching-learning processes, developing countries. and education system management), increased demand for education, and educational Archival reviews included the following: outputs (school completion), plus other enabling or moderating factors such as • Education Sector documents (sector policy improved donor cooperation, civil service and strategy documents, Project Appraisal and reform, support for complementary sectors Completion Reports, education sector retro- and subsectors, labor force (employment) spectives [annual reports], research and pol- services, and actions in the private sector and icy dialogue reports) by civil society. • Independent Evaluation Group (IEG) evalua- The framework specifies educational tion reports on other education subsectors outcomes to be both learning outcomes (basic (adult literacy; secondary education); social knowledge and skills acquisition) and employ- funds; health, nutrition, and population; ment/welfare outcomes; however, given the HIV/AIDS; community-driven/ community- scarcity of data, there was little coverage of the based development; capacity building in Africa, latter. This framework presents a “results chain” and Annual Reviews of Development Effec- through which Bank support is assumed to tiveness (2002–04). operate. As there are other factors influence educational outcomes (for example, family Portfolio Review status and home environment), this cannot be Trends in Bank financial support (International considered a fully elaborated model. Development Association [IDA] and Interna- tional Bank for Reconstruction and Develop- Evaluation Methods ment [IBRD] lending) for primary education were examined though a desk review of lending Literature Reviews documents and reports. The portfolio review As orientation and background material, the included an analysis of the full portfolio of study undertook reviews of literature, both projects with any funding for primary education thematic (covering relevant topics) and archival approved between fiscal 1963 and fiscal 2004, as (relevant documents). well as an in-depth study using a sample of 59 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Figure B.1: Channels of Bank Influence on Primary Education Outcomes Other Donors World Bank – Policy positions – Analytic work – Analytical work Government Policy – Policy dialogue – Magnitude/areas of and Capacity – Technical assistance financial support – Policies and EFA plans and/or capacity building (including curriculum – Lending (by expendi- reform, cost recovery, ture category)a decentralization, and the – Conditionalities like) – Public spending for education (particularly primary) – Institutional capacity Private Education Education Delivery Other Public Services and Civil Society – Coverage, by gender and – Civil service/CS reform – Coverage, costs, and socioeconomic status – Health and early child- quality of private – School quality (facilities, hood development education books, teacher status/ – Adult literacy – Civil society involvement moreale/quality, TLPb) – Secondary education in programs and policies – Internal efficiency (access and focus) – Education system – Higher education management (including (teacher training) M&E, community – Transportation involvement) – Labor force services Household Demand for Education (Subject to constraints of income, physical and human endowments, and economic environment) – Parental education – Educational expenditures – Decision to enroll/keep children in school (boys and girls) Outputs and Outcomesc – Access (enrollment and completion) – Educational outcomes (basic knowledge and skills acquisition, employ- ment and welfare outcomes) a. Including past, present, pipeline, and dropped. b. Teaching Learning Processes, including “time on task” and language of instruction. c. By gender and socioeconomic status. 60 APPENDIX B: STUDY METHODS projects from the full portfolio (called the The in-depth study reviewed all available “portfolio sample”). project documents for 50 projects to examine For Education Sector projects, data were in more detail: taken from one internal database. For projects managed by other sectors, data were obtained • Development objectives from a different database. All projects approved • Project activities from fiscal 1963 to fiscal 2004 having any alloca- • Monitoring arrangements tion to primary education were included.1 For • Effectiveness of completed primary education the in-depth study, a stratified random sample of projects, including both IEG ratings and the 50 primary education projects was taken from evaluation team’s assessment of how well proj- the full portfolio of education projects. The ects performed on their objectives.2 sample was stratified by managing sector, loan type, and portfolio status (active or closed) to Field-Based Studies represent the composition of the full portfolio. The primary education evaluation drew on field- based studies in 12 countries, either enhanced The full portfolio review used existing data on performance assessments (PPARs) of recent education projects to analyze the following: primary education projects in a country context or case, an impact study (Ghana), or case studies • Magnitude and focus of IDA and IBRD lending undertaken for this evaluation. The evaluation for primary education, especially since 1990, in sought to include examples from each of the nominal and real terms, through both invest- four categories in the matrix of educational ment and adjustment instruments and man- performance vis-à-vis income level (see table aged by different sectors (education and other) B.1, below). • Regional distribution of lending coverage of low-income countries Country Case Studies • Evolution of lending project objectives, in- Four country case studies were undertaken specif- cluding how well they addressed Education ically as part of the primary education evaluation, Sector policy objectives focusing on the development effectiveness of the • General IEG performance ratings of primary ed- Bank’s assistance to primary education at the ucation projects. country level, where Bank support to primary Table B.1: Selection of Countries for Country-Level Analysis Learning outcomes improvement Income level Poor Good Low income Mali Ghana Pakistan India Uganda Vietnama Niger Yemen, Republic of Middle or lower-middle income Peru Romaniab Honduras Uruguay Note: Countries in italics are subjects of enhanced PPARs on one or more primary education project; those in bold are case study or impact study countries. a. Vietnam has yet to report change scores on its standardized testing of fifth graders, but overall 2001 achievement levels in math and reading were considered to be quite high, espe- cially in math (World Bank 2004g). b. Romania was originally assessed to have positive learning outcomes based on the high (and recently increased) pass rate on its eighth-grade examinations; however, subsequent analy- sis of international assessment results have revealed a flat (non-increasing) pattern of learning outcomes. 61 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA education has been significant and longstanding. about 70 a year) through Project Performance The case study countries—Mali, Pakistan, Peru, Assessment Reports (PPARs). Projects are and Romania—were purposively selected to selected for evaluation through PPARs based on represent a variety of regions, income levels, and several criteria, including good potential for educational performance levels. learning (because of particularly good or bad The case study methodology included performance) and relevance to upcoming IEG interviews with Bank and local managers, benefici- sector or thematic evaluations. PPARs, which aries, donors and international agencies, and other normally include a field mission, rate projects in stakeholders. In addition, case studies collected terms of their outcome (taking into account official data on schools and learning outcomes, as relevance, efficacy, and efficiency), sustainabil- well as primary data in a small sample of schools in ity of results, and institutional development each country. Case studies reviewed the effective- impact. PPARs are similar to the completion ness of Bank-sponsored policy dialogue, analytic evaluations carried out by many development work, and lending, in the context of each country’s agencies and are the main project-level evalua- changing political economy. tions conducted by IEG. They are products in themselves but may also be intermediate inputs Impact Evaluation for other evaluations, such as this primary IEG completed an impact evaluation of World education evaluation. Bank support to basic education in Ghana In this case, primary education projects in during the primary education evaluation. The seven countries were to be evaluated by PPARs Ghana evaluation was based on a longitudinal during the period that this primary education study of achievement data from a household evaluation was under way: Honduras, India, survey (1988 and 2001) and school surveys for Niger, the Republic of Yemen, Uganda, Uruguay, both years in 85 districts. and Vietnam. For these, task managers were asked to address specific questions laid out in Project Performance Assessment Reports the Terms of Reference for the primary IEG assesses one in four completed projects (or education evaluation. 62 APPENDIX C: PROJECTS IN THE PORTFOLIO REVIEW SAMPLE Country Fiscal year Project ID Project name Education Sector investment projects having at least 50% primary: Closed since 1995 (n = 20) Bangladesh 1990 P009514 General Education Project Bosnia and Herzegovina 1998 P045313 Second Education Reconstruction Project Cape Verde 1988 P000424 Primary Education Upgrading Project Chad 1993 P000517 Basic Education Project Chile 1992 P006668 Primary Education Improvement Project China 1995 P003636 Basic Education in Poor and Minority Areas Project China 1996 P036950 Third Basic Education Project Guinea 1995 P001087 Equity and School Improvement Project Guinea-Bissau 1988 P000988 Basic Education Development Project India 1996 P035821 District Primary Education Project Lesotho 1992 P001392 Education Sector Development Macedonia, FYR 1998 P038391 Education Rehabilitation Project Mexico 1994 P007725 Second Primary Education Project Morocco 1989 P005480 Rural Primary Education Project Nepal 1989 P010335 Earthquake Emergency Schools Rehabilitation Project Pakistan 1987 P010280 Third Primary Education Project Pakistan 1990 P010346 Sindh Primary Education Development Program Project Tanzania 1990 P002790 Education Planning and Rehabilitation Project Trinidad and Tobago 1996 P035312 Basic Education Project Uruguay 1994 P008171 Basic Education Quality Improvement Project Education Sector investment projects having at least 50% primary: Active as of March 2004 (n = 10) Albania 2000 P069120 Education Reform Project Bangladesh 2004 P074966 Primary Education Development Program II Bolivia 1998 P006204 Education Quality Djibouti 2001 P044585 School Access And Improvement Program India 1998 P038021 DPEP III (BIHAR) India 2000 P050667 UP Third District Primary Education Indonesia 1999 P040196 Sumatra Basic Education Kenya 2003 P082378 Free Primary Education Support Project Lesotho 2004 P081269 ESDP II (PHASE 2) Education Sector investment projects having at least 50 percent primary: Closed since 1995 (n = 20) Bangladesh 1990 P009514 General Education Project Bosnia and Herzegovina 1998 P045313 Second Education Reconstruction Project (Continued on the following page.) 63 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Appendix C: Projects in the Portfolio Review Sample (continued) Country Fiscal year Project ID Project name Cape Verde 1988 P000424 Primary Education Upgrading Project Chad 1993 P000517 Basic Education Project Chile 1992 P006668 Primary Education Improvement Project China 1995 P003636 Basic Education in Poor and Minority Areas Project China 1996 P036950 Third Basic Education Project Guinea 1995 P001087 Equity and School Improvement Project Guinea-Bissau 1988 P000988 Basic Education Development Project India 1996 P035821 District Primary Education Project Lesotho 1992 P001392 Education Sector Development Macedonia, FYR 1998 P038391 Education Rehabilitation Project Mexico 1994 P007725 Second Primary Education Project Morocco 1989 P005480 Rural Primary Education Project Nepal 1989 P010335 Earthquake Emergency Schools Rehabilitation Project Pakistan 1987 P010280 Third Primary Education Project Pakistan 1990 P010346 Sindh Primary Education Development Program Project Tanzania 1990 P002790 Education Planning and Rehabilitation Project Trinidad and Tobago 1996 P035312 Basic Education Project Uruguay 1994 P008171 Basic Education Quality Improvement Project Education Sector investment projects having at least 50 percent primary: Active as of March 2004 (n = 10) Albania 2000 P069120 Education Reform Project Bangladesh 2004 P074966 Primary Education Development Program II Bolivia 1998 P006204 Education Quality Djibouti 2001 P044585 School Access And Improvement Program India 1998 P038021 DPEP III (BIHAR) India 2000 P050667 UP Third District Primary Education Indonesia 1999 P040196 Sumatra Basic Education Kenya 2003 P082378 Free Primary Education Support Project Lesotho 2004 P081269 ESDP II (Phase 2) Sri Lanka 1998 P010525 General Education II Non-Education Sector investment projects with the highest percent allocations to primary: Closed since 1995 (n = 5) El Salvador 1991 P007168 Social Sector Rehabilitation Mauritania 1995 P001857 General Education V Mongolia 1996 P036417 MN - Poverty Alleviation Togo 1999 P052263 Pilot Social Fund Uganda 1990 P002966 Poverty and Soc Costs Non-Education Sector investment projects with the highest percent allocations to primary: Active as of March 2004 (n = 5) Congo, Dem. Rep. of 2004 P082516 DRC Multisectoral HIV/Aids Project Ethiopia 2003 P077457 ESRDF I Supplemental Georgia 2003 P080376 Emergency Earthquake Rehabilitation/SIF Supplement Ghana 2003 P071399 Partnerships W/ Traditional Authorities Yemen, Republic of 2004 P082498 Ry-Social Fund For Development III 64 APPENDIX C: PROJECTS IN THE PORTFOLIO REVIEW SAMPLE Appendix C: Projects in the Portfolio Review Sample (continued) Country Fiscal year Project ID Project name Education Sector adjustment projects having some primary education: Closed since 1995 (n = 4) Côte d’Ivoire 1992 P001172 Human Resources Adjustment Program Ghana 1990 P000896 Education Sector Adjustment Credit II Kenya 1992 P001327 Education Sector Adjustment Credit Mali 1995 P035662 Education Sector Adjustment Loan Education Sector adjustment project having some primary education: Active as of March 2004 (n = 1) Tanzania 2002 P071012 Primary Education Development Program Non-Education Sector adjustment projects supporting primary education: Closed (n = 4) Chad 2002 P035594 Economic Recovery Credit India 1993 P009987 The Social Safety Net Sector Adjustment Program Pakistan 2001 P071463 Structural Adjustment Project Sierra Leone 1994 P074642 Economic Rehabilitation And Recovery Credit II Non-Education Sector adjustment project supporting primary education: Active as of March 2004 (n = 1) Nicaragua 2004 P082885 Nicaragua PRSC I 65 APPENDIX D: EDUCATION PROJECTS AND LENDING AMOUNTS,1963–2005, BY COUNTRY Projects with Projects with Total lending 25–49% of loan 50% or more for primary for primary of loan for Total education Number of education primary education lending Amount Percent of projects with % of all % of all for ($US total education Country/Region any education No. projects No. projects education millions) lending Afghanistan 8 4 50 1 13 141 64 46 Albania 8 1 13 1 13 37 17 46 Algeria 8 1 13 0 0 438 19 4 Angola 6 2 33 0 0 67 33 50 Argentina 21 0 0 0 0 1,137 200 18 Armenia 5 1 20 0 0 52 16 30 Azerbaijan 6 2 33 0 0 30 14 47 Bahamas, The 2 0 0 0 0 17 0 0 Bangladesh 23 1 4 5 22 1,074 538 50 Barbados 4 2 50 1 25 30 11 38 Belize 3 0 0 1 33 8 6 76 Benin 12 3 25 2 17 80 49 61 Bhutan 3 0 0 2 67 49 15 30 Bolivia 15 3 20 2 13 178 118 66 Bosnia and Herzegovina 9 1 11 2 22 50 23 46 Botswana 4 1 25 1 25 67 23 34 Brazil 32 3 9 12 38 2,824 1,486 53 Bulgaria 4 0 0 0 0 51 21 41 Burkina Faso 13 0 0 3 23 167 90 54 Burundi 10 2 20 1 10 107 43 40 Central African Republic 4 0 0 3 75 34 18 53 Cambodia 7 1 14 1 14 47 23 50 Cameroon 9 0 0 2 22 94 25 26 Cape Verde 4 1 25 2 50 25 13 53 Caribbean Region 1 0 0 0 0 6 0 0 Chad 10 1 10 4 40 112 72 65 Chile 10 0 0 1 10 478 131 27 China 33 1 3 5 15 1,903 446 23 Colombia 24 4 17 3 13 852 293 34 Comoros 5 2 40 1 20 26 13 50 Congo, Dem. Rep. of 8 1 13 2 25 123 68 55 (Continued on the following page.) 67 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Appendix D: Education Projects and Lending Amounts, 1963–2005, by Country (continued) Projects with Projects with Total lending 25–49% of loan 50% or more for primary for primary of loan for Total education Number of education primary education lending Amount Percent of projects with % of all % of all for ($US total education Country/Region any education No. projects No. projects education millions) lending Congo, Rep. of 6 3 50 1 17 48 22 46 Costa Rica 3 1 33 1 33 53 24 45 Côte d’Ivoire 13 3 23 1 8 478 35 7 Djibouti 6 1 17 2 33 23 19 82 Dominica 1 1 100 0 0 6 2 36 Dominican Republic 8 2 25 1 13 120 30 25 Ecuador 7 0 0 1 14 140 69 49 Egypt, Arab Republic of 13 0 0 1 8 454 51 11 El Salvador 10 2 20 3 30 272 119 44 Equatorial Guinea 1 0 0 1 100 5 4 85 Eritrea 8 0 0 1 13 122 40 32 Estonia 1 0 0 0 0 13 0 0 Ethiopia 17 3 18 1 6 423 128 30 Gabon 3 0 0 0 0 11 2 21 Gambia, The 5 0 0 2 40 47 23 50 Georgia 4 1 25 1 25 39 21 53 Ghana 18 2 11 2 11 460 136 30 Greece 4 0 0 0 0 142 1 1 Grenada 3 2 67 0 0 21 5 24 Guatemala 8 2 25 4 50 205 126 62 Guinea 11 1 9 1 9 174 76 43 Guinea-Bissau 4 0 0 2 50 22 17 74 Guyana 4 0 0 0 0 38 2 6 Haiti 6 3 50 2 33 55 33 59 Honduras 19 1 5 3 16 211 102 48 Hungary 4 0 0 0 0 344 4 1 India 30 1 3 9 30 3,450 1,923 56 Indonesia 58 2 3 4 7 3,193 379 12 Iran, Islamic Republic of 2 1 50 0 0 37 16 43 Ireland 2 0 0 0 0 38 0 0 Jamaica 10 0 0 0 0 162 15 9 Jordan 16 4 25 1 6 487 161 33 Kazakhstan 1 0 0 0 0 12 0 0 Kenya 18 1 6 2 11 484 77 16 Korea, Rep. of 15 0 0 0 0 756 1 0 Kosovo 2 1 50 0 0 6 2 41 Kyrgyz Republic 5 1 20 0 0 19 7 36 Lao PDR 9 0 0 2 22 40 24 60 Latvia 1 1 100 0 0 31 12 39 Lebanon 4 1 25 1 25 127 28 22 68 A P P E N D I X D : E D U C AT I O N P R O J E C T S A N D L E N D I N G A M O U N T S , 1 9 6 3 – 2 0 0 5 , B Y C O U N T R Y Appendix D: Education Projects and Lending Amounts, 1963–2005, by Country (continued) Projects with Projects with Total lending 25–49% of loan 50% or more for primary for primary of loan for Total education Number of education primary education lending Amount Percent of projects with % of all % of all for ($US total education Country any education No. projects No. projects education millions) lending Lesotho 7 1 14 4 57 99 50 50 Liberia 4 1 25 1 25 31 11 35 Lithuania 1 0 0 0 0 25 2 9 Macedonia, FYR 4 1 25 1 25 22 13 57 Madagascar 19 0 0 1 5 271 118 44 Malawi 20 3 15 3 15 341 120 35 Malaysia 15 3 20 1 7 1,007 182 18 Maldives 5 0 0 0 0 51 5 9 Mali 14 2 14 1 7 187 39 21 Mauritania 9 2 22 1 11 150 53 36 Mauritius 6 0 0 0 0 72 9 13 Mexico 20 0 0 5 25 3,389 984 29 Moldova 6 0 0 2 33 31 18 57 Mongolia 1 1 100 0 0 3 3 100 Morocco 16 1 6 2 13 772 148 19 Mozambique 9 0 0 3 33 262 99 38 Nepal 15 3 20 4 27 204 107 53 Nicaragua 12 3 25 2 17 181 106 58 Niger 10 1 10 2 20 157 78 50 Nigeria 14 1 7 2 14 591 253 43 OECS Countries 1 0 0 0 0 1 0 0 Oman 3 2 67 0 0 35 7 20 Pakistan 24 2 8 8 33 1,546 944 61 Panama 3 0 0 2 67 76 46 61 Papua New Guinea 7 0 0 1 14 132 17 13 Paraguay 8 1 13 1 13 93 20 21 Peru 9 0 0 2 22 357 230 64 Philippines 18 1 6 5 28 836 519 62 Poland 3 0 0 0 0 97 3 3 Portugal 4 0 0 0 0 124 5 4 Romania 6 2 33 1 17 249 97 39 Russian Federation 7 1 14 0 0 491 68 14 Rwanda 9 4 44 1 11 124 53 43 São Tomé and Principe 3 0 0 0 0 4 2 50 Senegal 17 1 6 1 6 198 51 26 Serbia and Montenegro 1 0 0 1 100 4 3 69 Sierra Leone 8 2 25 1 13 69 42 60 Singapore 2 0 0 0 0 29 0 0 Solomon Islands 4 0 0 1 25 28 7 24 (Continued on the following page.) 69 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Appendix D: Education Projects and Lending Amounts, 1963–2005, by Country (continued) Projects with Projects with Total lending 25–49% of loan 50% or more for primary for primary of loan for Total education Number of education primary education lending Amount Percent of projects with % of all % of all for ($US total education Country/Region any education No. projects No. projects education millions) lending Somalia 5 0 0 1 20 56 26 46 Spain 2 1 50 0 0 62 4 6 Sri Lanka 6 0 0 1 17 241 53 22 St. Kitts and Nevis 1 0 0 0 0 5 0 0 St. Lucia 2 0 0 1 50 19 7 36 St. Vincent and the Grenadines 1 0 0 0 0 5 1 13 Sudan 3 1 33 0 0 34 3 9 Swaziland 3 1 33 1 33 19 8 44 Syrian Arab Republic 2 0 0 0 0 36 3 9 Taiwan (China) 1 0 0 0 0 9 0 0 Tajikistan 3 0 0 2 67 26 20 78 Tanzania 20 3 15 2 10 494 227 46 Thailand 10 1 10 0 0 469 10 2 Timor-Leste 2 0 0 0 0 1 1 76 Togo 6 0 0 3 50 72 32 44 Tonga 1 1 100 0 0 1 0 50 Trinidad and Tobago 5 1 20 1 20 110 45 41 Tunisia 14 2 14 1 7 730 114 16 Turkey 12 0 0 2 17 1,203 436 36 Uganda 20 3 15 2 10 390 250 64 Ukraine 3 0 0 0 0 54 5 9 Uruguay 6 1 17 3 50 195 102 52 Vanuatu 2 0 0 1 50 12 2 15 Venezuela, República Bolivariana de 5 1 20 0 0 125 43 34 Vietnam 10 0 0 3 30 406 272 67 Western Africa 1 0 0 0 0 6 0 0 Yemen, Republic of 30 2 7 4 13 437 194 44 Yugoslavia (see Serbia and Montenegro) 1 1 100 0 0 10 3 28 Zambia 13 0 0 2 15 239 85 36 Zimbabwe 3 0 0 0 0 11 4 39 Source: Internal World Bank databases: one for Education Sector projects and one covering projects managed by other sectors. Note: This table includes projects with any commitments for education managed by the education and other sector boards. 70 APPENDIX E: WHAT ARE FAST-TRACK INITIATIVE COUNTRIES TARGETING? Applicants to the Fast-Track Initiative (FTI) are percent, and education spending as percent of asked to submit target benchmarks within the government recurrent costs, 10.9–28.3 percent. Indicative Framework (some of its main items are The guidelines for the use of the Indicative in the table below). The table shows the average Framework indicate that the benchmark “should on the framework items for 55 low-income not be applied rigidly.” Nevertheless, among the countries (on track and off track) and the average four FTI countries observed in this evaluation, for the 10 highest completion countries, which having very different characteristics and became the basis for the FTI benchmarks (goals, constraints, the target values in the indicative column “g”), 4 of which are also shown (c–f). framework appear to mirror the benchmarks There is considerable variation on the Indica- quite closely. For example, Niger’s targets tive Framework items across high-performance diverged from the benchmark on only two of countries; for example, on item 1 (average seven indicators: on average teacher salary teacher salary as percent of SDP per capita) the (where the difference was minor) and on range is from 1.2 to 6.1; on pupil:teacher ratio, average repetition rate, where Niger’s low target 20.6–45.3; average repetition rate, 2–18.3 reflects a policy of automatic promotion. 71 72 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Table E.1: FTI Indicative Framework: Some Benchmarks for Baseline and Partner Countries Baseline mean values 2015 targets in FTI, some partner countries FTI Whole Highest Indicative Republic sample completion Framework of (55 countries) countries Bolivia Lesotho Vietnam Zimbabwe benchmarks Honduras Niger Vietnam Yemen (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) Service delivery Average teacher salarya 4.4 3.6 2.2 6.6 1.2 6.1 3.5 3.5-4 3.9c 2.19 3.4 Pupil:teacher ratio 46 39.6 20.6 45.3 30.4 39 40 37.5 40 26.6 28 Non-teacher salary share of recurrent spending 23.8 25.2 19.4 30 45 25 33 32 33 35.6 32 Average repetition rate 16.6 8.2 3.7 18.3 3.5 2 10 2 5 0.8 6 Domestic resource mobilization Government revenues as % of GDP 19.6 20.7 21.2 35.9 22.3 27.4 14/16/18b 18 14 23 34 Education spending as % of recurrent budget 16.9 18.5 25.2 22.2 10.9 28.3 20 22 20 20 17 Primary as % of recurrent education spending 49.3 47.2 47.5 40.2 46.5 46.1 50 51 50 30 48 Source: Bruns, Mingat, and Rakotamalala 2003 and country FTI proposals. Note: GDP = gross domestic product. a. Expressed as a multiple of per capita GDP. b. Staggered targets proportional to per capita GDP. For a six-year primary cycle, these imply domestic expenditure on primary education equal to 1.4 percent, 1.6 percent, and 1.8 percent of GDP, respectively. c. This figure reflects remuneration for newly hired teachers. APPENDIX F: PRIMARY EDUCATION EVALUATION TERMS OF REFERENCE FOR COUNTRY-LEVEL ANALYSIS Background enrollment ratios, and completion rates). The World Bank’s Independent Evaluation Educational outcomes in the analysis will include Group (IEG) (formerly Operations Evaluation achievement gains in basic knowledge and skills Department) has undertaken an evaluation of (especially literacy and numeracy) and (time and the effectiveness of Bank support to primary data permitting) employment and welfare education. The purpose of the evaluation is to outcomes (improved health and nutrition and examine the effectiveness of the Bank’s reduced fertility). Equity with respect to assistance—its policy dialogue, analytic work, increased access to schooling and improved and lending—in support of country improve- learning outcomes will be measured by examin- ments in primary school access and educational ing the distribution across social groups, in partic- outcomes, particularly since the beginning of ular females, those from low-income families, the Education for All (EFA) movement in 1990, linguistic or cultural minorities, those who live with an eye toward influencing the design and remote or difficult areas, and the disabled. effectiveness of current and future Bank policies and programs. The evaluation has produced two Country-Level Analysis Instruments background papers (Boissiere 2004a, b) and a Country-level analysis will be of two types— desk review of the portfolio of World Bank “enhanced” PPARs and country case studies. lending for primary education. In the next PPARs are conducted by IEG on about one- phase, the evaluation will undertake a series of quarter of all completed Bank lending projects country-level analyses. as a means of ensuring the integrity of the Whereas the portfolio review covered only Bank’s self-evaluation process and developing lending and used the project as the unit of experience-based lessons for improved analysis, the country-level analyses will cover all directions, policies, and procedures. They are modalities of Bank influence: policy dialogue; often used as “building blocks” for larger evalua- analytical work; technical assistance and capacity tion studies, such as the current evaluation of building; lending; and conditionality. It views primary education. PPARs generally cover the these in an integrated manner vis-à-vis improved performance of one or more related projects in primary school access and learning outcomes, a single country. The PPARs contributing to the and in light of the country’s educational goals, country analysis for this evaluation will address policies, and EFA plans; and social, economic, standard IEG questions related to project political, and infrastructural conditions. outcomes, sustainability, and institutional development but will be “enhanced” with Objectives additional contextual information related to The objective of the country-level analysis is to country EFA plans and Millennium Development evaluate the effectiveness of Bank assistance to Goals (MDGs), general country indicators of countries in improving primary school access, progress on primary education access and educational outcomes, and equity. Primary school learning outcomes, the programs and contribu- access will be measured in terms of quantitative tions of other development agencies, and the expansion (increasing geographical coverage, objectives and features of ongoing Bank support 73 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA to primary education. The “enhanced” PPARs The question list will remain flexible enough will also comment on any lessons learned with to cover other salient questions and issues that respect to the key evaluation themes. might emerge from other boxes in the Country case studies are more comprehen- framework (for example, about private sive assessments, organized exclusively for this education or other levels of education) or evaluation, and will cover the full range of Bank country context (political economy, catastrophic supports related to primary education plus events) as appropriate. To facilitate analysis and country contextual factors related to their reporting, a standard set of appendixes will be success and failure. compiled for all PPARs and case studies covering Both the enhanced PPARs and the country the following: case studies will assess the effectiveness of the Bank’s assistance relative to the counterfactual • A timeline of key national educational policies (what would have happened if there had been and events, and key points of intervention of no Bank support?). the Bank and other donors • An inventory of World Bank assistance to the Questions to Be Addressed country, including lending and nonlending Evaluation questions for the country case supports to primary education; similar inven- studies stem from the main evaluative questions tories for other major development agencies specified in the evaluation Approach Paper. and their principal lines of support They also take into consideration the channels • Tables showing the trends in public expendi- of Bank influence represented in figure B.1 (see ture on primary education and education page 60). The figure depicts the main explicit overall: capital and recurrent expenditure; channels (that is, mentioned in project within recurrent, salary, and nonsalary ex- documents) through which Bank assistance can penditure influence educational outputs and outcomes, • Tables showing changes in major educational for example, through government policy and development indicators over time: enrollment, institutional capacity, specific features of the graduation, dropout and promotion, transi- educational system (facilities, books, teacher tion rates (primary to secondary), and learn- status and quality, and teaching-learning ing outcomes (scores on official, standardized processes), and increased demand for examinations) education, plus other enabling factors such as • Description of the political process of policy improved donor cooperation, civil service and program formulation, covering such items reform, support for complementary sectors, and as where the initiative for the program came actions in the private sector and by civil society. from, who was at the table during negotia- The main analytical questions refer to the items tions, what the main sources of conflict were, in the central boxes in figure B.1: who the winners and losers were, and who had ownership of the program/policy • Government policy and capacity • A listing of challenges and obstacles that pre- • Delivery of education services sented themselves in the implementation of • Household demand for education programs for improving access and outcomes • Outputs and outcomes. of primary education. The questions to be asked in all cases are: (a) Lessons Learned What changes have taken place since 1990? (b) In addition to assessing the effectiveness of the To what extent have Bank efforts (though Bank’s assistance to primary education, both the lending and nonlending channels) contributed PPARs and the country case studies will highlight to those changes? and (c) To what extent would the lessons learned from the Bank’s experiences the changes have taken place in the absence of in supporting primary education with respect to Bank support? the following: 74 A P P E N D I X F : P R I M A RY E D U C AT I O N E VA L U AT I O N T E R M S O F R E F E R E N C E F O R C O U N T RY- L E V E L A N A LY S I S • The kinds of trade-offs made between quanti- liamentary committees; local government of- tative expansion and quality improvement and ficials; actors playing significant roles in proj- the factors influencing them ect implementation; any relevant alliances or • The relation between the choice of lending coalitions of NGOs, teachers unions, social instrument (for example, investment versus movements, and parent-teacher associations; adjustment) and modalities (for example, community leaders, especially those involved Adaptable Program Loans [APLs], Sector Wide in school committees or councils; school teach- Approaches, and community-driven develop- ers; World Bank staff; other development ment—that is, use of “social funds” and simi- agency representatives; and private sector lar modalities) and both ownership and providers) effectiveness • Field visits to schools, school councils, teacher • The conditions under which decentralized or management training programs, and re- management in the delivery of primary edu- gional or local offices of education (preferably cation has been effective selected among outliers, good and bad) • Successful efforts to improve monitoring and • Any additional data collection or analysis, as evaluation of service delivery, student learning necessary. outcomes, and the use of this information to improve decision making Expected Outputs • Institutional development in the field of pri- The outputs of the two kinds of country-level mary education (problems and efforts to im- analyses will be in the form of individual study prove subsector governance and management) reports. • Donor coordination to enhance the effective- ness and efficiency of primary education as- Country Selection Criteria sistance. There will be six to seven enhanced PPARs, in addition to four country case studies and an Sources of Data impact study in Ghana already completed.1 The expanded PPARs and country case studies Countries have been selected on the basis of the will rely on the following sources of data: record of effectiveness for Bank-supported projects to include both those that have had • A review of the literature on primary education success in this regard and those that have not. in the country in question (covering research; Recognizing the positive correlation between policy studies; and previous evaluations by wealth and better learning outcomes, countries governments, the World Bank, and other de- were also stratified on the basis of their income velopment agencies) (low income or lower-middle income). Only • Analysis of data (enrollment and completion countries that were relatively large recipients of data, broken down by gender, Region, and so- Bank support were covered (at least $100 cioeconomic status of subjects; student learn- million borrowed);2 an attempt was made to ing outcome data, household survey cover both large and small countries and to information [for example, on educational at- represent all regions of the world. The countries tainment and/or achievement]; government proposed for enhanced PPARs (in italics) and and donor expenditures, service delivery data, country case studies (in bold) can be seen in data on private sector and nongovernmental table B.1. organization [NGO] activities) • Semistructured interviews with key actors (of- Staffing ficials in the Ministry of Education, particu- PPARs will be conducted by IEG Sector, larly “champions” of primary education Thematic, and Global Evaluation staff members. improvement and program entrepreneurs; Country case studies will be conducted by pairs technical staff who were at the table during of external consultants plus one or two domestic project/program design; parliaments or par- consultants from the case study country. 75 APPENDIX G: CASE STUDY SUMMARIES Pakistan factors such as traditional attitudes limiting This case study is one of four that aim to assess girls’ participation in schooling. the Bank’s support for primary education within A strong role for education was recognized by specific country contexts. This is a difficult task the founding fathers of independent Pakistan, in a country as large as Pakistan (population and UPE was established as a goal at the first about 148 million) with a complex federal National Education Conference in 1947. structure of government and a long history that However, military tensions with India and has resulted in a mosaic of ethnic and cultural perceptions of national security needs led to diversity, even though it is a majority Muslim relatively high military spending and relatively nation. low education spending—an unfortunate Nonetheless, it is important to attempt this spending pattern that continues up to now. task and to address major issues in primary By the end of the 1990s, the proportion of education such as access, quality, and equity. gross domestic product (GDP) spent on Although the Bank has been involved in education had not risen as expected and was 1.7 education in Pakistan since 1964, the time frame percent in 2001–02 versus 2.1 percent in for this study of primary education is from about 1991–92. Spending on primary education as a 1990, the year of the World Conference on share of GDP was low at about 0.8 percent of Education for All (EFA) in Jomtien, Thailand, and GDP in 2000–01 (see IEG 2004e). also the issue of the Bank’s major Primary Over the years the goal of UPE has been Education Policy Paper, to March 2005, the time repeated by a number of national conferences when the fieldwork for this study was and policy papers, shifting the goal further into undertaken in Pakistan. the future, and it is presently set for the year 2015 in agreement with the education MDGs. Primary Education in a National Context However, political instability since independ- The main story of primary education (grades ence has been a factor in holding back the 1–5) in Pakistan since independence from capacity of the primary education system to Great Britain in 1947 is that of the struggle to respond effectively in achieving this goal. Politi- achieve universal primary education (UPE) cal instability in the 1950s led to the first military within an adverse environment of severe regime of General Ayub Khan, who governed resource constraints, organization and manage- throughout most of the 1960s. The civilian ment problems, and inadequate institutional government of Zulfikar Ali Bhutto came to incentives. In addition, rapid population power in the 1970s after the civil war, resulting growth of about 3 percent throughout most of in the loss of West Pakistan and the creation of the post-independence period has put Bangladesh. Prime Minister Bhutto attempted pressure on the primary education system, many policy changes under the banner of an making it difficult to raise enrollment rates. Islamic Socialist regime, including the national- Only recently is population growth slowing to ization of many private educational institutions. somewhat above 2 percent. Demand-side The civilian government of Prime Minister factors are also come into play, including Bhutto was overthrown in 1978 by General Zia 77 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA ul Haq, who introduced privatization and and Pakistani society as a whole. General deregulation policies to counter Bhutto’s social- Musharraf aligned his government with the war ist policies. General Zia also introduced far- against terrorism and also against the Taliban reaching changes in education policy, including regime, previously an ally of Pakistan. Partly as a his version of Islamization of education. This result of this, aid flows have increased dramati- included establishing mosque/maktab primary cally and the economy has shown a marked schools, supporting madrassas (religious improvement compared with the stagnation seminaries beyond the primary school level), that characterized the 1990s. and revising all subjects in conformity with Islam and requiring teaching of Islamiyat up to grade World Bank and Other Donor Support 14 (early years of university). The Bank has been active in primary education, The opening of mosque/maktab schools was financing 25 projects since its first project in an attempt to increase dramatically access to 1964 for agricultural education. During the primary school by adding regular primary school 1960s the manpower planning approach was subjects to traditional religious instruction dominant within the Bank and with other provided to young children in the local mosque. donors, and during the 1970s that gave way to The program was abandoned later because it basic needs and then the rate of return to was not effective at teaching academic subjects, education approaches, both of which stressed due in part to the fact that the local imams were primary education. Primary education increas- not trained teachers. ingly became the focus of Bank education sector The regime of General Zia came to an end support in Pakistan. There have been 19 projects with his death in 1988, leading to a decade of and broader operations that have supported elected civilian governments. However, many primary education in Pakistan that total $1,365 Pakistanis refer to the 1990s as the “lost decade” million. The component activities of these because of political instability and economic projects were similar in most cases, involving stagnation. There was an alternation of elected teacher training, textbooks, and school civilian governments between Benazir Bhutto construction. (daughter of Zulfikar Ali Bhutto) and Nawaz Other donors were active in primary Sharif, leaders of the two main political parties. education in Pakistan, the largest including the Political corruption was also on the rise, Asian Development Bank (ADB), Department especially in the provinces, affecting primary for International Development (DfID) of the education through processes such as political United Kingdom, and the United States’ Agency patronage in the appointment and deployment for International Development (USAID). The of teachers. However, as a participant in the ADB has been especially active in teacher Jomtien EFA Conference (1990), Pakistan training. Overall, donor coordination has been committed itself to the goal of UPE by the year reasonable, with the Bank being the largest 2000. donor, thereby having the most influence. The decade of civilian rule came to an end in Looking back over Bank support, including 1999 with the military government of General economic and sector work (ESW) as well as Pervez Musharraf, who was Chief of the Army lending operations, the Bank can be seen to be Staff under Prime Minister Sharif. Broad struggling to come to grips with the complexi- education policy remained the same, and ties of working in Pakistan. First, there is the Pakistan participated actively in the EFA follow- federal structure of government, which often up conference in Dakar, Senegal, in April 2000, resulted in coordination problems between the again signing on to the goal of UPE, this time by federal level and provincial governments. The the year 2015. Bank thus tended to move toward more provin- The events of September 11, 2001, and the cial level projects and programs. Second, each ensuing war on terror proved to be defining province has its ethnic and cultural mix and events for the government of General Musharraf different alignments of political parties. Third, 78 APPENDIX G: CASE STUDY SUMMARIES each province has its own implementation tranche of $100 million with a substantial matrix problems, resulting in part from the first two of primary education policy reforms. That can factors. Thus the Bank was struggling to learn: work well with a province that is committed and (a) What is the right thing to do? and (b) What is capable, and it remains to be seen if this can the right way to do it? work in other provinces such as Sindh, which has expressed interest in such an approach. New Summary of Recent Changes in Primary comprehensive sector work is planned for that Education province, which can lead to an assessment of what can be appropriate there. Government policies and capacities Government policy for primary education has Delivery of education services always been aimed at UPE, with the target dates Despite the many implementation problems constantly shifting. Government policy also has that arose during the 1990s, schools were built always cited quality and equity goals, including and public primary enrollment did increase at the poor and girls’ education. During the 1990s an annualized rate of about 6 percent (10.8 a number of Bank-supported provincial million in 1990 to 19.5 million in 2000). Such education projects were aimed at creating the enrollment growth would compare favorably capacity to achieve these policy goals. Thus with Indonesia’s experience during its well- there were the Sindh Primary Education known school construction program in the Project (1990), Balochistan Primary Education 1970s and 1980s. However, Pakistan started from Project (1993), the Northwest Frontier a very low base level of about 16 percent gross Province Project (1995), and the Northern enrollment rate at the time of independence and Education Project (1998). experienced high population growth of about 3 The big push to achieve this policy goal was percent, so progress in improving enrollment the first Social Action Program Project (SAPP, rates was slow. 1994), which aimed to support a significant part The delivery of quality primary education was (about 10 percent) of the government-initiated negatively affected by the availability of teachers Social Action Program (SAP) that aimed to make and the low quality of those available. Teacher a rapid improvement in Pakistan’s social indica- absenteeism was constantly cited as a problem in tors (education, health, and poverty). This was rural areas. Also, many teachers of low qualifica- followed by a second SAPP (1998). The results of tions were appointed, sometimes as a result of both were disappointing. political patronage. Research studies by Warwick Many people interviewed look at the 1990s as and Reimers (1995) document the poor quality of the SAP phase of government and Bank involve- teacher training as well, with many primary ment and the period since 2000 as the post-SAP school teachers not performing much better than phase. These two projects were too large pupils on grade 5 reading and mathematics tests. (together $550 million in Bank support), too The curriculum and textbooks also leave complex, and poorly designed in terms of much to be desired. They rely heavily on rote mechanisms for implementation. For example, learning without real understanding and, there were many donors involved, and supervi- according to many observers, contain excessive sion missions were large and unmanageable. ideological material and religious indoctrination Disbursement mechanisms were too complex resulting from the time of General Zia’s drive to and placed an unduly large reporting and Islamize education. This situation is not unique documentation responsibility on an educational to Pakistan; Bank research is beginning to administration that could barely handle its examine in more detail the academic and social normal day-to-day responsibilities. content of primary school textbooks in a Based on these lessons, the more recent number of countries (see the latest Bank Punjab education adjustment credits (2004 and Education Sector Strategy Update [World Bank 2005) have simplified disbursement of a single 2005b]). 79 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA The two most significant changes in the and 2001–02, respectively. These figures are delivery of primary of education are the based on household surveys, and some decentralization of government services and the observers are puzzled by the data, given that the rising demand for private primary education. ambitious SAP projects were expected to have The Musharraf Government introduced an some impact around this period. ambitious program of decentralization in 2001, Also puzzling is that fact that two major partly to improve service delivery and partly to household surveys do not agree for some restore civilian politics by going around the measures. The Pakistan Social and Economic established national political parties, thus Survey gives 84.3 percent for the GER in holding elections at the local levels without 2000–01 while the Pakistan Integrated candidates declaring affiliation with a national Household Survey gives 72 percent in 2001–02. party. The recent trend toward private primary However, the two surveys are closer for net education is seen by many people as response enrollment rates (NERs), with the Pakistan to the poor quality of public primary education, Social and Economic Survey giving 48.6 percent with the result that many poor families struggle and the household survey giving 42 percent for to pay fees in private primary education based the same years. The reasons for these differing on the belief that their children will receive a measurements are not clear. better education in private schools, though Many education analysts now favor using the some are very low cost and quality is not yet primary completion rate, defined as the ratio of proven. number of children completing primary education over the number of children of Household demand for education primary completion age, as an outcome indica- Nearly all research shows that the level of tor for measuring the success of EFA. While parents’ education has a direct impact on the there is no time series for this measure in education of their children. However, in the Pakistan, there is an estimate for the year patriarchal household structures of Pakistan, in 2000–01, giving an overall primary completion which men are socially assigned a strong role as rate of 51.3 percent. For male and female, the the head of the family, there is attenuation of this breakdown is 69.4 percent and 64.6 percent for insofar as many parents prefer to invest in the urban versus 51.6 percent and 34.8 percent in education of sons. rural areas. However, many officials now report that the There are no time series for learning achieve- acceptance of education for girls is growing. If ment over time, although the National schools lack boundary walls or require a very Education Assessment System Project (2003) is long walk, the demand for girls’ schooling introducing an assessment system for grades 4 decreases because of parental concerns about and 8. The first results of the grade 4 assess- their daughters’ safety. In addition, remnants of ments should be available in 2006. Pupils are to feudal structures in the rural areas of Pakistan be tested in four subject areas: reading, also constrain demand for primary education, mathematics, science, and social studies/ since feudal land owners still exercise much Islamiyat. influence and often do not encourage education Pakistan does not have a national examina- among their populace. tion for a certificate of primary school comple- tion, which could provide some insight into this Educational outputs and outcomes issue. However, tests given in some regions on a Trends in output indicators (enrollment rates) one-off basis as a part of various research are available, but outcome trends (learning projects and other more qualitative judgments achievement and employment) are not. Gross by informed observers indicate that the trend enrollment rates (GER) have been somewhat for learning achievement overall would be flat at stagnant in recent years, being 75 percent, 71 best, or probably even declining, for public percent, and 72 percent in 1995–96, 1998–99, primary schools. 80 APPENDIX G: CASE STUDY SUMMARIES Finally, in terms of labor market and welfare practice the federal and provincial education outcomes related to primary education, there administrations were overwhelmed by the large are a few research studies that show that the number of small transactions to be documented wider economic and social benefits to primary and were often not clear as to what expenditures education in Pakistan are much the same as in were actually eligible for reimbursement. comparable developing countries. Other specific investment projects for primary education had a variety of implementa- World Bank Contribution tion difficulties and delays. Slow disbursement The Bank’s ESW has been an important factor in thus became the norm, and the overall disburse- the education dialogue with the Pakistani ment percentage after project completion for government and in recent years with civil society primary education projects in Pakistan was 72 organizations and NGOs. As explained by one of percent of the approved amount, compared the interviewees, this is where the Bank was with the Bank-wide average of 93 percent for strongest and helped the government to focus IDA-financed primary education projects (see on policy objectives. Although the last compre- IEG 2004d). However, despite these difficulties, hensive piece of sector work, covering all levels without the Bank’s persistent efforts to keep of the system, was the Education Sector Report access, quality, and equity issues on the agenda, in 1988, that report was influential in setting the it is likely that even less progress would have framework for the provincial primary education been made in increasing school enrollment, projects of the 1990s. especially for girls and the poor. Subsequent ESW , although generally of high quality, focused on specific analytical issues, Lessons such as girls’ education. As a result, there was a Virtually all the Bank’s projects and ESW noticeable change in the attitudes of govern- emphasized quality of learning in primary ment to problems and benefits of more education, but in practice there was no way of education for girls. The Bank also had an impact measuring this during the projects of the 1990s. on emphasizing the monitoring of quality and For example, the Sindh Primary Education learning achievement, resulting from a long Development Program (1990) financed activities period of dialogue in the National Education and inputs that were believed to be important Assessment System Project. for improving learning, but it was not until the Implementation of Bank-supported projects National Education Assessment System Project encountered many difficulties over the past 20 (2003) that a system was put in place to monitor years, perhaps more than the average level of learning achievement at the primary school difficulty for the Bank’s primary education level. In practice, as the mission was informed projects as a whole. This was especially true for by some provincial-level officials, there was a the two large SAP projects—SAPP1 in 1994 for trade-off made between quality and quantity, $200 million (60 percent to primary education) and only now are they really turning attention to and SAPP2 in 1998 for $250 million (60 percent quality. to primary education). Most government and There are also lessons about which instru- NGO persons interviewed expressed the view ments might best achieve objectives and how that this project was poorly designed and that decentralization of government can relate to the Bank and other donors put too much of a instruments. Large programs such as the SAPPs burden on a system that lacked the capacity to that try to pump large sums of money through effectively absorb such large infusions of funds. an incompetent and sometimes corrupt bureau- For example, the disbursement procedures cracy cannot work. A careful mixture of specific involved the government through prefinance investment and sectorwide approaches must be expenditures and to seek reimbursement considered for the education portfolio. Good through submitting Statements of Expenditures. specific investment projects can lead to building Although this appeared simple in principle, in the capacity that can later be used in a more 81 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA decentralized approach in the provinces and program approaches, each one being used in with the flexibility and speed that can come from the appropriate circumstances. a sectorwide approach. Also, the need for • The Bank needs to think carefully about how intensive training for decentralization to work is to engage the government about sensitive top- apparent in Pakistan, as it is in many other ics related to curriculum reform and textbook countries that have tried it. provision. In the cognitive domain there is ex- The need for donor coordination is important, cessive reliance on rote learning to the detri- but an important lesson of SAPP1 and 2 is to avoid ment of genuine cognitive development. In donor coordination becoming an undue burden the social domain there is inappropriate ide- on the counterpart agencies. While the formation ological material in the curriculum about reli- of the Multi-Donor Support Unit for the SAPPs gion and other political issues that are was a positive development, it is important to detrimental to promoting social cohesion. The build up the capacity of the government so it can recent Education Sector Strategy Update take the lead in donor coordination. (World Bank 2005b) points out that the Bank is starting to address this issue in other coun- Conclusions tries and could provide a starting point for ini- Overall, the development effectiveness of Bank tiating dialogue about these complex and support for primary education can be rated as sensitive issues. marginally satisfactory during the 1990s. Since • Perhaps the most important area of policy and then, the effectiveness of Bank support has program support for Bank assistance is to help improved to a more solid satisfactory level, Pakistan craft a realistic strategy for achieving based on the lessons learned during the 1990s quality EFA. Recent Bank reports indicate that and an improved understanding of the difficult Pakistan is far from this goal and not likely to implementation environment in Pakistan. A few achieve it by the year 2015 as specified in the conclusions in the way of broad directions to education MDGs. If it turns out that the year explore would include the following: 2015 is unrealistic, then a careful analysis is needed of what is possible, what resources • The Bank should return to the more compre- are required, and when it can be achieved. hensive approach to sector work taken in the 1988 Education Sector Report covering all lev- Peru els of education and their interactions, but do Since 1990, the World Bank developed and this in a province-specific way. It is important launched two major education loans in Peru, one that the analysis of primary education be em- (1995) aimed at improving urban primary bedded in the whole education system to take education and the second (2002) at improving account of fiscal interactions and the inter-re- rural primary education. The first loan totaled lationships of quality at different levels. It is im- US$146.5 million, and along with counterpart portant to do this in a province-specific way funds, invested nearly US$300 million in building because of the large variance in capacity and urban primary schools, developing and distribut- ethnic/cultural mix among the provinces. ing school textbooks, and improving classroom • While the ESW should become more com- teaching. The second loan is still in progress, but prehensive, the Bank would do well to focus it will end up investing roughly $170 million over its lending operations in those areas where it four years (with counterpart funds nearly can make the most impact. This would have to US$350 million) in improving rural primary be done in coordination with other donors teaching, testing incentive systems to improve and with the government taking a lead role. At- teacher and student attendance, and developing tention would have to be paid to how to en- a secondary school distance education system. hance the government’s capacity to take this The objective of this case study is to evaluate lead role. There should also be a judicious mix the relevance and effectiveness of World Bank of specific investment projects and sectorwide efforts in supporting primary education in Peru. 82 APPENDIX G: CASE STUDY SUMMARIES To carry out the task, the mission team between urban and rural areas. The past 15 interviewed 7 of the 16 current and former years of primary school expansion have Ministers of Education from 1990 to 2005, a produced near universal access to full primary number of key past and present educational education. The majority of urban youth are policy makers who have been involved in the also likely to finish secondary education (64 negotiations and implementation of the two percent of urban 16- to 18-year-olds have loans, the local World Bank education represen- completed secondary school), but the vast tative, and representatives of other international majority of rural youth do not (only 24 percent agencies who lend for or provide technical of 16- to 18-year-olds have competed second- assistance to education in Peru, including the ary). In urban areas, a relatively high percent- Inter-American Development Bank and the age of youth also attends some years of post- German Agency for Technical Cooperation secondary school. (Deutsche Gesellschaft für Technische Peru expanded education largely by making it Zusammenarbeit). The mission also visited a less expensive—principally by reducing teacher number of schools, where mission members salaries in real terms. Except for 1985–87 and an interviewed administrators, teachers, and earlier spending jump in 1980–81, educational parents and observed classes. spending per student fell steadily since the early During the period analyzed by this report, 1970s. Indeed, by 1990, spending per student 1990–2005, the World Bank lent only for primary had fallen about 60 percent from 1973–74 levels, education (grades 1–6) in Peru, although the whereas GDP had risen about 14 percent and rural primary loan does include a component GDP per capita had fallen about 23 percent. This for secondary distance education. The Bank has necessarily meant steep declines in teachers’ been a major force in stimulating primary real salaries. education improvement in Peru, largely because Teachers earned about 25–30 percent more the Ministry—aside from counterpart funds for than per capita income in the early 1970s and Bank loans—uses essentially its entire available earned about 23 percent less than per capita primary education budget to pay salaries and to income in 1990, a drop of about 50 percent meet other usual current expenses. Further, relative to the average Peruvian’s economic Peru has changed education ministers annually, situation. Part of this fall in teachers’ relative on average, over the past 15 years. Thus, the position is due to an increase in average Bank has ended up being an important shaper education in Peru’s labor force, but part is due (as well as institutional memory) of many, if not to a fall in teachers’ wages relative to those of most, Peruvian primary educational improve- other professionals. ment efforts during this period. Quality of education, as measured by pupils’ scores on international tests, is at the low end in Primary Education in the National Economic Latin America, much below the results in and Political Context Mexico, Chile, Argentina, and Colombia on the The context for these efforts was an economy same tests. This is not just an artifact of Peruvian that suffered serious setbacks in the 1980s (GDP students’ lower average socioeconomic decline and rapid inflation), a political system background. The top 10 percent of achievers in threatened in the 1980s and 1990s by terrorists, Peru on the Programme for International assaults on the Constitution by the elected Student Assessment (PISA) scored at about the president in the late 1990s, and the undermin- same level as the 60th percentile in Argentina. ing of the political system by drug cartels. In On UNESCO’s LLECE test, higher socioeco- education, beginning in the 1970s, a series of nomic background Peruvian pupils also scored governments emphasized expanding access much lower than their counterparts in many more than improving quality. other Latin American countries, and rural Educational attainment is relatively high in Peruvian students scored among the lowest in Peru but still very unequally distributed Latin America. 83 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA World Bank Support for Expanding and education, and social protection). Each of the Improving Primary Education Programmatic Social Reform Loans I-IV (PSRLs) The initial history of World Bank support for was signed in the amount of US$100 million Peruvian education mirrors that of many other (except for PSRL III, in the amount of US$150 Latin American countries: loans for primary million). Through the PSRLs, the Bank financed education in Peru started only in the mid-1980s, the publication of both international following the 1970s (technical and vocational) (UNESCO/LLECE) and national assessment and 1960s (tertiary) project cycles. In 1984, a loan results; establishment of monitoring and to improve and expand primary education was supervision systems, including creation of a signed, with the goal of supporting the first 3- payroll system to track the problem of ghost year phase of a 10-year education program teachers and compare teaching responsibilities designed to do the following: (a) provide with payroll amounts; piloting of a program of sufficient and adequate student places for local control in the distribution of salary school-age children, (b) improve the quality of incentives for rural teachers, guaranteeing primary education, and (c) improve primary budgetary allocations for counterpart funds for education management. The loan became finalizing MECEP; and development of a effective in June 1985; less than two years later monitoring and evaluation system designed to the Bank suspended disbursements to Peru. The provide transparency of information during the project outcome was rated as unsatisfactory. decentralization process. In 2004, a Technical In 1993, the Fujimori Government, with Bank Assistance Loan in the amount of US$7.8 million support, developed an extensive diagnostic of was signed to support the development of an Peruvian education and called for actions to accountability system for decentralization in the improve educational quality, efficiency, and social sectors, particularly with regard to equity. That report led to the design of the improved monitoring and evaluation activities. Primary Education Quality Project (MECEP). The In 2003, the Bank and the Toledo Government report pointed to key issues of instructional realized a long-in-gestation Rural Education materials, teacher training, public school Project. The first-phase PEAR APL was signed in autonomy and accountability, school infrastruc- the amount of US$52.5 million (with a govern- ture, and bilingual-intercultural education. ment contribution of US$29.5 million and an Together, the first four issues became the Inter-American Development Bank contribution basis for the broad 1994 MECEP Loan in the of US$12.2 million). The total program amount amount of US$146.4 million (with a government of the 10-year, three-phase APL is expected to be contribution of US$107.5). Though not initially US$347.2, of which US$172.5 is a World Bank contemplated in the project design, school loan. Project components include expanding infrastructure became the project’s largest access for rural children, improving quality in component (nearly half of project funds). This rural primary school, and reforming teacher was the direct result of President Fujimori’s policy and education management. Expansion of insistence on school construction as the access under the first project component focuses project’s main goal. To ensure that school on both preschool and secondary education. buildings did not take priority over “soft” invest- These loans represent significant amounts of ments, Bank staff set specific yearly targets for money in the context of Peruvian educational textbooks and training. Achievement of these spending. The $300 million of the primary targets triggered the release of funds for the education loan in 1995–2000 represented about construction component. 5–6 percent of the total education budget for Beginning in 2001 the Bank signed a series of those 5 years and almost 20 percent of the total Programmatic Structural Adjustment Loans, budget for primary education. The rural designed to transfer funds directly to the education loan now under way is much smaller Ministry of Finance in exchange for a broad array but also represents a significant fraction of the of social sector policy reforms (including health, money going to rural primary education. 84 APPENDIX G: CASE STUDY SUMMARIES The Bank’s Contribution to Sectoral Changes in component was included; in exchange for the Past 15 Years guaranteeing advances in other areas, Bank staff Each of the two Bank-supported projects and included school infrastructure. In hindsight, PSRLs implemented during this period there was considerable need for physical school (1990–2005) has generally been based on improvements, though, as discussed below, recommendations from detailed research-based these likely would have occurred even without diagnostics. These diagnostics were important an MECEP project. in shaping the direction of the projects and The second caveat relates to the low level of helped to build consensus around the institutional capacity building in project activities. challenges and potential solutions for the The Bank did help modernize the Ministry MECEP in the 1990s and the rural project. The through financing the technical assistance, activities outlined in the MECEP and PSRL hardware, and software to install information projects seemed appropriate to education systems for payroll and record keeping. The Bank sector conditions in the country at the time and financed the technical assistance to make the usually focused on areas where the Bank could Ministry more cost efficient through the elimina- contribute with extensive experience and tion of many superfluous payroll positions. The technical assistance, for example, in textbook Bank also supported the ministry in developing production, teacher training, and teacher and sustaining the Quality Measurement Unit, incentive pilots, and distance education in rural which has done excellent work in achievement areas. measurement and analysis over the past 10 years. Although the design of MECEP was relevant Yet at the same time, the Bank-created and - to the needs identified in the diagnostic, as financed Project Management Unit in the ministry highlighted in the IEG review, the institutional has had little impact on training people in the rest development component was overly ambitious, of the ministry or in departmental offices or especially given the volatile nature of the politi- installing management systems that permanently cal context and the lack of specific project become part of the ministry’s mode of operation. measures to help the Ministry develop and build One of the main problems in this regard has consensus around proposed reforms of school not been under the Bank’s control—the almost governance (especially autonomy) and adminis- constant change of ministers in the past 15 tration (for example, decentralization reforms). years. It is telling that the implementation of the Though proposed in the original project MECEP and PLANCAD (the National Plan for design, which did not adequately take into Teacher Training) is largely due to the fact that account issues of political will, neither school one minister, Domingo Palermo, lasted three autonomy nor regional decentralization was years during the Fujimori regime. implemented under the MECEP design, though The third caveat concerns the absence of some aspects of the original design would evaluation and monitoring of project activities appear both in later projects (such as the rural and impact. There have been no ex post evalua- project) and through independent ministry tions of project impact even though data are or actions, such as the new teacher-hiring process could have been available for assessing the effect implemented at the beginning of the Toledo of textbooks and teacher training on student presidency. achievement over the five-year period There are three important caveats to the 1996–2001. There is some indication that test overall positive assessment of the relevance of scores for primary school children have Bank-supported project activities. The first is the remained relatively constant throughout the inclusion of the construction component in the period, and this at a very low level compared MECEP, which was not originally seen as a with other large Latin American countries. priority in the sector diagnostic. The Fujimori However, this indication is not based on strict Government, however, had threatened not to comparisons of like items on tests at the fourth- have a project at all unless the construction grade level, for example, which would have been 85 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA possible if Bank or Bank-financed ministry staff marginalized urban and rural youth. This makes had built project evaluation into the Bank- it somewhat atypical for a lower-middle-income supported project. In the absence of further country. It is also atypical in the financial effort it evaluation, we do not even know if teachers has expended to accomplish these goals. Peru changed their practice. We do know that spends relatively little on its primary education thousands of teachers received training of system. Its costs per pupil are among the lowest varying quality from a variety of agencies in Latin America, and its teachers are paid contracted by the Ministry of Education. among the lowest salaries in the region relative Each of the PSRLs was highly relevant in to per capita income and compared with other establishing key administrative and legislative public servants. benchmarks for improvements within the Peru is typical of countries investing so little education sector as well as protecting key social per pupil in public primary education (Central sector antipoverty measures from budgetary American countries, for example) that its cuts during the transition period. Highly students score very low on international relevant measures include laying the administra- achievement tests, both at the primary level tive groundwork for the Rural Education Project, (LLECE) and in middle school (PISA), even when reforming the payroll system, and creating adjusted for socioeconomic class differences. additional transparency within the Ministry of Peru is also typical of most developing countries Education budget system. in that the teaching supervision system and Regarding the Rural Education Project, teacher and school accountability systems are project activities are relevant to the sector, essentially nonexistent. Finally, Peru shares with especially given the advances achieved under most countries a fundamental lack of capacity the PSRLs in terms of the creation of school for managing a massive and highly spread out councils, more autonomous regions, and primary education system. That is one more schools. There is, however, a question that the reason why the quality of these services is so low. rural project may be doing too much (that is, it These underlying conditions suggest that is spread too thin across a variety of activities). improving teacher capacity and the governance Some of the elements of the Rural Education of primary (and secondary) education are Project are being evaluated carefully, using crucial to improving quality and to increasing comparison groups. But it does not appear that efficiently the amount of schooling taken by an evaluation component using learning each student. The Peruvian case suggests that outcomes data has been built into the overall management capacity building, from ministry to effort to improve the quality of rural education. school to classroom, should be a priority for A recent progress report on this project shows governments and for agencies lending for some major problems, especially the lack of an primary education in developing countries. implementation strategy, an overall monitor- ing, and an evaluation plan; and a communica- Development Effectiveness of Bank Support tions strategy aimed mostly at parents, The Bank strategy under such conditions seems teachers, and administrative personnel linked to have been to invest in projects that with the project. emphasized successful delivery of educational inputs rather than the delivery of educational Lessons outcomes. In the 1996 urban primary education Peru’s history of progress in primary education loan, MECEP focused on two inputs—textbook is typical of developing countries in some ways distribution and improved classroom pedagogy. and very atypical in others. Peru has reached In theory, the delivery of these inputs should high levels of incorporating its population into produce higher student outcomes, but this is primary education, even in poor rural areas, and not what the Bank emphasized. rather high completion rates for primary school- Under programs that emphasize input ing (and secondary school attendance) for delivery, managers are considered successful if 86 APPENDIX G: CASE STUDY SUMMARIES they deliver so many repaired buildings and however, emphasize outcomes, such as pilot textbooks or train so many teachers. In a Latin community incentive programs to improve rural American country it should be expected that teacher attendance and teacher accountability projects could go to the level of outcomes: to local rural communities. delivering textbooks that are used in instruction On the Bank’s most input delivery-oriented and changing teacher and management project there was questionably a large amount behaviors. It appears that the project took the spent on school construction in the 1990s less-demanding road and focused on the inputs under pressure from President Fujimori but not on actual textbook use, teacher behavior himself. As long as school construction itself is in the classroom, or learning outcomes. monitored (the Bank claims that the 450 The Bank’s strategy implicitly assumed that if schools’ construction that it controlled met textbooks arrive at the school, teachers and specifications), the finished school is the students would use them effectively, and that if product itself, and it does provide additional teachers learned better teaching techniques, or at least improved classroom space for they would utilize them effectively. Although educational activities. there was slippage in textbook distribution and Thus, the Bank seems to invest in input some teacher corruption in taking commissions improvements whose delivery alone signals from competing publishers to not use the free project success, regardless of whether student textbooks, the presence of textbooks and outcomes increase as a result of such invest- exercise books probably did contribute ments. This is a prudent choice for the Bank in a positively to pupils’ learning. But a greater low management-capacity environment, al- emphasis on the effectiveness of textbook use though there exist some serious questions would have had to include considerable invest- regarding whether because of poor manage- ment in management capacity. Teachers did ment at the school level, the investments turned apparently use at least some of what they out to have relatively low yield in terms of learned in the in-service training courses, and, improving student learning significantly. In the based on teacher interviews, teachers who took long term, educational improvement will the courses considered them valuable. Contract depend on the ability of projects to influence teachers who were not eligible for the courses management and teacher behaviors and to also wanted very much to take them. improve educational outcomes. But even though investing in such inputs is a With constant changes in ministers, it is correct strategy, the question is whether without admittedly difficult to maintain continuity in supportive investments in supervision and reform efforts. The Bank has been fairly success- content knowledge their yield is high enough to ful in Peru despite this difficulty because of the justify spending considerable sums on them personnel in the local office of the Bank and the (particularly the much more expensive fact that the Bank’s Education Sector specialist pedagogical training part). It does not appear has been in place for 10 years. that the yield on pedagogical improvement was Thus, the Bank has been an important part of very high in the context of teachers’ low content the institutional memory for reform, and has, by knowledge, but an emphasis on outcomes may being firm in not changing the shape of its loan have forced a more effective investment strategy. agreements once signed, been able to get most of In the Rural Education Loan (2003), the what it wants in the agreement and implementa- emphasis is also on improvements that tion of the loan. This is not always a good thing, emphasize input delivery, such as expanding but for the most part, keeping implementation on access to rural education, nonformal preschools course has worked reasonably well. All in all, the run by community implementation agents, and Bank should be more aware of the longer-term providing direct access to distance secondary nature of successful educational reforms, particu- education based on programming from a larly in a country in which the educational system centralized location. Other improvements do, requires long-term improvements in quality. 87 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Mali ranging from home-grown policies such as This case study examines the impact of World ruralization, the Nouvelle Ecole Fondamentale, Bank assistance to the education sector in Mali and the Programme Décennal de l’Education from 1990 to 2005. It also examines the ways in (PRODEC) to international initiatives such as which government, donors, NGOs, and civil EFA; and (iii) direct investment in discrete society have responded to the enormous elements such as teachers, curriculum, challenges in the sector. It also suggests a variety infrastructure, and textbooks. of ways in which the support from all actors, and particularly the Bank, can be improved. Case Study Organization Malian children are among the poorest in the The study team reviewed progress in quantita- world. In 2001, 239 children per 1,000 died tive expansion, educational outcomes, and before reaching age five; 83 percent of children equity (regional, rural/urban, economic, and had anemia. Those children who make it to gender). Data on student achievement were school are confronted with a system ill suited to limited, as were data on links between schooling their needs. There are not enough chairs, books, and other outcomes, such as employment, fertil- pencils, or teachers, let alone more modern ity, and productivity. teaching materials. For most children, much The team attempted to contact all World instruction is in a language they do not Bank staff involved with the Malian education understand. Not surprisingly, a very high sector since 1990, as well as all ministers of percentage of children in the Malian school education and ministers from other critical system fail. Repetition rates averaged 19 percent ministries (primarily finance and planning) from per year in 2002. The pass rate for the sixth- the same period. The team met with at least one grade primary school exam is about 50 percent; central-level representative from all critical sixth-grade students are frequently incapable of technical areas, such as textbooks, teacher decoding single sentences in their textbooks. education, and school construction, and with Significant investments over the past 30 years numerous staff at all decentralized levels. Donor have improved the technical functioning of the and NGO interview lists were constructed based sector. Services such as statistics, curriculum on interventions during the case study time development, and in-service teacher education period. The team also met with members of have clearly improved, and there is a healthy Parliament, parents’ organization representa- policy dialogue about substantive issues such as tives, a former leader of the student union teacher hiring, public-private partnerships for movement, other union leaders, and leaders of service delivery, use of national languages, private school organizations. The case study curriculum reform, and textbook policy. team visited 18 schools. However, the basic story of Malian education over the past 30 years is the triumph of form World Bank Support for Expanding and over function, of expansion over quality, of Improving Primary Education inertia over reform. Neither the government of Mali nor the donors active in the sector have Early World Bank support succeeded in bringing either access or achieve- World Bank lending to the education sector in ment to acceptable levels. Mali started in 1973 with support for nonformal The World Bank was very active in Mali during education. The Third Education Project in 1984 this time, with a total of 78 approved IDA credits. provided support for the first time to formal Assistance to education was continuous and can education through teacher education for be divided into three distinct approaches: (i) primary school teachers, along with support for financial assistance, including structural adjust- nonformal adult programs. ment credits, the Highly Indebted Poor Country The implementation of the Third Education (HIPC) Initiative, and Poverty Reduction Project occurred in a difficult economic context Support Credits; (ii) education policy initiatives (see chapter 2), with the country undergoing a 88 APPENDIX G: CASE STUDY SUMMARIES series of adjustment and stabilization programs. the Improving Learning in Primary Schools These programs dominated Bank assistance to Learning and Innovation Loan (Cr. 33180, US$3.8 Mali in the late 1980s and early 1990s. The million) was approved. Under this project, the conditionality that they contained still affects the number of pédagogie convergente (bilingual) education sector and is still negatively referred classrooms increased from 300 to 2,056. to by Malians. One of the most disliked In 2000, the EdSIP opened with the support measures was the “voluntary departure of the Bank and 15 other donors and a total cost program,” through which about 1,000 teachers of over US$0.5 billion. Activities included: (i) left the sector, representing about 12.5 percent increasing GER from 47 percent in 1997 to 75 of the teaching force. percent by 2008 through school construction, preservice teacher education, and increased World Bank support from 1990 to 2005 involvement of communities and the private World Bank assistance to the education sector sector in school financing and management; (ii) during this period included policy dialogue, improving educational quality by expanding analytic work, lending, technical assistance, and pédagogie convergente, increasing expenditure capacity building. The Country Assistance Strate- on textbooks and other learning materials, and gies (CAS) of 1994, 1998, and 2003 cite education decentralizing personnel and budget manage- as a key poverty-alleviation strategy, and there ment; and (iii) making the education system was continued emphasis on policy dialogue and more cost effective by hiring public school funding for the sector during this time. teachers outside the civil service to reduce wage The Fourth Education Project (Education costs, and redirecting scholarship resources for Sector Consolidation Project, Cr. 2054-MLI, secondary students toward activities that would US$26 million) opened in 1989, followed by the improve learning conditions at the secondary Education Sector Adjustment Project (ESAP , Cr. school level. The EdSIP started slowly. In March 2673, US$50 million, 1995), the Learning and 2004, disbursement was only 45 percent and the Innovation Loan (Cr. 33180, US$3.8 million, credit’s closing date was extended from 2000), and the Education Sector Investment December 2004 to December 2005. Program (EdSIP , Cr. 3449, US$45 million). HIPC Initiative and Poverty Reduction Strategy ESAP (Cr. US$25 million) consisted of an Papers. The Bank’s support to the education adjustment component of US$3 million and an sector in Mali continued with collaboration of investment component of US$22 million. The the education and economic teams on the HIPC investment component aimed at increasing and PRSP initiatives. Mali became eligible for primary enrollment in three Regions and HIPC in 1999 due to its external debt burden, its improving quality nationwide. Enrollment vulnerability to external shocks, and its good targets were met and classroom construction track record of adjustment. The Poverty exceeded targets, but the adjustment Reduction Strategy Paper (PRSP) process began component was unsuccessful and two-thirds of in 1998. The PRSP was to be Mali’s medium-term the adjustment funding was cancelled. ESAP was framework for poverty-reduction policies and a repackaging of the Education Sector Consoli- strategies. Human development, including dation Project’s adjustment conditions. It education, was one of three priority areas. The achieved most of its objectives. Sustainability PRSP is partially financed by HIPC funds, which seemed likely for the less-controversial reforms, totaled an estimated FCFA 75 billion in 2002–04. but a few, including the ceiling on scholarship Forty-five percent of this amount was allocated expenditures, were reversed prior to the credit to education and adult literacy. closing date. Concurrently with the implementation of Contributions from Other Donors to the these latter projects, the government of Mali, Education Sector with Bank assistance, was preparing its 10-year Many donors have been involved in Mali’s education strategy. During program preparation, education system over the past 15 years, includ- 89 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA ing through cofinancing of World Bank-led tions. Education’s current budget as a percentage projects. A range of national and international of the national current budget rose from 20 NGOs have also contributed to sector develop- percent in 1991 to 30 percent in 2005 (MEN 2005) ment, either with their own resources or as and the budget for basic education as a percent- implementing agencies for donor programs. Save age of the education budget rose from 36 percent the Children (United States and the United in 1993 to 65 percent in 2004 (MEN 2005). Kingdom), Care International, World Vision, To improve educational quality, the Bank World Education, and Plan International are the supported teacher education (preservice and largest NGOs working in Mali. Many smaller professional development), textbooks, and, NGOs have supported the sector: Santé Sud has since 2000, the expansion of pédagogie conver- built community schools; Paul Gérin Lajoie has gente. The Bank has also been a consistent supported school management committees; supporter of gender equity in access to Action Mopti works with schools; and the education and of decentralization. The case Groupement des Retraités Educateurs sans study mission was told that education is now the Frontières conducts teacher education. most decentralized sector in Mali. The Bank has supported a variety of capacity- EdSIP building activities since 1990, including training The PRODEC/EdSIP marked a new beginning for in school mapping, information and communi- coordinated donor financing. Most donors cations technology, procurement, budgeting, participate today, making donor participation far the use of software for educational planning and more coherent and reducing the administrative modeling, and a variety of technical areas such burden on the government of Mali. Because as curriculum and textbook development. EdSIP is sufficiently descriptive and includes a However, capacity as measured by impact on the clear list of short-term activities, donors can now sector continues to be low. From 1990 until better direct their support. The program has 1998, Bank-financed projects were implemented improved Ministry of Education leadership, by the Bureau des Projets Éducation, an although it has not resulted in uniform independent unit housed in the Ministry of implementation procedures. National Education and staffed with civil servants relieved from their usual responsibili- The World Bank’s Contribution ties. The suppression of the Bureau in 2001 In a context where partners have unequal forced the administration to be more involved power, it is difficult to separate Bank policy from in policy dialogue and program implementation. Malian policy. At times (for instance, with higher education reforms) the Bank influence was so Efficiency and Sustainability of Changes strong that the government of Mali changed its Supported by the Bank policies even knowing that they could not be Bank support had a strong effect on Mali’s GER, implemented. The government was so in need which increased from 26 percent in 1990 to 71 of funding that it imposed reforms until finances percent in 2004. The most important activities were released and then rescinded the reforms. were the creation of the Basic Education The Bank often guided policy development Support Fund (Cr. 2054), the introduction of through the offering and withholding of double-shift teaching, the redeployment of resources, along with continuous policy advice. teachers from administrative positions to The adjustment projects stressed increasing classroom teaching positions, and the emphasis resources for the education sector, and especially on the recruitment of contractual teachers. In for basic education. The Bank took the lead addition, the Bank’s leadership of policy among donors in policy dialogue with the govern- dialogue in the mid 1990s encouraged ment of Mali, raising issues such as scholarships, additional donors to contribute to the sector. enrollment in upper secondary and higher Prior to the beginning of PRODEC, the Bank education, and budget increases and realloca- was minimally involved in educational quality 90 APPENDIX G: CASE STUDY SUMMARIES issues, other than its emphasis on resources for decreased salary expenditures, but had a major teaching and learning materials and the recruit- negative impact on educational quality. ment of pedagogic advisors. After 1998, the Last, although the Bank financed many Bank contributed to improving educational textbooks, inefficient distribution and training quality by financing textbooks, recruiting have kept Malian students from having even one pedagogic advisors from among experienced book in each core subject at the end of the case teachers, supporting curriculum reform and study period. associated teacher professional development, The education system will be dependent on and supporting the expansion of the bilingual donor aid for the long term. This was ensured education reform. by the government of Mali and Bank policy shift The Bank’s interventions have not been away from low-cost nonformal programs in the effective at improving classroom-level con- 1970s and the continued globalization of the ditions or student learning. Investments were system in the 1990s. The government and made in relevant areas (textbooks, teacher donors are pushing the system to an even higher education, and school health), but procurement level of dependence, with the attempted delays and the focus on the central level rather computerization of operations at all administra- than the service delivery level have kept them tive levels, ongoing purchases of expensive from having a substantial impact. The devolu- vehicles, and planned technology centers in tion of funds directly to schools—focusing more teacher education institutions. The functioning attention on the needs of teachers, allowing of some services within the ministry has schools to “contract” for services with local improved, but none of it will be sustainable education offices, providing professional without further external support. development opportunities to teachers and The counterfactual is impossible to know linking them with career and salary advance- fully. If the Bank had not provided resources ment—are alternative strategies that might have after the collapse of the Soviet bloc, the been more effective. economy could have collapsed, resulting in The efficacy of Bank support was lessened by violence and chaos. However, the adjustment contradictions in policy recommendations and program had a very negative and pervasive effect project-financed activities. Structural adjust- on the opinions of Malians toward the Bank. ment measures reduced the number of Perhaps, in the absence of Bank and other donor teachers in the sector. Contract teacher recruit- support, the government of Mali would have ment was unorganized, and the Bank did not been forced to be more effective. More likely, contribute substantially to contract teacher though, the country would have been even training. These policy contradictions slowed poorer and the system even less functional. progress in improving educational quality. The Based on the results of the case study, the more Bank pressured the government of Mali to useful question at this point in the development introduce double-shift teaching, which permit- of the sector is how the Bank can make better ted rapid increases in enrollment but decreased choices and provide better support. instructional time per student, a major factor in student achievement. Lessons In the early 1990s, the Bank pressured the The story of the World Bank’s support to the government of Mali to limit access to teacher Mali education sector over the past 15 years is a education to high school graduates, but the cautionary tale about the fragility of policy reluctance of graduates to enter teaching dialogue. It underscores how challenging it is to resulted in the near closure of the teacher establish and sustain a partnership whose education program. The government recruited members are unequal in strength and change contract teachers with little preservice teacher frequently. In addition to a number of country- education and struggled without Bank support specific lessons, the Mali experience highlights to provide them with short-term training. This the need for evidence-based dialogue that 91 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA incorporates country perspectives while at exit. As projects or programs come to an end, maintaining core principles about the how will policy dialogue be continued? How can importance of educational quality, equity, and children be made less vulnerable to changing access. priorities? For example, when USAID pulled Country-specific policy dialogue in education back from its community school intervention, must receive sustained attention from sector an absolutely critical program of support in managers, country directors, and regional largely rural areas of the country over the past management. During this period, the Bank was decade, there was little discussion among the too often dismissive of flawed but promising donors, including the World Bank, about how to reform initiatives championed by the govern- mitigate the potential negative impact among ment of Mali that could have been the beginning the most marginalized children. of country-led systemic change. The Bank also did not provide a consistent evidence-based Participation and ownership reform argument. It also appeared indifferent to The approach used by the Bank and other fundamentally important programs managed donors while preparing EdSIP significantly and negotiated by other donor partners, includ- contributed to the development of Mali’s ing most notably the NGO-managed community capacity in policy design. Malians developed the schools program that was the cornerstone of 10-year education policy largely by themselves, USAID support through the late 90s and early with external funding. The process ended with 2000s. the presentation and defense of the program to Support for national languages and preservice the National Assembly, which was a learning and teacher education provides further examples of legitimizing experience for the sector. the Bank’s inconsistent policy advice. The use of national languages as a medium of instruction Conclusions was supported by the World Bank in the mid ’70s The Bank has been a major contributor to and then dropped until 2000. Similarly, pre- educational finance and policy change since service teacher education, initially supported 1990. Without its involvement, system ex- heavily by the Bank, went through a long period pansion would not have occurred as rapidly as it of neglect and hostility that resulted in the did, and educational policy may have remained closing of much of Mali’s preservice teacher incoherent and based on disparate, donor- education program. Only under EdSIP did the driven interventions. Nevertheless, the quality Bank begin to rethink its decisive earlier move of Bank intervention has been inconsistent and away from preservice teacher education; no has also had negative effects. The Bank has significant reform has yet occurred. generally been unsuccessful in leveraging its lending program into direct impact on student Donor coordination experiences at the classroom level. Ultimately, donor coordination may be less The careful choice of approaches would important than donor coherence. Over the past increase efficiency and improve children’s lives. 10 years, donors in Mali have taken advantage of This approach can be based on strong in- frequent consultations and dialogue to improve country and regional ongoing analytic work; the coherence of their interventions. While reading of recent, peer-reviewed education harmonization of procedures has proven elusive, literature and studying of successful programs; the increased cooperation among donors has led and using impact analysis, combined with to a far greater sense of shared purpose and goals achievement-focused incentive systems for among donors and government. teachers, technicians, and administrators. Business as usual is unlikely to deliver the Quality at exit breakthroughs needed to reach sector goals. An Much is made of the importance of quality at intensified sector dialogue, widespread entry. Similar attention must be given to quality adoption of policy initiatives such as the 92 APPENDIX G: CASE STUDY SUMMARIES bilingual education program, and a significant household virtually eliminates the poverty gap increase in resource availability at the classroom between rural and urban households, indicating level are the only avenues likely to lead in the that rural areas have a greater concentration of medium term to significant change in the Malian households with lower economic potential. educational context. Employment Romania The dynamics and configuration of employment affect families’ demand for education. Since the Context transition Romania has seen the emergence of Over the last 15 years, two huge events have three perverse labor market trends: (1) impacted all dimensions of Romania’s migration of active workers into subsistence economic, political, and social life: the 1989 agriculture and other low-productivity/low- collapse of communism and of Romania’s earnings activities; (2) declining participation in autarkic regime and Romania’s drive to join the the labor force through retirements, early retire- European Union (EU). ments, and discouraged workers; and (3) flows out of employment and into long-term Economic trends unemployment. In the 1990s the government of Romania’s Romania started the transition with the commitment to reform, especially to economic largest share of employment in low-skill agricul- reforms such as privatizing the large number of ture among the central Eastern European state-owned enterprises, was vacillating and countries, and its employment structure deteri- ineffectual. By the end of the 1990s, real GDP orated in 2001 compared with the distortions was 83 percent of its 1990 level; the total poverty observed in 1989. The lack of labor reallocation rate had peaked at about 36 percent; and the opportunities in the nonagricultural sectors has extreme poverty rate was 14 percent. In turned agriculture into the labor employer of response to a greater commitment to economic last resort. Natural resources and unskilled labor reform, GDP finally regained and exceeded its also dominate (76 percent) the input composi- 1990 level in 2001. By 2004 per capita income tion of exports. was estimated to have returned to its 1989–1990 Romania’s employment structure and the level; poverty and extreme poverty had factor intensity of its exports suggests that declined; inflation had declined dramatically; during the review period Romania straddled two the banking sector was on firmer ground; and stages of economic growth—factor-driven and privatization of state-owned enterprises had investment-driven—and was in a low-skills, low- accelerated. wage equilibrium. These realities imply a dampening effect on family and business Poverty trends demand for education and on the government’s Even by 2002, 3 out of every 10 Romanians were motivations to reform education at least until poor and 1 out of 10, extremely poor. At the the late ’90s. Moving further into investment- same time, there is a strong positive association driven growth and innovation-driven growth will between economic growth and poverty accelerate business demand for skills and family reduction. Several variables predict poverty, but demand for education. multivariate regressions show that the key correlate of poverty is education; Roma ethnic- Demographic trends ity and being unemployed are second and third Between 1990 and 2035 the number of basic in importance, respectively. Rural residents have school-age children is projected to decline from more than double the probability of being poor 3.3 million to 1.8 million—a 45 percent decline. than urban residents, and rural areas account for However, most of this decline will have occurred 67 percent of total poverty. However, controlling by 2005, with the population of basic education for types of primary income earners in the age in 2005 being 37 percent less than it was in 93 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA 1990. Although the basic school-age population education. The period of 1990–1995 was a is projected to continue to decline until 2030, clarifying phase, with the reform gathering force the downward trend is much more gradual in the 1995–1999 period. The 2000–2003 period between 2005 and 2030. saw some reversal of the reform. EU accession process World Bank Support for Expanding and Romania applied for EU membership in June Improving Basic Education 1995; its entry into the EU is scheduled for The World Bank started helping Romania reform January 1, 2007. Government policies and basic education from early in the transition. The actions are driven by the EU accession process, Bank’s 1991 sector work showed that that the specifically by the EU’s Acquis Communautaire sector needed a comprehensive approach to that is designed to bring new entrants’ practices reform, with key priorities being pedagogy and in line with those of EU countries. The Acquis curriculum. This analysis led to a tightly does not provide a coherent strategy for social connected set of projects: the Education Reform policy issues, including health, education, and Project, approved in 1994; the School Rehabili- poverty reduction. It leaves these issues mostly tation Project, approved in 1997; a pilot focused to the individual countries. on rural schools, funded by reallocating The run-up to accession has consequently Education Reform Project funds; and a Rural “crowded out” the government’s attention to a Education Project, approved in 2003. number of areas outside the Acquis that are critical for Romania’s sustainable development, Summary of Recent Changes in Basic including education. EU accession poses a Education in Romania significant unfinished education agenda for Several players were involved in Romania’s basic Romania, as evidenced by the fact that its 15- education reform during the review period: the year-olds performed poorly on an Organisation Ministry of Education and Research (MER), the for Economic Co-operation and Development Minister of Finance, Parliament, semiautonomous learning assessment particularly relevant to EU agencies and NGOs, local governments, school accession. About 70 percent scored below the staff, parents, and students. They differed in their level that seems required to function in a commitment to the reform and in the “rules of modern workplace, in contrast to 37 percent of the game” (formal and informal), the organiza- EU 15-year-olds. tional infrastructure, and the skills and knowledge that they built to sustain and deepen National Goals for Basic Education it. For example, the semiautonomous agencies Romania was among the first Central and and NGOs provided the steadiest commitment to Eastern European countries to initiate compre- the reform and constituted its early technical hensive, large-scale education reform. Reform leadership. Over the review period they built a goals included introducing a flexible national cadre of professionals competent in curriculum curriculum, alternative textbooks, a private design, educational measurement, teacher textbook publishing industry, the teacher development, textbook quality, educational training required to change classroom practice, management, and rural education. the head teacher/principal and school inspector However, the MER still betrays a tension training required to institute quality-focused between the old concepts of command and management of the schools, a national assess- control and those of providing policy ment and examination service, and rationalized frameworks and oversight. Accordingly, it still management of the school infrastructure and lacks capacities required to function effectively reducing the quality gap between rural and in its evolving role under decentralization: data urban schools. on sector performance, policy analysis, evalua- Commitment to reform varied with the politi- tion, and strategic planning that are used in cal party in power and the specific minister of policymaking; a strong financial management 94 APPENDIX G: CASE STUDY SUMMARIES capacity that can give the MER an advantageous [TIMSS] and PISA) show a more negative seat at the table in Ministry of Finance budget picture. Romania’s eighth graders participated negotiations; and a modern human resource in the TIMSS in 1995 (before the reform), 1999 management system. (the reform was being vigorously implemented), and 2003 (conclusion of the 2000–03 period in Basic Education Outputs, Outcomes, and which efforts were made to stop or reverse Their Implications for Household Demand many aspects of the reform). Romania’s TIMSS results are virtually flat across the eight years, Enrollment and graduation rates for basic education and Romanian students performed less well in Romania started the transition with respectable mathematics and science for each of the three GERs for basic education and has managed to rounds than the average for all participating increase them steadily across the review period European and Central Asian countries. to about 100 percent. Total net rates are very It is not clear what the TIMSS series tells us close to gross rates; graduation rates from about the reform. Romania’s reform could not eighth grade are solid at about 96 percent but have started to affect the schools until the remained relatively flat from 1994–95 to curriculum reform started being implemented 2003–04. in the 1998–99 school year. Thus, there was no Enrollment rates vary by individual and reason to have expected an effect for the first household characteristics. Gender has no effect, two rounds of TIMSS. “No effect” can also signal and moving from the second through the top an unevenly or poorly implemented reform. consumption decile has only a modest effect. From 2000 to 2003 the MER minister tried to Rural residence depresses enrollment rates, but reverse aspects of the curriculum reform, the effect is not large, whereas other variables creating confusion at the school level that have significant negative effects: being Roma, vitiated or clouded the learning impact of the being handicapped, being extremely poor or in reform. The problems with in-service training of a household in the lowest consumption decile, teachers undercut the planned link between the having no parent that has attained more than new curriculum and teachers’ actual classroom primary education, or coming from a household practices. Finally, international evidence shows with a large number of people 0–14 years of age. that student performance often drops in the first years of a major reform simply because any big Repetition and dropout rates for basic education reform is inevitably “messy” as teachers and Consistent with the Region, Romania’s repeti- students struggle to grasp its implications. tion rates for basic education are low, stabilizing PISA is especially relevant to Romania’s at around 3.5 percent. From 1990–91 to 2003–04 aspirations to join the EU because it measures dropout rates, defined as the ratio between the skills valued in innovation-based economies. difference in the number of students enrolled at European and Central Asian countries generally the beginning and at the end of the school year, did not perform well on PISA, but Romania remained at or below 1.5 percent. tested below the Europe and Central Asia Region average and well below the EU average. Learning outcomes Particularly disturbing is that about 70 percent The 8th-grade (capacitate) and 12th-grade of Romania’s 15-year-olds performed below (baccalaureate) exams, structured to measure level 3—that is, at levels 0, 1, or 2. Scoring at the achievement of the curriculum’s learning level 3 or higher generally seems required to standards by subject and grade, have function in a modern workplace. EU students respectable and relatively stable pass rates: each had double Romania’s chance of performing at year about 90 percent pass the capacitate, and levels 3–5 (63 percent). about 96 percent, the baccalaureate. Romania has a substantial amount of work to The international assessments (Third do if its schools are to create the human capital International Mathematics and Science Study that Romania needs to compete economically in 95 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA the EU and its citizens need to avail themselves has supported education, but the education of higher wage job opportunities in the EU. sector has needed intersectoral attention that thus far has failed to materialize—for example, Labor market outcomes public administration, public expenditure and In 2004 the unemployment rates for primary financial management, decentralization, labor education graduates were relatively low, which markets, and rural development. may be correlated with the low unemployment Relative to the projects’ objectives, the rates in rural areas. The most vulnerable to Bank’s education assistance performed well unemployment are those in the middle of the except for teacher training, a problem that educational attainment distribution, that is, reflected more on the borrower than the Bank. individuals with lower-secondary, some high Despite efforts in 2000–03 to reverse the school, or high school and vocational (secondary reform, it has had measurable impact on or post-secondary) education. However, multiple concepts, incentives, and capacities. The regression estimates do not yield much evidence Implementation Completion Reports (ICRs) that schooling is systematically correlated with the assign solid ratings for outcomes, institutional likelihood of unemployment, regardless of age. development, and Bank performance; IEG The returns to an additional year of schooling ratings are consistent with or higher than across the period 1960–2000 were fairly flat those assigned by the ICRs. At the same time, between 1966 and 1989, but they more than the Bank significantly underestimated the doubled between 1989 and 2000. The data are magnitude of conceptual changes (“habits of not consistent with standard explanations of this thought”) that European and Central Asian upward trend, such as constrained supply of countries had to undergo if they were to better-educated workers, product shifts, or skill- establish market economies and democracies. biased technical change. Under communism The design of the Education Reform Project wages were compressed—that is, wages did not unfortunately did not include a sustained reflect variations in human capital. It is possible public relations campaign around its that the trend since 1990 simply signals the objectives. The concepts behind the reform predictable decompression of wages that occurs were alien to players conditioned to a highly with the introduction of prices. centralized command and control system. Interviews with Romanian counterparts left Household demand for education no doubt that Romania’s basic education would Household demand for basic education is not have made the progress achieved in the last generally high, as evidenced by enrollment 15 years without the consistent support of the rates, graduation rates, dropout rates, and Bank. One particular interview revealed the learning performances on the eighth-grade basis for these shared views: examination. Demand is variable. Being Roma has an independent and highly negative effect It was not just the money that was important. on demand, especially in urban areas. Although If we learned anything, we learned it from rural areas have lower enrollment rates than the World Bank teams. These teams helped urban areas, it is characteristics of households the country understand the concepts behind more prevalent in rural areas, not rural the project and helped us design and residence itself, that depress demand. implement the project. This is a major differ- ence between the Bank and other donors. World Bank’s Contribution to Sectoral Changes Other donors usually send consultants who (1990–2004) do their work and then leave. The World The Bank’s education lending has been Bank, on the other hand, builds groups of coherent and highly relevant. The relevance of Romanian specialists that can contribute to the Bank’s work in other sectors that affect activities other than World Bank activities. education is another story. Every CAS since 1993 Romania will need the Bank’s support even 96 APPENDIX G: CASE STUDY SUMMARIES after Romania joins the EU. You can’t find the needed to rationalize reforms of Romania’s World Bank’s expertise in the EU. World education system. Bank staff is highly committed and fine The failure to include the sector in any specialists…. There are only good lessons completed or planned public expenditure from the Bank. reviews is inexplicable. The MER is struggling ineffectually with the sector’s fiscal issues, and Romania’s drive to join the EU raises the neither the government of Romania nor the question of whether the Bank will continue to Bank has grounds for evaluating intersectoral have a role in education in Romania. The QAG allocations as they affect the education sector. Country Lending Enhancement Review (World The sector needs help with its fiscal framework Bank 2004c) concluded, “After accession the for decentralization, but decisions here have to social sectors—broadly defined—may be the be aligned with a larger decision framework. The most important niche for Bank involvement, as lack of a country team/government agreement the substantial EU resources will be focused on on a rural strategy undercuts efforts to support other sectors.” rural education and the Roma who live in rural areas. The work on EU integration has done little Lessons to pursue the human capital demands of Romania’s integration into the Union—for • Firmness and flexibility need to be balanced in example, rural-urban gaps in educational project negotiations. achievements or the implications of the PISA • Complexity can advance implementation if the results. design is coherent. The cross-sectoral problem is not unique to • The Bank may have a role in sustaining proj- the Education Sector or to Romania. IEG and ect achievements. QAG have conducted a number of country • Creating new partners among NGOs and semi- performance assessments that reveal that the autonomous agencies builds capacity that tends Bank’s matrix system is not working well. The to be sustained. country team was expected to create cross- • The government of Romania will need to exert sectoral collaboration around agreed-upon more donor coordination, especially for Roma problems that the country needed to solve. It projects. has proved very difficult to make this concept • Building support for reform is especially needed work well. for projects with long time frames or those There is agreement that even after Romania that are implemented under decentralization. joins the EU, the World Bank has a role in • The meso-level is important to successful ed- education. Romania has skill-level problems to ucation reform. solve to enable its entry into the Union. It cannot afford to focus solely on the Acquis, ignoring Conclusions factors outside of the Acquis that directly affect Although the Bank’s education team has had a its chances of solving problems within the significant and positive effect on Romania’s basic Acquis. Education is one of those factors. If the education system across the 15 years reviewed, Bank’s management for Romania chooses to fill the overall Bank gets a lower grade. Heretofore, the vacuum created by the Acquis, the Bank has the Bank’s management has not solved the problems of intersectoral collaboration to solve admittedly difficult “silo” problem that if it is to help the country address challenges in undermines the cross-sectoral collaboration the education sector. 97 APPENDIX H: EXTERNAL ADVISORY PANEL COMMENTS Response 1: Merilee S. Grindle, Beatrice planning, monitoring, and assessment. The Okyere, and Paulo Renato Souza report indicates that project designers need to The external panel welcomes this report. The pay more attention to the inclusion of appropri- report emphasizes the importance of the ate management incentives. Additionally, the Education for All (EFA) agenda established in report focuses attention on the all-too-frequent 1990 and reaffirmed in 2000, as well as the signif- failure of projects to include appropriate politi- icant place of primary education in the Millen- cal and institutional analyses as part of planning, nium Development Goals (MDGs) and, more monitoring, and assessment processes. broadly, their importance to the process of Overall, the report makes a strong statement development and poverty alleviation. In about the increasing importance of emphasizing addition, the report makes very clear that access the quality of education through project to education is not sufficient for meeting objectives that include important outcome important goals of equity and fairness in promot- measures. This is an important emphasis, and ing the life chances of the poor, girls, those who one that is particularly difficult for many govern- live in remote areas, and other disadvantaged ments and education experts to attend to, given groups. Too frequently, access has been the pressure of achieving the EFA and MDG promoted at the expense of quality in education. goals. The report acknowledges the difficulty of This Independent Evaluation Group report promoting quality at the same time that access is rightly emphasizes the importance of ensuring being expanded. As the World Bank considers that children not only attend school, but develop the general recommendation of the report, it the skills and knowledge base that will allow must address how access/quality tensions can be them to live productive and rewarding lives. effectively managed without sacrificing The report appropriately suggests that important equity goals. The report urges the greater attention needs to be given to improving simultaneous pursuit of both goals; the experi- school outcomes and using outcome measures ence of most countries, however, is that this is as centrally important vehicles for determining extraordinarily difficult. Pressures for access program success and for making adjustments to strongly tend to crowd out a focus on quality, projects as they are being implemented. In and, although there is less experience with this, addition, the report is valuable for its finding a focus on quality can easily increase inequity in that World Bank projects that were focused the delivery of education. We strongly urge Bank specifically on education tended to perform education specialists to address this issue better than multisector projects that included through research and innovative initiatives. education along with other reform activities. It is An issue related to the report’s emphasis on also important that the report emphasizes the quality and outcomes measurement raises importance of educational management, partic- another issue that is not directly addressed in ularly the need for governments to invest more the report—that of the time required for reform in the acquisition of pertinent and up-to-date projects and programs to produce effective information about schooling in their countries results. It may, in fact, take 5–10 years before and to use this information more effectively for improvements in the quality of education begin 99 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA to be clearly visible. This means that projects and important goals of the EFA and MDGs are to be programs may need to be based on longer-term reached or even approximated. commitments. We strongly urge Bank education specialists to consider if the timing of the Bank’s Response 2: David Archer projects is realistic when outcome measures I welcome this evaluation of World Bank invest- become a more important objective in those ments in primary education but feel that the projects. final report fails to capture some of the signifi- The emphasis in the report on quality, cant insights gained from the preparatory work welcomed by this external panel (along with very and country studies. Moreover, some of the real concerns about the remaining large gaps in concerns raised by the external panel over the access in a large number of countries, among past 18 months have not been adequately them the poorest in the world), stresses the addressed in this final report, which is importance of outcome measures and school somewhat too single minded in its focus on management. At the same time, however, the learning outcomes. report provides only brief insights into what most Clearly, learning outcomes are important— education experts agree are factors central to no one will disagree with this. The question is good-quality education—the teacher and the how to operationalize this new focus, and the classroom. Teachers—their recruitment, training, evaluation gives few orientations for this. Does it deployment, ongoing professionalization, and mean less attention should be paid to access? representation in the political sphere through The report claims not—asserting that expand- their unions and associations—are, in the final ing access and improvements in learning analysis, probably as important as factors relating outcomes do not have to be traded off against to the efficiency of educational management. The each other—but it is not very strong on this conditions, training, and incentives that affect point. I would go further and say they must not their performance linked to student achievement be traded off. When 100 million of the poorest should be placed at the center of any project that children remain out of school, shifting our focus purports to improve the quality of education. from access to outcomes would have serious Similarly, curriculum materials, class size, and implications for equity. The report should have hours of instruction should be much more paid more attention to the remaining challenges central to projects than they appear to have been. in achieving universal access to primary We encourage Bank education specialists to focus education. more attention on efforts to work with teachers Part of the concern here comes from a and their associations, to facilitate the profession- worrying subtext that suggests that measures alization of the teaching corps, and to increase such as the abolition of user fees are bad ideas that group’s ownership of education reform because they impact negatively on quality (that initiatives. is, progress on access has undermined These two concerns—the importance of outcomes). This may be the case, but it ignores quality and the centrality of teachers—suggest the fact that education is a fundamental right that the report could argue more forcefully for (embodied in most national constitutions as the importance of increased spending on well as international treaties such as CRC) and education. While there are undoubtedly efficien- that charging children to go to primary school is cies that can be achieved in many education the most blatant violation of that right. The systems through better management and use of sooner fees are abolished the better, and this resources, it is unlikely that such improvements should have been stated simply and clearly could provide sufficient funding for the rather than urging caution. No single measure infrastructure, salaries, materials, and other has such a dramatic impact on equity within an inputs into education that are needed. We education system—bringing millions of poor strongly urge the World Bank to acknowledge children into school. Rather than questioning the need to increase funding for education if the the wisdom of governments (or political 100 A P P E N D I X H : E X T E R N A L A D V I S O R Y PA N E L C O M M E N T S leaders) taking such abrupt measures, the The term “local teachers” that creeps in seems emphasis should be on ensuring a rapid to be an attempt at rebranding para teachers or response (with coordinated international aid contract teachers. In fact, the spread of these through mechanisms such as the Fast-Track “non-professional teachers” (a more accurate Initiative [FTI]) to situations where fees are term) is something that the Bank has actively abolished so that quality is not affected. supported in recent years, often with a very Unfortunately, this evaluation, spanning 16 negative impact on learning quality. In the final years, ignores the Bank’s own role in the contro- report there is no analysis of Bank interventions versial issue of user fees in education. It should in this area or of how they have sometimes have done a more systematic job in scrutinizing actively undermined the teaching profession. the Bank’s positions as they have shifted over For example, the Mali study documented how the period—and it should have been unequivo- the Bank’s Voluntary Departure Program led to cal in calling for abolition of all costs that prevent the loss of 12.5 percent of the teaching poor children from going to school. workforce (even at a time of expanding There is a danger that the shift of attention to enrolments), and the Bank did nothing to stop outcomes will be seen as a substitute for much- the closure of teacher education institutions. needed attention to inputs. One effect of this Instead, the Bank explicitly supported the hiring attention on outcomes may be to massively of unqualified non-public-service teachers and increase investment in testing of pupils (which in did nothing to support their training. There are itself does not contribute to learning) rather than many other examples of the Bank promoting to focus on inputs that might really improve non-professionals, and these should have been learning. Most inputs are obvious: ensuring that more closely documented in the final report. there are sufficient numbers of well-trained On the positive side, the final report does teachers who are teaching classes with manage- include a qualifying refrain that calls for more able numbers, and sufficient books and learning local teachers “as long as those teachers have materials in enough classrooms. The report fails access to professional growth opportunities to highlight the extent to which the Bank’s focus and job security”—something the Bank has since 1990 has been too narrowly focused on the failed to do in the past. It is also good to see the last of these—infrastructure—often at the call for more “evaluative research” on contract expense of other inputs. teaching and to see some of the concerns raised Perhaps the biggest omission in this report is about whether it is cost-effective, equitable, or in regard to the most important input: teachers. sustainable in all settings. But if this report is Many of the country evaluations documented serious in its call for quality learning outcomes, the deterioration in teacher quality and teacher then it should have been much more systematic conditions in recent years—and the failure of in looking at the teaching profession and the Bank to pay sufficient attention to this. The challenging the introduction of unqualified call for focusing on quality outcomes should teachers. The moderating clause calling for naturally lead to a call for a renewed focus on “professional growth and job security” feels quality teachers, but it does not. As it is, very tokenistic in this regard, failing to call for little attention is paid in this final report to the minimum requirements or time-bound critical issues of teacher recruitment, training, processes of qualification. In practice, non- retention, or deployment. professional teachers are being seen as a long- Rather, in places the report seems to do the term cheap labor solution in many countries, opposite, promoting the hiring of “local and this has a devastating impact on the teachers” as an effective measure. The Executive teaching profession as a whole—undermining Summary says “recruitment of local, often status and morale and destroying teacher associ- untrained, youth” is one of the “most promis- ations and unions. This is probably the biggest ing” measures, and elsewhere the “high dedica- single threat to achieving quality learning tion” of these contract teachers is celebrated. outcomes for all children. 101 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA It is self-evident that “what matters in the freezing on hiring of teachers in Pakistan or education is what happens in the classroom.” If low spending in Peru linked to IMF policies. This the Bank accepts this, then the quality of fits with the experiences of many other teachers should be at the center of its countries (see Marphatia and Archer 2005). attention—that is, unless the Bank is ready to Governments cannot even contemplate the take a dose of its own medicine and start hiring “trade-offs” between a rise of one percent in para-economists…. inflation and the recruitment of more teachers, One of the interesting elements in this evalua- as the inflation target is sacrosanct. The IMF tion is the recognition that decentralization talks openly of the “sacrifice ratio,” whereby policies and programs seem to have led to investments in education and health are “increases in inequities across income and social sacrificed in the name of macroeconomic stabil- groups.” There also seems to be a new recogni- ity. It is important for the World Bank to take a tion of the need for strong management in stand on these contradictions and to use its central ministries. This certainly warrants further influence with the IMF to seek solutions. research. Unfortunately, no effort is made to Building new schools is of little value if govern- address the evident tensions between these ments are at the same time blocked from observations and the call for “local teachers.” employing new teachers. The Bank should be More research is also needed on the impact of championing the benefits of investment in private schooling, the spread of which (as inputs education and helping countries remove the to this evaluation have clearly suggested) is constraints that prevent them from making such undermining equity gains in the public sector a sound investment. (especially in relation to gender inequity). It is On a related issue, I welcome the recognition time for the Bank to be explicit in its support for in the report that an increased focus on learning public education and to acknowledge that the outcomes will “raise the unit costs of primary achievement of education goals will not come education.” There is a call for the FTI “to through the spreading of private provision. develop cost and funding gap estimates” that One underlying problem here is that the recognize this increased cost. This coincides Bank has failed to address the contradictions with the commitment, in Abuja in May 2006, by between International Monetary Fund (IMF) ministers of finance from 20 African countries to macroeconomic prescriptions and achieving develop ambitious 10-year plans to get all education goals—and this final report fails to children into school. There is growing explore this critical issue. Some of the country momentum here, building on the British studies commissioned for this evaluation government’s recent pledge of $15 billion in showed these contradictions clearly, for predictable aid to education. One key element example, where the Bank built schools but, of all this, which the report fails to pick up on, is because of IMF limits on public sector wages, predictability. In the past, aid to education, there were no teachers to teach in the schools including from the Bank, has not been long term (for instance, in Pakistan, Peru, and Mali). The or predictable, so it has not been possible for recruitment of non-professionals as cheap labor countries to spend the money on what they is presented as unavoidable in situations of need: the recurrent costs, particularly teacher increasing enrolment, when new teachers are salaries, which are the vast bulk of primary needed but the government cannot increase its education spending. As aid to education spending on salaries. In fact, there should be becomes more predictable, countries should be more attention paid to why wage bills are able to spend it on recruiting more teachers— capped in the first place. but this will be impossible unless wage bill caps The country studies done for this evaluation and other macroeconomic conditions are show again and again that Bank investments in removed. education have been undermined by macroeco- I welcome the considerable attention paid by nomic constraints on governments, whether it is this report (at least in its recommendations) to 102 A P P E N D I X H : E X T E R N A L A D V I S O R Y PA N E L C O M M E N T S the FTI—but regret that the report fails to call agenda. There is an increasing investment in directly for the Bank to put its own money into secondary and particularly higher education, it! In line with the Paris Aid Effectiveness and the policy attention to these areas suggests guidelines, the FTI is an important means for that they will increasingly attract a larger share coordinating donor responses to education, and of the resources from the existing education the report should logically call for the Bank to budget. The focus on the knowledge economy align International Development Association/ is already attracting significant staff time and Poverty Reduction Support Credit funding resources that would previously have been behind FTI-approved national education plans. focused on primary education. Lending to Rather, the emphasis is placed on influencing primary education has actually fallen in the the FTI to include learning outcomes as indica- period 2000–2004 compared with 1995–1999. tors/benchmarks/targets. As it is, the report does Moreover, direct lending to primary education not give sufficient evidence to argue that indica- has fallen significantly. It is only lending from tors such as instructional time, teacher other sectors (that include some component of attendance, and availability of textbooks are the education work) that prevents this decline from key ones for improving learning outcomes. The being very dramatic and evident. This indirect more important reforms of FTI lie in ensuring support for education from other sectors is that it can make long-term commitments (for often very narrowly focused on infrastructure example, moving beyond the short-term aid of and is likely to have no impact on learning the catalytic fund), that it addresses the full EFA outcomes. The fall in spending on primary agenda, and that all donors live up to their education should be explicitly opposed. The promises to increase and better coordinate their achievement of quality universal primary aid to education. education (UPE) must remain the first and most The country studies show that, despite fundamental priority for the Bank’s education widespread rhetoric about donor coordination, work. in practice donors have not been good at this Of course the real constraint here lies in the (and the Bank has not helped) and that donor fact that the Bank continues to underinvest in power has often diminished the accountability the education sector as a whole. The FTI of governments to their own parliaments/ recommends countries should invest 20 percent citizens. This needs to change, and this should of their funding in education (and the Bank have been at the center of recommendations widely supports this position) —yet the Bank from this evaluation. itself spends just 7 percent of its own budget on One reason for the Bank to channel more of education. Why not 20 percent? its own support through the FTI is that it has An increase in the Bank’s spending on not been very successful in allocating money education will certainly be needed if it is to where it is most needed. Since 1990 the most respond to learning outcomes—and it would rapid growth in borrowing for primary also be essential if the Bank were to take on the education has been in East and Central Europe; full EFA agenda. The World Bank was cosponsor the greatest volume of borrowing now is in of the Jomtien and Dakar conferences— Latin America. But the greatest need is in Africa apparently buying into the EFA framework. Yet a (where increases have been slow and still fall defining part of the Bank’s education narrative short) and in South Asia (where commitments since 1990 has been a repeatedly reductive focus are now reducing). Bank support for FTI- on UPE—sidelining and ignoring adult literacy approved plans in Africa should be a particular and early childhood education. In many priority. documents, including those prepared for this Unfortunately, from my participation in this evaluation, UPE and EFA are conflated. This final external panel, I see an alarming shift in World report should have done more to acknowledge Bank investment away from primary the impact of this, addressing how the Bank’s education—effectively abandoning the MDG focus on primary education has impacted other 103 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA parts of the EFA agenda. It is an unsatisfactory without the Bank adding to the deafening fudge (and a denial of the Bank’s power) to say silence. that the Bank’s contribution to EFA has been In conclusion, I welcome the fact that the through UPE. This is of particular importance Bank has conducted this evaluation, but I feel given the widespread evidence of interdepend- that this final report is incomplete. Critical ency in the EFA goals. The impact of early issues do not find enough space, particularly childhood education and the home environ- issues around the teaching profession, ment on learning outcomes in schools are macroeconomic policies, and the failure of the recurrent themes in the country studies. It is Bank to target resources where they are most clear that little progress can be made on learning needed. The implications of the call for greater outcomes if we fail to consider the role of early attention to learning outcomes are not made childhood education and adult literacy (which clear enough, and the Bank’s past, present, and are key to the home environment). future global role in basic education is not Another key gap in this report concerns adequately analyzed. From earlier discussions I HIV/AIDS. It is shocking that most country gained the impression that this evaluation studies did not raise HIV/AIDS as an issue, would call for greater engagement by the Bank despite this being in the terms of reference. The with the IMF on questions of fiscal space—so impact of HIV/AIDS on education in the past 15 that countries are facilitated in making the long- years is one of the biggest developments in the term investments in education that will yield sector, particularly in Africa. The role that long-term economic returns. This remains the education plays in helping respond to HIV/AIDS central challenge in a world increasingly driven is crucial, yet still underregarded. The final by short-term financial planning. But sadly, this evaluation report should, at the very least, make has not materialized in the final report. It is a a big issue of the fact that the reports did not missed opportunity to address the strategic touch on HIV/AIDS. There is enough ignoring of issues that are undermining progress on quality HIV/AIDS in the education sector already, education. David Archer Head of International Education, ActionAid, and Chair, Commonwealth Education Fund, London, U.K. Merilee S. Grindle Edward S. Mason Professor of International Development, Kennedy School of Government, Harvard University, Boston, MA, U.S. Beatrice Okyere Senior Lecturer in Special Education/Teacher Education at the University of Cape Coast, Ghana Paulo Renato Souza President, Paulo Renato Souza Consultants, and Former Minister of Education, São Paulo, Brazil 104 APPENDIX I: MANAGEMENT RESPONSE Introduction countries themselves have collectively moved Management welcomes this IEG review of World along the continuum from supporting invest- Bank support to primary education. The ments in educational infrastructure and objectives of the review are to (a) assess World educational inputs to increasing education Bank assistance to countries working to improve system outputs, improving instructional quality, their basic knowledge and skills base by provid- and raising learning outcomes. While manage- ing primary education, and (b) provide lessons ment acknowledges the many challenges that for countries in their development strategies hinder the achievement of higher learning and for the Bank in its support to those strate- outcomes, it believes that the case studies gies. Early findings of the IEG review have been examined in the IEG review demonstrate that it incorporated into the 2005 Education Sector is addressing the right issues and has made Strategy Update. measurable progress in focusing attention on instructional quality and learning outcomes. Coverage. The evaluation covers the last 42 years, 1963–2005, a period during which lending Main Findings and Recommendations for primary education amounted to about $14 The IEG evaluation makes three key recommen- billion. Nearly 90 percent of Bank lending for dations: (a) primary education efforts need to primary education has occurred since the focus on improving learning outcomes, particu- beginning of the Education for All (EFA) larly among poor and other disadvantaged movement, which started in Jomtien, Thailand, children; (b) efforts are urgently needed to in 1990; and about two-thirds of this lending has improve the performance of sector management been in the form of IDA credits. A growing share, in supporting learning outcomes; and (c) the currently about one-third, of primary education Bank needs to work with its development lending has been through components of partners to reorient the Fast-Track Initiative (FTI) projects managed by sectors other than the to support improved learning outcomes, in Education Sector. parallel with the Millennium Development Goals’ emphasis on primary completion. Management Management Views. Management concurs agrees with these recommendations. At the same with the report’s conclusion that “to the extent time, to emphasize the evolution of Bank that public investments in primary education are assistance, management would like to highlight effective in conveying learning outcomes, several issues for further consideration. support for primary education is central to the World Bank’s mandate of poverty reduction.” Attention to Learning Quality. Management Management suggests, however, that the report agrees that Bank assistance and analytic work could have paid greater attention to the extent need to give more consistent attention to to which Bank assistance has evolved over time learning quality and to the measurement of toward a more focused emphasis on results. learning outcomes. On the other hand, manage- Over the 40 years of the review period, the ment suggests that the report would be World Bank, other donor partners, and the client strengthened by more clearly acknowledging 105 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA how the Education Sector has been moving should be clear that the inferences that can be consistently and systematically in this direction, made about an individual’s basic knowledge and while remaining cognizant of country ownership skill from such an assessment are quite limited. of the programs. The report states that “only The important point to be kept in mind is that one in five projects aims to improve learning projects aimed at improving student learning outcomes.” This bold statement does not outcomes need to invest appropriately in adequately capture the changing emphasis in measurement, analysis of results, and use of this the sector. For example, in Latin America in 1990 analysis to improve quality. there were three countries with functioning student assessment systems; by 1999, largely Quality Enhancing Elements in because of World Bank support, 18 countries Operations. Finally, the Education Sector has had functioning assessment systems, and several also been increasing its support for quality- were carrying out analysis of the results for enhancing inputs. Of 24 projects approved in feedback into instructional quality. Of 24 fiscal year 2006, 21 included provision for education projects approved in fiscal year 2006, teacher training: the Djibouti School Access and 17 provided support to improve capacity to Improvement Project, for instance, will “support measure student learning. In addition, because (i) training of teachers and (ii) in-service teacher of the Bank’s efforts (through the Development training (upgrading of skill improvements Grant Facility) for the International Association including modules on how to identify and of Education Progress (IEA) and the UNESCO address the problems of children with learning Institute for Statistics, IEA’s two assessments— difficulties and/or with special needs).” Eleven TIMSS (the largest international comparative of the 24 fiscal year 2006 education projects assessment of student outcomes in mathemat- include elements of school development grants; ics and science) and PIRLS (the internationally 12 provide support for research studies on recognized leader in studies of reading literacy current education issues, and all include achievement)—are supporting the participation management improvement training. Since fiscal of low-income countries in their comparative 2004 the Africa Region has used the analyses in international assessment programs. Country Status Reports—showing the weak relation between learning outcomes and Differing Views on Rapid Assessments. The spending—to motivate a whole new work report may overstate the case for low-cost, rapid program on improving education management assessments. There is clearly a need for such in African countries (known by its French assessments as part of a broad toolkit of instru- acronym, AGEPA, or Amélioration de la Gestion ments available to education planners, but it de l’Education dans les Pays Africains). 106 APPENDIX I: MANAGEMENT RESPONSE Management Action Record Major IEG Recommendation Management Response Primary education efforts need to focus on improving learning Management agrees with IEG’s recommendation to ensure outcomes, particularly among the poor and other disadvantaged that the Bank’s primary education assistance, whether led by children. the Education Sector or not, focuses to an even greater degree on factors directly related to improving learning outcomes. Management will build on ongoing efforts to strengthen or es- tablish learning assessments at the earliest grades and sup- port the use of these tools to set outcome targets, monitor results across different demographic groups, and use the as- sessment results and other impact evaluations to identify the most cost-effective strategies and interventions to raise learn- ing outcomes. Management will also encourage countries to increase their monitoring of schooling quality standards and unit costs of primary education and to target educational resources so as to reduce disparities in schooling quality standards, in- structional quality, and learning outcomes across different groups of students. In addition, management will ensure that education projects not managed by the Education Sector will attend to instructional quality and learning outcomes. The 2005 Education Sector Strategy Update (ESSU) incorporated ear- lier results from this IEG evaluation, which are reflected in the results framework (ESSU Annexes 8 and 9) and specify how Bank assistance will help countries shift to a substantially greater focus on results and learning outcomes. In addition, manage- ment will introduce in fiscal year 2007 a Quality Review Frame- work for education programs to benchmark and report annually on the degree to which lending and analytic activities meas- ure learning outcomes and focus on improving the quality of teaching and learning. Efforts are urgently needed to improve the performance of Management concurs with the need to raise the quality of in- sector management in support of learning outcomes. country Education Sector management and capacity, including at decentralized service levels, which Bank research has shown to be critical to improving instructional quality and learning out- comes. Management will build on and further propagate the successful experiences of countries benefiting from Bank as- sistance that are implementing actions to (a) improve the ca- pacity of the institutions responsible for recruitment and preservice and in-service training of teachers and school prin- cipals; (b) provide career development and other incentives for improving teacher performance and introduce better recruitment and career development policies; (c) empower schools to effi- (Continued on the following page.) 107 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Major IEG Recommendation Management Response ciently manage their own resources and pursue agreed targets for learning outcomes; (d) use indicators and evaluation tools to diagnose problems in instructional quality and student learn- ing outcomes; and (e) design interventions to overcome these problems. Management will also monitor and report on whether new results-based Country Assistance Strategies include learn- ing outcome indicators. In addition, management has initiated several activities to implement the 2004 World Development Report, Making Services Work for Poor People, including ac- tivities to strengthen governance and accountability and ana- lytic work on school-based management. As detailed in the ESSU (Annex 10), management will develop guidelines, toolk- its, instruments, and software for capacity building and will carry out targeted training programs to disseminate and help coun- tries use this guidance to strengthen their capacity to sys- tematically measure learning outcomes against baselines and targets, evaluate the impact of their programs and interven- tions, and focus their resources more effectively on improv- ing instructional quality and educational results. The World Bank Institute education core course was revamped in fiscal 2006 to focus more directly on results and improved service delivery. In fiscal 2007, management will publish and dis- seminate at least three impact evaluations, a set of guidelines for establishing a system for measuring, reporting on, and uti- lizing measures of learning outcomes to improve educational quality, and two toolkits for rapid reading assessments and school-based management. The Bank needs to work with its development partners to re- Management will continue to work with the FTI development orient the FTI to support improved learning outcomes, in par- partners to help all FTI-endorsed countries strengthen the at- allel with the MDG emphasis on primary completion. tention they give to instructional quality in their programs, ad- minister assessment instruments to measure learning outcomes, and use results from student learning assessments to improve instructional quality in all grades. Management agrees with IEG that the major challenge in rapid scale-up is maintaining and improving quality. Management will work with the FTI devel- opment partners to revise the FTI assessment guidelines and Indicative Framework to promote the use of additional indica- tors of fundamental schooling quality standards, such as in- tended and actual instructional time, presence and use of textbooks or supplementary reading materials, teacher quali- fication and attendance, minimum physical quality standards, 108 APPENDIX I: MANAGEMENT RESPONSE Major IEG Recommendation Management Response student promotion and dropout rates, and learning outcomes. In fiscal 2007, this will include working with the 20 currently endorsed FTI countries to help them implement a rapid read- ing assessment, and track implemented (versus intended) hours of instruction. In subsequent years, FTI-endorsed countries would be expected to report trends in learning outcomes against their baseline measures and report hours of instruction as part of the annual joint donor review process. 109 APPENDIX J: CHAIRMAN’S SUMMARY: COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) On July 5, 2006, the Committee on Develop- discussion by CODE. The current report From ment Effectiveness (CODE) met to discuss the Schooling Access to Learning Outcomes: An report From Schooling Access to Learning Unfinished Agenda is aimed at informing the Outcomes: An Unfinished Agenda—An Evalua- implementation of the ESSU (2005). tion of World Bank Support to Primary Education, prepared by the Independent IEG evaluation. The evaluation broadly covers Evaluation Group (IEG), and the Draft Manage- the 1963–2005 period, with an emphasis on the ment Response. The Statements of the External last 15 years, in which lending for primary Advisory Panel (CODE2006-0063) on the IEG education amounted to about $14 billion. IEG report were circulated as a background makes three key recommendations: document. • Primary education efforts need to focus on Background. A World Bank Education Sector improving learning outcomes, particularly Strategy, prepared in 1999, highlighted basic among the poor and other disadvantaged chil- education for the poorest and for girls; early dren. childhood interventions; innovative delivery; • Efforts are urgently needed to improve the and systemic reform. On June 15, 2005, CODE performance of sector management in the discussed an Update of the 1999 Sector Strategy countries assisted by the Bank in order to im- Paper (Draft Education Sector Strategy Update prove learning outcomes. (ESSU): Broadening Perspective, Maximizing • The Bank needs to encourage the EFA–Fast- Our Effectiveness), which was later endorsed by Track Initiative (FTI) to strengthen its focus on the Board. It highlighted three strategic themes raising learning outcomes in parallel with the to help the Bank meet the diverse challenges MDG emphasis on primary completion. facing the sector: (a) integrating education into a countrywide perspective; (b) broadening the Draft management response. Overall, manage- strategic agenda through a sectorwide ment welcomes IEG’s evaluation and agrees with approach; and (c) becoming more results- most IEG recommendations. It suggests, however, oriented. The updated strategy focused on that the Bank assistance has evolved over time helping client countries (i) attain the Education toward more emphasis on results (education for All (EFA) and the Millennium Development system outputs, instructional quality, learning Goals (MDGs), and (ii) strengthen education for outcomes). In this vein, management believes the the knowledge economy, by building higher case studies examined in the IEG review skills and knowledge needed to compete in demonstrate that the Bank is addressing the right global markets and foster economic growth. The issues, and that the report reflects the trend in report The World Bank’s Assistance to Primary Bank support for educational quality and reforms Education—An OED Portfolio Review was in governance and management to address quality. prepared in 2004 as an input into the prepara- tion of the Bank’s strategy and circulated for Overall conclusions. The Committee broadly information and as background material for its endorsed the IEG findings and recommenda- 111 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA tions and welcomed management’s constructive Next steps. Management will introduce in fiscal and forward-looking draft response, noting that year 2007 a Quality Review Framework for the recently approved ESSU focuses on education programs and report annually on the education quality and results. It also appreciated degree to which lending and analytic activities the staff comments on country experiences, and measure school outcomes. It will also publish was encouraged by the current innovative works and disseminate at least three impact evalua- that are undertaken in the Regions to enhance tions, a set of guidelines for establishing a the focus on quality and learning outcomes. system for measuring and reporting on learning There was broad agreement on the importance outcomes, and two toolkits for rapid reading of learning outcomes, while equal access, partic- assessments and school-based management. ularly for girls and other disadvantaged children, The Bank and its FTI development partners will remained relevant. In this regard, members continue working with the 20 currently took note of IEG’s recommendations to encour- endorsed FTI countries to help them implement age EFA-FTI to strengthen its focus on improv- a rapid reading assessment and track im- ing learning outcomes and universal plemented hours of instruction. completion in its partner countries. They Members raised the following issues during commented on the challenges ahead to achieve the meeting: the MDG calling for universal completion of primary education by 2015. General comments. Speakers welcomed the Overall, there was support for taking a more different perspectives of the External Advisory comprehensive approach to education, as Panel. Although a few speakers commented on emphasized in the ESSU. Such approach would the timing of the IEG report in relation to the include greater integration of multisectoral Bank’s ESSU, it was also noted that earlier factors affecting school attendance and learning versions of the report had been shared and such as infrastructure (that is, the roads, electric- discussed with Bank staff, which served as an ity, transportation) and student health and instrumental input to the ESSU. One member nutrition, as well as more attention to post- asked about the budget implications of primary options (that is, vocational and techni- implementing the IEG recommendations. cal education), and links to the labor market and Others felt the evaluation report could have had demand for skills in a knowledge economy. more detailed analysis of different approaches Members noted the need to further strengthen to improve primary education, structured Bank support to improve education sector according to different categories of countries, management and governance, while under- for example, low-income and middle-income standing the country political and institutional countries, or in urban and rural areas. IEG dimensions. Other comments related to the commented that the report gives broad sugges- high number of primary education initiatives tions on the kinds of intervention needed for supported through multisectoral projects, countries at different levels of development but including development policy lending (DPL) suggests that solutions must be customized to and sectorwide approaches (SWAps); availability meet unique country conditions. of financial and human resources; private sector participation; the role of teachers; and measure- Overall support for primary education. ment and results in the education sector. The Several members noted that some countries are issue of communication and dissemination of currently halfway on the road to achieving the the IEG report, the opinions of the External MDGs. They urged the international community Advisory Panel and the draft management to accelerate progress to reach the universal response were also addressed. In this regard, completion of primary education by 2015. the need to externally communicate balanced Management reiterated its support to the messages emerging from the IEG report to avoid achievement of the MDGs. Some members misinterpretations was underlined. noted the shortfall in financial resources, includ- 112 A P P E N D I X J : C H A I R M A N ’ S S U M M A R Y: C O M M I T T E E O N D E V E L O P M E N T E F F E C T I V E N E S S ( C O D E ) ing for the EFA-FTI, to support the achievement teachers in improving education quality and the of the MDG in primary education. Some need to consider teachers’ training, accredita- speakers also noted the need to consider the tion, and performance incentives. A member private sector role, fiscal space for education, thought that the IEG report could have provided and sustainability issues in the long term. The more analysis of performance of teachers, both importance of coordination and partnership regular and contract. IEG commented that the with other donors was emphasized. report includes references to adequate teacher supply and incentives and issues regarding the Expanding access and strengthening sustainability of contract teachers. A few learning outcomes. The Committee discus- members stressed that the Bank’s interventions sion also addressed the complementary nature should be focused on institutional reform, partic- of education access and quality and the ularly on the reform of labor market for public challenges of doing both. Members strongly school teachers. At a broader level, some emphasized the importance of ensuring quality members noted that more efforts were needed education for all children. At the same time, a to enhance countries’ sector management and few speakers commented on potential tradeoffs governance through DPL and SWAps, while between education access and quality, and recognizing the challenges associated with proposed more research on how to address cultural, institutional, and political dimensions. access/quality tensions; one member cautioned Other speakers also noted the importance of against a wholesale shift away from equity of data and assessment tools for decision making access. Some speakers observed that expanding and strong ownership for reforms. A few access and improving quality will require questions were asked about the effects of increased unit costs in reaching EFA goals in decentralization on education management and country programs supported by the Bank. the fragmentation of education management Others asked about the timing and sequencing capacity building in Africa. to provide education support. In the opinion of one member, costly tradeoffs, while scaling up Bank operations. Several speakers noted the outcomes, could be avoided through scaling up growing share of lending for primary education resources and improving efficiency. Another through multisectoral projects. One member member commented that it was too early to welcomed this trend, while others asked assess the impact of ESSU on project design and whether there were any differences in learning implementation including increased attention outcomes of education components in multisec- to learning outcomes. One member felt that key toral projects compared to single-sector edu- factors to good quality education such as the cation projects. IEG commented that it was too classroom (more precisely, what happens in the early to assess differences in terms of learning classroom—school books, other learning outcomes, and moreover, very few such materials, incentives for pedagogic innovation, components had learning outcomes as primary etc.) need to be better analyzed and addressed. objectives. A member was interested in the impact of Bank support in the form of recurrent Education sector management. A number of financing for primary education, while another speakers emphasized the important role of member noted the risk of aid dependence. Pietro Veglio Chairman 113 ENDNOTES Chapter 1 eral assistance and the balance multilateral. Interna- 1. See, for example, the reviews by Lockheed and tional Development Association (IDA) credits were the Verspoor (1991) and Schultz (1993) on the returns to single largest source of funding for basic education that women’s education. Primary schooling is also asso- year. International Bank for Reconstruction and De- ciated with higher knowledge of HIV/AIDS and in- velopment (IBRD) lending was not counted, but if it creased condom use, compared with people with no had been, the Bank’s share would have been even schooling (Deheneffe, Carael, and Noumbissi 1998, higher. As with the World Bank, many bilateral de- Filmer 1998). For a review of the rationale for public velopment agencies also showed increases in the investment in primary education, see Boissiere share of their aid going to basic education since 1990, (2004a). but over most of the period the overall bilateral fund- 2. Many studies examining the relationship be- ing commitments to education were on a downward tween primary education and economic growth (micro trend (UNESCO 2004). and macro) have used years of education attained 4. There is also significant inequity in educational (educational attainment) as the education variable achievement. On the United Nations Educational, Sci- and have come up with few unequivocal findings entific, and Cultural Organization (UNESCO)–spon- (Pritchett 2001; Harmon, Oosterbeek, and Walker sored Latin American Laboratory for Assessment of 2000; Venniker 2001). More robust connections have Quality of Education (LLECE) exam in Latin America, been found when primary education has been meas- all participating countries except Cuba showed sig- ured in terms of knowledge and skills acquired. nificantly lower literacy scores for children of parents Glewwe (2002) showed high private returns to cog- with few compared with more years of education. For nitive skills (literacy and numeracy). Hanushek and example, in Peru the average score for children of Kimo (2000) conducted a cross-national study with the parents having only 3 years of education was around Third International Mathematics and Science Study 220, and that for those with parents having 14 years (TIMSS) and found strong connections between learn- was about 260 (the range of group averages on the test ing and economic growth. Likewise, Coloumbe and across all participating counties went from 220 to 350). others (2004) found a strong relationship between lev- 5. The evolution of World Bank policies on primary els of literacy in the labor force and economic growth education is summarized in Appendix A. across 16 Organisation for Economic Cooperation 6. “These needs comprise both essential learning and Development (OECD) countries (now being ex- tools (such as literacy, oral expression, numeracy, and tended to less-developed countries). For more details problem solving) and the basic learning content, see Boissiere (2004a). (such as knowledge, skills, values, and attitudes)” 3. Data from the OECD Development Assistance (Secretariate of the International Consultative Forum Committee indicate how Bank support compares with on World Declaration on Education for All 1990, Ar- that of other development agencies; however, Com- ticle I). mittee records only cover basic education—a head- 7. World Declaration on Education for All (1990), ing that includes mostly primary education but other Article IV. subsectors as well (see box 1.2). In 2003 a total of 8. Although it is titled Education for All—Fast- $2,412 million was committed globally by donors for Track Initiative, the FTI focuses on completion and basic education, of which $1,429 million was bilat- not specifically on learning. 115 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA 9. The proposals from participating countries are Evaluation was more focused on processes of exter- mainly intended to be financed directly by donors, nal assistance (client ownership, the trade-offs be- rather than from a centralized FTI fund. However, tween project and program support, donor the FTI does dispose of a “Catalytic Fund” designed harmonization, and so forth). Because the Bank was to temporarily assist countries having solid plans but a participant in that evaluation, such processes are not little current donor agency support. It is assumed highlighted in this study. that the performance using this temporary funding source will attract new donors to continue the assis- Chapter 2 tance. In addition, FTI has created a small Education 1. For a complete list of all projects covering pri- Program Development Fund to assist countries in mary education approved between fiscal 1963 and their attempts to create solid and fundable educa- 2005, by country, see Appendix D. tion plans. The FTI has been recognized as an effec- 2. The number of projects approved in 2005 (70) tive mechanism for encouraging country ownership was higher than in fiscal 2004 (54) and well above the of external assistance agendas and donor agency co- annual average (45) for fiscal 2000–04. New commit- operation and (sometimes) harmonization. ments in fiscal 2005 ($818 million) were substantially 10. Education for All—Fast-Track Initiative: lower than in fiscal 2004 ($1.04 billion), albeit slightly Framework (2004). These benchmarks include more higher than the annual average for fiscal 2000–04 than a dozen targets based on empirical analysis of a (about $800 million per year). set of low-income countries that are “on track” to 3. In nominal dollar commitments; in 2003 constant achieve primary school completion—such as a 20 dollars, the increase was 82 percent. percent share for education in the government re- 4. Primary education also has risen dramatically as current budget, a 42–64 percent share for primary ed- a share of total World Bank education commitments, ucation in the education budget, a pupil:teacher ratio from 20 percent before 1990 to 43 percent during the of 40:1, an annual teacher salary equal to 3.5 times 1990s and 53 percent in 2000–04 (IEG 2004d) gross domestic product (GDP) per capita, and one- 5. Investment lending has a long-term focus (5–10 third of primary education recurrent spending for years) and finances goods, works, and services, such items other than teacher remuneration. However, as improvements in physical and social infrastruc- Bruns, Mingat, and Rakotamalala (2003), whose work ture, and sector-specific inputs, such as learning ma- informed these targets, note that they “should not be terials and training. Development policy lending has applied rigidly” (see Appendix E). a short-term focus (1–3 years) and provides quick-dis- 11. As of January 2006, 20 countries had joined the bursing financing to support policy and institutional FTI: Burkina Faso, Djibouti, Ethiopia, The Gambia, reforms. Ghana, Guinea, Guyana, Honduras, Kenya, Lesotho, 6. Of the 68 projects managed by the Education Sec- Madagascar, Mauritania, Moldova, Mozambique, tor for fiscal 2000–04 with any commitments for pri- Nicaragua, Niger, Tajikistan, Timor-Leste, Vietnam, mary education, only two were sector adjustment-type and the Republic of Yemen. It is expected that 40 projects. Sixty percent were standard investment-type countries will be receiving FTI support by the end of operations, 29 percent were adaptable program lend- 2007 (World Bank 2006). ing, and the remainder consisted of learning and in- 12. This report sometime uses universal access novation lending, or emergency lending. Thus, within synonymously with universal enrollment/completion. the sector there has not been a shift to adjust- 13. While there have been studies on the rela- ment/development policy lending. tionship between primary education outputs and wel- 7. A PRSC is a recently developed World Bank fund- fare and employment outcomes in many countries ing instrument that provides budget support to coun- (see, for example, IEG 2004d), very few Bank-sup- tries for poverty-reduction efforts in a multisectoral ported primary education programs and projects in- package, usually based on a country-led PRSP, which cluded these as objectives, so this part of the results is vetted by the Bank. chain was not covered in the evaluation. 8. More than three-quarters of the 70 projects ap- 14. Whereas the current evaluation is mainly ori- proved in fiscal 2005 with any primary education ex- ented toward results of Bank assistance, the Joint penditure were managed by other sectors. Of the 116 ENDNOTES share of primary education commitments managed by has led the Education Sector in its 2005 sector strat- other sectors (53.5 percent), 26.4 percentage points egy update to press for more of a systemwide ap- were for development policy lending (including proach to planning development assistance in PRSCs), 10.9 percent for social fund/community-driven education at the country level. development investment projects, 9.0 percent for 13. The proportion on ongoing projects that plan other investment projects, and 7.2 percent for emer- to use learning outcomes as performance indicators gency projects. is about 80 percent, but this proportion should be 9. The average primary education commitments per viewed with caution. It is not clear how many of these project managed by other sectors has remained sta- projects have solid assessment designs, including ble over time: $8.0 million prior to 1990 (there were good benchmark data. Also, IEG has reported that only six such projects), $7.3 million in 1990–95, $8.7 among completed projects specifying learning out- million in 1995–99, and $8.2 million in 2000–04. Among comes as indicators, 35 percent never followed projects managed by the Education Sector, the aver- through with their planned assessments (IEG 2004d). age commitment to primary education per project 14. As shown in the portfolio review for this eval- prior to 1990 was only $9.4 million (174 projects). This uation, the share of projects with a female education jumped to $55.2 million per project for 1990–94, $46.9 feature rose from 20 to more than 40 percent over the million for 1995–99, and $40.1 million in 2000–04. period from before 1990 to 2004 (IEG 2004d, p. 16). 10. The Education Sector has reviewed the dis- Increases were particularly strong in South Asia from bursements to Ministries of Education (in some in- before 1990 to 1995 and in Sub-Saharan Africa be- stances explicitly for primary education) by the roughly tween 1990 and 1994 and 1995 and 2000. Among the two dozen projects in the Africa Multi-Country AIDS most recently approved projects (since 2000), more Program, which are managed by other sectors (Bak- than 85 percent of those in Africa and South Asia ad- ilana and others 2005). Disbursements to ministries dress female education, while in the other Regions were low—only about $4.4 million, or 2.1 percent of (some of which have already attained or nearly attained total disbursements by the end of fiscal 2004, among gender parity) the share is half or less. Chapter 3 in- the 13 projects effective in fiscal 2001–02 (Bakilana and dicates how successful recent projects have been in others 2005, table 2). Factors that have contributed to closing gender gaps. low disbursements for education activities and that 15. Reduced repetition and dropout rates—even presumably would also affect their efficacy include lack without any expansion of enrollment, improved qual- of recognition by AIDS and health authorities of the ity, or learning outcomes—would result in higher importance of the education sector; lack of commit- rates of completion of primary school. ment and capacity by ministry officials to fight AIDS; 16. This includes all primary education projects ap- the limited engagement of education specialists in proved from July 1, 2004, through May 23, 2006, and preparation or supervision of the multisectoral proj- managed by the Education Sector, with the exception ects; and the lack of an implementation plan for ed- of two that were emergency rehabilitation projects. ucation activities prior to project effectiveness. 17. Ten such projects were examined in detail. 11. In a recent policy change (mid-2006), PRSCs are Five were randomly selected from among Education not to be fully evaluated until a series has been com- Sector adjustment projects spending at least half of pleted. their funding on primary education, and the other five 12. Accepting and fulfilling these conditions was not were purposively selected from among projects man- without controversy in many places. The decision to aged by other sectors that have the highest percent- increase allocations to primary education was often ages of funding committed to primary education. See accompanied by agreements to cut scholarships for IEG (2004d) for an explanation of the sampling pro- upper secondary and higher education students, cedure. based on data that show that elites were receiving a 18. See Boissiere (2004a) for a review of current lit- disproportionate share of educational benefits. Such erature on the determinants of educational outcomes. moves led to protests and student unrest in places 19. The World Bank coined the term economic and such as Ghana, Mali, and Niger. A concern for possi- sector work (ESW) to refer to its analytical reports and ble imbalances across the subsectors in some places policy notes. Recently, however, that phrase has been 117 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA superseded by the term analytic and advisory ac- and sustainable use of its human, financial, and nat- tivities, which refers to analytical work and policy ural resources.” notes plus conferences, workshops, policy dialogue, 24. No projects approved after 1999 had been and technical assistance. As this report only covers an- completed as of the end of fiscal 2004. alytical work and policy notes, it will use the older term, ESW, or simply analytic work. The Bank also Chapter 3 conducts research, some relevant to primary educa- 1. Hanushek (2005) shows that schooling effects tion, within its Development Economics Vice Presi- on learning outcomes are stronger in developing dency. According to information provided by the countries than in higher-income countries, where so- Research Committee Secretariat, from 1992 to 2005 cial status factors are more important. But in addition the World Bank research support budget financed to schooling variables there are many other factors at roughly 32 research projects related to primary edu- play, including family background, health and nutri- cation, amounting to $2.69 million (excluding the tion, and demands for skills in the labor force. time of World Bank research staff). Because the Edu- 2. The Peru case study indicates that the school con- cation Sector does not count this research among its struction/renovation done in the project eventually reported analytic work, it is not covered here. may have been undertaken by the government with 20. These included an evaluation of Brazil’s Bolsa its own funds, but the Primary Education Project con- Escola program (fiscal 2000), an evaluation of India’s siderably increased the speed at which the renovations District Primary Education Project (DPEP , fiscal 2003), were done. and a Vietnam report on learning outcomes among 3. This is in Uttar Pradesh, where 45,000 contract fifth-grade students in math and Vietnamese. teachers were hired in 2004 alone. 21. These include a policy note, “Determinants of 4. In Ghana, the proportion increased from 50 to Learning in Mexico,” and reviews of educational 80 percent from 1980 to the mid-1990s, but since achievement in the Caribbean and in Central America. then it has fallen again, largely because of the rapid 22. Even this misses much analytic work, as an in- growth of private schools, whose teachers are generally creasing amount is now being financed through lend- less well trained. ing programs and is embedded within them. 5. The Republic of Yemen has been an exception. 23. Outcome is defined as “the extent to which the Arrangements were made for classrooms constructed project’s major relevant objectives were achieved, or under public works and social funds projects to be pro- are expected to be achieved, efficiently.” An outcome vided with adequate numbers of textbooks and trained has three components: relevance, efficacy, and effi- teachers (World Bank 2003a). ciency. The relevance of objectives is the “extent to 6. The PRSPs in Uganda are exceptional in that at least which the project’s objectives are consistent with the one of them was task managed by an Education Sector country’s current development priorities and with specialist. This is one plausible explanation for the in- current Bank country and sectoral assistance strate- clusion of learning outcomes in the Uganda PRSCs. gies and corporate goals.” Efficacy is defined as the “ex- 7. This lack of attention to learning results can tent to which the project’s objectives were achieved, produce a system that thinks it is producing edu- or are expected to be achieved, taking into account cated children when it is not. This is epitomized by their relative importance.” Efficiency is the “extent to Niger, where those merely enrolled in school are la- which the project achieved, or is expected to achieve, beled scolarisés (schooled), regardless of whether a return higher than the opportunity cost of capital they can read or write (IEG 2005d). and benefits at least cost compared with alternatives.” 8. See PPARs in Niger (IEG 2005d) and Uganda Sustainability is defined as the “resilience to risk of (IEG 2004c) and the case study on Mali. The pattern net benefits flows over time—taking into account fac- in Niger is typical. Classes having enrollments above tors such as technical resilience, financial resilience, 70 were split into morning and afternoon sessions, social support, government ownership, and institu- taught by the same teacher, with additional instruc- tional support.” Institutional development impact is tion given on Saturday. In this way, each child gets the “extent to which a project improves the ability of about 40 percent less scheduled time than in a regu- a country or region to make more efficient, equitable lar classroom. 118 ENDNOTES 9. Demand factors are also at play with respect to tion rate of 100 percent by 2015 and estimated the learning outcomes. The case studies for Romania and funding needed to reach that goal plus to close cur- Peru both show how the low-skills-low-wage equilib- rent funding gaps. In Niger, the poorest of the four, rium in the labor force is placing few demands on the only 23.7 percent of students “attained” grade 6 in school system for improved skills. This is something 2001, implying a huge expansion effort. Yet in 2000 the Bank and its partners could deal with in the future only about 54 percent of the few children who did through more cross-sectoral research and planning. graduate reported that they could read easily, and 10. Other strategies used by the government (but various national assessments showed very weak lan- not funded by the Bank) to increase enrollment and guage and math achievement. The new 10-year plan attendance were a midday meal scheme, covering all does include curricular revision, teacher training, and students, and free textbooks, uniforms, and scholar- better textbook provision, but does not include any ships for girls and low-caste/tribal people. While all of learning improvement target. The perception that them appear to have had some effect, the midday expansion has overwhelmed learning goals is rein- meal program was found to be the most effective of forced by the fact that in the 2003 PRSC progress re- all demand-side programs in motivating both enroll- port enrollment indicators were tracked, but there was ment and daily attendance (World Bank 2004e). no mention of extremely weak learning outcomes. 11. The fourth target of the Millennium Declaration 16. The four projects effectively supporting learn- was to “eliminate gender disparity in primary and sec- ing outcomes gains were Chile: Primary Education Im- ondary education, preferably by 2005, and to all lev- provement Project (1991); India: District Primary els of education by 2015.” Much progress was made Education Project II (1996); Mexico: Second Primary before the 2005 target date, with many areas of the Education Project (1994); and Uruguay: Basic Edu- world achieving gender parity in primary education by cation Quality Improvement Project (1994). The proj- the target date; however, South Asia, the Middle East ects from India and Uruguay were examined by IEG and North Africa, and Sub-Saharan Africa continue to in the field (PPARs). have ratios of girls’ to boys’ enrollment below 90. 17. Most of these projects focused on determinants These regions did not achieve gender parity in primary of student learning (inputs and processes) such as re- education by 2005 and may not even do so by the 2015 vised curriculum, improved textbooks and textbook dis- deadline (World Bank 2004g). tribution, teacher training (in-service and preservice), 12. According to World Bank data, the Republic of community involvement, and improved supervision. Yemen’s GER for girls increased to 72 percent in 2004, Learning outcomes were used as performance indica- but that for boys also increased (to 102). The gender tors in about a third of these projects. A larger pro- gap in 2004 stood at 30, little changed from that of portion of ongoing primary education projects (about 2000, when it was 34. 80 percent) have plans to do so, but past experience 13. In Mexico, access was not an issue. In Uruguay, shows that such plans are often not carried out. Also, which also has near universal enrollment in primary IEG could not verify whether the planned learning as- education, the Bank supported a project to expand ac- sessments were appropriately designed (had a baseline cess to pre-primary education for the poor. and adequate measurement methods). 14. This low attention to the distribution of learn- 18. Among those projects without learning out- ing outcomes across socioeconomic groups continues comes indicators, most proposed to develop new in the most recent Education Sector projects. Of the measures. While doing so is important for the ability 23 primary education projects approved in fiscal 2005 to monitor the impact of future primary education in- and 2006, none had an explicit objective of improv- vestments, it means that “baseline” will likely take ing learning outcomes for the poor as a group, al- place well into the project, reducing the likelihood of though a few did target learning outcomes repeated tracking during the project’s lifetime. Another improvement among children living in poor areas. option would be to establish a simple baseline by 15. Four of the evaluation’s field-based study coun- project start-up to be bolstered by more complete tries—Honduras, Niger, Vietnam, and the Republic of measures later on. Yemen—have become FTI partners. All of them have 19. Based on the national NGO Pratham’s Annual put forward proposals to reach a primary comple- Status of Education Report, 2005, released in early 119 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA 2006 and based on household surveys in 485 rural dis- the world have decentralized education management tricts throughout India. to varying degrees—and with different ends in 20. In rapidly expanding school systems, declines in mind…”) and provides no evidence for its impact on average achievement levels do not necessarily indicate improved educational outcomes for the poor, the falling individual performance levels. Instead, such lev- paper does end up listing decentralization as one of els may be more a reflection of the relatively low school its preferred “policy directions” (World Bank 1999). readiness of the new learners, who will mostly be from 2 . The 2004 Punjab Sector Reform Project has disadvantaged backgrounds (for example, having non- clarified much of this for the province. literate parents and poor learning conditions at home). 3. This is the “short route” of accountability described During periods of rapid expansion, educational service in the World Development Report 2004, in which client delivery would have to be improved even to maintain power is applied directly to the frontline service providers learning outcomes at previous levels. (schools and teachers) (World Bank 2003d). 21. This term has been used by Ayyar and Bashir 4. This study controlled for the decision as to (2004) to refer to processes of curricular reform, text- whether a child enrolled in an EDUCO or traditional book revision, teacher training, and improved su- school (“participation characteristics”). Unfortunately, pervision, based on a paradigm of student-centered because of data limitations, the authors were unable learning and a focus on learning outcomes. to take into account the decision on whether to en- 22. This is clearly not a full-blown model or an ex- roll a child in any school. haustive set of stages. It is simply an example of how 5. However, children attending EDUCO schools different kinds of treatments are relevant to coun- come from more disadvantaged backgrounds. Thus, tries at different stages of development. the unconditional mean test scores for children at- 23. Bank support in Uganda did influence better re- tending EDUCO schools are lower than for children source planning and the accelerated provision of in- in traditional schools (though the differences are not puts during the rapid expansion period; had it not been statistically significant). for Bank support, the situation would have been much 6. In the state of Rajastan in 1995–96, 32 inspection worse. Also, it is not clear how much of the decline is officers were listed for 3,680 schools. In 1999–2000 the due to deterioration of learning conditions and how number of schools had expanded to 4,124 but the much to the characteristics of the newly enrolled. number of supervisors had fallen to 31 (Clarke and Jha 24. In Vietnam, in response to a public expenditure 2006). The government of India’s ambitious DPEP review conducted in 2000 showing a high degree of established local and block (subdistrict) resource cen- inequity in resources spent on education and conse- ters, which included (ideally) monthly school visits by quent variations in student enrollment and achieve- mentors. However, as these were not connected to the ment, the government established a policy and official teacher management systems in any way program (supported by the Bank and others) to make (teacher promotions, transfers, pensions, and so sure that all schools, even the most remote, are forth), they have had less influence over certain aspects equipped to meet the country’s basic “fundamental of teacher professional growth than the supervision school quality level” standards (World Bank 2003e). system. The two systems have tended to exist in par- 25. Unfortunately, project evaluators only con- allel. Innovation under DPEP also includes school ducted univariate analyses in presenting their out- oversight by village education committees, which comes and thus did not assess the differential impact might have been expected to improve education qual- of the various interventions provided. The 2001 PPAR ity. In practice, they have been much more active in recommended the future use of more advanced sta- overseeing the quality of buildings and grounds than tistical techniques in showing the impact of govern- the quality of instruction, in which they have felt re- ment interventions (IEG 2001). luctant and unqualified to intervene. 7. A recent review of teacher incentives in Latin Chapter 4 America shows that rewards to teachers are rarely 1. Even though the Bank’s 1999 Education Sector based on any measure of performance (Vegas 2005). Strategy paper’s treatment of decentralization is de- 8. In India, where locally appointed teachers are scriptive and neutral (“Central governments around used in many states, some states are more proactive 120 ENDNOTES than others in designing career tracks for locally hired egy Update, all new education lending projects in the teachers (personal communication, Prema Clarke). Latin America and Caribbean Region will have built- 9. It is assumed that project managers need out- in impact evaluations (World Bank 2005b). come evaluation to help guide them toward im- 13. In the primary education portfolio review analy- provements in program and project results (see Kusek sis, 11 primary school projects were found to have and Rist 2004). plans for outcomes evaluation. Only 7 of them actu- 10. The QAG results pertain to education projects ally implemented these plans. In contrast, all but one in general, not just those containing primary edu- of the 25 projects planning evaluation of outputs im- cation features. To assess “quality at entry” (QAE) plemented their plans. (See IEG 2004c, box 4.) QAG took a random sample of new projects—50 in all (roughly 20 percent of all projects), stratified by Appendix B region and network. QEA5 included projects ap- 1. Following Education Sector practices, the eval- proved by the Bank in fiscal 2002; “education proj- uation assumed that half of commitments to “general ects” were those managed by the Education Sector. education” were for primary education. In total, 730 Ratings were based on a review of project design doc- projects having some allocation to primary educa- uments and interviews of relevant staff members. tion were counted; of those, 198 projects committed Given the relatively small number of projects in one at least 50 percent of funding to primary education (re- QAE round, regional and sectoral trends were de- ferred to in the evaluation as “primary education proj- termined using combined QAE scores from QEA3 ects”). (calendar year 1999), QAE4 (January 2000–June 2. IEG ratings based on Implementation Comple- 2001), and QAE5 (fiscal 2002). In five other sectors tion Reports (ICRs) give a single outcome rating for the the rating on evaluating impact/outcomes was as entire project. The evaluation team used project doc- low or lower than in the education sector; in three uments to create a rating for each project objective. the rating was higher. 11. The use of outcome indicators was even less Appendix F prevalent in the projects in other sectors with primary 1. The Ghana case study was conducted as one of education components. Of those, only two proposed a series of World Bank/Department for International the use of outcome measures. Development “impact studies.” 12. According to the 2005 Education Sector Strat- 2. As of 2003 there were 56 of these. 121 BIBLIOGRAPHY Abadzi, Helen. 2005. “Monitoring Basic Skills Ac- Bruns, Barbara, Alain Mingat, and Rmahatra Rako- quisition Through Rapid Learning Assess- tamalala. 2003. Achieving Universal Primary ments: A Case Study from Peru.” Prospects 35 Education for All by 2015: A Chance for Every (2): 137–56. Child. Washington, DC: World Bank. ———. 2003. “Adult Literacy: A Review of Im- Busto, Marit, Robert Langley Smith, and Roald plementation Experience.” IEG Working Paper, Skoelv. 2006. “Building Capacity for the Edu- World Bank, Washington, DC. cation Sector in Africa.” Report from the 8th Alexander, Karl L., Doris R. Entwistle, and Linda Annual Norway Education Trust Fund Seminar, S. Olson. 2001. “Schools, Achievement and In- Oslo, Norway, October 12–14, 2005. equality: A Seasonal Perspective.” Educational Clarke, P., and J. Jha. 2006. “Ragasthan’s Experi- Evaluation and Policy Analysis 23 (2): 171–91. ence in Improving Service Delivery in Edu- Ayyar, R.V.V., and Sajitha Bashir. 2004. “District Pri- cation.” In Reinventing Public Service mary Education Programme.” In Ency- Delivery in India: Selected Case Studies, ed. clopaedia of Indian Education, vol. 1, ed. J.S. V. Chand. Thousand Oaks, CA: Sage. Rajput. New Delhi: National Council of Edu- Coulombe, Serge, Jean-François Tremblay, and cational Research and Training. Sylvie Marchand. 2004. “Literacy Scores, Azim Premji Foundation. 2004. “Status of Learn- Human Capital and Growth across Fourteen ing Achievements in India: A Review of Em- OECD Countries.” International Adult Lit- pirical Research.” Bangalore, India. Photocopy. eracy Survey Monograph No. 1. Ottawa: Sta- Bakilana, Anne, Donald Bundy, Jonathan Brown, tistics Canada. and Birger Fredrickson. 2005. “Accelerating the Deheneffe, Jean-Claude, Michel Carael, and Education Sector Response to AIDS in Africa: Amadou Noumbissi. 1998. “Socioeconomic A Review of World Bank Assistance.” Working Determinants of Sexual Behaviour and Con- Paper, World Bank, Washington, DC. dom Use.” In Confronting AIDS: Evidence Behrman, J.R., P . Sengupta, and P . Todd. 2000. The from the Developing World, eds. Martha Impact of PROGRESA on Achievement of Test Ainsworth, Lieve Fransen, and Mead Over. Scores in the First Year. Washington, DC: In- Brussels: European Commission. ternational Food Policy Research Institute. Durston, J. 1999. La Gestión Comunitaria de la Berryman, Sue E., N. Boyle, F. Golladay, M. Educación Rural en PROHECO . Paris: Holmes, P . Keefer, and K. Sigrist. 1997. “Guide- UNDP/CEPAL. lines for Assessing Institutional Capability.” Filmer, Deon. 1998. “The Socioeconomic Corre- Photocopy. lates of Sexual Behaviour: A Summary of Results Boissiere, Maurice. 2004a. “The Determinants from an Analysis of DHS Data.” In Confronting of Primary Education Outcomes in Develop- AIDS: Evidence from the Developing World, ing Countries.” IEG Background Paper, World eds. Martha Ainsworth, Lieve Fransen, and Bank, Washington, DC. Mead Over. Brussels: European Commission. ———. 2004b. “Rationale for Public Investment Fuller, Bruce. 1987. “What Factors Raise School in Primary Education.” IEG Background Paper, Achievement in the Third World.” Review of World Bank, Washington, DC. Educational Research 57 (3): 255–92. 123 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA G-8 Group. 2006. “Statement of G-8 Education Education Sector Project Performance Assess- Ministers Meeting.” http://en.g8russia.ru/news/ ment Report, World Bank, Washington, DC. 2006 0602/1151807.html. ———. 2005e. “Yemen: Basic Education Pro- Glewwe, Paul. 2002. “Schools and Skills in De- ject.” IEG Project Performance Assessment veloping Countries: Education Policies and Report, World Bank, Washington, DC. Socioeconomic Outcomes.” Journal of Eco- ———. 2004a. Books, Buildings, and Learning nomic Literature 40 (2): 436–82. Outcomes: An Impact Evaluation of World Govinda, R., and Y. Josephine. 2005. “Para Teach- Bank Support to Basic Education in Ghana. ers in India: A Review.” Contemporary Edu- Washington, DC: World Bank. cation Dialogue 2 (2). ———. 2004b. “Honduras. Rural Primary Edu- Grindle, Merilee S. 2004. Despite the Odds: The cation Management Project, Basic Education Contentious Politics of Education Reform. Project.” IEG Project Performance Assessment Princeton, NJ: Princeton University Press. Report, World Bank, Washington, DC. Hanushek, Eric. 2005. “Why Quality Matters in ———. 2004c. “Uganda: Primary Education and Education.” Finance and Development 42 Teacher Development Project, and Education (2): 15–19. Sector Adjustment Credit.” IEG Project Per- Hanushek, Eric, and Dennis Kimko. 2000. formance Assessment Report, World Bank, “Schooling, Labor Force Quality and the Washington, DC. Growth of Nations.” American Economic Re- ———. 2004d. The World Bank’s Assistance to view 95 (5): 1184–208. Primary Education: An OED Portfolio Re- Harmon, C., H. Oosterbeek, and I. Walker. 2000. view. Washington, DC: World Bank. The Returns to Education: A Review of Evi- ———. 2002. Social Funds: Assessing Effective- dence, Issues and Deficiencies in the Litera- ness. Washington, DC: Workd Bank. ture. London: Center for the Economics of ———. 2001. “Chile: Primary Education Im- Education, London School of Economics and provement Project.” Performance Audit Re- Political Science. port, World Bank, Washington, DC. IEG (Independent Evaluation Group, World Jalan, Jyotsna, and Elena Glinskaya. 2003. Im- Bank). 2006a. “Uruguay. Vocational Training proving Primary School Education in India: and Technological Development Project, Basic An Impact Assessment of DPEP-Phase 1. New Education Quality Improvement Project, Sec- Delhi/Washington DC: Indian Statistical In- ond Basic Education Quality Improvement stitute/World Bank. Project.” IEG Project Performance Assessment Jimenez, E., and Y. Sawada. 1999. “Do Commu- Report, World Bank, Washington, DC. nity-Managed Schools Work? An Evaluation ———. 2006b. “Vietnam: Primary Education Pro- of El Salvador’s EDUCO Program.” World ject.” IEG Project Performance Assessment Bank Economic Review 13 (3): 415–41. Report, World Bank, Washington, DC. Joint Evaluation (Joint Evaluation of External Sup- ———. 2005a. Capacity Building in Africa: An port to Basic Education in Developing Coun- OED Evaluation of World Bank Support. tries). 2003. Final Report: Local Solutions to Washington, DC: World Bank. Global Challenges: Towards Effective Part- ———. 2005b. Committing to Results: Improv- nership in Basic Education. The Hague: ing the Effectiveness of HIV/AIDS Assistance. Netherlands Ministry of Foreign Affairs. An OED Evaluation of the World Bank’s As- Kattan, Raja Bentaouet, and Nicholas Burnett. sistance for HIV/AIDS Control. Washington, 2004. User Fees in Primary Education. Wash- DC: World Bank. ington, DC: World Bank. ———. 2005c. The Effectiveness of World Bank Kusek, Jodi Zall, and Ray C. Rist. 2004. Ten Steps Support for Community-Based and -Driven to a Results-Based Monitoring and Evalua- Development. Washington, DC: World Bank. tion System: A Handbook for Development ———. 2005d. “Niger: First Education Project, Practitioners. Washington, DC: World Bank. Primary Education Development Project.” Basic Lockheed, Marlaine, and Adrian Verspoor. 1991. Im- 124 BIBLIOGRAPHY proving Primary Education in Developing oping Countries: Barriers, Benefits, and Poli- Countries. Oxford, U.K.: Oxford University Press. cies, eds. Elizabeth M. King and M. Anne Hill. Marphatia, Akanksha A., and David Archer. 2005. Baltimore, MD: Johns Hopkins University. Contradicting Commitments: How the Achieve- Secretariat of the International Consultative ment of Education for All Is Being Under- Forum on Education for All. 1990. World De- mined by the International Monetary Fund. claration on Education for All: Meeting Basic London: ActionAid. Learning Needs. Paris: UNESCO. MEN (Ministre de l’Education Nationale). 2005. SIEMAT (State Institute of Educational Manage- Repartition Fonctionnelle des Depenses ment and Training). 2005. Research Report, 2000–2005. Bamako, Mali : Direction Admin- Uttar Pradesh. istrative et Financiere. Tietjen, Karen, Arifur Rahman, and Seth Spauld- OED (Operations Evaluation Department, World ing. 2004. Bangladesh Educational Assess- Bank). See IEG. ment. Time to Learn: Teachers’ and Students’ Patrinos, Harry Anthony, Joseph Shapiro, and Jorge Use of Time in Government Primary Schools Moreno Trevino. 2004. “Education for All: Com- in Bangladesh. Washington DC: United States pensating for Disadvantage in Mexico.” Edu- Agency for International Development. cation Note, May 2004, http://siteresources. UNDP (United Nations Development Pro- worldbank.org/EDUCATION/Resources/Edu gramme). 2005. Human Development Report cation-Notes/EduNotes_Mexico2.pdf. 2005: International Cooperation at a Cross- Perkins, Gillian. 2004. Rationales and Results roads: Aid, Trade, and Security in an Un- in Secondary Education Investments: A Re- equal World. New York: UNDP. view of the World Bank’s Portfolio, 1990–2001. UNESCO (United Nations Educational, Scientific, Washington, DC: World Bank and Cultural Organization). 2004. EFA Global Pratham. 2006. Annual Status of Education Re- Monitoring Report 2005: Education for All— port, 2005. New Delhi: Pratham. The Quality Imperative. Paris: UNESCO. Pritchett, Lant. 2001. “Where Has All the Edu- ———. 2003. EFA Global Monitoring Report cation Gone?” World Bank Economic Review 2003/4: Gender and Education for All—The 15 (3): 367–91. Leap to Equality? Paris: UNESCO. Psacharopoulos, George, and Maureen Woodhall. ———. 2002. EFA Global Monitoring Report 1985. Education for Development: An Analy- 2002: Education for All—Is the World on sis of Investment Choices. New York: Pub- Track? Paris: UNESCO. lished for the World Bank [by] Oxford ———. 2000. The Dakar Framework for Ac- University Press. tion. Paris: UNESCO. Rawlings, L.B., and G.M. Rubio. 2003. “Evaluat- ———. 1999. UNESCO Statistical Yearbook. Mon- ing the Impact of Conditional Cash Transfer treal: UNESCO Institute for Statistics. Programs. Lessons from Latin America.” Pol- Vegas, Emiliana, ed. 2005. Incentives to Improve icy Research Working Paper No. 3119, World Teaching. Lessons from Latin America. Wash- Bank, Washington, DC. ington, DC: World Bank. Roberts, J. 2005. “What Happens When Public Ex- Venniker, Richard. 2001. Social Returns to Edu- penditure Is Scaled Up? An Enquiry into the cation: A Survey of Recent Literature on Costs and Cost-Effectiveness of Expenditure in Human Capital Externalities. CPB Report Phases of Expansion.” Draft ESAU Working Paper, 00/1. Amsterdam: Netherlands Central Bu- Overseas Development Institute, London. reau of Planning. Roberts-Schweitzer, Eluned, Andrei Markov, and Warwick, D.P., and F. Reimers. 1995. Hope or Alexander Tretyakov. 2002. “Achieving Edu- Despair? Learning in Pakistan’s Primary cation for All Goals: School Grants.” Staff Schools. Westport, CT: Praeger. Working Paper, World Bank, Washington, DC. Watkins, K. 2001. The Oxfam Education Report. Schultz, T. Paul. 1993. “Returns to Women’s Ed- Bath, U.K.: Redwood Books/Oxfam GB for ucation.” In Women’s Education in Devel- Oxfam International. 125 FROM SCHOOLING ACCESS TO LEARNING OUTCOMES: AN UNFINISHED AGENDA Winkler, Don. 2001. ESW in the Education Sec- ———. 2004e. Pakistan Public Expenditure tor. Photocopy. Management: Strategic Issues and Reform World Bank. 2005a. Education Fiscal Year 2005 Agenda. Washington, DC: World Bank. Retrospective: Summary of Key Achievements. ———. 2004f. “Snakes and Ladders. Factors In- Washington, DC: World Bank. fluencing Successful Primary School Com- ———. 2005b. Broadening our Perspective, pletion for Children in Poverty Contexts.” Maximizing our Effectiveness. Education Discussion Paper Series, Report No. 6, World Sector Strategy Update. Washington, DC: Bank, Washington, DC. World Bank. ———. 2004g. “Vietnam: Reading and Mathe- ———. 2005c. Gender Factsheet. Department matics Assessment Study, Vol. 2.” Human De- for International Development Policy Divi- velopment Sector, East Asia and Pacific Region, sion Info series No. 048. Washington, DC: World Bank, Washington, DC. World Bank. ———. 2004h. World Development Indicators. ———. 2005d. Implementation Completion Re- Washington, DC: World Bank. port for Ghana Second Poverty Reduction ———. 2004i. World Development Report 2005: Support Credit. Poverty Reduction and Eco- A Better Investment Climate for Everyone. nomic Management 4, AFC10. Washington, Washington, DC: World Bank. DC: World Bank. ———. 2003a. Education Fiscal Year 2003 Ret- ———. 2005e. Islamic Republic of Iran: Achieving rospective Review. Washington, DC: World Universal Access to Education for the Knowl- Bank. edge Economy. Washington, DC: World Bank. ———. 2003b. “EFA in Indonesia: Hard Lessons ———. 2005f. Mexico: Determinants of Learn- about Quality.” Education Notes, World Bank, ing Policy Note. Washington, DC: World Bank. Washington, DC. ———. 2005g. “Vietnam: Targeted Budget Sup- ———. 2003c. “Yemen: Social Fund for Devel- port for National Education for All Plan Im- opment.” Implementation Completion Re- plementation Program.” Project Appraisal port, World Bank, Washington, DC. Document, World Bank, Washington, DC. ———. 2003d. Making Services Work for Poor ———. 2005h. Simplified Implementation Com- People: World Development Report 2003/2004. pletion Report for Madagascar First Poverty Washington, DC: World Bank. Reduction Support Operation. Washington, ———. 2003e. Quality at Entry in FY02 DC: World Bank. (QEA5)—A QAG Assessment. Washington, DC: ———. 2005i. “Uganda: Fifth Poverty Reduction World Bank. Operation.” Program Document, World Bank, ———. 2003f. “A Review of Educational Progress Washington, DC. and Reform in the District Primary Education ———. 2005j. World Development Report 2006: Programs (Phases I & II).” Discussion Paper Se- Equity and Development. Washington, DC: ries Report No. 1, World Bank, Washington, DC. World Bank. ———. 2003g. “Vietnam: Primary Education for ———. 2004a. Education Fiscal Year 2004 Ret- Disadvantaged Children.” Project Appraisal rospective Review. Washington, DC: World Document, World Bank, Washington, DC. Bank. ———. 2002a. Education Fiscal Year 2002 Ret- ———. 2004b. “Education for All: Compensating rospective Review. Washington, DC: World for Disadvantage in Mexico.” Education Notes, Bank. World Bank, Washington, DC. ———. 2002b. Education for Dynamic ———. 2004c. Fourth Quality-at-Entry Assess- Economies: Action Plan to Accelerate ment. (QEA4) January 1, 2000–June 30, 2001. Progress Toward Education for All. Wash- Washington, DC: World Bank. ington, DC: World Bank/IMF Development ———. 2004d. Framework Document: Educa- Committee. tion for All–Fast-Track Initiative. Washing- ———. 2002c. Opening Doors: Education and ton, DC: World Bank. the World Bank. Washington, DC: World Bank. 126 BIBLIOGRAPHY ———. 2001a. Implementation Completion Re- ———. 2000c. Quality at Entry of Operations in port for Chile: Primary Education Improve- CY99 (QEA3)—A QAG Assessment. Washing- ment Project. Washington, DC: World Bank. ton, DC: World Bank. ———. 2001b. “Honduras: Community-Based ———. 1999. Education Sector Strategy. Wash- Education Project.” Project Appraisal Docu- ington, DC: World Bank. ment, World Bank, Washington, DC. ———. 1995. Priorities and Strategies for Edu- ———. 2000a. Hidden Challenges to Educa- cation: A World Bank Review. Washington, tion Systems in Transition Economies: Edu- DC: World Bank. cation Sector Strategy Paper for Europe and ———. 1990. Primary Education: A World Bank Central Asia Region. Washington, DC: World Policy Paper. Washington, DC: World Bank. Bank. ———. 1988. Pakistan Education Sector Strat- ———. 2000b. Project Appraisal Document for egy Review. Washington, DC: World Bank. Mali Education Sector Expenditure Program. ———. 1980. Education Sector Policy Paper. Washington, DC: World Bank. Washington, DC: World Bank. 127 IEG PUBLICATIONS Study Series 2004 Annual Review of Development Effectiveness: The Bank’s Contributions to Poverty Reduction Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs Agricultural Extension: The Kenya Experience Assisting Russia’s Transition: An Unprecedented Challenge Bangladesh: Progress Through Partnership Brazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support The CIGAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An OED Review of the HIPC Initiative Developing Towns and Cities: Lessons from Brazil and the Philippines The Drive to Partnership: Aid Coordination and the World Bank Economies in Transition: An OED Evaluation of World Bank Assistance The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990–99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002 Financial Sector Reform: A Review of World Bank Assistance Financing the Global Benefits of Forests: The Bank’s GEF Portfolio and the 1991 Forest Strategy and Its Implementation Fiscal Management in Adjustment Lending IDA’s Partnership for Poverty Reduction Improving the Lives of the Poor Through Investment in Cities India: The Dairy Revolution Information Infrastructure: The World Bank Group’s Experience Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector Jordan: Supporting Stable Development in a Challenging Region Lesotho: Development in a Challenging Environment Mainstreaming Gender in World Bank Lending: An Update Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, Pakistan Nongovernmental Organizations in World Bank–Supported Projects: A Review Poland Country Assistance Review: Partnership in a Transition Economy Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003 Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the Electricity Sector Promoting Environmental Sustainability in Development Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Reforming Agriculture: The World Bank Goes to Market Sharing Knowledge: Innovations and Remaining Challenges Social Funds: Assessing Effectiveness Tunisia: Understanding Successful Socioeconomic Development Uganda: Policy, Participation, People The World Bank’s Experience with Post-Conflict Reconstruction The World Bank’s Forest Strategy: Striking the Right Balance Zambia Country Assistance Review: Turning an Economy Around Evaluation Country Case Series Bosnia and Herzegovina: Post-Conflict Reconstruction Brazil: Forests in the Balance: Challenges of Conservation with Development Cameroon: Forest Sector Development in a Difficult Political Economy China: From Afforestation to Poverty Alleviation and Natural Forest Management Costa Rica: Forest Strategy and the Evolution of Land Use El Salvador: Post-Conflict Reconstruction India: Alleviating Poverty through Forest Development Indonesia: The Challenges of World Bank Involvement in Forests The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Uganda: Post-Conflict Reconstruction Proceedings Global Public Policies and Programs: Implications for Financing and Evaluation Lessons of Fiscal Adjustment Lesson from Urban Transport Evaluating the Gender Impact of World Bank Assistance Evaluation and Development: The Institutional Dimension (Transaction Publishers) Evaluation and Poverty Reduction Monitoring & Evaluation Capacity Development in Africa Public Sector Performance—The Critical Role of Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg THE WORLD BANK ISBN 0-8213-6792-7