68671 INDEPENDENT EVALUATION GROUP Work Program and Budget (FY13) and Indicative Plan (FY14–15) May 7, 2012 CONTENTS Abbreviations and Acronyms 2 Introduction 3 Independent Evaluation Group Strategy 3 Highlights of the FY12 Work Program 5 Coverage of IEG Evaluative Work 7 FY13–15 Work Program 10 The Challenge of Inclusion 10 Thematic Clusters of Evaluative Work 10 Strategic Focus of IEG’s Project-Level Work 13 Methodology 13 Engagement and Following Up on IEG Recommendations 14 Learning, Openness, and Transparency 15 Building Client Monitoring and Evaluation Capacity 16 Staffing and Budget 18 Human Resources 18 Budget Proposal 18 Accountability 21 External Assessment 21 Conclusion 22 Appendix 1: Proposed Evaluation Program FY13–15 23 Appendix 2: Selectivity Frameworks 25 Appendix 3: IEG Results and Measurement Framework 48 Appendix 4: Global and Regional Programs with Recently Completed Self-Evaluations 54 Appendix 5: FY10–15 Consolidated Trends by Expense Category and Income Source 55 Appendix 6: Consolidated Trends by (i) Department and (ii) Service Category 56 Box 1: Project Evaluation Coverage of World Bank Lending Operations 9 Box 2: Staff Learning Going Global 16 Table 1: IEG Institution Contributions, FY10–15 19 Table 2: Summary of Sources (including Trust Funds) and Uses and Key Spending Ratios, FY11–15 20 1 ABBREVIATIONS AND ACRONYMS AAA Analytic and advisory activities CLEAR Centers for Learning Evaluation and Results CODE Committee on Development Effectiveness ECD Early child development ECG Evaluation Cooperation Group FCS Fragile and conflict-affected states IAD Internal Audit Department (World Bank) ICR Implementation Completion Report IDA International Development Agency IEG Independent Evaluation Group IFC International Finance Corporation INT Department of Institutional Integrity IPDET International Program for Development Evaluation Training MAR Management Action Record M&E Monitoring and evaluation MDG Millennium Development Goal MIGA Multilateral Investment Guarantee Agency OPCS Operations Policy and Country Services PPAR Project Performance Assessment Report (of World Bank Projects and Knowledge Products) RAP Results and Performance (IEG annual report) WBG World Bank Group XPSR Expanded Project Supervision Report (for IFC Investment Operations) 2 Draft IEG WORK PROGRAM AND BUDGET (FY13) AND INDICATIVE PLAN (FY14–15) Introduction 1. This document presents the work program and budget proposal for FY13 for the Independent Evaluation Group (IEG) and indicative plan for FY14 and FY15. This proposal comes at a time when significant rethinking of the international development architecture is under way as the world is beginning to emerge from the global financial and economic crisis that began in 2008, donor relations are being reshaped in the aftermath of the Busan High-Level Forum, and the target date for reaching the Millennium Development Goals (MDGs) is just three years away. The FY13–15 work program builds on strategic directions over the past two years and the organizational changes that have taken place within IEG, takes the Post-Crisis Directions Paper1 as a point of departure, and builds a work program that balances appropriate coverage of World Bank Group (WBG) activities with a thematic approach to draw synergies and achieve greater cohesion. 2. A number of elements in the proposed work program respond directly to several inputs, including IEG’s self-evaluation,2 concluded in 2011; the External Review of Oversight and Accountability Units at the World Bank Group3 (subsequently referred to as the 5 I’s Review); and the guidance received by members of the Board, as well wide-ranging consultations with WBG management and direct feedback received through IEG’s client survey. As in past years, the document also benefited from an External Review of IEG’s Budget for FY13 and Linkage to Its Work Program, commissioned by the Committee on Development Effectiveness (CODE) and based on a draft of this paper.4 3. This document outlines the strategic context for decisions and trade-offs in the work program and work program details and sets out the resource framework in the form of staffing and budget. The appendixes to this document include (i) a summary work program table; (ii) detailed descriptions for each of the proposed major evaluations, using a common selectivity framework, and other work program areas; (iii) an updated IEG’s Results and Measurement Framework; and (iv) a summary of spending plans. Independent Evaluation Group Strategy 4. The proposed work program builds on and extends the strategic directions that IEG introduced three years ago.5 In addition to responding rapidly to emerging issues, forging 1 DC2010-0002. 2 See ―Self-Evaluation of the Independent Evaluation Group,‖ February 24, 2011, CODE2011-0013. 3 COGAM2011-0014. 4 ―External Review of Independent Evaluation Group’s (IEG) Budget for FY13 and Linkage to its Work Program,‖ BC2012-0003, CODE2012-0011. 5 See Independent Evaluation Group Work Program and Budget (FY10) and Indicative Plan (FY11–12), April 29, 2009, R2009-0094. 3 stronger links with WBG directions, and promoting greater IEG integration, IEG is proposing a theme for its three-year rolling work program to increase the linkages across IEG’s work and thereby improve the effectiveness of the whole program. As a result, the work program will take full advantage of both micro and macro evaluations—respectively, project-level evaluations and evaluations of sectors, themes, countries programs, or corporate issues—and generate a cohesive set of insights and recommendations that will reinforce each other. 5. The most recent United Nations projections indicate that the world is largely ―on track‖ to meet poverty reduction goals, thanks to strong growth in many middle-income countries and a significant number of countries in Africa. Progress has been uneven across and within countries, and many countries are greatly concerned about future growth, in part because of a lack of inclusiveness and sustainability. Global challenges, experienced at local levels, combine interrelated factors such as financial, fuel, and food crises, environmental disasters, stress on natural resources, population growth, and stability and security, to name a few. 6. As the 2015 deadline for reaching the internationally agreed-on MDGs draws near, debate will increase about the approaches and solutions to poverty reduction: what works, what doesn’t, and why. These discussions will take place in an environment that is vastly different from the one in 2000. The aid architecture is transforming into a development partnership that includes many additional stakeholders: new donor countries, foundations, the private sector, and civil society. Private capital flows, including those between developing countries, although varying widely, have surpassed the level of official development aid by a large margin. When implemented in a sound and sustainable manner, such investments have demonstrated positive development results. The relationship between governments and citizens has changed in many countries; citizens now demand more accountability, openness, and public debate. All of these stakeholders are keen to have evidence and lessons from experience, especially from institutions such as the WBG that have been central to the development agenda and poverty reduction. 7. IEG aims to draw insights from experience of success and failure in ways that inform decisions and actions for the future. Evaluation asks critical questions about the effectiveness of the development endeavor and provides accountability for whether intended results were achieved. 8. The same questions apply to IEG: Are we achieving the results we set out to achieve? The evaluation profession has grown considerably over the past 10 years, with greater and greater demand for evidence and lessons about what works, how to achieve development results, and how to avoid common mistakes. This demand has led to an expansion of the evaluation field, stimulating debates about methods, professional standards, competences, and many other issues. One such issue—the effectiveness of evaluation—has often taken the back seat as the profession grappled with other important questions. 9. As indicated in last year’s work program document, IEG has therefore further developed its results framework and included in this document as an appendix a framework for selecting each proposed evaluation. Both will be developed further to integrate more fully the thematic approach described below. 4 10. IEG is structuring its three-year rolling plan for its FY13–15 evaluative program around four clusters that, combined, contribute to lessons relevant to the mission of the WBG, as articulated in New World, New World Bank Group, endorsed by the Bank Group’s Board of Governors in April 2010, and the modernization agenda, endorsed by the Development Committee in the April 2012.6,7 11. These evaluation clusters will be:  The WBG’s contribution to growth and poverty reduction (MDG1 and beyond)  The WBG’s contribution to achieving other welfare outcomes captured by the MDGs  The challenge of achieving development results in different country contexts  Implementing the development effectiveness agenda. 12. IEG’s accountability role requires that the full range of the WBG’s work—lending, investments, guarantees, knowledge work, advisory services, and partnerships—is adequately evaluated. This document begins to analyze the overall work of the WBG compared with the areas covered by IEG evaluations. This work will carry over into FY13 and is intended to lead to informed discussions about the degree to which IEG adequately covers the WBG’s most important activities and any gaps in coverage. Highlights of the FY12 Work Program 13. In recent years the global economy has experienced a number of shocks that risk setting back the results of previous development efforts, accompanied by an ongoing transformation in the development world. Together these factors increase the demand for evidence-based decision making and raise the bar for deriving lessons from a past that might no longer be a guide for the future. 14. During FY12, IEG focused on providing timely input to the development agenda. This has included the evaluation of World Bank Group’s Response to the Global Economic Crisis: Phase II. Three additional thematic evaluations are well advanced and expected to be completed in FY12, namely the evaluation of youth employment, the evaluation World Bank Group Experience in Managing Forest Resources for Sustainable Development, and the third and final phase of IEG’s work on climate change: Climate Change and the World Bank Group: Climate Adaptation. 15. On other important topics where evaluations were completed in late FY11, IEG engaged with the broader development community, including participating in the High-Level 6 See DC2010-0003, April 2010. The vision of the WBG endorsed by the Governors has since been ―to overcome poverty—by supporting an inclusive and sustainable globalization, enhancing growth with care for the environment, and creating individual opportunity and hope.‖ 7 See DC 2012-0005, April 2012. ―Update on the Bank’s Business and Modernization: Results, Openness, and Accountability. ― 5 Forum in Busan, based on IEG’s evaluations of World Bank progress in harmonization and alignment in low-income countries, the evaluation of the world bank’s trust fund portfolio and its biennial review of global and regional partnership program reviews. IEG also continued its outreach and engagement on its evaluation of the World Bank country-level engagement in governance and anticorruption. 16. IEG continued its practice of focusing its micro work, particularly the Project Performance Assessment Reports (PPARs), on themes that would support subsequent evaluations. These included, among others, work on sustainability of infrastructure in India, Mozambique, and Poland; response to the food crisis in Burundi, Djibouti, and Sierra Leone; and financial sector development in the Arab Republic of Egypt, Guatemala, Morocco, and Pakistan.8 In addition, IEG is completing reviews of more than 450 project-level investment and advisory activities across the WBG. 17. At the country level, IEG is completing two more Country Program Evaluations in fragile and conflict-affected states (Afghanistan and Liberia) to complement earlier evaluations (Timor-Lest and West Bank and Gaza) as a foundation for IEG’s forthcoming evaluation on the World Bank Group in fragile and conflict-affected states (FY13). In addition, the review of CAS Completion Reports (CASCRs) is done for all CASs and Country Partnership Strategies that are presented to the Board with a CASCR. 18. IEG is completing four reviews of global and regional partnerships this year. These include the Global Fund to Fight Aids, Tuberculosis, and Malaria and the World Bank’s Engagement with the Global Fund. Two additional partnership reviews will be completed by the end of FY12, the Global Fund for Disaster Reduction and Recovery and the Forest Carbon Partnership. Further insights into the effectiveness of approaches toward specific development challenges were examined in IEG’s Review of Impact Evaluations. 19. At the institutional level, the FY12 Results and Performance (RAP) report will further contribute to the WBG’s focus on results, while The Matrix System at Work: An Evaluation of the World Bank Group’s Organizational Effectiveness contributed to the discussion the modernization and reform agenda for the World Bank. The evaluation of the system for private sector operations is scheduled for completion this fiscal year. 20. A new feature in IEG’s work program was a series of notes to the Board addressing updates on Regional Strategies. These proved to be very effective and will be continued in the future. In addition, Executive Directors requested similar notes to accompany future network updates; those will be included in the FY13 work program and beyond. Additional IEG products based on earlier evaluations included notes on Southern Sudan, Bulgaria, and the financial sector. 21. Beyond the evaluative work program, and the accompanying communications, outreach, and learning efforts,9 IEG took significant steps to implement recommendations stemming 8 A more comprehensive list is available in IEG’s quarterly reports to the Board. 9 Further details on communication, outreach, and learning activities are provided in the subsequent sections, as much of these efforts bridge fiscal years and are ongoing in nature. 6 from the 5 I’s Review, the 2011 self-evaluation, and recommendations from IEG’s External Advisory Group (see relevant boxes). 22. In response to the specific recommendations in the 5 I’s Review and, IEG will provide a written update on its actions taken during the course of May 2012 in follow-up to the CODE discussion. As one of the follow-up actions to the 5 I’s Review, IEG has developed an annual report that provides an account of IEG’s work in the previous fiscal year. The first such report, covering FY11, is a prototype and will be submitted in FY12. As of FY13, this report will be part of IEG’s regular reporting to CODE and the Board, building on the quarterly reports. The IEG Annual Report will be finalized and distributed routinely during the first quarter of each fiscal year. That report will take stock and account for work during the fiscal year and reflect on future directions for IEG. It will complement the RAP, which focuses on the findings of IEG micro and macro evaluations, and will provide a synthesis of evaluation insights and an aggregation of lessons derived from them to report on the achievements of the WBG. 23. Among the new features in this work program is the application of a selectivity framework developed from a study of IEG’s influence and IEG’s results framework—each innovations in the evaluation field. The framework will be included in all new evaluations proposed for FY13–15.10 The selectivity framework builds on closer engagement with management and stakeholders, as well as on the areas emphasized for follow-up from the implementation of previous IEG recommendations captured in the Management Action Record.11 Coverage of IEG Evaluative Work 24. The breadth of development programs and services covered in the post-crisis directions paper directly reflects the growing demand for traditional WBG assistance in the form of financial and knowledge services, as well as the need to develop new lines of business commensurate with the new challenges the WBG’s clients are facing in the 21st century. IEG’s evaluative work has traditionally built on the WBG’s self-evaluation system, with in-depth analyses of specific thematic and corporate areas. As the 2011 Self-Evaluation indicated, this has led to pressure on IEG to expand its accountability and learning roles to new products that have been mainstreamed in recent years (as in the case for evaluations of IFC’s Advisory Services). 25. IEG will therefore review (i) the services provided by the WBG to clients by thematic area and volume, compared with IEG’s recent evaluations; (ii) other WBG development activities that are currently outside of IEG’s evaluations, such as its engagement with the G20; and (iii) partnerships and subsidiaries that involve the WBG and that are large in scale, yet have a relatively small impact on WBG resources. This analysis is at a preliminary stage and will be developed further in the coming months. Its results and implications will be discussed in the next work program and budget document. 10 Some will be included in the final work program document. 11 See below for updates on Management Action Record reform and engagement with management. 7 26. There is a strong linkage between self-evaluation and independent review in the areas of World Bank lending, International Finance Corporation (IFC) financial and advisory services, and increasingly in Multilateral Investment Guarantee Agency (MIGA) guarantee activities. Still, the World Bank’s knowledge and technical assistance services are not systematically covered (nor is it clear how best to cover them) and rely on targeted efforts by IEG through coverage under country, thematic, or clustered PPARs. 27. However, significant areas of institutional activities are not covered by IEG (or self- evaluated by the respective institutions):  Short-term trade finance investments by the IFC, including the $3 billion Global Trade Finance Program, which had already been identified in the previous work program and where work is under way to both integrate trade finance operations in the project evaluation framework and evaluate their overall relevance, effectiveness, and efficiency so far.  At the World Bank, a systematic inclusion of knowledge, technical assistance, and analytic work amounting to about $500 million in annual staff resources that directly benefit clients or contribute to global public goods.12  IFC’s role in affiliated entities, which have a small impact on IFC’s financial commitments but have grown to about $5.3 billion in available funds.13  WBG partnership programs, beyond those falling under the already existing Global and Regional Partnerships Programs.14 These include work principally conducted by the World Bank on global issues and special topics with both Bank resources and Bank-executed Trust Funds. Taken together, these account for about $200 million annually in staff resources that go largely unevaluated.  Finally, a large number of institutional processes and systems that have a direct bearing on the organization’s development effectiveness (such as the process for raising International Development Association (IDA) and Trust Fund resources), amounting to 30–50 percent of total resources across the three WBG institutions. 28. Coverage on a sector and regional basis as well as by institution is strong, as IEG reviews 100 percent of World Bank Implementation Completion and Results Reports (ICRs) of completed projects, a 50 percent sample of IFC’s operationally mature investment projects for which Expanded Project Supervision Reports are prepared, a 70 percent sample 12 See The State of World Bank Knowledge Services 2012. These products are being assessed in the context of country and sector/thematic evaluations, however. These include the Asset Management Company, the African Enterprise Fund, the Small Enterprise Fund, the 13 Venture Capital Fund, the Performance-Based Grant Initiative/Output-Based Aid, and the Funding Mechanism for Technical Assistance and Advisory Services. 14 Appendix 5 lists the formal partnership with recently completed self-evaluations. 8 of Project Completion Reports prepared for IFC advisory services operations, and 100 percent of MIGA’s regular guarantee projects.15 Box 1: Project Evaluation Coverage of World Bank Lending Operations IEG reviews 100 percent of received Implementation Completion and Results Reports, the self- evaluations prepared for all IBRD, IDA, Global Environment Facility operations, and most operations funded by recipient executed trust funds.a Traditionally, IEG has aimed to conduct independent field evaluations (Project Performance Assessment Rrports, PPARs) on 20-25 percent of these operations to enhance accountability and learning and serve as building blocks for major evaluations. However, with the steep rise in recipient executed trust fund operations, as well as the piloting of field reviews of analytical and advisory activities (AAA), the effective coverage of lending operations has declined in recent years to under 10 percent. Contributing to that decline is a dramatic increase in the number of recipient-executed trust fund operations and a doubling of GEF operations. IEG is reviewing the purpose, coverage, and funding of this work to reflect the distinct trend in the operational services and to ensure that the proper trade-offs are being made. Declining Share of Lending Operations Evaluated in the Field, by Fiscal Year of Exit 500 450 # of PPARs needed to reach 25% 400 350 # of Actual PPARs Number of exiting 300 GEF 250 projects 200 Recipient Executed 150 IBRD/IDA 100 50 0 a. ICRs are completed for all operations funded by recipient-executed trust funds over US$5 million. 29. IEG is committed to working with Operations Policy and Country Services (OPCS) to develop a framework for the evaluation of World Bank knowledge products, based on a number of pilots that have been undertaken by IEG, as well as the 2008 report Using Knowledge to Improve Development Effectiveness and the World Bank’s own knowledge report. Similarly, IEG has agreed with OPCS to develop an evaluation framework for the Program for Results instrument, and the agreed follow-up to the 2012 matrix evaluation provides an opportunity to periodically assess progress in the organizational reform agenda. Similar 15 Excluding guarantees underwritten under MIGA’s Small Investment Program, which will be evaluated on a programmatic basis. 9 arrangements are being discussed to cover other areas of IFC, such as the Asset Management Company and other affiliated subsidiaries. FY13–15 Work Program The Challenge of Inclusion 30. The challenge for the coming years is to deliver a work program that is well adapted to the new range of WBG services, generates insights that can help the WBG and its clients be more effective in achieving inclusive growth and poverty reduction, and transforms business process and partnerships. Partners also look to IEG leadership in the evaluation profession and thereby directly contribute to the goal of a New Global Development Partnership that emerged out of Busan.16 31. Against this backdrop, with the MDGs viewed in an applied context, IEG intends to examine from a variety of perspectives how effective the WBG has been in addressing the challenge of inclusion through its work. Although there is no universally accepted definition of inclusiveness, the challenges of reaching the poorest segments of the population, continued gender disparities, the inclusion of minority groups, pressures brought about by population growth and migration to urban areas, reaching people in isolated areas, and protecting those who are most vulnerable to shocks all form part of a picture of inclusive growth and poverty reduction. In short, the approach IEG will use to address inclusiveness could be described as equality of opportunity for fairness of outcomes. 32. Using this lens, IEG will consistently take advantage of opportunities to both select evaluations and to ask pertinent questions within relevant evaluations that shed further light on the WBG’s contribution to achieving development results—the MDGs and beyond. The proposed work program places these evaluations within a four-pillared structure. The structure is meant to allow for questions to be addressed across a set of evaluations and provide useful insights for the post-2015 development challenges. Thematic Clusters of Evaluative Work17 33. The WBG Contribution to Growth and Poverty Reduction: MDG1 and Beyond. This set of evaluations will focus on aspects of the WBG’s global contribution—through financial services, knowledge and advisory services, and partnerships—to poverty reduction from the angle of inclusive growth through economic and social opportunities, including distributional effects and ways of reaching the most impoverished groups. Managing risks and also addressing the challenges of the ―new poor‖ will feature in this work.  IEG intends to conduct a series of evaluations that look at the role of the financial and private sector in providing sustainable services that build the foundation for growth and access and equality also at the base of the economic and social pyramid. See Busan Declaration, 4th High-Level forum on Aid Effectiveness, November 29–December 1, 2011, 16 Busan, Korea. 17 See Appendix 1 for details of IEG’s evaluative work program. This section does not necessarily provide a comprehensive listing of all evaluative products. 10 This work is proposed to be conducted in three phases: innovation and entrepreneurship (FY13), investment climate reform (FY14), and financial inclusion (FY15). It is complemented by a look at WBG support to small and medium enterprises focusing on IFC and MIGA (FY13) and public-private-partnerships for access and sustainability (FY14).  A second series of evaluations looks at the provision and sustainability of basic infrastructure services. Phase I looks at transport (FY13); Phase II will focus on energy and water (FY14). Access and equity considerations for the lowest quintiles of the income distribution will be important considerations.  A further element for this cluster of work relates to the need to protect vulnerable groups from falling into poverty as a result of shocks. IEG will evaluate the WBG’s global food crisis response program (FY13) as well as the WBG’s impact on urban poverty (FY15) and rural non-farm economic growth (FY15). 34. WBG contribution to other MDGs. This set of evaluations will focus on the health and education outcomes and environmental issues that form the major part of the MDGs. Taking stock of existing evaluations and identifying the gaps in evidence will help generate insights of what else needs to be done to advance this agenda.  Earlier reviews of impact evaluations, including nutrition and social safety nets, will be complemented with meta-reviews of impact evaluations on maternal and child health and poverty and gender (FY14). Evaluations of the reform of health systems (FY14) and of early child development (FY15) will deepen insights into the WBG’s effectiveness in these critical areas related to the MDGs.  In addition, Global Program Reviews of the Global Alliance for Vaccines and Immunization and the Global Partnership for Education18 will provide insights into the effectiveness of these partnerships.  In addition, a number of evaluations will address environmental issues, complementing IEG’s past work in this area and filling knowledge gaps. In particular, IEG will conduct a Global Program Review of the WBG’s partnership with the Global Environment Facility (FY13), an impact evaluation on environment (FY14), and an evaluation of green growth analytics (FY15). 35. The Challenge of Achieving Development Results in Different Country Contexts. This set of evaluations will address how country-specific factors affect the opportunities for inclusive growth and poverty reduction and the effectiveness of WBG interventions in different client segments.  For Country Program Evaluations, IEG will focus on middle-income countries (Brazil and Tunisia in FY13) to gain a deeper understanding of their long-term 18 To be confirmed for future years. 11 development challenges, such as governance, distributional impacts of public policies, and the different roles the three WBG institutions are playing.  Building on country and project work, IEG will evaluate the WBG’s activities in the context of knowledge-based country programs (FY13) and in fragile and conflict- affected situations (FY13), followed by an in-depth look at the WBG’s focus on poverty and results in low-income countries (FY14) ahead of the IDA17 replenishment round. A full review of the WBG’s work in middle-income countries is planned for FY15, together with an evaluation of the WBG’s role in promoting South-South development (FY15). 36. Implementing the Development Effectiveness Agenda: Resolutions of the high-level forums in Paris, Accra, and Busan define principles for working with partner countries—ownership, harmonization, alignment, use of country systems—that aim to increase development effectiveness. IEG has evaluated the WBG’s performance against parameters of harmonization and alignment and is starting an evaluation of procurement. An increasing focus on linking these principles to actual outcomes for inclusive and sustainable growth can contribute to informing discussions on development effectiveness.  Building on the matrix evaluation, and to contribute to the further reform efforts of the WBG, IEG will look at the development effectiveness of World Bank procurement policies (FY13) and learn from the different approaches taken to client focus and decentralization by the World Bank and the IFC (FY14).  IEG will also look at the development effectiveness of so-far unevaluated instruments, including trade finance (FY13), and the initial implementation of the new Program for Results instrument (FY15). 37. Much of the work program over the coming years is geared toward making a significant contribution to the global discourse on development and poverty reduction that is likely to take place as 2015 approaches. IEG will have a work program for the outer years that remains to be shaped to achieve synergies across evaluations. IEG will examine the WBG’s role in addressing long-term challenges with a direct poverty impact. IEG will also study the engagement of the WBG with middle-income countries, as many of these are foreshadowing the role the WBG will increasingly have to play in today’s better-performing low-income countries. 38. In the spirit of the ―new development partnership‖ agreed at Busan, IEG will take a look at the role of the private sector in reaching development goals, the role of South-South cooperation and review the WBG’s self-evaluation system within this new development context. Finally, the annual RAP reports will track closely the WBG’s results agenda, including progress in operationalizing the WBG’s Corporate Scorecard. By 2015, the RAP will attempt to develop an overall assessment of progress in the WBG’s contribution to achieving development results. 12 Strategic Focus of IEG’s Project-Level Work19 39. Much of the work mentioned above, which is the majority of what the Board typically reviews, is underpinned by a strong system of project-level evaluations that increasingly include knowledge and advisory services. Beyond the established system of self- evaluation and independent review for financial (Bank, IFC, and MIGA) and advisory (IFC) services, IEG intends to increasingly draw on results from PPARs and other field evaluations of lending and knowledge products. 40. Pending the review of PPARs mentioned in Box 1, several strategic themes will be introduced in the coming years. One will focus on the mega projects that have received substantial financial support from the WBG. Many of these share features such as risk taking and have the potential to be transformative. These PPARs will look for lessons similar to those put forth in IEG’s evaluation of the set of programs in support of the Chad- Cameroon pipeline. 41. A second strategic theme relates to the use of public expenditure analysis to inform policy dialogue and decisions around the allocation of public resources. This is a particularly important question in low-income countries and will feed into the evaluation of the WBG’s focus on poverty and results in low-income countries. 42. IEG has started developing project cluster evaluations covering evaluative findings on IFC and MIGA projects in a sector or theme; these intend to draw out findings and lessons for operational staff. In FY13, these cluster evaluations covered themes such as the poverty impacts of IFC’s projects, extractive industries, and support to South-South investments. 43. Furthermore, IEG intends to expand the concept of in-depth reviews of projects to the private sector, as well as mainstream recently conducted cluster evaluations of private sector projects. Further work will be done to assess the potential for extracting added learning value from IEG’s micro work while fully satisfying IEG’s accountability function. Methodology 44. IEG has been continuously engaged in strengthening and expanding its methodologies, including working with the Evaluation Cooperation Group (ECG) to develop and refine evaluation methods and good practice standards for project evaluation in multilateral development banks. IEG’s recently completed self-evaluation highlighted the need for IEG to develop explicit evaluation standards and to become more transparent with respect to its methods. IEG remains strongly committed to the professional development of its staff and has initiated work with Human Resources to establish evaluation competencies consistent with WBG competency frameworks. A major effort to increase the provision of 19 The references to thematic clusters and PPARs should be considered preliminary until the review of PPARs indicated in Box 3 has been completed. 13 formal evaluation training for staff at all levels has already resulted in substantial uptake.20 The methods work program priorities for FY13 are as follows:  Developing a comprehensive Evaluation Policy that draws together, codifies, and consolidates the current evaluation architecture that was developed separately for evaluation of the World Bank, IFC, and MIGA. This will be based on the need for a clear and transparent policy statement on the principles and norms that govern the WBG’s independent evaluation function, which articulates the purpose and roles of evaluation within the WBG. It will outline organizational roles and responsibilities for the evaluation function and set out the role of self-evaluation. It will also provide principles and norms to guide the conduct of evaluations so as to ensure their quality, objectivity, and utility, consistent with internationally accepted principles and standards. The Evaluation Policy will be delivered in FY13 for discussion and endorsement by CODE and approval by the Board.  Implementing IEG’s recently developed quality standards for Approach Papers based on experience with the ongoing pilot. These standards are particularly important for IEG’s macro evaluations and will complement the evaluation standards that are already in place for IEG’s project evaluations and Global Program Reviews, among others. Similarly, principles for IEG’s evaluation reports have been developed and piloted in FY12 and will be rolled out in FY13.  Developing (i) supplemental guidelines for Country Assistance Strategy Completion Report Reviews tailored to joint (World Bank and IFC) strategies, (ii) a methodology for evaluating IFC’s trade finance program, and (iii) a manual for PPARs.  Piloting the use of ―inclusive data collection‖ methods that are now possible through the use of geographical information systems and other technology in selected evaluations. Engagement and Following Up on IEG Recommendations 45. Building on the work done in FY12, particularly the development of a results framework and reform of the Management Action Record (MAR), strategy work in FY13 will focus on implementation in close coordination with the methodology work program described above. Key elements of the MAR reform include stepping up engagement between IEG and WBG management, establishing clearer linkages between findings and recommendations in the Management Response, and clearly prioritizing IEG recommendations. In addition, the MAR would be finalized 90 days after the CODE discussion to take that discussion into account, including laying out more specific actions and timelines when necessary and feasible. IEG will use the final MAR to report on implementation. 46. WBG management and IEG have jointly undertaken three MAR reform pilots: Assessing IFC’s Poverty Focus and Results; Social Safety Nets—An Evaluation of World Bank Support, 20 Several of the work program areas in this section relate directly to IEG’s results and measurement framework (see Appendix 3). 14 2000–2010; and Capturing Technology for Development—An Evaluation of WBG Activities in Information and Communications Technologies. 47. At the December 12, 2011, meeting to update CODE on MAR reform (CODE/GS2012-0001), IEG and management reported that the MAR pilots have demonstrated positive results: more interaction between WBG management and IEG when drafting recommendations, better understanding of the links between findings and recommendations, increased clarity in actionable recommendations, and increased ownership by WBG management of agreed recommendations. The engagement on recommendations prior to CODE discussion is being rolled out in all IEG evaluations that include recommendations, which may require additional time for engagement throughout the evaluation cycle. 48. In addition, the stock of recommendations for IFC and MIGA has been reviewed, and outstanding recommendations that were no longer relevant have been retired. A similar process will be carried out for the World Bank. A new standardized reporting system is also being developed for tracking recommendations for consistency across institutions; this will be operational in FY13. The reporting and analysis of recommendations has taken place both via the quarterly reports to CODE and the RAP. This year’s RAP will provide an assessment of the WBG’s adoption and implementation of IEG recommendations from a risk perspective. Learning, Openness, and Transparency 49. Under its new Access to Information Policy, IEG released to the public the complete collection of World Bank project performance ratings. The more than 8,000 project assessments correspond to about 6,000 completed projects since IEG was established in 1967.21 It is the longest running development project performance data collection of its kind. IEG also declassified several project- and country-level evaluations and provided a streamlined means to address information requests from the public. 50. In coordination with WBG management, IEG is introducing a new MAR tracking system building on industry-leading software. The new system was piloted in March 2012 and is currently being customized for implementation in FY13. IEG expects it will provide greater expediency and transparency to the recommendation follow-up and reporting process. 51. With respect to project evaluation, IEG responded to demand from the Bank and IFC staff for training on how to produce high-quality ICRs and Expanded Project Supervision Reports (XPSRs), tailoring its findings and lessons more directly to operational teams and reaching more than 300 World Bank and IFC staff so far in FY12. IEG will continue to deliver training to Bank Group staff upon request and will also seek opportunities to do develop e-learning programs based on existing platforms developed by the World Bank Institute. 21 This includes PPAR ratings on projects that had already been rated as part of the ICR Review process. 15 52. IEG has planned further investments to align its information management systems with new strategic priorities. A new WBG-wide integrated platform will facilitate collaboration and improve work program coordination, streamline access to evaluation- related data resources and analytics, and better support learning from IEG evaluations. Being a centralized unit within an increasingly decentralized organization, IEG is also investing in technologies to bring learning programs to country office staff on a much more frequent basis. 53. In a recent (2011) client satisfaction survey, WBG staff confirmed their interest in project-level evaluation and in lessons derived from experience. To this end, IEG began working with OPCS on a joint ―good practices‖ pilot database, which included lessons contained in PPARs. The database could be linked into the Operations Portal, the World Bank’s core portfolio management system, thus enhancing field staff access to IEG findings and lessons. A similar system, eLRN (―e-Learn‖), has been developed for IFC staff and will be extended to include MIGA projects next year. Box 2: Staff Learning Going Global Complementing IEG’s new Access to Information policy, IEG is able to produce learning materials and open training programs up to non-WBG staff. Evaluation Week, which is now in its eighth year, is much different than when it first began. In October 2011, IEG hosted more than 30 learning sessions that ranged from sharing knowledge on how to write good ICRs and Expanded Project Supervision Reports to why it is important to mainstream M&E results into public budgeting decisions. The events were open to external participants who learned about them through IEG’s dedicated website, social media, or promotional emails. The dedicated website generated 5,095 page views and 3,813 unique visitors in October from 126 countries. Facebook updates highlighting the live-stream videos and real-time updates from Evaluation Week generated more than 7,000 impressions. Fifteen sessions were streamed live to external audiences in 58 countries and then saved on IEG’s video server. Increasingly, IEG staff are sought out for their expertise on evaluation topics. Evaluation Week is now drawing several hundred participants from the WBG and many external evaluation units. Over the past year, IEG has been asked to take a major role in learning events that were organized for staff at the African and Islamic Development Banks, the Organisation for Economic Co-operation and Development–Development Assistance Committee, the European Development Finance Institution meeting, and the International Program for Development Effectiveness Training. IEG staff also participated in events organized by the American and African Evaluation Associations to share evaluation findings and methodologies with development practitioners. Building Client Monitoring and Evaluation Capacity 54. IEG actively engages with partners to support evaluation capacity development in client countries and to share evaluation findings with the broader development community. FY12 was the second full year of the implementation of the CLEAR (Centers for Learning on Evaluation and Results) program. CLEAR was established in response to the commitments under the Paris Declaration (2005) and the Accra Agenda for Action (2008), and it finds renewed impetus under the declarations made at the Busan High-Level Forum (2011). IEG also continued its flagship International Program for Development Effectiveness Training (IPDET) and the development of M&E knowledge products and advisory services. IEG’s evaluation capacity development program collectively supports the foundations of 16 transparency, accountability, and evidence-based decision making, which are needed for catalyzing inclusive and sustainable growth and poverty reduction at the country level. 55. In FY12, working closely with the Bank’s regional vice-presidential units, CLEAR solidified its presence in Africa and South Asia by establishing the two competitively selected regional institutions—the University of Witwatersrand in South Africa and JPAL-South Asia at the Institute for Financial and Management Research in India. Both institutions received grants and began implementing their programs, working with key clients such as the South African Presidency’s Department for Performance Management and Evaluation and Pratham, a key nongovernmental organization in education, in India. In addition, CLEAR selected two more institutions: Centre Africain d’Etudes Supérieures en Gestion, in Senegal for Francophone Africa and Centro de Investigación y Docencia Económicas in Mexico for Spanish-speaking Latin America. The collaboration with the Asia-Pacific Finance and Development Center in China continued, with an expanded set of programs for East Asia and piloting of a key workshop on impact evaluation. IEG also began linking learning from its evaluation work with this initiative, thereby creating win-win opportunities for learning among all partners. IEG and CLEAR Center Africa held the first joint workshop on IEG’s evaluation, Assessing IFC’s Poverty Focus and Results. 56. CLEAR’s donor partners expanded and now include the African Development Bank, the Asian Development Bank, the Australian Agency for International Development, the Belgian Development Cooperation, the Inter-American Development Bank, the Rockefeller Foundation, the Swedish International Development Agency, the Swiss Agency for Development Cooperation, and the United Kingdom Department for International Development. The total contributions to date are approximately $6.7 million. 57. In FY13 CLEAR will go through a consolidation phase, with the five centers spanning four regions establishing their programs more firmly and building regional networks to bring quality know-how and learning opportunities close to country clients. They will implement a range of approaches to evaluation capacity development and work with a variety of clients at all three levels of capacity—individual, organizational, and institutional. The indicators of program success include the centers expanding the range of their capacity development services, the quality of their services being rated highly by clients (as evidenced through evaluations), and increasing revenues for their services. In addition, IEG will seek appropriate opportunities to advance joint client learning based on IEG’s and other partners’ evaluations. 58. Over the past 10 years, IPDET has provided training to more than 2,000 practitioners from developing countries. IEG is continuing this program in a joint partnership with Carleton University. Agreement was reached to develop a sister program in French with the École nationale d’administration publique in Quebec. 59. IEG also continued its unique knowledge publication series on how governments develop, maintain, and use M&E systems. In FY12, IEG published case studies on China, Chile, Korea, and the United States. IEG also continued its advisory services and technical assistance on a strategic, demand-driven basis, including sharing knowledge on M&E with South Africa and the Dominican Republic. In addition, IEG continued the Bank-wide 17 practice group on M&E in collaboration with the Poverty Reduction and Economic Management Network. Staffing and Budget Human Resources 60. IEG will continue to keep staff skills closely aligned with the needs of its work program. As of end March 2012, staffing levels in IEG stood at 101, below its full complement of about 115. Internal mobility in IEG compares favorably with other operational units and the Bank as a whole, and IEG is taking advantage of staff turnover to refresh and renew its staff skills base. IEG management is conscious of the need to draw both on WBG experience and on experience from other organizations. The proportion of professional staff (GE+) at the end of February 2012 that were recruited from outside the WBG stood at 49 percent, with 77 percent of FY12 hires at the GE+ level having had no previous WBG employment. IEG will continue to maintain staff and other fixed costs at a level consistent with a fixed cost ratio of around 70 percent to allow for adequate flexibility. Budget Proposal 61. Sources. IEG’s projected resources to deliver its FY13 work program total $35.3 million, comprising a $33.2 million regular budget to be approved by the Board, $1.2 million in trust funds, and $0.2 million in other revenues. This resource envelope is consistent with the budget framework agreed at last year’s work program discussion with the Board. IEG is also requesting the reinstatement of carryover for FY13, as detailed below; if approved, this would amount up to $0.7 million of IEG’s approved budget in carryover resources in FY13, based on a projected under-run for FY12. 62. At the time this budget proposal is being submitted, IEG is aware of changes to the Bank’s chargeback methodology, which will be cost neutral at the institutional level but may not be cost neutral at the level of the vice-presidential unit. IEG has not been apprised of the full impact of these changes; accordingly it has assumed that they will be cost neutral and that any changes will result in a commensurate increase or decrease in the World Bank contribution to IEG. 63. The regular budget is set out in Table 1, which shows recent trends as well as projections in real terms using FY12 prices. IEG’s budget for FY13 is based on the following:  No change is proposed for the World Bank contribution, which comprises almost 80 percent of IEG’s regular budget sources and has remained flat in real terms since FY05.  No increases are proposed for IFC and MIGA either. In FY11, MIGA agreed to absorb $0.12 million in headcount-based nondiscretionary fees and charges; this continues to be reflected in a lower IEG funding requirement, as set out in Table 1. In the event that these fees and charges rise, MIGA’s contribution to IEG will be increased by a commensurate amount. 18 Table 1: IEG Institution Contributions, FY10–15 (in constant dollars using FY12 prices) FY10 FY11 FY12 FY13 FY14 FY15 Budget Budget Budget Proposed Indicative Indicative $m $m $m $m $m $m WB Contribution 25.7 25.8 25.8 25.8 25.8 25.8 IFC Contribution 5.9 5.7 5.9 5.9 5.9 5.9 MIGA contrubution 1/ 1.5 1.4 1.4 1.4 1.4 1.4 Total IEG 33.2 33.0 33.2 33.2 33.2 33.1 IEG as % of parent institution budget 2/, 3/ WB 1.4% 1.4% 1.4% 1.4% 1.4% 1.4% IFC 1.1% 1.0% 1.0% 1.0% 1.0% 1.0% MIGA 3.7% 3.7% 3.2% 3.2% 3.2% 3.2% Total 1.4% 1.3% 1.3% 1.3% 1.3% 1.3% 1/ The approved contribution from MIGA assumes that it will continue to absorb $0.12m in IEG related non-discretionary fees and charges. In the event that MIGA chosed to depart from this practice which it initiated in FY11, or these fees and charges rise, MIGA's contribution to IEG will be increased by a commensurate amount. 2/ In the case of the World Bank and IFC, the Net Administrative Budget and Regular Administrative Expenses budgets have been used as the denominator. In the case of MIGA, for comparison purposes, the denominator comprises the Regular Administrative Budget plus FIAS contributions, Contingencies, Rent Increase allocations, CAO/Ombudsman office, and Depreciation. For consistency purposes, the numerators have also been adjusted, with an adjustment being added to the IFC contribution for space and IT, and with the MIGA contribution being reduced in the amount of overheads and pension contributions. 3/ Percentage calculations assume that budgets for the World Bank, IFC, and MIGA will remain at FY12 levels in constant dollars 64. Carryover. IEG is requesting authority to carry over to FY13 unspent FY12-approved funds of up to a maximum of 2 percent of its total spending authority, or $0.7 million. IEG expects an under-run of $1.4 million in FY12, owing to a higher than anticipated level of staff vacancies, which has challenged the timely delivery of the FY12 work program. In response, IEG management focused on ensuring that FY12 commitments were delivered in full, redeploying resources as necessary to ensure that this was achieved. As a result, the start of evaluations planned for FY13 has been delayed, with likely bunching in the future. To address the bunching issue, IEG is therefore requesting the carryover to accelerate work on the FY13 work program, with a view to ensuring a better distributed delivery of output to CODE by FY14. Any carryover would be deemed to derive from all contribution sources pro rata. 65. Trust funds. IEG expects to receive $1.2 million in trust funds. These are incorporated into IEG’s spending plans in Table 2 and Appendixes 5 and 6. Of these, $0.8 million will finance studies and $0.4 million will finance IEG’s evaluation capacity development program, including the CLEAR secretariat and a scholarship program for IPDET. In addition, donors have contributed around $5 million to the CLEAR program to fund regional centers. Although these funds are also managed by IEG, they cannot be used by IEG to support its work program. 66. Recognizing the imperative of protecting independence from misperceptions that could be caused by reliance on trust funds, IEG has put in place clear procedures on the use of such funds, including a requirement that aggregate use of trust funds should not exceed 15 percent of total spending on products and should also not exceed 25 percent of the cost of any individual product. In addition, IEG’s procedures require that trust funds be used for noncore evaluation activities, such as literature reviews, case studies, focus groups, and in- 19 country workshops, but not for core activities, such as strategy development, portfolio reviews, and approach paper preparation. 67. IEG also receives $0.2 million in other income, including $80,000 from the Global Environment Facility for ICR and PPAR work on Facility projects, and $40,000 in trust fund fee income. 68. Capital budget. A capital budget request of $450,000 was approved last year to fund several ongoing projects. These include an upgrade of IEG’s product-tracking tools— tracking progress on the micro products work program—and facilitating the follow up on recommendations of IEG evaluations and progress against IEG key performance indicators. IEG is not requesting a capital budget authorization this year. Implementation of these programs is on track and should be completed as envisaged during the course of FY13. 69. Spending trends. Table 2 sets out trends in resource use over the FY10–14 period, comprising not just regular budget resources but also trust funds and other income. Table 2. Summary of Sources (including Trust Funds) and Uses and Key Spending Ratios, FY11–15 (nominal $) FY11 (act) FY12 (plan) FY12 (proj) FY13 (est) FY14 (est) FY15 (est) $m $m $m $m $m $m Total Sources 33.9 34.0 34.3 35.3 34.5 34.5 Uses Project/AAA Evaluations 6.0 6.7 6.3 6.3 6.3 6.3 Thematic/Sector/Corporate 10.0 8.3 9.3 10.5 9.7 9.7 Country and Global 2.2 4.1 3.2 3.3 3.4 3.4 Knowledge Activities 4.9 4.9 4.8 4.7 4.7 4.7 Rapid Response 0.6 0.6 0.3 0.6 0.6 0.6 Sustaining 5.7 5.1 4.4 5.1 5.1 5.1 Indirects 4.4 4.4 4.6 4.7 4.6 4.6 Total Uses 33.8 34.0 32.9 35.3 34.5 34.5 Proportion of spending on Products 56% 58% 58% 59% 58% 58% Proportion of Spending on Knowledge Activities 15% 14% 15% 13% 14% 14% Proportion of Spending on Sustaining Activities 17% 15% 13% 15% 15% 15% Proportion of Spending on Indirect Costs 13% 13% 14% 13% 13% 13% Nominal Spending Growth 1% -3% 7% -2% 0% Cost flexibility - Fixed Cost Ratio 70% 69% 63% 70% 72% 72% Note: Variation in nominal spending growth between FY11 andFY15 is primarily related to the carryover request 70. IEG’s strategic directions are reflected in the spending proposals above. In FY12 IEG made a commitment to strengthen its country and global work to include Regional reviews. This effort will be sustained in FY13–15, albeit at somewhat lower cost than was planned for FY12. IEG is committed to delivering 11 major studies each year, including the annual RAP. In addition, IEG has committed to provide Network notes at the time of the annual Network Updates, as the Board has requested. A particular focus of IEG’s efforts in FY13 will be addressing a longstanding bunching issue that was made more acute this year by high staff 20 vacancies. Over the past two years, IEG has set aside resources to fund rapid response activities that address urgent issues and strategic priorities not planned for at the time of budget formulation. IEG has included $0.6 million in its plans for this purpose again in FY13–15. 71. Consistent with FY12, IEG has programmed two weeks per staff in FY13–15 for both learning and institutional priorities to support staff learning, greater collaboration across the WBG, and engagement in corporate initiatives, reflected in a planned increase in spending in knowledge activities and indirect costs. 72. The aggregate size of IEG’s budget was reviewed in the context of the self- evaluation (CODE2011-0013) conducted in FY11. The report noted that IEG has been operating in a severely constrained budget environment and that efficiency levels have increased. Contributions from the World Bank, IFC, and MIGA were compared with ECG figures. The conclusion was that ―After allowing for scale, the ratios for the WBG are not out of line with ECG comparators, although IFC is still at the low end.‖ Accountability 73. IEG adheres to WBG budget rules and procedures. IEG is subject to Controller’s Quality Assurance reviews of selected expenses and has consistently received favorable ratings on adherence to budget rules, procedures, and policies. IEG’s control environment continues to be ranked as one of the strongest in its peer group. 74. In regard to external audit, the contributions by World Bank, IFC, and MIGA are audited as part of regular external audits of those institutions. External Assessment 75. In recent years, CODE and IEG have followed a procedure by which IEG has undertaken self-evaluations of its budget procedures and arrangements. CODE has supplemented these with an external independent review.22 In accordance with the agreed procedure, CODE commissioned an external review of an earlier draft of this paper.23 Overall, the review provided a favorable assessment of IEG’s proposal, including IEG’s $0.7 million carryover request. IEG values these periodic external reviews, as they provide valuable third-party input into continuous improvements in IEG’s processes and monitoring systems. 76. As noted, IEG will work in particular on reducing the bunching over the coming years. The guidance from the external review is helpful, and IEG will report in the quarterly reports on forthcoming approach papers and evaluations to track the degree of preparedness and de-bunching. Although ―de-bunching‖ may not fully occur in FY13, the carryover should significantly help in supporting additional work during FY13 that will put IEG in a better position to deliver its major evaluations in a timely fashion for policy input (for 22 Procedure for CODE review of the Independent Evaluation Group’s annual budget proposal and supplemental note (R2006-0196, R2006-0196/1). 23 External Review of Independent Evaluation Group’s Budget for FY13 and Linkage to its Work Program (BC2012-0003, CODE2012-0011). 21 example, IDA replenishment) and to space them more equally throughout the fiscal year in FY14 and beyond. 77. There are a number of recommendations related to the tracking of expenditures against plans. These will be incorporated as part of IEG’s internal tracking mechanisms, including the resources devoted to IEG’s communications, outreach, and learning efforts to support IEG’s evaluative work. Some of these changes are already reflected in IEG’s quality standards for approach papers. Conclusion 78. Through independent evaluation, IEG can play a vital role in informing Board and CODE discussions on progress toward WBG objectives. IEG’s proposed FY13 work program and FY14–15 indicative plan reflect a comprehensive and balanced set of evaluations across the World Bank, IFC, and MIGA that will provide relevant and timely information to the Board that should help in decision making. Accordingly, IEG would like to recommend that the Executive Directors approve:  IEG’s proposed work program for FY13  Funding in the amount of (i) $25.78 million from the World Bank; (ii) $5.92 million from IFC; and (iii) $1.42 million from MIGA, toward the cost of delivering IEG’s approved work program, with commensurate adjustments that reflect the Board approved price factor, and institutional chargeback adjustments  Approval of carryover authority of unspent FY12 budget (amounting up to $0.7m of IEG’s approved budget) to FY13  Continued pooling of World Bank, IFC, and MIGA funding in one location under the Director-General, Evaluation’s IEG-wide direction, with continued accountability for the use of resources across the World Bank, IFC, and MIGA. 22 Appendix 1: Proposed Evaluation Program FY13–15 FY12 FY13 FY14 FY15 1. PROJECT-LEVEL REVIEWS AND EVALUATIONS WORLD ICR Reviews ICR Reviews ICR Reviews (100% ICR Reviews BANK (100% of (100% of of completed (100% of completed completed lendinglending operations completed lending operations with with available self- lending operations with available self- evaluations) operations with available self- evaluations) available self- evaluations) evaluations) IFC XPSRs IFC XPSRs IFC XPSRs IFC XPSRs IFC (random 50% (random 50% (random 50% (random 50% coverage), PCRs coverage), PCRs coverage), PCRs coverage), PCRs (100% coverage) (70% random (70% random (70% random coverage) coverage) coverage) MIGA Project MIGA Project MIGA Project MIGA Project MIGA Evaluations Evaluations Evaluations Evaluations Direct evaluation Direct evaluation Direct evaluation Direct evaluation and validation of and validation of and validation of and validation of PERs (100% of PERs (100% of PERs (100% of PERs (100% of regular guarantee regular guarantee regular guarantee regular guarantee projects projects projects projects PPARs—for PPARs—including PPARs—including PPARs—including WBG24 about 20–25% of series of clustered series of clustered series of clustered completed reports on public reports on Public reports and projects, and private sector Expenditure IFC/MIGA including AAA; Mega projects; and Analysis and sector project and IFC/MIGA IFC/MIGA sector Equity; and cluster sector project project cluster IFC/MIGA sector evaluations cluster evaluations project cluster evaluations evaluations 2. PROGRAM, COUNTRY, AND THEMATIC EVALUATIONS The WBG’s Crisis Response Infrastructure Infrastructure Financial Contribution Phase II Sustainability I Sustainability II Inclusion to Growth Youth (Transport) (Water and Energy) Urban Poverty and Poverty Employment Innovation and Investment Climate Rural Non-Farm Reduction; Forest Resources Entrepreneur- Reform Economic MDG 1 and Climate Change ship PPPs for Access Growth Beyond Phase III Global Food Crisis and Sustainability (Adaptation) Response Support to SMEs (focus on IFC and MIGA) 24 As mentioned above, IEG is currently reviewing opportunities for greater learning from project-level work in the public and private sector. The clusters and themes indicated in this table should be considered preliminary until the review is completed. 23 FY12 FY13 FY14 FY15 The WBG’s Global Program Global Program Global Program Global Program Contribution Reviews (Global Reviews (GEF, Reviews (tbd) Reviews (tbd) to Achieving Fund, Global GAVI) Environment Early Child Other Fund for Disaster Maternal and Impact Evaluation Development MDGs Reduction and Child Health Reform of Health Green Growth Recovery, Forest Meta Impact Systems Analytics Carbon Evaluation Poverty and Gender Private Sector Partnership) Meta Impact Role in Review of Impact Evaluation Achieving Evaluations MDGs The Country Program Country Program Country Program Country Program Challenge of Evaluations Evaluations Evaluations Evaluations Achieving (Liberia, (Brazil, Tunisia), (focusing on (countries tbd), Development Afghanistan), including CASCR MICs), including including CASCR Results in including CASCR Reviews CASCR Reviews Reviews Different Reviews Knowledge-based WBG Focus on WBG in Middle- Country Country Poverty & Results Income Contexts Programs in Low-Income Countries WBG in Fragile Countries WBG Role in and Conflict- Promoting affected South-South Situations Development Implementing Matrix Procurement Client Focus and Evaluation the Evaluation Trade Finance Decentralization System for Development Evaluation Private and Effectiveness System for Public Sector Agenda Private Sector Operations Operations Evaluation of the P4R Instrument 3. ANNUAL AND QUICK RESPONSE NOTES AND EVALUATIONS Results and Results and Results and Results and Performance Performance Performance 2014 Performance 2012 2013 IEG Annual Report 2015 IEG Annual IEG Annual Report Regional Updates IEG Annual Report Regional Updates Network Updates Report Regional Updates Network Updates Quick Response Regional Updates Southern Sudan Quick Response Notes Network Updates Note Notes Quick Response Bulgaria Note IDA16 Mid-Term Notes Financial Sector Note Note Note: AAA = analytic and advisory activities; CASCR = Country Assistance Strategy Completion Report; GAVI = Global Alliance for Vaccines and Immunization; GEF = Global Environment Facility; IDA = International Development Association; MIC = middle-income country; P4R = Program for Results; PCR = Project Completion Report; PER = Project Evaluation Report; PPAR = Project Performance Assessment Report; PPP = public-private partnerships; SME = small and medium enterprise; XPSR = Expanded Project Supervision Report. . 24 Appendix 2: Selectivity Frameworks25 State-Building and Peace-Building in Fragile and Conflict-Affected States: An Evaluation of World Bank Support—FY13/14 Proposed Evaluation Coverage, Rationale, and Expected Outcome (from IEG Work Program and Budget):  This evaluation will focus on World Bank support for state-building and peace-building efforts in fragile and conflict-affected states (FCS). It will be wholly concerned with Bank operations in FCS and will not address corporate issues of aid allocation (in particular IDA) to FCS or review the internal organizational set of the Bank’s FCS work (for example, the recent establishment of OPCS’s FCS Nairobi hub).  In its 2006 evaluation of Bank support for low-income countries under stress, IEG emphasized the need for a clear operational strategy on state building and peace building. State building was considered a critical element to promoting stability, legitimacy, and ultimately poverty reduction in these countries. Since then the Bank has embarked on various initiatives to define and operationalize state-building efforts, including a 2007 strategy on responding to emergencies, an approach to staffing in FCS settings, various aspects of the Governance and Anticorruption Strategy, and the 2011 World Development Report.  The relevance, efficacy, and sustainability of these efforts are key evaluative concerns, which should inform ongoing efforts of the Bank and key development partners. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Since  Who is the specific audience in the  How does this evaluation fit the 2006, several Board and CODE members have expressed strong WBG group for this evaluation and its within IEG’s mandate? The interest in learning more about FCS issues in IEG’s sector, recommendations? Given the evaluation is wholly concerned with corporate, and global reviews. Several civil society organizations operational focus of this evaluation, the the relevance and efficacy of Bank currently involved in FCS would like to know how the Bank is main audience should be operational operations in FCS, and therefore doing in this important area. teams in the Regions. Anchor units are well within IEG’s mandate.  Is this in line with WBG strategic priorities? ―Addressing the also key including OPCS FCS Dept and  Is it within the scope of another special challenges of post-conflict and fragile states‖ is one the Bank its Nairobi hub (OPCFC). oversight group (IAD, INT, IPN, Group’s six strategic themes. Issues of state-building in FCS were  Who will be the WBG ―champion‖? At CAO)? There are certainly risk given special prominence in the recent World Development Report, a leadership level, it will be important to management issues in FCS and they are considered ―foundational‖ to the Bank’s new Africa identify an operational MD that begins to operations that are within the strategy. own the FCS agenda. At the technical purview of IAD and INT. Similarly,  Does this address a risk to WBG development effectiveness? level, OPCFN, PREM and its some budget related issues will need State weakness in FCS pose significant risks for IDA, IFC, and Governance and Anticorruption to be addressed. However, IEG is MIGA operations—all of which have expanded over the past Secretariat, and Social Development the only unit with the mandate to decade. should serve as champions. integrate these concerns with to the  Is this aligned with key WBG corporate deliverables, decision  What is the evaluation expected to goal of improving the Bank’s overall points? Progress on state and peace building in FCS is pivotal to impact? Bank operations in FCS. operational impact. 25 Selectivity frameworks have not been prepared for Country Program Evaluations or other evaluations that follow an agreed format (such as GRPPs, annual reports, such as the RAP, or evaluations where an Approach Paper has already been submitted to CODE). 25 the success of IDA, and to meeting IDA16 commitments.  Are there critical decision points  Does IEG have the staff to  Is there need for follow-up based on the MAR (unfinished within the WBG which would drive the conduct this evaluation? Yes but business)? The MAR for IEG’s 2006 review of low-income timing of the evaluation? The IDA16 will need to recruit consultants and a countries under stress recommends an operational strategy on state- Mid-term Review in 2012 should provide strong advisory panel with relevant building in FCS. Yet the recommendation has been only partially an opportunity for IEG’s early inputs. A expertise. addressed and requires a nuts and bolts understanding of what more substantial platform would be the  What other IEG deliverables works and what does not. IDA17 Replenishment. would this evaluation draw on  Will this address a knowledge gap in IEG? Yes. Since 2006, and/or contribute to? The five IEG’s treatment of FCS in its sector and corporate reviews has Country Program Evaluations been sparse. However, important Country Program Evaluations in recently conducted in FCSs will Nepal, Timor Leste, West Bank and Gaza, and more recently, serve as building blocks as well as Liberia and Afghanistan, offer an opportunity to develop an the large data set and country cases integrative view of WBG performance. prepared for the governance and  Are there other reasons why IEG should proceed with this anticorruption evaluation. Similarly evaluation? relevant sector and corporate Such a review could benefit the entire development community, reviews will be mind in preparing particularly those agencies that are currently partnering with the this evaluation. WBG. 26 Procurement in Bank Operations—FY13 Proposed Evaluation Coverage, Rationale, and Expected Outcome from IEG Work Program and Budget):  This evaluation will consider the role that WBG procurement policies and procedures play in three areas: (i) fiduciary responsibility and integrity in the discharge of World Bank project financing, leading to transparent and cost-effective implementation of World Bank-supported projects; (ii) efficiency in the implementation of WBG-supported projects, from the point of view of prompt, cost-effective, and quality procurement of goods and services for the benefit of the client country; and (iii) the extent to which the Bank’s procurement systems have contributed to strengthening country procurement systems and institutions and helped countries develop their own capacity in this area. It is likely that the evaluation will focus in particular on the last aspect. In this context, it will also review the quality of diagnostic work on procurement undertaken by the Bank, as well as the quality of follow up to diagnostic findings. It will look in particular at the variation in results across different country groupings and consider the question of whether the present ―one-size-fits-all‖ set of policies and procedures is appropriate. The evaluation will also review the Bank’s procurement policies vis-à-vis those of other multilateral development banks and financial institutions in order to better understand the extent to which the Bank has maintained a good- or best practice approach to procurement in global terms.  The overall balance between these topics is likely to be determined in tandem with the scope of work being undertaken elsewhere in the WBG, notably IAD and INT work on fiduciary aspects of compliance with procurement policies and procedures.  The findings and recommendations of the evaluation would help improve the contribution of procurement rules to the development effectiveness of World Bank- financed projects, as well as their contribution to building procurement-related institutions and capacity in Bank client countries. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG  Who is the specific audience in  How does this evaluation fit stakeholders?Procurement is an area of particular interest to the WBG group for this within IEG’s mandate? IEG’s the Bank’s Board and management in the context of the evaluation and its mandate of ―assessing whether current discussions on the review and anticipated reform of recommendations? The audience the WBG’s programs and procurement under Bank-financed projects. It is also of interest would include Bank senior activities are producing expected to the scores of private companies that are contractors and management, which will steer results‖ would appear very clearly suppliers under Bank-supported projects. procurement reform and the to encompass procurement, since  Is this in line with WBG strategic priorities? As the modernization agenda more the latter is a key step in the upcoming internal review suggests, procurement is a key area broadly, as well as the Board, who implementation of Bank- where Bank management has acknowledged that the current will oversee these efforts. supported projects and an policies and procedures are outdated and reform is widely  Who will be the WBG important driver of outcomes. believed to be necessary. ―champion‖?  Is it within the scope of another  Does this address a risk to WBG development  The obvious WBG champion is oversight group (IAD, INT, effectiveness? Failure to adequately overhaul the current OPCS, as well as WBG senior IPN, CAO)? INT and IAD have procurement policies and procedures would pose a risk for management, who are steering the both conducted procurement- development effectiveness in three respects: fiduciary internal reform agenda. In terms of related work (both are ongoing), safeguards, efficient implementation of Bank-supported oversight, several Board members although INT’s specific focus is projects, and development of client country procurement have already shown signs that they on the integrity of procurement 27 institutions and capacity. consider review and reform of transactions, and IAD’s on  Is this aligned with key WBG corporate deliverables, procurement to be a priority. compliance with Bank policies decision points? The IEG evaluation would run in parallel to  What is the evaluation expected and procedures. In defining the management’s own procurement review. The evaluation would to impact? The evaluation is scope of work in the evaluation, inform a second phase of management review, which would expected to impact operational IEG will consult closely with focus on adapting the Bank’s operational rules and guidance on policies and guidelines on these two oversight groups, as procurement in order to reflect needed reforms. procurement and is also likely to well as more broadly, to ensure  Is there need for follow-up based on the MAR (unfinished impact diagnostic tools for complementarity of efforts. business)? A recommendation from a prior evaluation of procurement-related knowledge as  Do we have the staff to conduct fiduciary diagnostics—relating to the need for diagnostics to be well as approaches to building this evaluation? Although a task complemented by an integrated view of necessary reforms in procurement-related institutions team leader has been identified, public expenditure management (which encompasses PFM and and capacity in client countries. the team is still being put together. procurement) at the country level—remains outstanding. This  Are there critical decision points This is posing considerable evaluation can help shed light on the degree of progress that within the WBG which would challenges because of the timing has been made in practice. drive the timing of the in the fiscal year (where the work  Will this address a knowledge gap in IEG? Even though evaluation? It would be important programs for most staff and procurement diagnostics were reviewed under an evaluation of for the findings and consultants are already fiduciary diagnostics in 2007, and procurement was included in recommendations of the evaluation determined) and especially to the the IDA controls and GAC evaluations, there has been no in- to be available to management recent change in policy depth focus on how well current procurement policies and when decisions are made regarding concerning the use of retirees as procedures serve the goals of fiduciary safeguards, efficient changes to be made to the Bank’s short-term consultants (which implementation of Bank-supported projects, and development procurement-related operational renders several knowledgeable of client country procurement institutions and capacity. policies and procedures (currently potential consultants ineligible to  Are there other reasons why IEG should proceed with this expected to be early in the second work for one year as of July 1). evaluation? Bank management requested this evaluation some half of FY13), and to the Board  What other IEG deliverables time ago and regularly reaffirmed interest. Underlying this is when these changes are presented would this evaluation draw on the view that an independent evaluation in parallel with for Board consideration (later in and/or contribute to? The Management’s own review would help focus and strengthen FY13). This points to the need to evaluation is expected to draw on the basis for subsequent procurement reform. The second coordinate closely with several recent IEG evaluations reason is interest by several key Bank shareholder management on the schedule for (for example, governance and constituencies that have urged IEG to review the procurement its own phase I review. anticorruption). It would function to help lay the groundwork for any needed reforms. contribute to several others, notably country evaluations, as well as to project reviews. However, no major ―critical path‖ issues are foreseen. 28 Knowledge-Based Country Program—FY13 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  This evaluation will look at the relevance, quality, and results of the WBG’s knowledge-based country programs, where knowledge services have become the main conduit for assistance in the partnership strategy.  As country programs, especially in middle-income countries, become increasingly based on knowledge services, the WBG needs to understand when is it that it can be the best possible partner to client countries and what are the knowledge products where the WBG’s comparative advantage and swiftness of response is the most valued by client countries that do not primarily rely on its financing services. The findings and lessons would help the WBG structure its partnership strategies with countries that are demanding higher knowledge content and that are not in urgent need of the WBG’s lending products. They would also help the WBG effectively leverage its global knowledge to further help client countries that have become less dependent on external finance make further strides in the reduction of poverty. This is particularly the case in middle-income countries, where the more pressing needs lie in finding innovative solutions to specific development challenges, which often have global ramifications, such as climate change and the food and financial crises Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders?  Who is the specific audience in  How does this evaluation fit Cear interest expressed by management in light of Post-Crisis Directions Paper] the WBG group for this evaluation within IEG’s mandate?  Is this in line with WBG strategic priorities? Knowledge is a critical element of and its recommendations? Within mandate—―Assessing the WBG’s internal reform agenda—―The knowledge agenda is an integral part of Operational managers (Regional vice whether the WBG’s programs the overall reform effort and essential to supporting all five WBG priorities‖. (New presidencies, Directors of and activities are producing World, New World Bank Group: Internal Reform Agenda, SecM2010-0147). In its 2011 Operations, Regional Chief expected results.‖ Regional Strategy Update, East Asia and Pacific identified ―promoting a knowledge Economists, Country Directors,  Is it within the scope of economy‖ as one of the main challenges for middle-income countries. SDs), staff working on middle- another oversight group  Does this address a risk to WBG development effectiveness? Knowledge income countries and OPCS. (IAD, INT, IPN, CAO)? services are essential to Bank effectiveness. In FY11, the Bank allocated 31 percent  Who will be the WBG Other units have not yet been of its administrative budget on ―core knowledge services.‖ Lessons will be drawn on ―champion‖? Regional Directors of consulted, but they would not how the WBG can improve its development effectiveness by conveying customized, Operations; Regional Chief likely look at AAA practical, and timely solutions to MICs that rely increasingly on its knowledge Economists. performance in the context of services.  What is the evaluation expected to country programs.  Is this aligned with key WBG corporate deliverables, decision points? The impact? Improved CASs/CPSs in  Does IEG have the staff to evaluation will inform the design of Country Partnership Strategies in important knowledge-based country programs. conduct this evaluation? MIC clients that are demanding higher knowledge content from the WBG, and Methodology for assessing the Team has already been contribute to developing a more robust methodology for the assessment of analytic relevance, quality, and results of assembled. and advisory activities (AAA) effectiveness by the WBG. AAA. Design of World Bank AAA  What other IEG deliverables  Is there need for follow-up based on the MAR (unfinished business)? The and IFC Advisory Services so as to would this evaluation draw AAA evaluation (2008) found that government receptivity influenced effectiveness maximize results for private sector on and/or contribute to? and recommended that the Bank ―build on client preferences‖ when developing development in MICs. The evaluation will build on 29 economic and sector work/technical assistance. In the 2011 MAR, both  Are there critical decision points IEG’s work to date on AAA management and IEG rated implementation on the recommendation low because within the WBG which would reviews: Performance Assessment capturing structured client feedback is still in the planning stage. drive the timing of the evaluation? Review—Growth Diagnostics in  Will this address a knowledge gap in IEG? Although IEG is piloting evaluation No. Four African Countries (Rep No. of selected knowledge products, the last Bank-wide review of the effectiveness of 55404); World Bank Support for knowledge services was the evaluation of analytic and advisory activities carried out Revenue Reforms in ECA (Rep. in 2008. This evaluation will fill a gap as it will assess the performance of knowledge No. 62944?)—Investment Climate services in the context of country programs that revolve primarily around such Assessments in Five Transforming services, and it will also assess how the bundling of such activities and lending Economies (Rep No 62874). affects the performance of knowledge services by reviewing differences between knowledge-based and lending-based country programs.  Are there other reasons why IEG should proceed with this evaluation? The evaluation is expected to contribute to the development of WBG’s methodology for the evaluation of the development impact of knowledge services. It is also expected to provide an input to the revision of fee-based policies and procedures that has been initiated by OPCS (September 2011). 30 Evaluation of IFC’s Trade Finance Programs—FY13 Proposed Evaluation Coverage, Rationale & Expected Outcome (from IEG Work Program & Budget):  This evaluation will review IFC’s trade finance activities since 2006; assess their relevance, efficacy, and efficiency; and provide an independent assessment of their development effectiveness.  The study is response to specific requests from Members of the Board of Directors. Short-term trade finance is an area of rapid past and potential growth in IFC that has not been evaluated to date.  The evaluation will inform the discussion and decisions on the extent and nature of IFC’s engagement in short-term trade finance in the future. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes, it  Who is the specific audience in the  How does this evaluation fit within has been specifically requested by Board members. WBG group for this evaluation and IEG’s mandate? Within mandate—  Is this in line with WBG strategic priorities? IFC has been its recommendations? Board, IFC ―Assessing whether the WBG’s rapidly expanding its engagement in short-term trade finance since management, IFC staff engaged in trade programs and activities are producing 2006 and plans significant further expansion. The evaluation will be finance programs. expected results‖. of strategic importance to this direction in IFC.  Who will be the WBG ―champion‖?  Is it within the scope of another  Does this address a risk to WBG development effectiveness? Management of IFC’s short-term oversight group (IAD, INT, IPN, Although IFC’s trade finance programs have been growing rapidly finance programs. CAO)? No, there is no other group and currently represent 40 percent of its annual commitments, there  What is the evaluation expected to looking at development effectiveness has been no independent evaluation of them to date. impact? The extent and nature of of IFC’s trade finance programs.  Is this aligned with key WBG corporate deliverables, decision IFC’s short-term trade finance products  Does IEG have the staff to conduct points? IFC expects to request a further increase in the authorized that have rapidly grown over the last six this evaluation? ceiling of the Global Trade Finance Program in FY13Q2, which will years. Yes, there is a task team leader and represent a key decision point on the extent and nature of IFC’s  Are there critical decision points analyst staff and IEG is contracting future short-term trade finance programs. within the WBG which would drive expertise as needed.  Is there need for follow-up based on the MAR (unfinished the timing of the evaluation? Yes,  What other IEG deliverables would business)? No, IEG has not done an evaluation of IFC’s trade IFC expects to request a further this evaluation draw on and/or finance programs in the past. increase in the authorized ceiling of the contribute to? This evaluation will  Will this address a knowledge gap in IEG? Yes, IEG has not Global Trade Finance Program in both draw on and contribute to efforts done an evaluation of IFC’s trade finance programs in the past. FY13Q2. It is also launching a range of to develop project level evaluation There is a large perceived gap among stakeholders in the availability new short-term trade finance products. criteria for trade finance transactions. of evaluation information on trade finance programs. The evaluation will inform the  Are there other reasons why IEG should proceed with this discussion of these directions for IFC. evaluation? There is a strong demand among the evaluation agencies of MDBs to collaborate on developing the methodology and process of evaluating short-term trade finance programs and operations. 31 WBG Focus on Poverty and Results in Low-Income Countries—FY14 Proposed Evaluation Coverage, Rationale, and Expected Outcome (from FY12 IEG Work Program and Budget):  Reducing poverty—broadly defined to include income, opportunity, and access – is the WBG’s overarching goal yet few evaluations address this. The evaluation will focus on World Bank support that targets poverty reduction through macro and fiscal policy frameworks in IDA countries. How has the Bank—through Development Policy Loans, economic and sector work, and policy dialogue—engaged countries to target poverty, inequality, and growth through public expenditure and policy?  Budget support operations and related macro and fiscal analysis in AAA are major instruments for World Band engagement in IDA countries, and they have important implications for fiscal policy, fiscal sustainability, and poverty reduction.  Lessons on the effectiveness and relevance of Bank-supported fiscal frameworks to medium and long term fiscal sustainability and efforts to combat poverty and inequality. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Evaluation of the  Who is the specific  How does this evaluation fit poverty and results from a macro policy and targeting perspective is highly audience in the WBG group within IEG’s mandate? It is relevant to all IEG stakeholders and central to the institution’s core mission. for this evaluation and its aligned with ―Assessing whether the  Is this in line with WBG strategic priorities? Poverty alleviation is a core recommendations? WBG’s programs and activities are goal and in line with the WBG’s Post-Crisis Directions, Targeting the Poor and Development practitioners, producing expected results.‖ Vulnerable. This work on public finance and macro framework complements development agencies, and  Is it within the scope of another the usual focus on specific interventions (that is, social safety nets and the client countries. oversight group (IAD, INT, IPN, access to basic services).  Who will be the WBG CAO)? Other groups are not  Does this address a risk to WBG development effectiveness? Consistency ―champion‖? PREM looking at WBG poverty approaches of the macro framework with effective poverty reduction strategies is a highly Anchor, DEC, country teams. per se. IEG will consult at the relevant risk factor and important to strategic management.  What is the evaluation approach paper stage.  Is this aligned with key WBG corporate deliverables, decision points? It expected to impact? It will  Does IEG have the staff to is aligned with work under PREM and DEC (for example, the Global inform policy makers, conduct this evaluation? Yes, and Monitoring Report) as the 2015 MDG deadline approaches. management and staff IEG is further staffing up now for  Is there need for follow-up based on the MAR (unfinished business)? No regarding macro policy another appropriate lead specialist. recent tracked evaluation has focused on the macro/fiscal policy framework relevance and consistency in the design of country  What other IEG deliverables and poverty in IDA countries. programs. would this evaluation draw on  Will this address a knowledge gap in IEG? Yes. The macro-fiscal-service and/or contribute to? Country delivery nexus with poverty outcomes has not been addressed in recent IEG  Are there critical decision points within the WBG that Assistance Evaluations, CASCRs, work. other major thematic reports with  Are there other reasons why IEG should proceed with this evaluation? would drive the timing of the evaluation? Yes, the 2015 attention to public expenditure and Yes. Growing international attention to inequality and attention to redefining allocation. deadline for MDG the MDGs for the next decade will make this analysis of relevance. discussions. 32 Gender Equity and Poverty IE Meta-Analysis—FY14 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  Poverty reduction is the WBG’s overarching objective, reflected in commitment to the MDGs: both reducing poverty and enhancing inclusiveness and equality of opportunity through improved gender-related outcomes. This meta-analysis will contribute to understanding of the impact the World Bank's gender-targeted poverty-reduction tools can expect to have on improving gender equity in income, opportunity, and access. It is another contribution to IEG’s meta-IE analysis series, covering an important and newly emerging area for impact evaluation.  Project, Development Policy Loans, and technical assistance are all important instruments for World Bank engagement in gender-related issues in member countries. A deeper understanding of what lessons can be drawn from rigorous impact evaluation literature will be valuable to informing these efforts.  WBG projects can be classified into three categories: poverty reduction activities that have gender equity implications, gender equity projects that have poverty effects, and projects explicitly designed to affect both poverty and gender equity. This evaluation will focus principally on the last category to assess the effectiveness of both Bank and non-Bank interventions in improving gender equity and gender-equity-linked poverty reduction.  The review will synthesize lessons and identify existing gaps and priorities for further impact evaluation work. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? This line of  Who is the specific audience in  How does this evaluation fit inquiry follows recent attention to issues in poverty and gender seen in the WBG group for this evaluation within IEG’s mandate? This the 2012 World Development Report, new operational initiatives within and its recommendations? activity falls squarely within IEG’s the WBG, major bilateral aid flows, and client demand. The MDG Development practitioners and mandate of ―assessing whether the deadline makes this analysis more salient. economists both in and outside the WBG’s programs and activities are  Is this in line with WBG strategic priorities? Gender equity is a core WBG. producing expected results.‖ element of WBG poverty reduction strategy and to the MDGs.  Who will be the WBG  Is it within the scope of another  Does this address a risk to WBG development effectiveness? Risks ―champion‖? PREM Gender oversight group (IAD, INT, to WBG effectiveness are reflected in failure to address issues of equity Anchor, Development Economics, IPN, CAO)? No. and shared opportunity for half of the developing world. The 2006/07 and country teams.  Does IEG have the staff to WDRs identified equity and gender as collinear with development.  What is the evaluation expected to conduct this evaluation? IEG  Is this aligned with key WBG corporate deliverables, decision impact? Bank management, task has skilled consultants and is points? The WBG will be assessing its own contribution and strategy to team leaders, and economists bringing gender-equity skills on the MDGs over the next three years. contributing to the design of policies board.  Is there need for follow-up based on the MAR (unfinished aimed at the gender/poverty nexus.  What other IEG deliverables business)? The MAR for IEG’s 2010 gender equity evaluation continues  Are there critical decision points would this evaluation draw on to be monitored but this does not include assessing evidence from IEs. within the WBG that would drive and/or contribute to? Current  Will this address a knowledge gap in IEG? Yes, IEG has not the timing of the evaluation? Yes, maternal and child health impact explored this particular intersection of poverty and gender. 2015 marks the internationally evaluation work; Proposed  Are there other reasons why IEG should proceed with this targeted completion date for the ―Poverty and Results in Low- evaluation? This builds on a growing foundation of IE knowledge about MDGs. Income Countries‖ evaluation. development effectiveness. 33 Reform of Health Systems—FY14 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  This evaluation will examine Bank support to health system strengthening as identified as a core objective in the Bank’s 2007 Health Sector Strategy. It will look at how effectively the health systems approach has helped countries deal with systemic issues of equity in access, efficiency, and financial sustainability of the health system and what role the Bank has played in this respect.  Over the past decade the focus of in the international donor community has increasingly been on health systems as a whole, moving away from a disease-specific approach. The Bank’s current Health Sector Strategy embraces a health system approach, but the effectiveness of this approach has not yet evaluated. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? The evaluation can help  Who is the specific audience in the  How does this evaluation fit clarify understanding about the health systems approach among stakeholders and WBG group for this evaluation and within IEG’s mandate? It identify what can realistically be achieved and ways to address constraints. its recommendations? The HNP assesses ―whether the WBG’s  Is this in line with WBG strategic priorities? Access to quality and affordable basic family within the Bank and other health programs and activities are health care are part of the Bank’s Post-Crisis Directions, Targeting the Poor and stakeholders (donors, governments, producing expected results.‖ Vulnerable. NGOs, private sector). Other sectors ,  Is it within the scope of  Does this address a risk to WBG development effectiveness? Health, Nutrition, like education, have also adopted a another oversight group (IAD, and Population projects have lower rate of satisfactory outcomes at exit (69 percent, ―system approach‖, and could learn INT, IPN, CAO)? No. compared with an average of 74 percent for FY08-10). lessons from a health system evaluation  Does IEG have the expertise  Is this aligned with key WBG corporate deliverables, decision points? It will  Who will be the WBG ―champion‖? to conduct this evaluation? inform a mid-term review of the corporate HNP strategy in FY14 and the international The HDN vice presidency, HNP IEG has a team with a lead HNP 2015 post-MDG dialogue. Anchor, and health units in Regions. evaluator, but would have to  Will this address a coverage or follow-up gap in IEG? IEG carried out an HNP  What is the evaluation expected to recruit health specialist evaluation in 2009. Bank strategy has moved away from disease-specific to a health impact? Inform stakeholders on the staff/consultants to support the systems approach. The new approach—adopted not only by the Bank but by the contribution of health systems approach evaluation. international community—has yet to be evaluated. to equity, efficiency, and financial  What other IEG deliverables  Is there need for follow-up based on the MAR? IEG’s 2009 HNP evaluation is still sustainability in health outcomes and would this evaluation draw on under implementation and review in the Management Action Record. However, until identify areas that need special attention. and/or contribute to? It would the third year of implementation, the MAR noted a relative lack of progress on the It would explain the critical factors draw on PPARs of closed health recommendation to enhance support from other sectors. Further follow up will be determining effectiveness of the systems operations and from the 2009 done in this year’s MAR to assess progress. approach. IEG health evaluation, as well as  Are there other reasons why IEG should proceed with this evaluation? There is  Are there critical decision points on other planned evaluations increasing interest among all health stakeholders about the effectiveness of the health within the WBG that would drive the (fragile and post-conflict states). systems approach. Most donors and countries are following it, but so far, its timing of the evaluation? It will It would contribute to future effectiveness in improving health outcomes has not been evaluated. inform a mid-term review of the HNP assessments of the Bank’s strategy in FY14. contribution to the MDGs. 34 Rural Non-Farm Economic Growth—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  This evaluation will examine WBG support to rural non-farm economic growth, one of the five pillars in the current sector strategy paper (FY2003 Reaching the Rural Poor). The evaluation will assess how effectively the Bank and IFC have been in stimulating income and employment growth in non-farm rural enterprises and reducing rural poverty, with a particular focus on improved access to financial services, skills upgrading, infrastructure development, and the development of land markets.  In FY2011, the non-farm part of Agriculture and Rural Development’s portfolio accounted for 10 percent of disbursements and 14 percent of commitments. In FY09-11, the Regions with the largest disbursements in this area were South Asia ($115 million), Africa ($52 million) and East Asia ($32 million). Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG  Who is the specific audience in the  How does this evaluation fit within stakeholders? Strategies for diversifying out of agriculture WBG group for this evaluation and its IEG’s mandate? It responds to the will become increasingly important and the rural non-farm recommendations? In the Bank, the following part of the DGE’s mandate: economy will have a bigger role to play because farming is Rural Family, plus clusters of staff working ―Incorporating evaluation assessments and largely a part-time activity in more developed countries. on infrastructure and financial services. In findings into recommendations designed to  Is this in line with WBG strategic priorities? The FY10 IFC, agribusiness, tourist, and investment help improve the development effectiveness position paper (New World, New World Bank Group) notes climate specialists. of the World Bank Group’s programs and that most low-income countries have small bases of  Who will be the WBG ―champion‖? The activities‖ productive wage employment, with extremely high rates of ARD Anchor and the ARD units in the  Is it within the scope of another self- and informal employment. In such environments, Regions. oversight group (IAD, INT, IPN, CAO)? policies that enhance the productivity of micro and small  What is the evaluation expected to No. enterprises will be required. Many of these enterprises will impact? The evaluation will show if the  Does IEG have the expertise to conduct be located in rural areas. rural non-farm economy is a useful this evaluation? Current staff and  Does this address a risk to WBG development organizing focus for Bank Group consultants have expertise in assessing effectiveness? Although the aggregate numbers for FY09- investments and AAA and which various aspects of the non-farm economy: 10 show that IEG outcome ratings are the same for interventions and instruments are most rural schools and vocational education, Agriculture and Rural Development as for other sectors effective in terms of job creation and community infrastructure, land (72 percent satisfactory), there are no data for rural non- poverty reduction. The evaluation will administration, rural finance. Other IEG farm projects specifically. examine to what extent rural non-farm staff have worked on agribusiness and  Is this aligned with key WBG corporate deliverables, growth is driven by local farming or by tourism. decision points? An FY14 delivery might contribute to remittances from further afield and what  What other IEG deliverables would this the mid-term review of the next ARD Strategy (FY13-15). institutional framework (including financial evaluation draw on and/or contribute to?  Will this address a knowledge gap in IEG? This is a and land administration services) is needed It would draw on project performance topic that has not been previously evaluated by IEG. for the non-farm sector to flourish. By assessments of closed ARD projects and 35 IEG’s most recent sector-themed studies address exploring the contexts in which non-farm relevant project assessments by IEG staff. agricultural productivity and forests. projects have performed most effectively it The area covered by the evaluation is  Is there need for follow-up based on the MAR? Not will be possible to make recommendations relevant to the MDG Goal 1 (Sub-Goal: applicable. about where the Group should focus its Promote productive and decent  Are there other reasons why IEG should proceed with investments. employment), and the evaluation will this evaluation? In general, this is a little evaluated topic.  Are there critical decision points within therefore contribute to IEG’s assessment of There are knowledge gaps: since FY06, only 2 percent of the WBG that would drive the timing of how effectively the Bank Group has all the AAA that the sector has conducted was mapped to the evaluation? The evaluation will be facilitated attainment of the MDGs. the non-farm pillar of the strategy. delivered in time for mid-term review of the FY13-15 ARD strategy and may also feed into preparation of the subsequent strategy. 36 Early Child Development—FY15 Proposed Evaluation Coverage, Rationale & Expected Outcome:  This evaluation will review the evidence and distill lessons World Bank investments in early child development (ECD) programs and analytic work. The goal of ECD programs is to improve young children’s capacity to develop and learn. Interventions include educating and supporting parents; delivering health, education, and nutrition services to children; and developing capacities of caregivers.  The research literature has pointed to ECD investments as extremely effective, which has raised their profile. However, much less is known about their effectiveness in the context of public programs in developing countries—specifically, the contextual factors affecting impact, whether pilots can be scaled up and remain effective, the sustainability of such programs, and their affordability, particularly in low-income countries.  The evaluation will take a critical look at what has been learned about the efficacy, cost-effectiveness, and sustainability of ECD programs in different developing country settings. A related issue is how to effectively implement cross-sectoral programs in an environment of sectoral silos, both in countries and in the Bank. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes,  Who is the specific audience in the WBG  How does this evaluation fit of major interest to the education and health sectors. group for this evaluation and its within IEG’s mandate? It  Is this in line with WBG strategic priorities? ECD is prominent recommendations? The education and assesses ―whether the WBG’s in the recent education sector strategy and is one of several health sectors of the World Bank. programs and activities are approaches to achieving the MDGs.  Who will be the WBG ―champion‖? The producing expected results‖ and  Does this address a risk to WBG development effectiveness? HDN vice presidency, education and health the lessons from experience. ECD has been a small but growing portfolio in the World Bank and sector boards.  Is it within the scope of an area that builds on cross-sectoral investments in education,  What is the evaluation expected to another oversight group (IAD, nutrition, and health. impact? It would inform decisions on the INT, IPN, CAO)? No.  Is this aligned with key WBG corporate deliverables, decision conditions under which these approaches are  Does IEG have the expertise points? It will inform the post-MDG dialogue, in particular cross- feasible, effective, and sustainable options. to conduct this evaluation? sectoral interventions.  Are there critical decision points within Yes, the HD team, supplemented  Will this address a knowledge gap in IEG? IEG carried out an the WBG that would drive the timing of by consultants. evaluation of primary education in 2006; an evaluation of health, the evaluation?  What other IEG deliverables nutrition, and population in 2009; a review of nutrition impact No specific decision points, but critical for would this evaluation draw on evaluations in 2010; and an education portfolio review in 2011. future of MDGs. and/or contribute to? It would These evaluations were aligned with sectors and did not focus on draw on the review of nutrition the impact of cross-sectoral interventions on the cognitive impact evaluations (2010), the development of preschool children. education portfolio note (2011),  Is there need for follow-up based on the MAR? Some ECD and the HNP evaluation (2009). programs were assessed in the nutrition impact evaluation review, It would also draw on proposed but they were not explicitly discussed in the recommendations of impact evaluations on maternal any recent evaluations. and child health. 37 Urban Poverty—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcome (from IEG Work Program and Budget):  The growth of world population is slowing, but that of the urban population is accelerating, a trend expected to continue in the foreseeable future.  Up to now, most poverty reduction strategies have focused on the poorest of the poor, who have been in rural areas. But poverty is likely to take on a more urban aspect because of demographic shifts. To what extent should these strategies shift?  This evaluation will examine whether the WBG has been effective in supporting efforts to alleviate poverty in urban areas. It will cover both lending and knowledge activities, at both the Regional and global levels.  Such an evaluation will be essential to ready the Bank Group to tackle new and emerging trends in the persistent problem of poverty. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders?  Who is the specific audience in the  How does this evaluation fit within  Further discussions needed with management. Recognized as an WBG group for this evaluation and IEG’s mandate? Within mandate— important development topic given demographic changes and its recommendations? PREM and ―Assessing whether the WBG’s increasing urbanization. SDN through poverty and urban programs and activities are producing  Is this in line with WBG strategic priorities? Poverty alleviation boards. But also HDN and other expected results‖. has been the core mission of the WBG. It is in line with the WBG’s boards because poverty is  Is it within the scope of another Post Crisis Directions, Targeting the Poor and Vulnerable. As the multidimensional. oversight group (IAD, INT, IPN, number of urban poor increases, the focus on their particular  Who will be the WBG ―champion‖? CAO)? Other groups are most likely problems will become more salient. Poverty and Urban Board Chairs in the not looking at an approach per se.  Does this address a risk to WBG development effectiveness? World Bank. IFC/MIGA not clear. IEG can consult at the approach paper The WBG’s has paid more attention to urban poverty in the past  What is the evaluation expected to stage. three to five years. But are these efforts adequate and effective? impact? WBG lending and knowledge  Does IEG have the staff to conduct  Is this aligned with key WBG corporate deliverables, decision services. this evaluation? Yes; poverty and points? It will prepare the Bank to engage better in the upcoming  Are there critical decision points SDN recruiting is ongoing. global discussion of the agenda beyond 2015’s MDGs. within the WBG that would drive the  What other IEG deliverables would  Is there need for follow-up based on the MAR (unfinished timing of the evaluation? Not clear. this evaluation draw on and/or business)? No. contribute to? IDA and FCS and  Will this address a knowledge gap in IEG? There has not been a infrastructure evaluations. review of urban issues in some time. There has never been a report on urban poverty (there was a report on rural poverty 10 years ago). 38 Green Growth Analytics—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  The evaluation will assess the construction, contribution, and effectiveness of the World Bank's use of ―green growth analytics‖ in strengthening policy formulation linked to natural resource exploitation and sustainability nationally and globally.  Application to sectors and resources of special importance to human development and growth, such as water and nonrenewable energy, will underpin critical findings of the report.  The 2011 World Bank initiative to expand and standardize the use of green growth analysis in economic and sector work and policy advice to governments targets countries for which dependence on natural resources is high and hence important to informing economic management and resource planning. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? There  Who is the specific audience in the  How does this evaluation fit within is global interest in environmental protection, sustainable WBG group for this evaluation and IEG’s mandate? The quality and development, and protecting the global commons. Relevant and its recommendations? Development application of knowledge on credible metrics is vital for this. practitioners and economists both in sustainable development is one  Is this in line with WBG strategic priorities? It is a priority and outside the WBG. important parameter of the Bank's following the 2010 World Development Report and the recent  Who will be the WBG ―champion‖? development effectiveness as a Bank initiative to expand use of the green growth analytic tool. SDN anchor environmental team. development partner.  Does this address a risk to WBG development effectiveness? A  What is the evaluation expected to  Is it within the scope of another central element of WBG support for poverty reduction is sound impact? Bank management, task team oversight group (IAD, INT, IPN, and sustainable macroeconomic strategies. leaders, and economists contributing to CAO)? No.  Is this aligned with key WBG corporate deliverables, decision macro analysis and ―mainstreaming‖ of  Does IEG have the staff to conduct points? The 2011 initiative for green growth analytics commenced growth and sustainability analysis. this evaluation? The environmental in 2011 with a [3] year funding profile.  Are there critical decision points team and macro teams have the  Is there need for follow-up based on the MAR (unfinished within the WBG that would drive the requisite skills. business)? No. timing of the evaluation? Not at  What other IEG deliverables would  Will this address a knowledge gap in IEG? Yes, because there present. this evaluation draw on and/or is no evaluation work that looks at the analytic underpinnings of contribute to? Climate change wealth accounting and growth in IEG. evaluations (for example, phase I on  Are there other reasons why IEG should proceed with this World Bank energy policies and evaluation? Not at present. support). 39 Evaluation System for Private and Public Sector Operations—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcome:  The evaluation would cover M&E systems across the WBG, building on the Biennial Report on Operations Evaluation for private sector operations and the Annual Report on Operations Evaluation for public sector operations.  Assessing the credibility, relevance, and effectiveness of M&E systems is a key step in the development of ensuring that data, processes, and systems are in place to adequately track and assess performance on progress toward achieving WBG strategic priorities and development effectiveness Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes,  Who is the specific audience in the  How does this evaluation fit within the development of sound M&E systems is a key component of WBG group for this evaluation and IEG’s mandate? Yes: ―Appraising the WBG results agenda. its recommendations? The Board, the World Bank Group’s operations  Is this in line with WBG strategic priorities? Robust systems WBG management, and staff working self-evaluation and development risk for tracking, measuring, and assessing performance are central to on results agenda. management systems and attesting to assessing progress on WBG strategic priorities.  Who will be the WBG ―champion‖? their adequacy to the Boards.‖  Does this address a risk to WBG development effectiveness? OPCS at the World Bank, IFC’s  Is it within the scope of another Weak M&E systems is a major risk to assessing the quality of Development Impact Department. oversight group (IAD, INT, IPN, WBG’s development effectiveness.  What is the evaluation expected to CAO)? No.  Is this aligned with key WBG corporate deliverables, impact? Data, processes, and systems  Does IEG have the staff to conduct decision points? A strong results orientation is at the core of for measuring results across the WBG. this evaluation? Yes. WBG deliverables. This evaluation will focus on the systems for  Are there critical decision points  What other IEG deliverables would tracking and measuring results. within the WBG that would drive this evaluation draw on and/or  Is there need for follow-up based on the MAR (unfinished the timing of the evaluation? contribute to? Biennial Reports on business)? IEG recommendations have focused consistently on Results tracking and measurement is Operations Evaluation and Annual the lack of M&E over the last years. ongoing and feeds into key decision Reports on Operations Evaluation, as  Will this address a knowledge gap in IEG? There have been a points during a project cycle. well as recent RAPs. number of relevant IEG reports over the last years with relevant information on both private and public sector evaluation systems.  Are there other reasons why IEG should proceed with this evaluation? The results agenda is at the core of assessing progress on global development objectives, such as the MDGs. 40 Private Sector Role in Achieving MDGs—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcomes:  This evaluation would cover a broad range of lending and nonlending private sector activities that have been used to make progress on poverty, hunger, and other MDG targets.  The MDGs are set to be achieved by 2015. Although the focus has mainly been on the public sector, a number of IEG evaluations have demonstrated the key role that the private sector can play in reducing poverty and hunger and improved delivery of services, such as health and education.  The evaluation would draw lessons from relevant IEG evaluations on the role of the private sector in achieving MDG goals with a view to strengthening private sector roles, as well as the synergies between public and private sector actions in post the MDGs. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes,  Who is the specific audience in the  How does this evaluation fit within the role of the private sector in development is being increasingly WBG group for this evaluation and IEG’s mandate? IEG’s mandate recognized in the development community. IEG stakeholders look its recommendations? The Board, includes assessing development to IEG to provide objective insights and lessons on how private management, and staff in WBG effectiveness at the WBG. This sits at sector interventions can be made more effective in achieving institutions, other multilateral the core of assessing progress on development outcomes. development banks, and development poverty reduction and related MDG  Is this in line with WBG strategic priorities? Yes, the strategic agencies with private sector operations. targets. priorities of WBG institutions all revolve around poverty reduction,  Who will be the WBG ―champion‖?  Is it within the scope of another providing opportunities, and improving lives. IFC’s Development Impact oversight group (IAD, INT, IPN,  Does this address a risk to WBG development effectiveness? Department; Strategy in FPD Network CAO)? No. Yes, by not exploiting lessons from the key role of private sector  What is the evaluation expected to  Does IEG have the staff to conduct operations, the development effectiveness of future development impact? Private sector lending and this evaluation? Yes. interventions would be compromised. non-ending operations; knowledge and  What other IEG deliverables would  Is this aligned with key WBG corporate deliverables, decision learning agenda on private sector and this evaluation draw on and/or points? Yes, as with most development institutions, the WBG development. contribute to? IEG’s evaluation on would be seeking to assess its progress on achieving the MDGs by  Are there critical decision points health and education; agriculture and 2015. within the WBG that would drive the agribusiness; environment; information  Is there need for follow-up based on the MAR (unfinished timing of the evaluation? This is and communication technology; IFC’s business)? An important part of the evaluation would involve an related to the broader WBG agenda on poverty focus; M&E systems for assessment of how WBG institutions have followed up on poverty progress toward MDGs. private sector operations. related actions from relevant interventions.  Will this address a knowledge gap in IEG? The recent IFC Poverty evaluation as well as the information and communication technology evaluation might have relevant information. The ongoing youth employment evaluation might also be relevant. In 41 addition, IEG can bring together the findings from previous micro synthesis and macro evaluations in this area.  Are there other reasons why IEG should proceed with this evaluation? The issue is very timely given that 2015 is only three years away. Taken together among all multilateral development banks, the WBG has the largest lending and non-lending program to the private sector. These lessons from this evaluation would therefore have very broad applicability. 42 WBG Role in Promoting South-South Development—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcomes:  The evaluation will assess trends and patterns of South-South investments, their relevance to the WBG mission of poverty reduction, and the extent to which IFC and MIGA support to South-South investors has been effective. It will distill factors that have been associated with better development outcomes and lessons for IFC and MIGA engagement in this area. The study would also review relevant World Bank strategies, programs, and initiatives that aim to foster cooperation, partnerships, and knowledge sharing among developing countries.  The international aid architecture is transforming into a development partnership that includes new stakeholders, including the private sector. In the last decade, investments from developing to developing countries have increased substantially (now comprising close to 30 percent of global foreign direct investment outflows). This trend has been driven by several factors, including the rise of emerging markets and new global players (especially in BRIC countries). Similarly, World Bank support to governments will increasingly evolve around a partnership model, especially for middle-income countries, sharing best practice and knowledge from across developing countries.  Based on experience to date, this evaluation would seek to identify challenges and opportunities for the WBG in supporting South-South investment and cooperation. The evaluation will present good practices and business models and lessons to allow the WBG to harness the potential of cooperation among developing countries to further achievement of its development goals—especially in an era where the Bank and IFC face capital constraints. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes,  Who is the specific audience in the  How does this evaluation fit within promoting South-South investments has been a strategic priority for WBG group for this evaluation and IEG’s mandate? Yes, accountability IFC and MIGA since 2004. However, the results and lessons in this its recommendations? for results and learning from past area have not been systematically evaluated, especially given the BG management and staff, especially in experience of support to South-South increasing prominence of this topic. IFC and MIGA, and the development investments.  Is this in line with WBG strategic priorities? Yes, strategic community at large.  Is it within the scope of another priority for IFC and MIGA.  Who will be the WBG ―champion‖? oversight group (IAD, INT, IPN,  Does this address a risk to WBG development effectiveness? Various departments, including CAO)? No. South-South investments have distinct characteristics that may pose economics, development effectiveness,  Does IEG have the staff to conduct greater challenges for developing countries than those from and strategy departments in IFC, this evaluation? Yes. Staff within developed countries. Compliance with environmental and social MIGA, and the Bank. IEG have familiarity with the IFC and performance of IFC South-South projects has been an issue.  What is the evaluation expected to MIGA aspects of this work. Other  Is this aligned with key WBG corporate deliverables, decision impact? Improved understanding and IEG staff have familiarity with aspects points? Yes, it is aligned with the annual IFC strategy update (Road knowledge of staff and stakeholders of pertaining to South-South cooperation Map) and MIGA’s Three-Year Strategy, as well as WBG strategic the challenges and opportunities in (knowledge, partnerships). directions. supporting South-South investment and  What other IEG deliverables would  Is there need for follow-up based on the MAR (unfinished initiatives. Ultimately, better integration this evaluation draw on and/or business)? No. of South-South components in contribute to? An IEG cluster note  Will this address a knowledge gap in IEG? The Busan programs offered to WBG clients to on South-South projects was prepared Partnership for Effective Development Co-operation has reiterated maximize impact from these capital in FY12. The study would build on 43 the importance of South-South cooperation. flows and cooperation. IFC and MIGA project evaluations,  Are there other reasons why IEG should proceed with this  Are there critical decision points and the IEG study on middle-income evaluation? As noted above, capital flows among developing within the WBG that would drive the countries, and on knowledge-based countries are increasing significantly, and it will be important to timing of the evaluation? No, but this country programs. understand better how to harness such flows for development is likely an important component of the impact, including how the institutions may need to change their post-financial crisis strategy of the delivery mechanisms to support these types of projects. WBG. 44 WBG in Middle-Income Countries—FY15 Proposed Evaluation Coverage, Rationale, and Expected Outcomes:  The evaluation will assess the relevance, effectiveness, and efficiency of WBG operations in middle-income countries. It would seek to identify development challenges of different types of middle-income countries and review the appropriateness of the suite of World Bank, IFC, and MIGA instruments to address them. In so doing, the evaluation may segment and focus the analysis on some types of middle-income countries.  Despite the progress toward meeting the MDGs, poverty reduction has been uneven among and within countries. Middle-income countries are a diverse group of countries but face some common development challenges, including the fact that most of the world’s poor live in middle-income countries and that inequalities have been rising. Despite this, the World Bank in particular has faced the issue of the relevance of its instruments in some upper-middle-income countries (evidenced by a decline in lending volume prior to the financial crisis), implying that it has been challenging for the Bank to engage in certain types of countries on a range of development issues. At the same time, a new development partnership has been emerging involving new players, including the private sector, that now play a larger role in development. Regarding IFC and MIGA, most of their projects have been in middle-income countries (including many IDA and frontier markets that are lower middle income). However, these two institutions were not included in the previous IEG evaluation. Given these considerations, it is timely to review the challenges and the adequacy of Bank Group instruments in assisting middle-income countries.  The evaluation will aim to inform strategy formulation and decision-making regarding the optimal WBG engagement and product mix in MICs. Relevance to WBG Development Effectiveness Potential Influence IEG Mandate and Value Added (Is this a crucial topic to evaluate?) (Who, how, and when?)  Is this an area of interest/priority for IEG stakeholders? Yes,  Who is the specific audience in the  How does this evaluation fit within the effectiveness of support to the countries is of utmost WBG group for this evaluation and IEG’s mandate? Assessing importance for achieving the WBG’s mandate and for ensuring that its recommendations? The Board, accountability for results and deriving it remains relevant to this group of countries. WBG management, and staff, as well as lessons for future programs and  Is this in line with WBG strategic priorities? Yes, achievement the development community at large. operations. of WBG strategic priorities (post crisis directions) depends crucially  Who will be the WBG ―champion‖?  Is it within the scope of another on the effectiveness of its program in MICs. WBG management, Regional vice oversight group (IAD, INT, IPN,  Does this address a risk to WBG development effectiveness? presidencies, and OPCS. CAO)? IEG is the only entity to Yes, as noted above, there is a risk that the WGB suite of  What is the evaluation expected to review development effectiveness. instruments may lose relevance in certain middle-income countries, impact? Decisions on optimal  Does IEG have the staff to conduct with implications for the ability of the WBG to engage with these engagement with middle-income this evaluation? Yes, a team would countries on developing challenges. countries and the WBG’s product mix. need to comprise staff from across  Is this aligned with key WBG corporate deliverables, decision  Are there critical decision points IEG. points? Yes, it would be input for periodic WBG strategy updates within the WBG that would drive the  What other IEG deliverables would and for decision making on the evolving WBG’s product mix. timing of the evaluation? No, but a this evaluation draw on and/or  Is there need for follow-up based on the MAR (unfinished decrease in global economic uncertainty contribute to? This evaluation would business)? In its 2007 evaluation on middle-income countries, would likely result in a return to complement the evaluation of poverty IEG called for a more pragmatic approach to cooperation across portfolio trends observed prior to FY08 focus and results in low-income the Bank Group—focusing on those few areas in a given country (that is, a decline in IBRD lending countries and will draw on findings 45 where cooperation might have a substantial impact on development volume in some upper-middle-income from the evaluation of South-South and tracing through activities to outcomes. In the concluding countries). development, Knowledge-Based comments of the 2011 MAR, management indicated that joint Country Programs, and the Private Bank/IFC guidance is being been widely disseminated based on Sector Role in Achieving MDGs, lessons from a pilot exercise for IDA countries, but it remains true global program reviews, country and that no comparable agenda has been developed for operations in regional evaluations, and project middle-income countries and it is unclear whether experiences in evaluations. IDA countries can be extended to them.  Will this address a knowledge gap in IEG? No.  Are there other reasons why IEG should proceed with this evaluation? No. 46 Appendix 3: IEG Results and Measurement Framework 1. IEG introduced a results and measurement framework as part of the FY11–13 Work Program and Budget Document. Members of CODE and the Budget Committee appreciated the development of the framework as this was not common practice for evaluation units. They encouraged IEG management to refine the framework to become less reliant on survey data—including using outside consultants to conduct the survey—and also to make a closer connection with development results and the emerging Corporate Scorecard. 2. The thematic approach outlined above provides an opportunity to make further refinements along the lines suggested by Executive Directors during the discussion last year. Specifically, a thematic approach could be linked throughout IEG’s Theory of Change and include more specific outcome indicators to which IEG’s impact could be linked. Although this will be a first for an evaluation unit among multilateral development banks and will require significant additional thinking, it would result in a more satisfying presentation of IEG’s impact on the overall development effectiveness of the WBG, at least within one thematic area. 3. As the evaluative work program is defined in more detail through approach papers, IEG will seem to systematically infuse key questions related to the four strategic clusters into its evaluations. From these, IEG will attempt to define common measurable indicators that could provide a longer-term view on how IEG’s FY13–15 work program has contributed to strengthening the development impact of the WBG’s work. 4. For the time being, IEG’s results framework is structured as follows with targets indicated, where appropriate, in the comment section:26  Section A of the results framework includes the key inputs used to deliver IEG’s work program—people and financial resources.  Section B includes indicators that reflect the timeliness with which IEG delivers on both the macro and the micro work program.  Section C captures the outputs/deliverables committed in the work program.  Section D includes indicators that would measure the quality of IEG’s work. These measure the extent to which IEG practices are consistent with the Good Practice Standards developed by the ECG, for evaluation products. They also include indicators on perception of quality of IEG’s work by key stakeholders—WBG staff, the Board, and the broader external audience, and ratings in independent third-party assessments.  Section E measures the influence of IEG evaluations on the WBG. The indicators here are perceptions of the use and influence of IEG evaluations based on the client survey. However, the MAR is the official tool for documenting the actions taken by WBG management in response to IEG recommendations. 26 A significant number of indicators are based on results from an annual client survey. This is carried out by an external firm, and the individual responses are held confidential. An overview analysis of the survey is posted on IEG’s website. 47  Section F reports on the extent of and feedback on the communication and dissemination activities of IEG, which includes direct and Web-based dissemination, as well as training to develop evaluation capacity.  Section G includes measures of impact of IEG’s evaluation on the WBG as well as the broader development community, based on responses gathered from the client survey, as well as citations in World Bank documents and external publications. BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year A. INPUTS 1. As % of WBG administrative 1.4% FY06-10 1.3% FY12 RM team Annual Updated during preparation of the IEG work budget (annual average) program and budget document. a. WB 1.4% FY06-10 1.4% FY12 b. IFC 1.0% FY06-10 1.0% FY12 c. MIGA 3.7% FY06-10 3.2% FY12 2. As % of WBG administrative 1.1% FY09-12 1.1% FY12 RM team Annual and trust fund budget (annual IEG has developed rules on the use of trust average) funds in evaluations and appointed a staff member to ensure appropriate usage and reporting on trust-funded activities. Staff 3. Share internally vs. externally — — 49% FY12 RM team Annual The proportion of professional staff (GE+) at recruited end February 2011 that were recruited to IEG from outside the WBG stood at 49 percent, with 77 percent of FY12 hires at the GE+ level having had no previous WBG employment. Target is 50%. 4. Share of women evaluators 53% FY12 53% FY12 RM team Annual Note: This number is women evaluators GE+ GE+ and does not include back-office functions and front office staff except for female staff with evaluation training (IEGCS, IEGDG). Target is 53%. Diversity indicator 5. Diversity Index Trend 0.62 FY10 FY12 0.56 RM team Annual IEG will strive to meet WBG diversity targets. B. EFFICIENCY NOTES BASELINE CURRENT SOURCE FREQ. INDICATOR Value Year Value Year 1. Percent of evaluations RM team Annual IEG will begin monitoring this for evaluation completed within timeline deliverables in FY13. Target is 80% specified in approach papers 2. Percent of micro evaluations RM team Annual IEG will begin monitoring this for evaluation completed within IEG deliverables in FY13. Target is 80% standards for timely delivery IEG Standards for delivery (days): ICR: 4.4 PPAR: 30.5 XPSRs: 10.8 PCRs: 4.0 48 C. OUTPUTS BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year 1. Product mix and quantity Micro a. Implementation 215 FY09-11 220 FY12 IEG work Annual 100% of ICRs Completion Report program Reviews b. Project Performance 55 FY09-11 70 FY12 IEG work Annual Target as per work program Assessment Reports program c. Extended Project 71 FY09-11 90 FY12 IEG work Annual Sample Supervision Reports program d. Project Completion 92 FY09-11 150 FY12 IEG work Annual Sample Reports program e. PER 2 FY11 2 FY12 IEG work Annual program f. PER validation 2 FY11 2 FY12 IEG work Annual program Country and Global g. Global Program 4 FY09-11 3 FY12 IEG work Annual Targets in annual work program Reviews program h. Country Assistance 16 FY09-11 20 FY12 IEG work Annual Targets in annual work program Strategy Completion program Report Reviews i. Country Program 2 FY09-11 2 FY12 IEG work Annual Target : 2 Evaluations program j. Sector/Thematic 9 FY09-11 4 FY12 IEG work Annual Targets in Annual work program Evaluations program k. Corporate Evaluations 2 FY09-11 3 FY12 IEG work Annual Targets in annual work program program D. QUALITY BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year 1. Percentage of ECG GPS ECG Good Practice Standards are currently adopted and undergoing review and revisions on both the implemented by IEG private and public sector to improve a. Independence 88% 2010 88% 2012 Internal Annual standards and reflect evolving evaluation review methodologies. IEG will have an external b. Evaluation of country 85% 2010 85% 2012 Internal Annual review of implementation every two years. assistance strategies review Target is 90% compliance as all standards c. Evaluation of private 93% (IFC) 2010 93% (IFC) 2012 Internal Annual may not apply. sector investment 73% 73% review (MIGA) (MIGA) d. Evaluation of public 74% 2010 74% 2012 Internal Annual sector projects review e. Evaluation of 83% 2010 83% 2012 Internal Annual development program review loans f. Evaluation of AAA TBD — TBD — Internal Annual Standards under development; will be and knowledge review applied as of FY13 products 2. Proportion of Client Most Recent Client Survey Survey respondents who November/December 2011 were satisfied (or better) with quality of IEG evaluations (perception of quality) a. Board 80% FY10 77% FY12 Client Annual survey b. WBG 60% FY10 58% FY12 Client Annual survey c. External 80% FY10 78% FY12 Client Annual survey 3. Overall Assessment of 10 dimensions of quality by client survey 49 D. QUALITY BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year respondents (rated on 6 point scale) a. Board 4.9 FY10 4.95 FY12 Client Annual survey b. WBG 4.5 FY10 4.49 FY12 Client Annual survey c. External 5.0 FY10 5.03 FY12 Client Annual survey Independent third-party assessment IEG will keep track of third party 4. Global Accountability evaluations—no standardized criteria are Assessment of IEG available a. WB 90+, tied 2008 for third among the 27 b. IFC 84, 6th out 2008 of the 30 organizatio ns BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year E. Use and influence of IEG products 1. Proportion of Client Survey respondents rating the aggregate influence of IEG products as “to some extent� (or better) a. Board 95% FY10 85% FY12 Client Annual survey b.WBG 80% FY10 70% FY12 Client Annual survey c. External 92% FY10 91% FY12 Client Annual survey 2. Proportion of Client Survey respondents rating the aggregate use of IEG products as “to some extent� (or better) a. Board 88% FY10 87% FY12 Client Annual survey b. WBG 69% FY10 63% FY12 Client Annual survey c. External 84% FY10 72% FY12 Client Annual survey F. Outreach and Dissemination BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year 1. 1. Number of page 513,441 FY12 Web Monthly views (monthly analytics average: 42, 786) 2. 2. Number of web 46,829 FY12 Web Monthly downloads (monthly analytics average - 3,902) 3. 3. Number of social Facebook FY 11 Facebook: FY12 Social Annual media followers : 1,006 (May 12,462 media data (Facebook, Twitter, Twitter: 2012) Twitter: YouTube) 535 1,660 YouTube YouTube video video views: 50 BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year views: 13,882 6,300 1. 4. Share of WBG staff Baseline 2013 934 WBG 2011/2 Sign-up Monthly with at least one direct to be staff 012 sheets participation in IEG set/decid attended 25 from IEG learning events ed upon IEG events events, log- from April ins, dial- 2011 until ins, etc. March 2012. The average is 37 WBG staff per event. 5. Percentage of IEG Weighted FY11 Weighted FY11 Client Annual Client survey data 2011: Question “How client survey Average: Average: survey would you rate IEG's outreach and respondents rating 85.2% 85.2% dissemination efforts in the following IEG‟s website as areas?� (Website). This number is a satisfactory (or higher) weighted average of three audiences: Board: 97%, WBG: 77%, External: 94% Effectiveness of evaluation capacity development 6. Percent of CLEAR 88% 2011 CLEAR Annual center users rating center effectiveness as survey satisfactory (or higher) 7. Percent of IPDET 82% 2009 87% 2011 Training Post- All IPDET evaluations are available at participants rating evaluations IPDET http://ipdet.org/page.aspx?pageId=downlo effectiveness of training ads as satisfactory (or higher) G. IMPACTS BASELINE CURRENT INDICATOR SOURCE FREQ. NOTES Value Year Value Year Impact on WBG development effectiveness 1. Proportion of respondents to Client Based on most recent client survey, Survey that rate IEG‟s impact on the November/December 2011 WBG‟s development effectiveness as Moderate (or better) a. Board 86% FY10 86% FY12 Client Annual survey b. WBG 67% FY10 65% FY12 Client Annual survey c. External 75% FY10 79% FY12 Client Annual survey Impact on broader development community effectiveness 2. Proportion of respondents to Client Survey that rate IEG‟s impact on the broader development community as Moderate (or better) a. Board 75% FY10 68% FY12 Client Annual survey b. WBG 55% FY10 53% FY12 Client Annual survey c. External 72% FY10 73% FY12 Client Annual survey Impact on knowledge and learning 3. Number of IEG citations in internal 111 2009- 92 2011 „Publish Annual WBG and external publications 2011 or average Perish‟ software program 51 4. Number of IEG citations/mentions in March FY12 eBoard Monthly IEG conducted a document keyword Board documents 2012: IEG search within all March 2012 eBoard mentioned documents (without IEG evaluations, in 65 approach papers, evaluation green Board sheets, and CODE minutes) documents 52 Appendix 4: Global and Regional Programs with Recently Completed Self- Evaluations In selecting programs for review, preference is given to (i) those that are innovative, large, or complex; (ii) those in which the Bank is sufficiently engaged to warrant a review; (iii) those that are relevant to upcoming IEG sector studies; (iv) those for which the Executive Directors or Bank management have requested reviews; and (v) those that are likely to generate important lessons. IEG also aims for a representative distribution of Global Program Reviews across sectors in each fiscal year.27  Forum for Agricultural Research in Africa, ARD, December 2011  Special Climate Change Fund, ENV, October 2011  Association for Strengthening Agricultural Research in Eastern and Central Africa, ARD, August 2011  Public Expenditure and Financial Accountability, PSG, July 2011  Forest Carbon Partnership Facility, ENV, May 2011  Financial Sector Reform and Strengthening Initiative, FPD, May 2011  The Association for the Development of Education in Africa, EDU, April 2011  The Access Initiative, ENV, January 2011  Small States Network for Economic Development, OS, January 2011  Stolen Assets Recovery Initiative, PSG, December 2010  Communities, Conservation and Markets, ARD, November 2010  African Programme for Onchocerciasis Control, HNP, October 2010  Global Alliance for Vaccines and Immunization, HNP, September 2010  Africa Stockpiles Program, ENV, May 2010  Critical Ecosystem Partnership Fund, ENV, April 2010  Caribbean Catastrophe Risk Insurance Facility, FPD, April 2010  Affiliated Network for Social Accountability—Africa, SDV, March 2010  Global Environment Facility, ENV, January 2010  Global Facility for Disaster Reduction and Recovery, UD, January 2010  Trust Fund for Statistical Capacity Building, PO, January 2010 27 For a full description of GPPRs and evaluation methods, please see IEG 2006 Source Book for Evaluating Global and Regional Partnership Programs. Washington, DC: World Bank. 53 Appendix 5: FY10–15 Consolidated Trends by Expense Category and Income Source 1/ (in nominal dollars) FY10 FY11 FY12 FY12 FY13 FY14 FY15 Actual Actual Plan Projected Plan Indicative Indicative $'m $m $'m $'m $'m $'m $'m Fixed Costs Staff Costs 18.8 20.1 20.1 18.4 21.3 21.3 21.3 Communications and IT 1.2 1.2 1.2 1.2 1.2 1.2 1.2 Equipment and Buildings 1.4 1.4 1.4 1.4 1.4 1.4 1.4 Total Fixed Costs 21.5 22.7 22.7 21.0 23.9 23.9 23.9 Variable Costs Consultants and Temps 7.3 6.6 6.4 7.5 6.9 6.1 6.1 ETC & ETT 0.8 0.9 0.8 0.8 0.8 0.8 0.8 Travel Costs 1.9 1.6 2.4 1.9 2.1 2.1 2.1 Representation and Hospitality 0.1 0.0 0.1 0.1 0.1 0.1 0.1 Contractual Services 0.7 0.8 0.7 0.8 0.7 0.7 0.7 Other Expenses 0.6 0.7 0.6 0.7 0.7 0.7 0.7 Total Variable Costs 11.4 10.8 11.0 11.7 11.1 10.3 10.3 SRP and RSBP Contributions 0.3 0.2 0.2 0.2 0.2 0.2 0.2 IBRD Services and Support Fee 2/ 0.2 0.1 0.1 0.1 0.1 0.1 0.1 Total Expenses 33.4 33.8 34.0 32.9 35.3 34.5 34.5 Total Budget 31.9 32.4 32.5 33.2 33.2 33.2 33.2 Carryover 0.7 0.0 0.0 Total BBGEF 0.1 0.1 0.1 0.1 0.1 0.1 0.1 Total Trust Funds 1.3 1.4 1.2 0.9 1.2 1.2 1.2 Total Other Sources 0.1 0.2 0.1 0.1 0.0 0.1 Total Sources 33.4 33.9 34.0 34.3 35.3 34.5 34.5 Underrun/Overrun 0.0 0.1 0.0 1.4 0.0 0.0 0.0 Underrun % 0% 0% 0% 4% 0% 0% 0% Fixed Cost Ratio (BB Resources only) 67% 70% 69% 63% 70% 72% 72% 1/ FY10, FY11, and FY12 numbers are expressed in nominal terms. FY13, FY14, and FY15 numbers are expressed in real FY12 terms. 2/ The Total Budget number has been reduced in FY11 by $0.1m and IBRD Service and Support Fee reduced by $0.1m to reflect a reduced budget and reduced IBRD service and support fee. 54 Appendix 6: Consolidated Trends by (i) Department and (ii) Service Category (in nominal dollars) IEG: Expenditure Trends by IEG Department - FY12-15 FY12 (plan) FY12 (proj) FY13 (plan) FY14 (est) FY15 (est) $m $m $m $m $m Office of the Director-General 3.3 2.7 2.6 2.3 2.3 Strategy, Communications, and Learning Department 5.9 6.3 5.8 5.8 5.8 Private Sector Evaluations Department 7.6 7.5 8.4 8.0 8.5 Public Sector Evaluations Department 10.7 9.6 10.8 11.0 11.1 Country, Corporate, and Global Evaluations Department 6.5 6.7 7.6 7.3 6.7 Total 34.0 32.9 35.3 34.5 34.5 IEG: Summary of Sources and Uses and Key Spending Ratios - FY11-15 FY11 (act) FY12 (plan) FY12 (proj) FY13 (est) FY14 (est) FY15 (est) $m $m $m $m $m $m Total Sources 33.9 34.0 34.3 35.3 34.5 34.5 Uses Project/AAA Evaluations 6.0 6.7 6.3 6.3 6.3 6.3 Thematic/Sector/Corporate 10.0 8.3 9.3 10.5 9.7 9.7 Country and Global 2.2 4.1 3.2 3.3 3.4 3.4 Knowledge Activities 4.9 4.9 4.8 4.7 4.7 4.7 Rapid Response 0.6 0.6 0.3 0.6 0.6 0.6 Sustaining 5.7 5.1 4.4 5.1 5.1 5.1 Indirects 4.4 4.4 4.6 4.7 4.6 4.6 Total Uses 33.8 34.0 32.9 35.3 34.5 34.5 Proportion of spending on Products 56% 58% 58% 59% 58% 58% Proportion of Spending on Knowledge Activities 15% 14% 15% 13% 14% 14% Proportion of Spending on Sustaining Activities 17% 15% 13% 15% 15% 15% Proportion of Spending on Indirect Costs 13% 13% 14% 13% 13% 13% Nominal Spending Growth 1% -3% 7% -2% 0% Cost flexibility - Fixed Cost Ratio 70% 69% 63% 70% 72% 72% 55 Appendix 6 (continued) IEG: Expenditure Trends by Affiliated Parent Institution - FY12-15 FY12 (plan) FY12 (proj) FY13 (plan) FY14 (est) FY15 (est) $m $m $m $m $m Project/AAA Evaluations 6.7 6.3 6.3 6.3 6.3 WB 3.5 3.3 3.3 3.3 3.3 IFC 1.8 1.7 1.7 1.7 1.7 MIGA 0.5 0.6 0.6 0.6 0.6 Joint 0.7 0.6 0.6 0.6 0.6 Thematic/Sector/Corporate 8.3 9.3 10.5 9.7 9.7 WB 2.2 0.7 0.6 0.9 0.6 IFC 1.7 0.4 1.1 0.2 0.5 MIGA 0.0 0.0 0.4 0.15 0.2 Joint 4.4 8.2 8.4 8.5 8.5 Country and Global - Joint 4.1 3.2 3.3 3.4 3.4 Knowledge Activities - Joint 4.9 4.8 4.7 4.7 4.7 Rapid Response - Joint 0.6 0.3 0.6 0.6 0.6 Sustaining - Joint 5.1 4.4 5.1 5.1 5.1 Indirects - Joint 4.4 4.6 4.7 4.6 4.6 Total 34.1 32.9 35.3 34.5 34.5 56