Approach Paper Data for Development An IEG Evaluation of World Bank-Support for Data and Statistical Capacity June 24, 2016 Background and Context 1. Data and evidence are the foundation of development policy and the effective implementation of programs. Countries need data to formulate policy and assess progress, and the World Bank needs data to frame policies and assess the outcomes of its efforts to help end extreme poverty, promote shared prosperity, and meet the Sustainable Development Goals (SDGs). Yet, half of all member countries lack the data necessary to measure progress against the Bank’s twin goals of ending extreme poverty and promoting shared prosperity (Serajuddin et al, 2015). Monitoring the 169 targets under the SDGs compared to the 21 targets under the Millennium Development Goals (MDGs) is also likely to heighten the need for data. 2. The Bank has been an early and strong supporter of development data, providing an important public good (IEG, 2015). Development data, used synonymously with data for development, is defined here as data pertaining to countries’ social, economic, and environmental issues produced by country systems, the Bank, or third parties. (This definition is broader than the Data Council’s definition in that it also includes data produced by the Bank and third parties irrespective of the involvement of country systems.1) Development data may be sourced from national statistical offices, ministries, and other agencies, or may be sourced from censuses, household surveys, and agricultural surveys. It also includes administrative data, civil registration and vital statistics, economic statistics such as labor force and establishment surveys and trade statistics, geospatial data, other environmental data such as real-time monitoring, and mobile and IT-user data. 3. The Bank is involved in development data in various capacities and through many different types of activities, complicating the identification of the portfolio for this evaluation. 1 The Data Council was established on July 28, 2014 to: (i) provide a governance framework for managing the WBG’s data assets and (ii) to make strategic decisions about support for data activities in WBG client countries and for allocation of financial and human resources for data activities within the institution. The Council is co-chaired by the WBG’s Managing Director/Chief Operating Officer and the WBG’s Chief Economist. The Council has two pillars, one for corporate data assets and another for development data assets. The Data Council defines development data as data “that are produced by country systems around social, economic, and environment issues, such as data from censuses, household surveys, administrative records and civil registration.” 1 Box 1 summarizes the forms of Bank support to development data and Attachment 2 provides details of the portfolio. Box 1. Forms of World Bank Support to Development Data IDA/IBRD financing. Bank financing for development data has typically aimed at improving the economic and social information for policy making and poverty reduction. Activities in support of this objective have included: strengthening statistical planning and legislation, data infrastructure, human resources, data collection, data processing and analysis, data archiving, and data dissemination; preparing national strategies for the development of statistics; supporting statistical capacity building; and conducting censuses and surveys. Contribution to or execution of trust funds. The Bank has contributed financing to or executed trust funds, the major Multi-Donor Trust Funds being the Trust Fund for Statistical Capacity Building (TFSCB), Statistical Capacity Building Program (STATCAP), and the Statistics for Results (SFR) program. Advisory Services and Analytics (ASA). The Bank has supported ASA related to data, including the development of statistical master plans and country statistical assessments. The Bank has also provided training and non-lending technical assistance for strengthening statistical capacity. Production of datasets and administration of surveys. The Bank has itself produced datasets such as Country Policy and Institutional Assessments and Atlas GNI per capita, and administered surveys such as the World Bank Enterprise surveys, Doing Business surveys, and the Living Standards Measurement Study (LSMS) surveys. Standard setting. The Bank has worked with other organizations on standard setting for new statistical methods, data collection activities, and statistical capacity-building programs, for example: (i) the Bank’s work with the IMF on the General Data Dissemination System (GDDS) which provides guidelines to countries on the dissemination of economic, financial and socio-demographic data to the public and establishes a broad framework for countries seeking improvements in their statistical systems, (ii) the development of guidelines for the preparation of metadata covering the following areas: population, education, health, poverty assessment and monitoring (iii) participation in the governing body of the International Household Survey Network (IHSN), and coordinating the IHSN secretariat. Compilation of indicators. The Bank undertakes the collection of development indicators, compiled from officially-recognized international sources, and produces indicators such as the World Development Indicators. Innovation and research. The Bank launched the Open Data Initiative (ODI) to provide free, open and easy access to over 5,000 indicators. It gives “anyone the freedom to use, re-use, or redistribute data.” The ODI website allows individuals, groups, and organizations to create applications, programs, visualizations, and other tools that will help monitor and measure progress of various development initiatives and projects. With partners, the Bank also supports client countries with open data. There is also a program on “Innovations in Big Data & Analytics for Development” and research on survey methodology and other topics. Source: IEG. 2 4. The World Bank’s Articles of Agreement do not explicitly define a role for the Bank in data. There is also no Bank-wide policy on data, although Bank Procedure 14.10 does mandate that the Development Economics Vice-Presidency Data Group (DECDG) undertakes specific data tasks such as those related to external debt reporting, and Operational Policy 3.10 assigns to DECDG responsibility for per capita income estimates. 5. Furthermore, Bank presidents have emphasized the importance of data in their statements from time to time. For example, James Wolfensohn said in 2004 about the Marrakech Action Plans for Statistics (MAPS): “This is more than a detail. This is at the center of our ability to demonstrate progress and we have been too long in giving this the appropriate weighting.” And Robert Zoellick said in 2010: “We need more core data across countries and time periods on health, education, infrastructure, and gender. We need more and better data on public finance, especially at sub-national levels, which is critical for better governance. We need more hands and minds to confront theory with evidence on major policy issues…This is the direction that I want the World Bank to take. This is democratizing development economics. This will forever change how we conduct development research.” 6. In addition, the Bank’s role in development data is implicit in its launch of the Country Policy and Institutional Assessment (CPIA) scores in 1977, the World Development Indicators (WDIs) in 1997, and the Doing Business Indicators in 2004. 7. The Bank’s leadership or participation in a number of partnerships for data also reveals the Bank’s de facto role in development data. The Bank was a founding member of PARIS21 in 1999 to help build and strengthen national statistical systems in developing countries. To complement PARIS21 activities, the Trust Fund for Statistical Capacity Building (TFSCB) was set up in 1999 as a World Bank-administered, multi-donor trust fund to provide financial resources to developing countries for statistical capacity building activities. The Bank was also a founding member of the Marrakech Action Plan for Statistics (MAPS) in 2004 which aimed to broaden the efforts at both national and international levels to help developing countries achieve stronger capacities in statistics. The Bank hosts the International Comparison Program for global price statistics. 8. More recently, the Bank’s role in development data is evident in the establishment of the Data Council in July 2014 and in the World Bank Group Strategic Actions Program for Addressing Development Data Gaps: 2016 - 2030 (World Bank, 2016). The latter aims to reduce the gaps in the quantity and quality of development data in Bank client countries, and to encourage Bank clients to adopt data innovations that are scalable and proven to reduce the cost and time of data production, while preserving data quality. Box 2 summarizes statements on the Bank’s role in development data. 3 9. The Bank’s strategic approach to development data for 2016-2030 forward has been informed by the challenges facing the Bank’s work on development data. Several recent external, Bank, and IEG documents provide overviews and diagnostics of these challenges, which are discussed below. Box 2. Statements on the Bank’s Role in Development Data Development Committee (2015) “We encourage the WBG [World Bank Group] to ensure the technical robustness of the [SDG] goals and targets and to strengthen countries’ data capacity, to enable development and to monitor progress towards the WBG’s goals and the SDGs.” Role of the Data Council (excerpts, 2015)  Establish the strategic data priorities for the WBG and agree on the annual action plans for development data and corporate data  Make recommendations to senior management regarding the appropriate institutional arrangements as well as proper allocation of financial and human resources  Consider and endorse relevant data policies/standards/protocols for data management and oversee their effective implementation  Ensure that country clients and WBG teams can use data for evidence-based systematic diagnostics to identify the most critical challenges and opportunities for meeting the twin goals, and for effectively monitoring progress towards the goals. Role of the Data Council’s Development Data Directors Groups (excerpts, 2015)  Play a visible leadership role in the Data Revolution for Sustainable Development  Support client countries in building statistical capacity  Foster partnership and innovation for production and effective use of relevant, reliable, open and timely development data  Ensure a coherent WBG Development Data agenda in support of the WBG Strategy and its twin goals  Strengthen statistical capacity in client countries  Improve the quality and consistency of WBG-produced development indicators  Maximize the accessibility and effective use of WBG-held development data by staff  Gain recognition as a leader in customized client services for data production, management and analysis to unlock revenue sources  Foster data skills and professional growth within the WBG. Three Pillars of the Development Data Framework Endorsed by the Data Council (2015)  Increase country capacity to produce, disseminate, and use data  Improve WBG capacity to access, manage, and disseminate data  Produce high quality statistical indicators and data products to improve development outcomes. The Data Council has put forward a World Bank Group Strategic Actions Program for Addressing Development Data Gaps 2016 - 2030 aiming to operationalize a long-term WBG development data strategy in support of the SDGs and the twin goals. Its primary development objectives are: (a) to reduce the gaps in the quantity and quality of development data in Bank client countries, which are critical for the core business of the WBG; and (b) to encourage Bank clients to adopt data innovations that are scalable and proven to reduce the costs and time of data production, while preserving data quality. Source: Development Committee Communiqué, April 2015; Terms of Reference, World Bank Group Data Council, World Bank, September 29, 2015; World Bank Group Strategic Actions Program for Addressing Development Data Gaps 2016 – 2030, World Bank, 2016; Terms of Reference, World Bank Group Data Directors Group; World Bank, September 2015. 4 10. According to the terms of reference for the Data Council: “Development Data assets held across the World Bank Group (WBG) are fragmented, incomplete and often difficult to find. The existing model of data production and data use does not serve the WBG well any more. It is costly and inefficient, hampering our ability to make full use of these increasingly valuable assets to generate new, transformative solutions to the world’s most difficult development challenges…Relevant, timely and open data are central to operationalizing the WBG Strategy through a new approach to country engagement. Significant data gaps undermine effective policy-making and program design, and prevent measuring progress against development goals, including the WBG’s overarching twin goals. Key Executive Directors have expressed concerns over these development data gaps.” 11. The recent external review of the Bank’s Development Economics Vice-Presidency (DEC) found that the World Bank is seen as a leader in open data, data access, and presentation, but also that the role of data collection was under-appreciated (Besley et al, 2015). Led by three eminent statistical leaders, the data portion of the recent DEC evaluation concluded that the role of the DEC Data Group is not a Bank priority, a complete central data repository is lacking, there are only policy guidelines for data quality, there is no mandate to collect standardized data, and there is no professional track for data management and statistics. DEC has proposed a plan to adopt several of the recommendations from this evaluation, including the creation of a new position of a Chief Statistician. 12. In addition, two reports by the Bank’s Internal Audit Department (IAD) found weaknesses in governance, coordination, and quality management of the various data initiatives (IAD 2012 and 2013). Furthermore, some reviews found that even as data availability had improved, data literacy was an issue and the uptake, understanding, and demand for data was low in some countries (Thomson, Eele, Schmieding, 2013; Ngo, 2015). Finally, IEG’s report on the poverty focus of country programs concluded that the World Bank provides an important public good in supporting and reporting global poverty data and diagnostics, and that its work on poverty data is generally robust. It also found some fragmentation and lack of standardization, and urged the Bank to invest more in sustainable data collection and to adopt data reporting standards (IEG, 2015). 13. Much statistical capacity building has been conducted or financed through partnership programs (Box 3) and has aimed to strengthen the capacity of developing countries’ national statistical systems through the design and implementation of National Statistics Development Strategies (NSDS). IEG’s global program review of the Partnership in Statistics for Development in the 21st Century (PARIS21), the Marrakech Action Plan for Statistics (MAPS), and the Trust Fund for Statistical Capacity Building (TFSCB) found that these programs are generally working well and that the Bank has a comparative advantage in statistical capacity building (IEG, 2011). 5 Bank support to these partnerships has increased the number of countries with NSDSs, improved access to data, coordination of survey instruments, and expanded the number of countries conducting censuses (IEG, 2011; IEG, 2015). A recent external evaluation found that the International Household Survey Network (IHSN) had only limited success in coordinating the collection of survey data and improving survey quality at the country level (Ngo, 2015). Box 3. Partnerships for Data The World Bank collaborates closely on data and statistics with numerous agencies and national statistical offices. There is a partnership with the IMF, UN, and Multilateral Development Banks under a Memorandum of Understanding on statistics, signed in April 2013. The Bank also participates in several partnership programs and benefits from dedicated trust funds, including: International Comparison Program (1968): A partnership of various statistical administrations of up to 199 countries housed at the World Bank. The Program produces internationally comparable price and volume measures for gross domestic product (GDP). Trust Fund for Statistical Capacity Building (1999): A multi-donor trust fund that aims to improve the capacity of developing countries to produce and use statistics with an overall objective of supporting effective decision-making for development. The TFSCB supports projects aiming at strengthening national statistical systems in priority areas and developing statistical capacity in a sustainable manner, including openness and accessibility of data in line with the Open Data Initiative and innovative approaches to improve data collection. Marrakech Action Plan for Statistics (2004): A global plan for improving development statistics, agreed at the Second International Roundtable on Managing for Development Results in Morocco in 2004. Eight programs have been developed with the UN and other international agencies to take the identified actions into practice. Partnership in Statistics for Development in the 21st Century (PARIS21, 1999): A partnership to promote better use and production of statistics throughout the developing world. A worldwide network, PARIS21 is committed to evidence-based decision making through the improvement of institutional and technical capacity, stimulating, meeting, and improving national demand through comprehensive national plans for improvement. Statistics for Results Facility (2009): A multi-donor initiative, managed by the World Bank, to support statistical development in developing countries. Along with its Catalytic Fund, this initiative promotes statistical capacity building and supports better policy formulation and decision-making through improvements in the production, availability, and use of official statistics. Living Standards Measurement Study (LSMS) Program (1980): A household survey program focused on generating high-quality data, improving survey methods, building capacity, and facilitating the use of household survey data. The Global Financial Inclusion (Global Findex) database (2011): A comprehensive database on financial inclusion. It provides in-depth data on how individuals save, borrow, make payments, and manage risks. The first Global Findex database was launched in 2011 in partnership with Gallup and with funding from the Bill & Melinda Gates foundation. A second edition was launched in 2014. Partnership for Open Data (2014): A tool designed to help developing countries use open data standards and understand and exploit the benefits of open data. Its objectives are to support developing countries to plan, execute, and run open data initiatives; increase the use of open data in developing countries; and grow the evidence-base on the impact of open data for development. 6 Global Partnership for Sustainable Development Data (2015): A network of governments, civil society, and businesses working together to strengthen the inclusivity, trust, and innovation in the way that data are used to address the world’s sustainable development efforts. Source: IEG. 14. The Independent Audit of Data Collection and Production at the World Bank found that the Bank had made significant progress at collecting and producing data and establishing a governance framework, but there was still room to improve the capacity of some countries to produce timely and quality data. The weak link in the work has been in implementing NSDSs and using data for evidence-based policy making and planning (IEG, 2015; Open Data Watch, 2015). Open Data Watch’s review of 27 evaluations found that the more successful statistical capacity building programs focused on the users of statistics and their needs; ensured flexibility in the design of programs; adapted to changing country priorities and maintained country ownership; and coordinated the activities of donors (Open Data Watch, 2015). Objectives and Audience 15. The objective of this evaluation is to inform the Board and the Data Council on the effectiveness of Bank support to country clients in producing, sharing, and using development data, and to offer recommendations on how to strengthen it. Accordingly, the evaluation will focus outward on client countries and assess the Bank’s own data-related work in so far as it is a means to support client countries. Data production, sharing, and use are the three focus areas within the first block of WBG’s Development Data Framework. 16. This evaluation will take stock of the Bank’s experience relating to the production, sharing, and use of development data since the Marrakech Action Plan for Statistics (MAPS) in 2004, and its evolution over the period of the Paris Declaration (2005), Accra Agenda for Action (2008), and SDGs. It will also provide early reflections on the extent to which the establishment of the Data Council in 2014 and the launch of the Strategic Actions Program for Addressing Development Data Gaps 2016 – 2030 are beginning to, or are likely to, help address the gaps and challenges surrounding development data highlighted in external, Bank, and IEG documents. 17. This is a corporate evaluation that responds to the second objective of IEG’s Work Program – it will generate evidence and learning regarding the early implementation experience of the WBG strategy (specifically, that of the Data Council) to enable course corrections. 18. The nature of development data makes for a diverse group of stakeholders for this evaluation. Besides CODE, the main stakeholders are the members of the Data Council and its Data Directors Groups (comprising Directors from DEC and select Global Practices) and all Bank Regional and other units working with country clients on the production, sharing, and use of development data. Other stakeholder groups comprise staff of donor agencies, partnership 7 programs, national statistical offices, and sector ministries collaborating with the Bank on development data. Evaluation Questions and Coverage/Scope 19. The scope and questions for this evaluation were identified in an independent manner following consultations with key counterparts from DEC, Global Practices with a major role in data, and thought leaders in the field. Reflecting its objective, this evaluation will focus on the following evaluation question: How effectively has the Bank supported country clients in producing, sharing, and using development data? Country data capacity is an issue that pervades data production, dissemination, and use and the coherence of Bank support to fostering country capacity to produce, share and use data will be part and parcel of the evaluation question. Also addressed will be the institutionalization of data production, dissemination, and use through a review of the Bank’s work in promoting institutional, financial, and legal frameworks for national statistical systems. 20. Development data are an essential global public good that may be under-produced if left to individual countries. Accordingly, this evaluation will also ask the question: What roles has the Bank played in supporting data as a global public good and how effectively has it performed those roles? The adoption of the SDGs has catalyzed numerous initiatives to improve data availability with the Bank leading in some areas and contributing in others, often in partnership with other agencies. The evaluation will assess the Bank’s role as a provider of data for the global community and for SDG monitoring, and its role as a partner in various data initiatives. This part of the assessment will include sub-questions on the Bank’s contributions and funding modalities, and on the relevance and effectiveness of select partnerships with Bank participation (especially those listed in Box 3; the team will not be able to inventory and assess all data-related initiatives and partnerships because of their large number). The assessment will also attempt to review whether there is a tension between support for data for MDG and SDG monitoring and support for data for domestic uses for policy design, targeting, and monitoring. 21. The assessment of data production will cover sub-questions such as what kinds of data has the Bank supported country clients to produce, how has the Bank helped clients to identify their data needs, how has it identified data gaps, how has it helped country clients bridge those gaps, to what extent the Bank has fostered ownership of data among clients through involving them in choosing what kinds of data to produce or in other ways, and what steps has the Bank taken to ensure data quality?2 22. The assessment of data sharing and dissemination will cover Open Data issues (public access, frequency, and user-friendliness of data sharing); and access by the Bank to data it 2 The idea of ‘fit for purpose,’ i.e., that some quality dimensions are more important for particular types of data use, will be explored. For example, data used to inform cross-country comparison needs to be 8 finances. It will look at data sharing and dissemination both by clients and the Bank itself, internally and externally. 23. The assessment of use will cover sub-questions such as which types of Bank-supported data have the various country clients (i.e., government officials, NGO staff, civil society, and researchers) used, what purposes (e.g., policy making, targeting benefits to the poor, holding governments or donors accountable, conducting research and innovation, and monitoring and evaluation of programs and policies) have they used it for, and what are the key supply- and demand-side drivers of usage, including political economy factors? The evaluation will be mindful of the possibility that data may be used selectively without taking a balanced view of the evidence, or that it may not be used sufficiently or at all for reasons unrelated to the quality of Bank support. The evaluation will seek to establish a typology of use of development data and make headway toward measuring use. 24. Given recent internal reviews of governance issues relating to the Bank’s data work, limited new primary work will be done on Bank internal organizational issues. The findings of existing reviews including the evaluation of DEC and recent Internal Audit reports will inform the evaluation, complemented by interview findings. 25. IEG’s 2015 evaluation of the poverty focus of country programs found that the lack of good-quality, timely poverty data and the issues of data accessibility remain major constraints to carrying out robust poverty diagnostics and policy dialogue. Building on that finding, this evaluation will seek to identify ways in which the Bank can strengthen the effectiveness of its support for closing such major data gaps. 26. The evaluation will cover Bank financial support, trust funds, Advisory Services and Analytics, production of datasets and administration of surveys, standard setting, compilation of indicators, and innovation and research. In terms of Bank lending for development data, the initially identified portfolio consists of 291 projects approved since FY06 for which the data components amount to US $1.03 billion. 27. The Bank supports many different types of data; not all can be covered here. Different case studies will cover the types of data that have received most of the Bank’s attention and support—something that varies across countries, initiatives, and partnerships. Gender data will be covered given its corporate priority. Certain aspects of the Living Standards Measurement Study (LSMS) will be covered given its long history (since 1980). comparable. Conversely, comparability is less important for data used in domestic policy design and planning, where consistency and comparability over time is often more important. The requirements for frequency of data collection, level of disaggregation, accessibility, and so on are all dimensions that vary with intended use. 9 28. While issues of the relevance and effectiveness of data use will be covered in-depth and through primary analysis, issues of efficiency or value for money will be addressed through existing studies, as available. 29. Regarding whether the Bank is keeping abreast of, or is involved in, innovations in the fields of ICT, mobile data, big data, and so on, and how these initiatives are influencing data use, this evaluation will conduct a rapid rather than in-depth assessment. 30. The focus of this evaluation will be on reviewing the Bank’s efforts to help countries produce, share, and use development data. To keep the scope manageable, the evaluation will not cover: how the Bank itself produces, shares, and uses data (for research, for example) except as a means to support countries to do so; IFC and MIGA; corporate data pertaining to the Bank Group itself such as data relating to Human Resources or Treasury, and so on; qualitative data; and project monitoring and evaluation.3 Evaluation Design and Evaluability Assessment 31. As noted earlier, this evaluation will assess Bank support to client countries for the production, sharing, and use of development data. While the Bank has only recently formally articulated a plan for doing so, it has promoted the production, sharing, and use of development data for decades. 32. The conceptual framework for the evaluation will be a theory of change. This theory of change will be developed in consultation with counterparts and will serve to guide the team’s work. It will cover the entire chain from statistical capacity building to data production, sharing and dissemination, to data use by country decision-makers. It will spell out the underlying assumptions and enabling factors for each link in the chain from data production, sharing, and use. It will cover the following steps:  Bank inputs such as internal administrative resources, organizational structure and governance arrangements, and management signals and incentives as they each relate to supporting country clients in the production, sharing, and use of development data will be assessed mainly using existing sources.  Bank outputs such as the projects and programs financed through IBRD/IDA financing and trust fund support, advisory services and analytics, datasets and surveys, standard setting initiatives, compilation of indicators, and innovation and research (as described in Box 1) will be assessed, and the links between these outputs and Bank inputs will be examined. 3 Data purely or mainly for project M&E will not form part of the scope of this evaluation, in contrast to sector data with a broader statistical scope and coverage which will. 10  The intended outcome in terms of the effectiveness of Bank support to country clients in producing, sharing, and using development data will be assessed, and the links between this outcome and Bank outputs will be examined.  Factors that can enable—or potentially derail—the successful production, sharing, and use of data will also be examined. On the supply-side such factors would include the institutional context and state of the statistical legal and governance systems, management of national statistical systems, level of resources, and the infrastructure for statistics. On the demand-side such factors would include the political economy surrounding data, the government’s results focus, influence of other donors, and level of engagement by citizens and researchers. 33. Adopting a mixed methods approach, this evaluation will combine various quantitative and qualitative approaches, instruments, and sources:  Assessing the theory of change.  Analysis of ICRs and ICR Reviews for closed data projects, in collaboration with the team conducting the RAP 2016, combined with a stocktaking of the active portfolio.  About six country case studies of the Bank’s development data work, combined with at least two Project Performance Assessment Reports for statistical capacity strengthening projects in Ghana and Kenya. The case study countries will be sampled purposefully based on the following criteria: (i) at least one Bank-funded statistical capacity-building initiative that is active or recently closed (e.g., STATCAP program); (ii) multiple other data-related projects and Advisory Services and Analytics as identified in the preliminary portfolio; and (iii) diversity of regions and of large and small countries, with deliberate oversampling of IDA countries. The team will conduct interviews and roundtable discussions with data users and producers, including country client officials, staff in partner agencies, and researchers. The suitability of applying “process tracing” for determining the use of data in decision making will be considered for a few of the country case studies.  A comprehensive literature review of academic and grey literature, evaluations, and donor reports will be undertaken to establish what is already known about the key evaluation questions.  The evaluation team will explore the feasibility of a quantitative analysis to identify patterns of associations between the World Bank’s contribution to data for development and existing measures of data capacity, access, and use at the country level.  An electronic survey of Bank staff representing key groups of data users and producers will be undertaken relating to both lending and analytical work.  Interviews of Bank staff and managers sampled purposefully will be conducted to represent key groups of data users and producers. 11  An electronic survey of country client officials, staff in partner agencies, and researchers is under consideration and will be undertaken if existing surveys do not provide the necessary information.  Background papers or desk reviews exploring key topics (e.g., the Bank’s Statistical Capacity Indicator, trade-off between data needed for global monitoring purposes versus for national policymaking, gender data, and Bank policies and practices for data sharing) will be undertaken. 34. This evaluation will provide a broad perspective on the roles the World Bank plays in fostering data for development through partnerships. Review of partnerships in data for development will be mainstreamed as part of the evaluation. Some of these have been instrumental in statistical strengthening, setting data standards, and providing critical information for development-related purposes (e.g., the International Comparison Program and the Living Standards Measurement Study). The evaluation will assess what roles the Bank plays in data for development in partnership with other agencies and institutions, how aligned those roles are with the expectations articulated by the Data Council, how partnership programs and trust funds with Bank participation address the different elements of the theory of change leading to clients’ and partners’ use of data, the effectiveness of programs, and how and why partners and stakeholders value the Bank’s role and contribution. The methodology for the partnership review will follow a nested approach and provide an overview of existing partnerships, conduct a deeper document- based review of multi-country programs, and conduct a full assessment of the largest funded programs hosted by the World Bank. The review will be based on existing evaluations of the major programs, surveys of partners, clients, and program staff, and interviews. Design Strengths and Limitations 35. The strength of this evaluation lies in its ability to draw evaluative lessons from country engagements and corporate initiatives to inform strategic directions on development data. The main limitation is that this is a rapidly evolving field and too little time may have lapsed to conclusively assess the outcomes of some recent initiatives such as the establishment of the Data Council. The team will, therefore, attempt to inform the work of the Data Council going forward, but does not expect to make a summative assessment of the Data Council’s effectiveness. Furthermore, capacity building and data use can be difficult to evaluate in a manner that establishes causal relationships and attribution. Quality Assurance Process 36. This evaluation will be quality controlled in various ways. First, it will go through IEG’s normal process with IEG management providing oversight at each stage, including at IEG’s internal One-Stop meetings. Second, peer reviewers will comment on draft. The Peer Reviewers 12 are Elizabeth King (Nonresident Senior Fellow, The Brookings Institution), Pali Lehohla (Statistician-General, South Africa), Martin Ravallion (Professor, Georgetown University), Stefan Schweinfest (Director, UN Statistics Division), and Michael Woolcock (Lead Social Development Specialist, The World Bank). Third, the evaluation will receive additional quality review through participatory engagement with World Bank staff and managers, including via informal participatory workshops to discuss the draft Approach Paper (which has already taken place) and, if feasible, the draft evaluation report. There will also be engagement with the partners involved in the partnership programs being evaluated in order to seek their feedback. Expected Outputs, Outreach and Tracking 37. The output of this task will be a report of 40 pages plus Appendixes and an Overview. Engagement and consultation for this evaluation will be conducted on an ongoing basis, and the task of creating awareness and buy-in around it will hence start well before the dissemination phase. Dynamic methods for stakeholder outreach will be used during and after the evaluation. A complete dissemination and outreach strategy will be presented at the One-Stop review meeting for the full report, and will include a session that discusses the disclosed report with the Data Council’s Development Data Directors Groups, a launch event at Bank headquarters, blogs, and presentations at relevant events/conferences through which external stakeholders can also be reached. Resources 38. The team will be led by Soniya Carvalho and Rasmus Heltberg, co-team leaders and lead authors, complemented by Javier Bronfman, Sankalpa Dashrath, Ann Flanagan, Nidhi Khattri, Chad Leechor, Eduardo Maldonado, Estelle Raimondo, Vivek Raman, Swizen Rubbani, and Bahar Salimova, all IEG. External consultants Andrew Flatt, Morten Jerven, Joan Nelson, and Brian Ngo will advise and/or prepare background papers. The proposed net budget of this evaluation is $991,905. 13 Attachment 1 References Banerjee, Abhijit, Deaton, Angus, Lustig, Nora, Rogoff, Ken and Edward Hsu. 2006. An Evaluation of World Bank Research, 1998-2005. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/17896 License: CC BY 3.0 IGO. Besley, T., Paxson, C., Peter, H., and C. Udry. 2015. Evaluation Panel Review of DEC. A Report to the Chief Economist and Senior Vice President. Washington, DC: World Bank Group. Ferreira, Francisco H. G., Lustig, Nora, and Daniel Chaim Teles. 2015. “Appraising cross-national income inequality databases: an introduction.” Policy Research Working Paper No. WPS 7489, World Bank Group, Washington, DC. http://documents.worldbank.org/curated/en/2015/11/25384034/appraising-cross-national- income-inequality-databases-introduction Ngo, Brian T., and Andrew J. Flatt. 2014. Statistics for results facility - catalytic fund (SRF-CF): evaluation report of the pilot phase. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/2014/02/19586528/statistics-results-facility-catalytic-fund-srf- cf-evaluation-report-pilot-phase Open Data Watch. 2015. “Partnerships and Financing for Statistics: Lessons Learned from Recent Evaluation.” Open Data Watch Study, Washington, DC. PARIS21 Secretariat. 2004. “A Guide to Designing a National Strategy for the Development of Statistics (NSDS).” The Partnership in Statistics for Development in the Twenty-first Century (PARIS21), Paris, France. Roberts, Richard, and Claudine Voyadzis. 2009. “Evaluation of Paris21: Final Report.” Paris21, Paris, France. Salop, Joanne, et al. 2006. Independent Evaluation of DECDG Final Report (the World Bank’s Development Data Group). Washington, DC: World Bank. Serajuddin, Umar, Uematsu, Hiroki, Wieser, Christina, Yoshida, Nobuo, and Andrew L. Dabalen. 2015. “Data deprivation: another deprivation to end.” Policy Research Working Paper No. WPS 7252, World Bank Group, Washington, DC. http://documents.worldbank.org/curated/en/2015/04/24426058/data-deprivation- another-deprivation-end Snorrason, Hallgrimur, et al. 2010. World Bank Trust Fund for Statistical Capacity Building Evaluation Report. Washington, DC: World Bank. Soromenho-Ramos, Sofia. 2015. “Light Evaluation of Paris21: Final Report: Improving lives through better statistics.” Paris21, Paris, France. Sustainable Development Network. 2015. “Data for Development: a Needs Assessment for SDG Monitoring and Statistical Capacity Development.” United Nations. Thomson, Anne, Graham Eele, and Felix Schmieding. 2013. “Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP) Final Report.” Oxford Policy Management, Oxford, United Kingdom. Willoughby, Christopher; Crook, Philip. 2008. Marrakech action plan for statistics: report of an independent evaluation. Washington DC: World Bank Group. http://documents.worldbank.org/curated/en/2008/12/19132167/marrakech-action-plan-statistics-report- independent-evaluation World Bank. 2011. Marrakech action plan for statistics: partnership in statistics for development in the 21st century - trust fund for statistical capacity building. Global Program Review; volume 5, no. 3. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/2011/06/15456111/marrakech-action-plan- statistics-partnership-statistics-development-21st-century-trust-fund-statistical-capacity-building 14 ______. 2015. “The poverty focus of country programs: lessons from World Bank experience.” Working Paper. World Bank Group: Washington, DC. http://documents.worldbank.org/curated/en/2015/07/24846931/poverty-focus-country-programs-lessons- world-bank-experience ______. 2016. “World Bank Group Strategic Actions Program for Addressing Development Data Gaps: 2016- 2030.” World Bank: Washington, DC. 15 Attachment 2 Preliminary Portfolio Review World Bank Support for Development Data, FY06-16 The IEG project identification exercise aimed at selecting a portfolio of World Bank interventions that have involved support for development data activities using the criteria identified in the paragraphs below. The identification of a relevant portfolio focused exclusively on IDA/IBRD lending (including DPLs), recipient executed trust fund grants and World Bank analytic work approved between fiscal year 2006 and fiscal year 2015.The review did not include Bank executed grants, projects that were terminated, dropped, or are still in the pipeline. Methodology and data sources Given the absence of a harmonized system for tracking World Bank support for development data activities, the portfolio was constructed through a process of triangulating data from the different sources listed below Statistical Capacity Building Programs/Trust Funds: The first step involved selecting all projects approved under the World Bank’s different statistical capacity building programs/trust funds. A partial list of the Bank’s statistical capacity building programs was retrieved from the World Bank website.4 Accordingly, a list of projects was identified based on data available on the World Bank websites of the respective initiatives including, STATCAP, ECASTAT, and the Statistics for Results Facility Catalytic Fund (SRF-CF). The commitments approved under the Trust Fund for Statistical Capacity Building were obtained separately from the World Bank’s Business Intelligence database. Project lists from World Bank teams: The above project lists were supplemented with project data compiled and provided by the Development Data Group (DECDG) of the World Bank. DECDG maintains a list of all development data Bank activities led by DECDG project staff.5 World Bank thematic codes: Also included into the portfolio were all the Bank activities (loans, grants & AAA) assigned theme code 22 (Economic Statistics, Modeling and Forecasting). 4 This partial list of World Bank statistical capacity programs only included the programs managed by the Statistical Development and Partnership Team of the Development Data Group. Other existing programs in other Bank departments were not included. This list is available at http://www.worldbank.org/en/data/statistical-capacity-building/trust-fund-for-statistical-capacity-building 5 The IEG team requested for similar lists from other World Bank departments/global practices but was informed that there is no systematic tracking of this data in other GPs. 16 Although the Bank does not have a theme/sector flag to identify data related activities, it was felt that this particular code is a close approximation. Keyword search of project titles: A keyword search of project titles was used to identify projects whose names indicated support for development data activities. The keywords used included; statistical capacity building, devstat, stats, survey, and census. Keyword search of relevant databases: The final step in the portfolio selection process involved conducting keyword searches of the prior actions database and components database.6 The keywords used in the search included among others; data, statistical, open government, statistics, websites, and open data, civil registry, living standards measurement, census, survey. The identification of relevant AAA activities was done on the basis of only theme codes and project title searches. Results Based on the selection criteria above, a portfolio of 610 World Bank interventions was identified. This portfolio consisted mainly of AAA interventions which comprised 52 percent of the overall number of development data interventions. Table 1. Number of WB Interventions by Nature of Support, FY06-15 Type of WB support Number of interventions Percent of overall total AAA 319 52 Financing 291 48 Total 610 100 Source: Business Intelligence Database, IEG calculations. The AAA support for data was delivered primarily through Non-Lending Technical Assistance (NLTA) which accounted for 60 percent of the number of delivered AAA activities. In terms of the number of commitments IDA/IBRD financing accounted for about half of the commitments with data related content. By value, IBRD/IDA accounted for a greater share (77 6 The World Bank prior actions database is a consolidated database on all prior actions associated with development policy operations approved since 1980. The database is maintained by the Operations Policy and Country Services (OPCS) Vice-Presidency of the World Bank. The project components database has information on the projects components of investment loans approved since 1997. The database was created and is maintained by IEG. 17 percent) of the volume of financing commitments. This is mostly due to the fact that trust fund grants, though numerous, tend to be of very small sizes usually amounting to less than US$ 0.5 million (Table 2). Table 2. World Bank Financing Support by Product Line, FY06-15 Percent of Size of data Product Number of commitments by component7 Percent by line commitments number US$ M value IDA/IBRD 146 50 797.0 77 Trust funds 145 50 236.4 23 Total 291 100 1033.4 100 Source: Business Intelligence Database, IEG calculations. World Bank support by Fiscal Year The number of World Bank interventions did not fluctuate much during the period but there has been a dramatic rise in interventions since 2014. A possible explanation for this increase could be the renewed focus on support for building country statistical capacity under IDA 17. Figure 1. Number of WB Interventions by Fiscal Year 78 80 Number of interventions 60 51 39 43 33 37 40 30 26 28 26 28 24 28 26 24 20 19 20 16 20 14 0 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Fiscal Year AAA Financing Source: World Bank Database. The volume of financing commitments displayed more fluctuation during the period but has also risen greatly since 2014. 7 The size of the data component in the identified portfolio is understated because it does not include the commitment amounts for several Development Policy Operations whose data component amount could not be established. 18 Figure 2. Volume of WB Financing by Fiscal Year 212 200 156 160 142 US$ millions 137 118 120 73 64 80 42 43 46 40 0 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Fiscal year Source: World Bank Database. World Bank Support by Global Practice Three WB global practices account for more than half of the value of Bank interventions supporting data for development. The Governance, HNP, and Poverty & equity global practice together constitute about 68 percent of the total value of interventions. Table 3. WB Support by Global Practice WB Global Practice US$ value of data component % Governance 238.7 23 Health, Nutrition & Population 228.9 22 Poverty and Equity 218.4 21 Macro Economics & Fiscal Management 121.5 12 Social Protection & Labor 103.3 10 Education 60.0 6 Transport & ICT 28.2 3 Environment & Natural Resources 9.3 1 Social, Urban, Rural and Resilience Global Practice 7.4 1 Water 6.4 1 Other 6.0 1 Finance & Markets 3.8 0 Agriculture 1.5 0 Total 1033.4 100 Source: Business Intelligence Database, IEG calculations. 19 World Bank Support by Region The bulk of the Bank support for development data was concentrated in the Africa region which accounted for about 33 percent of the number and 49 percent of the value of financing interventions. Figure 3. World Bank Interventions by Region 60% 49% 50% 40% 33% Percent 30% 23% 20% 16% 15% 11% 12% 9% 9% 9% 10% 7% 3% 4% 0% 0% AFR SAR EAP ECA LCR MNA OTH Region By value By number Source: World Bank Database. Performance of Bank supported data for development projects Out of the 291 financing projects in the data for development portfolio, only 65 projects have already been rated by IEG. The IEG rated projects represent about 22 percent of the data portfolio by number of projects and about 34 percent by commitment value (Table 4). Table 4. Number and Value of Development Data Portfolio Value of data Data for development # of Percent by Percent by component8 portfolio projects no. value (US$ M) IEG rated 65 22 347.1 34 Unrated 226 78 686.3 66 Total 291 100.0 1033.4 100 Source: Business Intelligence Database, IEG calculations 8 The value of the data component is understated and excludes the commitment amounts for several DPL projects for which the size of the data component could not be determined. 20 About 62 percent of the rated projects received a moderately satisfactory development outcome rating or better while 39 percent of rated commitments received a high or significant risk to development outcome rating (Table 5). Table 5. IEG Ratings of Development Data Portfolio Percent in terms IEG Outcome Rating Number of projects of numbers HIGHLY SATISFACTORY 2 3 SATISFACTORY 16 25 MODERATELY SATISFACTORY 22 34 MODERATELY UNSATISFACTORY 18 28 UNSATISFACTORY 5 8 HIGHLY UNSATISFACTORY 1 2 NOT APPLICABLE 1 2 Total 65 100 Source: Business Intelligence Database, IEG calculations. 21