92623 The World Bank Group TOWARDS OPEN KNOWLEDGE: THE STATE OF WORLD BANK KNOWLEDGE SERVICES Operations Policy and Country Services Reform Secretariat July 18, 2011 ABBREVIATIONS AND ACRONYMS AAA Analytic and Advisory Activities ADePT Automated DEC Poverty Tables BB Bank Budget BEEPS Business Environment and Enterprise Performance Survey BW Business Warehouse CO Country Office CODE Committee on Development Effectiveness DALYs Disability Adjusted Life Years DEC Development Economics Vice Presidency (the Research Group) DFID Department for International Development DIME Development Impact Evaluation Initiative ESSD Environmentally and Socially Sustainable Development ESW Economic and Sector Work GDLN Global Development Learning Network GDN Global Development Network HDN Human Development Network HR Human Resources HRSLO Leadership and Organizational Effectiveness IBRD International Bank for Reconstruction and Development ICT Information and Communication Technologies IDA International Development Association IEG Independent Evaluation Group IT Information Technology JBIC Japan Bank for International Cooperation KSG Knowledge Strategy Group LAC Latin America and Caribbean Region LSE London School of Economics LSM Living Standards Measurement M&E Monitoring and Evaluation MENA Middle East and North Africa Region MIC Middle-Income Country MIS Management Information System ODA Official Development Assistance OED Operations Evaluation Department OETF Organizational Effectiveness Task Force OPCS Operational Policies and Country Services PERs Public Expenditure Reviews PREM Poverty Reduction and Economic Management QAG Quality Assurance Group RVPU Regional Vice Presidential Unit TA Technical Assistance TE External Training VPU Vice President Unit WBI World Bank Institute WDI World Development Indicators WDR World Development Report ACKNOWLEDGMENTS The report was produced under the overall guidance of Shanta Devarajan, Ann Harrison and Ani Dasgupta. Preparation of the report was co-led by Kene Ezemenari and Christos Kostopoulos, with the following core team members: Matthew Glasser, Frederick Swartzendruber, Jean-Jacques Dethier, Sharon Felzer, Alma Hajrovic, Melissa Fossberg, Alua Satybaldina, Fahrettin Yagci, Chukwuma Obidegwu, Michael Haws, and Raymond Boumbouya (OPCRS); Samuel Otoo and Violaine Le Rouzic (WBICR); Jill Armstrong (OPCDR) and Kamal Siblini (OPCRX); Augusto de la Torre and Laura Chioda (LCRCE); Maninder Gill (LCSSO); Bill Maloney (DECMG); Mapi Buitano (AFRCE); John Panzer (AFTP1); and Macdonald Benjamin (AFCSN). The report also benefited from inputs from the technical working group on various issues related to reform in the knowledge agenda. In addition, to the names above, these specialists included Klaus Lorch, Avjeet Singh, Rita El Ali, Marie H. R. Bakker, Bruce J. Courtney, Lesley-Ann Shneier, Andrei V. Tolstopiatenko, Leena Chaukulkar, Marco Nicoli, Michael Stumpf, Nidhi Khattri, Mohini Malhotra, Victoria Chang, and Theresa Bradley. Other contributors included Klaus Tilmes, Ramin Aliyev, Marisela Montoliu Munoz, Marijn Veerhoeven, Jaime Saavedra, Ambar Narayan, Nobuo Yoshida, Omer Karasapan, Catherine Gwin. The Reform Secretariat wishes to thank the distinguished external reviewers for their valuable comments: Martine Haas, Associate Professor of Management, Wharton School, University of Pennsylvania; Dirk Messner, German Development Institute, Vice Chair of the German Advisory Council in Global Change; Professor Wadan Narsey, Global Development Network Regional Partner for the South-Pacific, University of the South Pacific; Ravi Kanbur, T.H. Lee Professor of World Affairs, International Professor of Applied Economics and Management, Professor of Economics, Cornell University; Raquel Fernandez, Professor of Economics, Department of Economics, New York University; Roberto Steiner, Professor of Economics, Universidad de Los Andes in Bogotá, Researcher at the Economics Research Department of Colombia’s Central Bank; Mauricio Cardenas, Former Minister of Economic Development and Transportation, Director of National Planning of Colombia, Senior Fellow and Director of the Latin American Initiative at the Brookings Institute; Eduardo Moron, Former Peruvian Vice Minister of Finance, Professor of Economics at the Univesidad del Pacífico in Lima, Perú; Ahmed Galal, Managing Director, Economic Research Forum of the Middle East, former Executive Director and Lead Economist, Egyptian Center for Economic Studies; and Tchaboure Aime Gogue, International Consultant, former Minister of Planning, former Minister of Education, former Rector of the University of Togo, visiting Professor at the University of Quebec. The document was processed by Oxana Bricha and Ena Loureiro-Aguilar. Deborah Davis was the overall editor and made substantive contributions to the report. PREFACE This first World Bank Knowledge Report launches what is intended as a series of World Bank Knowledge Reports. Although knowledge is a fundamental to the Bank’s overall mission, and informs all of the Bank’s development activities, it has not received the same degree of attention from management that has been accorded to its lending activities. By paying attention to knowledge work as a distinct service, the Bank and our stakeholders will be able to better understand the results of that work. This will lead to better management of the knowledge production process, more strategic partnerships with other knowledge institutions, and greater operational and policy impact. As the first in the series, this report spends some time surveying the horizon—describing what kind of knowledge activities the World Bank undertakes and what its experience with this type of work has been. Consultations within the Bank and with external reviewers and partners during preparation of the report have enabled the institution to speak more clearly about the full range of these activities, and how they are evolving in response to changing conditions in the world and the changing demands of clients. In terms of the way forward, this first Knowledge Report makes some specific recommendations for measuring results and opening many of the Bank’s processes to more external inputs. The report also identifies important issues that will become part of ongoing conversations, both within the Bank and with clients and external partners. This volume is a synthesis of inputs from a variety of studies and units across the Bank. These are covered in significantly greater detail in a Background Report, which may be of greatest interest to Bank staff and managers who want a more in-depth understanding. The Background Report is available to any interested party on-line at __________________ or in hard copy upon request to the Bank’s Reform Secretariat. TOWARDS OPEN KNOWLEDGE: THE STATE OF WORLD BANK KNOWLEDGE SERVICES Contents EXECUTIVE SUMMARY ................................................................................................................................. i Chapter 1. A Changing World of Development Knowledge .................................................................. 1 Increasing number of globally connected knowledge providers............................................................... 1 Increasingly sophisticated clients.............................................................................................................. 2 Spreading and democratizing technology: ................................................................................................ 3 Changes in the nature of development knowledge ................................................................................... 4 The World Bank’s knowledge services in this changing world ................................................................ 5 Becoming more open and collaborative........................................................................................... 5 The Bank’s three knowledge roles ................................................................................................... 7 The Bank’s knowledge ecosystem ................................................................................................... 8 Chapter 2. The State of the World Bank’s Knowledge Services ......................................................... 10 The Bank’s knowledge services .............................................................................................................. 10 The core knowledge products .............................................................................................................. 13 Category 1: Knowledge for external clients .................................................................................. 14 Category 2: Global public goods .................................................................................................. 19 Category 3: Knowledge for internal use ........................................................................................ 20 Key trends .............................................................................................................................................. 22 Trust funds are increasing resources and shaping priorities .......................................................... 23 Shifts within the knowledge portfolio............................................................................................ 24 Knowledge is becoming more open ............................................................................................... 25 The Bank’s knowledge work going forward....................................................................................... 27 Chapter 3. Managing for Results............................................................................................................ 29 Measuring results and ensuring quality.............................................................................................. 29 Knowledge work for external clients ............................................................................................. 31 Global public goods ....................................................................................................................... 33 Internal Knowledge ........................................................................................................................ 34 Managing knowledge as a portfolio..................................................................................................... 35 Connectivity ................................................................................................................................... 36 Chapter 4. Conclusion ............................................................................................................................. 39 Recommendations ................................................................................................................................. 39 Looking ahead ....................................................................................................................................... 40 EXECUTIVE SUMMARY I. Introduction i. Just as knowledge is central to development, knowledge services are central to the value the World Bank brings to its clients. Knowledge is at the core of all of the Bank’s activities – lending, advice and convening – and provides insights for decision making and solving problems on the ground. This report focuses on knowledge services provided by the Bank. This report focuses on these services. On the one hand, the Bank provides country-level analysis, engages on specific policy issues, training, and technical assistance related to our clients’ particular development challenges . On the other hand, the Bank maintains the world’s most comprehensive, multi-decade data on a wide range of global, regional, country-level, and local-level development issues. In addition, the Bank is increasingly playing an important role in disseminating knowledge produced by other institutions. This report, the first in a series of World Bank Knowledge Reports, presents an overview of this complex and inter-related set of activities, and proposes ways for these activities to be better managed, evaluated, and used more effectively to meet changing global development challenges. ii. This report, which covers the whole spectrum of the Bank’s knowledge services, has three objectives. First, to try to understand how the changing landscape of development knowledge is significantly reshaping the Bank’s knowledge services. Given the increasing number of development knowledge providers, the growing sophistication of our clients, and the rapid changes in knowledge sharing technology, the Bank is rethinking how it can continue to bring the highest value to clients. The Bank has two unmatched capabilities—an in-house capacity to combine data, research, policy analysis, and practice to create customized solutions to specific development challenges; and deep and broad cross- sectoral and cross-country learning from ongoing engagement in project and policy implementation in more than 120 countries. In discussing the various components of the knowledge services and how they are responding to the changing landscape, as well as the less concrete aspects of the Bank’s institutional knowledge, this report aims to facilitate a critical discussion on how the Bank can best integrate and combine its vast array of knowledge resources to provide the highest value service to our clients and the global development community. iii. Second, to begin to address the Bank’s “knowledge paradox.� On the one hand, the Bank’s knowledge is highly valued and appreciated, and seen as a comparative advantage, by clients, as consistently shown by client surveys. In one survey in 2010, clients cited the Bank’s knowledge work as its most valuable contribution more than twice as often as its financial resources. On the other hand, staff regularly perceive that the Bank’s knowledge work is consistently undervalued by management, compared to lending. Surveys show staff dissatisfaction with the lack of incentives or recognition for knowledge work. In addition, evaluations by the Independent Evaluation Group (IEG) and Quality Assurance Group (QAG), have concluded that the Bank’s knowledge work is often more focused on production of a high quality report than on impact. The Knowledge Report argues that the root cause of both these problems is the lack of systematic measurement of how each component of the Bank’s knowledge services contributes to results on the ground. Attributing knowledge inputs to results is an inherently difficult exercise, which requires the development of robust yet practical procedures for evaluating the quality and effectiveness of each piece of knowledge work. The report argues for a practical way forward that includes upfront focus on objectives and clients, internal quality control, i measurement of intermediate outcomes and regular client feedback. The recent AAA reforms1were a first step forward in this agenda. iv. Third, to highlight the need for the Bank to embrace a comprehensive open knowledge agenda. In the 1950s and 1960s, the Bank focused on the reconstruction of infrastructure, and provided knowledge services that were technical in nature. During the structural adjustment period of the 1970s and 1980s, the Bank’s knowledge services primarily dealt with economic policies and their reform. Since the 1990s, it has become clear that technical analysis alone—be it of projects or policies—was not sufficient, because development required the building and strengthening of institutions, a process that was specific to the country’s social, political and cultural setting. Today, after the greatest financial crisis since the Great Depression—which many developing countries weathered better than their more developed counterparts—it has become clear that consensus is not the same as wisdom; but rather, that evidence- based debate and discussion, in the context of particular challenges, offers the best hope for progress. The events of the past year have shown that open debate and discussion can bring about real and profound change within very different societies. The Bank can help ensure that these changes take hold, and can support effective dialogue and policymaking going forward, by making its cutting edge and operationally relevant knowledge more accessible to client governments, civil society, and all other development actors. The Bank’s recent launch of the Open Data Initiative and other knowledge-sharing initiatives have been important steps in this direction (see box). Other innovative work is ongoing, including the creation and sharing of analytical tools, and the capacity building of change agents. But much more will need to be done in the coming years. The Bank’s knowledge services are responding to the global changes  The Bank is becoming more collaborative in its knowledge services. The Bank’s best-known knowledge activities—classified as “economic and sector work� and “technical assistance�—are evolving into closer collaborations with our clients. For example, the Vietnam Public Expenditure Review evolved from being Bank-led in 1996 to a joint product with government in 2005; the Brazil early childhood development and teacher quality work brought together local academics and local governments; and the Lao PDR Development Report, prepared in partnership with the Government and bilateral donors, fed into Government’s medium-term strategy; the Africa Infrastructure Country Diagnostic, a joint exercise with the African Development Bank and the Africa Union, documented the continent’s infrastructure deficit and how to address it.  The Bank is mainstreaming successful experiments, such as the South-South knowledge exchange, and undertaking new ones. The World Bank Institute (WBI) now offers a range of knowledge tools and support services, including a searchable South-South knowledge portal containing results stories and good practice examples. The knowledge platforms are a brand new experiment in collaborative knowledge creation through the sharing of policy and research experience across countries and among development institutions. The first three knowledge platforms on urbanization, green growth, and the role of ICT have just been launched.  The Bank has also adopted new policies: Open Data, Open Knowledge, and Open Solutions, on top of a new foundation, a new Access to Information Policy. Beginning in April 2010, the Bank released its development data free of charge. The Open Data work continues to move forward, as the Bank expands the data available for download from the Web, develops new applications to enable easy access to data across platforms and devices, and strengthens its capacity-building activities in developing countries to help ensure the quality and ready availability of relevant data. The new Access to Information policy is based on the principle that the World Bank will disclose any information in its possession that is not on a list of exceptions. It makes available new information on projects under preparation, projects under implementation, analytic and advisory activities, and Board proceedings. 1 Analytical and Advisory Activities is the term applied to the core set of knowledge services provided by the Bank as summarized in Figure 1 below. ii II. The state of the Bank’s knowledge services v. A significant portion of the Bank’s knowledge services are oriented to country clients and are a central component of country engagements. Knowledge services for country clients are typically designed as part of country partnership strategies, usually part of the same process that forms the lending portfolio. Knowledge services for clients are varied, and help the Bank fulfill many roles. In some cases, some reports, such as Country Economic Memoranda or Poverty Assessments, are used to inform Country Partnership Strategies. In other cases, a sector review (i.e. health sector review) or a strategy (i.e. an urban transport strategy) may inform the design of lending operations. In many instances, technical assistance services to countries (i.e. how to design a household survey or how to design a public investment appraisal system) lead to important policy discussions with counterparts. Sometimes the Bank uses knowledge services to simply connect clients from one country to another, in the belief that meetings between practitioners are sometimes the most efficient means of transferring knowledge. Taken together, with its knowledge services, the Bank performs the triple roles of knowledge producer, knowledge customizer, and knowledge connector. The Bank spends about half of all spending on core knowledge (defined below) for country clients in these roles. The other half of Bank spending on core knowledge is geared towards global and regional clients, and is done by networks, by regions, and by DEC and WBI. This report is about the knowledge activities that the Bank uses to fulfill these three roles. vi. The Bank’s knowledge services serve three distinct functions. They provide: (a) knowledge for clients: (b) knowledge for public goods; and (c) knowledge for internal use. These services are classified along of nine product lines, each often produced by a different part of the Bank and each serving a different purpose in the knowledge ecosystem.2 Figure 1 provides an overview of the nine product categories and the parts of the Bank that produce them. Figure 1. Distribution of core knowledge activities in FY10 2 The importance of other, “non-core� knowledge work, including strategy documents, evaluation and quality assurance, and some of the work with partnerships, will be considered in future knowledge reports. iii vii. These core product lines comprise:  Economic and sector work (ESW), to provide analysis that informs policymaking  Non-lending technical assistance (TA), to support clients in implementing reforms  External client training (TE), to develop the capacity of country-level actors  Impact evaluation (IE), to measure and assess the outcomes of interventions  Global monitoring, to provide data on global development trends  Research services, to deepen understanding of key development issues  World Development Report, to consolidate state-of-the-art knowledge and advance the development debate  Internal knowledge products and tools, to provide staff with cutting-edge knowledge  New product development and piloting of new Bank business lines. viii. In FY10, the Bank spent a total of $606 million on all three major categories of knowledge products, of which about 40 percent came from trust funds. Each knowledge product line serves a different purpose: (a) Knowledge for clients a. Economic and sector work is the best-known of the Bank’s knowledge activities, produced in various forms almost from the inception of Bank operations. It provides analysis and information that helps clients to develop effective policies. Examples include the Senegal Country Economic Memorandum: Looking for Work - The Road to Prosperity and Transport Prices and Costs in Africa: A Review of the Main International Corridor. Today an increasing share of ESW is in the form of global or regional studies, sometimes prepared with other organizations. Country-focused ESW includes poverty assessments; public expenditure reviews (PER); debt and creditworthiness studies; sector reviews (environment, infrastructure, energy, health, education, gender); and sharply focused policy notes addressing a specific issue identified by a client Government. A significant trend over the last decade has been a steady decrease in the number of country level core diagnostic work (economic memoranda, PER, poverty assessments, etc). In 2010, the Bank for the first time spent less on economic and sector work than on technical assistance. ESW is produced mainly in the Bank’s Regions, generally to support or guide country operations and policy dialogue. In FY10 it accounted for 30 percent of the Bank’s expenditure on knowledge services. b. Non-lending technical assistance supports clients’ efforts to implement reforms and strengthen institutions. One successful example cited by an IEG evaluation is the Bank’s “Aid for Trade� work in 20063, which helped the Government of Mauritius to define its reform program. A Bank mission provided analysis of possible reforms, including estimates of the adjustment costs of opening up the economy, identified tax reforms, and articulated strategies to lower the cost of key services. Many of the mission’s recommendations were incorporated into the Government’s reform program, which was supported by a follow-up loan. More recently, a programmatic technical assistance activity is helping Turkey to modernize its agri-food supply chains, in preparation for EU accession. Surveys show that clients place a high value on technical assistance, and because the nature of the work requires engagement with clients, collaboration is a natural part of technical assistance work. The Quality Assurance Group’s assessments of ESW and TA found that TA activities tended to have better defined the objectives at the inception stage (than ESW activities) and more effective arrangements to disseminate the knowledge gained to stakeholders (via 3 This report includes examples of work completed several years ago. The goal was to include examples that have had demonstrated impact on the ground, a process which often takes several years to materialize. iv workshops, local publications, or other means). Like ESW, TA is also primarily carried out within the Regions, and accounted for just over 30 percent of the Bank’s knowledge services expenditure in FY10. c. External client training. The World Bank Institute, which is the main provider of external training, is now focusing less on training individuals and more on leadership and coalition building, to help change leaders navigate the political economy of reform. WBI has started to deliver this training at the wholesale level to regional training institutions. WBI also works to build the capacity of both government and non-governmental actors (parliaments, civil society, private sector, youth), and helps convene platforms for this diverse set of actors to coalesce around policy and institutional reforms. Recent client training programs have included governance in extractive industries, policies to support innovation, and the management of agricultural water in the context of climate change. External client training represented about 9 percent of the Bank’s spending on knowledge services in FY10. d. Impact evaluation is the newest type of client-focused knowledge product, established in 2006 as part of the Development Impact Evaluation (DIME) initiative. Impact evaluation measures the effect of a development intervention relative to a specified counterfactual. Recent impact evaluations have examined early childhood development programs in Brazil, and malaria control efforts in Nigeria. The DIME initiative is led by DEC and most impact evaluations are conducted in the Regions, and represented about 2 percent of all knowledge service expenditure in FY10. (b) Knowledge as a public good e. The World Development Report, published once a year, is the Bank’s flagship knowledge product. Each WDR provides an in-depth discussion and analysis of a specific aspect of development, based on state-of-the-art global knowledge. The 2010 WDR focused on Development and Climate Change, and the 2011 WDR focuses on Conflict, Security and Development. The 2012 WDR will be on Gender Equality and Development, while the 2013 WDR will focus on Jobs. WDRs bring together research, case studies, and the results of extensive consultations with leaders and development practitioners throughout the world. An independent evaluation pointed to the "lasting importance and influence" of several WDR reports. The WDR, managed by the Development Economics Vice Presidency (DEC), represented about 2 percent of total knowledge services expenditure in FY10. f. Research is carried out primarily by DEC (also known as the Research Group). It is an original analytical effort with wide applicability across countries or sectors, carried out to advance knowledge or stimulate debate. Unlike work for clients, research products do not necessarily represent Bank views (work is owned by the authors, not the unit). Recent research topics have included conditional cash transfers, the impact of citizen report cards, and the effectiveness of antiretroviral drug delivery programs. In recent years DEC has expanded its collaboration with other research organizations and is in the process of developing a series of innovative tools to enable wholesaling of analytical work. In FY10 research accounted for about 6 percent of all knowledge service expenditure. g. Global monitoring products allow the Bank to combine expertise in regional issues, market trends (e.g., world trade, financial conditions), and forecasting methods to construct a consistent view of global economic prospects and the implications for macroeconomic trends in developing countries. An example is the Little Data Book series and the World Development Indicators (WDI). The Bank’s new Open Data initiative has geometrically expanded the number of users accessing the Bank’s databases –from about 140,000 paid subscribers prior to April 2010, when the policy was v implemented, to 20 million page views since then. Global monitoring is centered in DEC and accounts for about 3 percent of the Bank’s knowledge spending. Another example of a global monitoring tool is the Doing Business database, which makes the FPD website the second most visited site after DEC’s in the Bank. (c) Knowledge for internal use h. Internal knowledge products are aimed at one important goal—ensuring that staff are well connected across the globe, and have access to the ideas, experiences, research, and global knowledge needed to be as effective as possible. A 2009 QAG report4 found that the Bank’s internal knowledge products are of significantly lower quality than our knowledge products for clients, and that they are supply driven and undermanaged. Networks have been working to improve the quality of knowledge products for internal use. For example, the SDN network is now applying the quality assurance framework used for economic and sector work to internal knowledge products. Internal knowledge products are primarily produced in the Networks, and accounted for about 13 percent of the Bank knowledge expenditure in FY10. Internal knowledge products include studies, operational guides or toolkits, and sectoral or thematic databases used by Bank staff. Recent studies include Policy Briefs on the Financial Crisis, and Costing for Nutrition Interventions; the Pharmaceutical Policy Book is an operational guide; and an example of an internally-focused database is Logistics Perception Indicators. i. New products include pilots and prototypes of new Bank products, approaches, and business lines. These are produced mainly by the Networks and other central units, and represent about 3 percent of knowledge spending Examples of new products include the Prototype Carbon Fund, and development of new sub-national finance products for Bank clients. Key cross-cutting issues for knowledge services ix. Composition of knowledge activities is changing; trust funds are an important factor5. The Bank administers nearly 3,500 separate trust funds, each with its own objectives. There needs to be careful consideration of how to ensure that trust-funded work is consistent with the Bank’s strategic priorities.  Although trust funds make it possible to do more work than can be funded through the Bank’s budget, these funds also influence the composition of the Bank’s knowledge portfolio and the type of staff that are hired. Yet, trust funds have enabled the Bank to provide much more TA that it would have otherwise in their absence. Trust funds contributed about 40 percent of funding for knowledge services in 2010, up from 27 percent in 2002. By product line, trust funds accounted for 81 percent of funding for impact evaluation, 53 percent of technical assistance, and less than 30 percent of ESW.  Trust funds have played a significant role in the shift from ESW to technical assistance. Ten years ago, twice as much was spent on economic and sector work as on technical assistance. In 2010, there is more trust-funded technical assistance than other category of knowledge activity. One category of TA, “how-to� guidance for clients, more than tripled from FY05 to FY10 and now accounts for over half of all TA. With regard to ESW, trust-funded spending 4 Synthesis of Past Assessments of Analytical and Advisory Activities (AAA): a QAG Review, QAG, 2009. 5 There is an ongoing, larger discussion on trust fund reform. This report only addresses how trust funds affect knowledge services. vi on country work has been declining relative to regional and global work; and spending on core diagnostic work has been declining most significantly. The Bank may want to discuss whether sufficient ESW work at the country level, either core diagnostics or other ESW, is being undertaken to (a) inform the Bank country strategies and the lending portfolio, and (b) to maintain a level of readiness to support clients in future crises, in response to the Post Crisis Directions. In doing so, the Bank may wish to survey the degree to which country teams are able to collaborate with other donors and jointly undertake diagnostic work. x. The large number of knowledge activities has become difficult to manage. In FY02, the Bank carried out 3,672 separate pieces of knowledge work. By FY10, in response to client demand and the availability of trust funds, this number had increased to 5,192. Given the decentralized nature of the Bank, the responsiveness to country demand, and the differentiated priorities of each of the trust funds, Networks, and Regions, as well as WBI and DEC, it is difficult for all of these services to be maanged as a cohesive portfolio. In response to this challenge, consolidations are ongoing both in regions (e.g. SAR) and at the center. For example, within its own knowledge operations, WBI has grouped its knowledge activities into defined categories and dramatically reduced the number of tasks from 1,677 in 2005 to 292 in 2010. xi. The Bank does not systematically measure the impact of its knowledge services . The sheer number of knowledge tasks makes it difficult to provide adequate management attention. Peer reviews — a critical component of the Bank’s quality assurance system—are carried out only inconsistently. An Activity Completion Summary (a self-assessment by the task leader) is required for only three of the nine knowledge product lines. There is also no systematic arrangement to taken an an independent view on quality of knowledge services. IEG did carry out an extensive, and so far, the only, survey of impact of ESW and TA in 2008. Thus, the Bank has limited tools to monitor and assess the effectiveness of its knowledge portfolio. xii. Knowledge is becoming more open. The Open Data, Open Knowledge and Open Solutions initiatives, along with the new Access to Information Policy represent significant departures from the way the Bank thought of its knowledge services in the past. The new initiatives, in part based on practices that had been developing, and in part based on the need for a new vision in light of the changing landscape for development knowledge, are already influencing the way the Bank works. Hits on data websites the highest in the Bank increased significantly. The three knowledge platforms were launched in FY11, while another three are pending for FY12, solidifying the central nature collaborative knowledge generation will have for the Bank. The Bank's Office of the Publisher is making the transition from a traditional book publishing model to a new open model for the dissemination of the Bank's formal research outputs. Open debate is a more inclusive and expanding model of engagement with clients, which involves civil society, NGOs, and the private sector in evidence-based debate on policy and investment choices. xiii. The survey of the nine components of the Banks knowledge services illustrates, first and foremost, that clients value Bank knowledge services (see box below). It also identifies the need for the Bank to improve regular measurement of results of knowledge services in order to manage these activities better and the systems that allow integration and connection among various knowledge services. As discussed in chapters 2 and 3 of this document, attention to results is critical for ensuring that the Bank’s knowledge portfolio is meeting clients’ needs and making the best use of Bank resources. The report proposes that the quality of knowledge services be evaluated in three dimensions: technical quality of the work; its relevance, as defined by the needs of client countries and alignment with the Bank’s strategic objectives; and results, in terms of development impact. Chapter 3 outlines the importance of a knowledge results framework to improve the overall quality and governance of knowledge activities, as well as to strengthen the knowledge culture within the Bank. vii III. Managing for results xiv. Measuring impact of knowledge is inherently difficult. Besides the challenges of attribution and causality, knowledge work often takes a long time to show results. The central challenge is to clearly and systematically define the goal of each knowledge activity, and then to think through how to measure progress toward that objective. When the goals of a knowledge activity are clearly stated, it then becomes possible to assess progress by measuring intermediate outcomes. This approach has already been adopted by some parts of the Bank. The recently completed AAA Review (by OPCS in FY11) has proposed that teams be required to define, in collaboration with clients, the change objectives to which a knowledge activity or series of knowledge activities will contribute. The review also proposes that knowledge activities be designed using multi-year programmatic approaches, to enable greater flexibility than is possible if they continue to be tied to the project cycle. xv. The use of multiple approaches would enable the Bank to make a rounded assessment of progress. One approach would involve maintaining and strengthening the current self-assessment mechanism by adopting the measures proposed in the AAA Review. Another would be to systematically use client surveys as an important source of information gathering. Two approaches to client surveys have been piloted in the course of preparing this report. Other approaches to eliciting the client’s assessment of the impact of knowledge activities within the context of the overall country portfolio will need to be piloted to inform the way forward. Storytelling or case studies can also be used (i.e. MENA knowledge notes), as well as web-based metrics such as number of views and downloads; and references to knowledge products in policy documents, the popular press, and academic journals. Piloting Client survey of economic and sector work and technical assistance activities (2011). As part of the preparation of this report, a large survey was conducted of knowledge activities completed between FY08 and FY10. The survey was designed to be representative of ESW and TA work conducted during that period. The responses received do not yet form a representative sample 6. Despite this, it gives an indication of client perceptions, which are largely consistent with IEG’s findings. In general, respondents considered the Bank's ESW and TA products to be effective in supporting policy decisions and implementation. The survey also showed an almost inverse relationship between cost and impact on policy—smaller reports (costing less than $50,000) were considered to have had the most impact, and larger reports (costing more than $500,000) the least impact. Clients also reported significant preference for non-lending TA (NLTA) over ESW in terms of providing support for policy decisions, timeliness, relevance, and being a source of relevant information on global good practices7. These findings indicate that the traditional role of ESW and TA may be blurred; TA appears to inform client policies as well as ESW. xvi. The governance of knowledge activities is hampered by shortcomings in connecting knowledge across product lines, regions, and central units of the Bank. A recent assessment by the 6 The survey was planned and conducted year or two after the ESW and TA was completed. Often task teams had difficulty locating the contact information of their clients. One of the recommendations of this report is to make it mandatory to identify and agree with management the list of clients at the beginning of any activity. 7 Non-lending Technical Assistance (NLTA) captures services provided by the Bank to clients as a way to assist them in policy / program implementation, developing / strengthening institutions, and facilitating knowledge exchange; it has no Bank financing attached to it. In contrast, a Technical Assistance Loan (TAL) is used to build institutional capacity in a borrower country. It may focus on organizational arrangements, staffing methods, and technical, physical, or financial resources in key agencies. Since TALs are tied to Bank financing, they typically require agreement on specific action programs to strengthen organizations, and on terms of reference for the appointment of consultants and local counterparts. viii Finance and Private Sector Development Network (FPD)8 noted that the scale, budget, and strategic alignment of knowledge activities are limited by the fact that the Networks and Regions operate in silos. In the FPD Network, a proposed restructuring (to be implemented in July 2012) will address these issues by putting in place new processes, structures, and information infrastructure to reward knowledge sharing and promote collaboration across regional units. Strong interaction across units, particularly between the Regions and Networks, brings to bear global experience in the particular sector or technical field and offers assurance of the knowledge activity’s relevance and quality. External connectivity is equally important and the above box outlines some of the initiatives the Bank has launched to strengthen it. xvii. The Bank will need to put greater emphasis on managing knowledge services as a portfolio to ensure connectivity and knowledge flow. At present, decentralized knowledge production enables the Bank to be responsive to the needs of individual clients, primarily through country units; and to contribute to global knowledge through various network councils and sector boards. This decentralization, however, limits management’s opportunities to increase the overall impact of the knowledge portfolio; instead, synergies among the different knowledge services—and the units that provide them—happen informally, with minimal coordination. xviii. Different units across the Bank are already engaged in initiatives to strengthen the connectivity of knowledge services and promote greater movement of information across the institution. At the sector board level, aside from restructuring of the FPD network to improve the flow of knowledge across the institution, other sector boards are developing a 10 point program to strengthen quality, human resource management, and knowledge flow across the institution. At the regional level, one approach to encouraging knowledge flow is the LCR Chief Economist’s Office coordinating regional knowledge services with country management units; another example is MNA’s important effort to draw lessons from knowledge services embedded in lending operations (these are funded by borrowers and carry lessons applicable across the Bank); and yet another example is SAR’s effort to structure the numerous regional knowledge services into portfolios with clear objectives. xix. Finally, the effective governance and assessment of knowledge activities will depend on the existence of a coherent results framework. This will enable management to determine how well each part of the knowledge system is contributing to the overall effectiveness of the Bank’s knowledge work. Such a framework will also show where resources are adequate and where they are lacking, and enable a discussion about the allocation of resources, including trust fund resources. IV. Summary of Recommendations xx. In order to address these challenges, the Bank needs to engage in a conversation about the most effective and practical ways to measure results of knowledge activities and about how to strengthen the connectivity of various components of the knowledge portfolio. This would allow the Bank to ensure that individual knowledge services to client countries have the greatest possible impact, not only by being better focused on results, but by better drawing on the knowledge and experience from other countries. This would go a long way to resolving the knowledge paradox. It would also allow the Bank to have the greatest possible impact in the areas of global public goods that it engages in, since with better connectivity, it could rely on one of its important comparative advantages, its presence in 120 locations. However, the changing landscape for development knowledge is forcing the Bank to change not only the issues it works on, but the way it works –across the nine types of knowledge activities. The new approach of democratizing knowledge proposes that the Bank’s scarce resources can be used much more effectively when aimed to a much broader audience. 8 FPD Global Practices – Prototyping Global Practices in the FPD Network. June 25, 2010. ix Measuring results and strengthening quality xxi. While it is essential to strengthen results measurement for the Bank’s knowledge work, many steps have already been taken. The report recommends that the current plans for measuring the results of economic and sector work and technical assistance be expanded to require a systematic approach to assessing the results of all core knowledge work. This requires a few basic elements: a. a clear objective when the knowledge activity is initiated; b. agreement on intermediate outcomes and measurable indicators, including wherever possible information on citations, page views, and downloads; c. budgeted plans for assessing outcomes; and d. mechanisms to ensure lessons are captured and fed back into the design and implementation of future activities. xxii. Inferring the impact of knowledge services will require the adoption of multiple evaluation methods (or triangulation). This will have to complement the current self-assessments and IEG evaluations with client and stakeholder surveys, and with periodic independent and retrospective evaluations. To ensure timely responses from clients, task managers should be mandated to identify possible clients for each task as a prerequisite for doing completion reports. To ensure that explicit knowledge products have a lasting impact, task managers should be required to indicate in completion reports where data sources and analysis can be found on Bank websites for replicating analysis. xxiii. Quality assurance, support and monitoring of knowledge services at other levels could be enhanced as follows: e. Peer review practices should be strengthened and systematized, with external experts and client perspectives included in peer review panels. The role of sector boards could be enhanced in ensuring that global experience is taken into account in the peer review process. f. Management arrangements should be made consistent. It is essential that management have appropriate tools to ensure quality, results, and relevance. This may include clearer “sponsorship� for all product lines, where the sponsor, would responsible for providing guidelines, monitoring the portfolio , and arranging post-review of quality, relevance, and results within each product line. Managing knowledge as a portfolio and strengthening connectivity xxiv. The Bank’s primary building block for delivering knowledge services is the country level, even though just about half of funding spent on core knowledge is for clients . In order to ensure that country clients get the maximum impact from Bank knowledge services, the institution needs a forum or a process to ensure that knowledge from the rest of the Bank (via networks), new knowledge from networks, knowledge from DEC and from WBI is made readily available and supports the country programs. Likewise, the Bank’s engagement in global public goods, driven by the networks, DEC, and WBI, would have greater impact if it could be linked to country programs, especially given the local presence in 120 countries. However, Bank resources are scarce, and the Bank cannot afford to be heading in multiple directions at once, even if trust funds play a significant role in alleviating resource constraints. As such, this report argues that knowledge also requires a corporate perspective because while the current approach produces locally optimized outcomes with regard to both priorities and quality assurance, this system limits connectivity and opportunities for synergies and economies of scale. x Moving toward conscious priority setting will require debate and discussion. The comparable business lines across networks, DEC, and WBI, recommended by CFR, are one instrument to do this. The report recommends creating explicit mechanisms for discussion of work programs and priorities among networks and regions. One possibility would be to use more extensively current review bodies, such as the KLC and the CEC, in reviewing work programs across networks, regions, WBI, and DEC. This has the potential to strengthen quality, relevance, and results; identify avoidable overlaps and duplication; and achieve genuine synergies. We can improve our quality and results by inviting external discussion and review of the quality of our knowledge services, how we prioritize our knowledge activities, and how we can bring the best minds in the world to bear on the most pressing development challenges. xxv. The networks are expected to play an important role in strengthening connectivity across the Bank. This is envisioned in the 10 point strategy to strengthen sector boards. Evaluating the results from knowledge products (as one of the nine knowledge activities), as proposed in this report, would be important. Networks will have to be transparent about their own plans to encourage and to support the flow of knowledge, particularly in light of the fact that FPD considered it important to restructure its organization and invest heavily in a work collaboration system to succeed in increasing the flow of knowledge across the institution. As part of the work collaboration system, document capture will have to play a critical role. All knowledge outputs should be captured in the Bank’s document systems, in a form that is retrievable and searchable to meet continually changing client demand, and benchmark these systems with practices in the private sector to ensure timely responsiveness. xxvi. Another important dimension of knowledge flow is staff incentives . This too is critical to resolving the knowledge paradox. However, it can be argued that the most important factor to improving staff incentives for knowledge flow is signals from management – demonstrated leadership and behavior change in sharing knowledge. Incorporating knowledge services in the corporate scorecard will also be critical to that end. In addition, emphasizing the achievement of an intermediate output of a knowledge activity, instead of the production of new knowledge against end of FY deadlines can help change incentives. At the same time, the Bank may wish to review the current approach to incorporating knowledge flow in the staff performance evaluation, and take lessons from other organizations. Open Knowledge xxvii. The Bank has rolled out the Open Data, Open Knowledge, and Open Solutions initiatives, and has already developed experience at the corporate level with implementation, for example with DEC and WBI, and in the Africa Region. The next steps for the Bank are two-fold. First, to track the ‘Open’ programs, and to monitor user satisfaction, and ensure these knowledge services are accessible and user friendly. Second, to think through how these initiatives can be implemented at the country level, taking into consideration each country’s context. Looking ahead xxviii. This Knowledge Report is the first in a series. The research and consultations that led to this report have helped us to understand the knowledge landscape, and we have mapped a portion of that vast territory. Within the territory covered by this report, we have recommended specific work to bring about more coherence, oversight, and accountability. Beyond the scope of this report, we have prepared a working list of topics that might usefully be covered in future reports. That list will continue to evolve through further research and consultations, and we will report each year on both the implementation and evolution of this year’s recommendations, and the agenda for further consideration. xxix. The next Knowledge Report could focus on knowledge flow. The discussion in this Report has largely been grounded in the reality that the Bank manages knowledge products much the same as it xi manages lending projects. The management information system we use to manage knowledge actually has its roots in enterprise software developed to manage industrial processes. This makes it convenient to speak about knowledge products, and to manage the creation and completion of a discrete knowledge product. This report has mostly dealt with explicit, codified knowledge products. We have not looked at tacit knowledge, i.e., the stock of knowledge that Bank staff, clients, and partners gain from development experience; or the transfer of tacit knowledge to colleagues and clients. Nor have we looked at the transfer of explicit knowledge across the organization and externally—an area in which the Bank may have slipped in the last decade, after taking an early lead in the late 1990s. These aspects may be explored further going forward. xii Chapter 1. A Changing World of Development Knowledge Increasing number of globally connected knowledge providers 1.1 A growing and diverse group of institutions and individuals produces development knowledge and provides knowledge services. Knowledge is created and held by individuals, small groups, and large institutions, ranging from universities and governments to multilateral development agencies. Knowledge and expertise based on that knowledge are globally distributed. In today’s world, tools such as the internet, email and social media have provided that full range of actors with affordable pathways for supply-side dissemination of knowledge, and have opened up new ways of providing demand-driven access to knowledge. These developments are leading to the democratization of knowledge and are energizing a movement for open knowledge. 1.2 The democratization of knowledge through the new forms of media has led to the demand for even more access to knowledge, and to a growing number of institutions and individual experts focused on knowledge production and dissemination, or on knowledge services, as a core business activity. Knowledge is created, stored, and used to inform decision-making by universities, governments, and development agencies; as well as by the private sector, NGOs, citizens groups, advocacy organizations, and individual researchers. The global distribution of those knowledge products and services means that the World Bank is one of many institutions in the global knowledge marketplace. 1.3 The private sector is playing an increasingly important role in creating knowledge to guide public policy. The McKinsey Global Institute defines its mission as helping “leaders in the commercial, public, and social sectors develop a deeper understanding of the evolution of the global economy and to provide a fact base that contributes to decision making on critical management and policy is sues.� The Economist Intelligence Unit bills itself as “the world's leading resource for economic and business research, forecasting and analysis.� 1.4 Non-governmental organizations are also actively creating and disseminating knowledge. The Bertelsmann Foundation helps “policymakers, government and society find solutions for the future�—its “Megatrend Meta-analysis� information management project is one example. The International Growth Centre, based at the London School of Economics (LSE) and Oxford University, and funded by DFID, has about 100 research projects underway on growth-related issues. Its research supports in-country engagements with decisionmakers and informs policymaking. The web site of Foreign Policy magazine notes 228 nominees for “global go-to� think tanks with specializations in politics, economics, health, and other areas.9 1.5 Civil society organizations, which often operate on a limited budget, have found economical ways to share their knowledge by forming information networks—i.e., “loose arrangements with or without a formal structure…which are primarily focused on helping members to communicate and share 9 http://blog.foreignpolicy.com/posts/2008/01/09/the_worlds_top_think_tanks 1 reliable research and up-to-date information.�10 There are also easy channels for individual experts and development practitioners to communicate views and information that have a policy impact, and for practitioners in different parts of the world to create virtual communities of practice. Such communities have clustered collective knowledge in hubs—and these hubs are slowly being connected with each other. Increasingly sophisticated clients 1.6 The Bank’s clients have become more knowledgeable and more specific in their requests for the Bank’s expertise. Client country counterparts have often attended top schools, earned advanced degrees, and now demand top-quality knowledge services. Clients often ask Bank teams for information on how other countries have handled particular development issues, and for specific examples and advice in terms of implementing reforms. For example, one middle-income country (MIC) that wanted to establish a national planning commission asked the Bank to quickly summarize other countries’ experience with similar commissions, and to identify the lessons from these practical examples. This demand for knowledge is not limited to MICs—across the income spectrum, government officials and those advising them are asking for practical and specific implementation experiences, and not just general policy advice. 1.7 Rather than pre-formulated prescriptions and standardized policy advice, clients expect customized, integrated solutions to their problems. Such solutions require Bank staff to have and make available a combination of data and operational expertise, including in-depth operationally relevant knowledge on particular issues (which could range from sugar subsidies to pension reform); relevant experiences from other countries facing similar problems (South-South connections are frequently of interest); and local expertise (often involving experts outside the Bank); and which often involves identifying and working in partnership with local experts. 1.8 In the creation and dissemination of knowledge, the Bank is one institution among a growing number of global and local consulting firms that can provide global expertise quickly, on demand; and many governments have the financial ability to hire such firms. Front-line managers increasingly report that clients ask about the education, qualifications, and experience of World Bank staff. Client country counterparts increasingly have the same level of training and education as Bank staff. The World Bank imprimatur no longer confers automatic authority, and the Bank is seldom the only source of information. 1.9 Finally, although there was a surge in World Bank loan origination during the recent financial crisis, in the longer-term we expect the Bank’s role as a provider of finance to become relatively smaller, even though it may grow in absolute terms. With Bank lending accounting for a smaller share of transfers to developing countries, the Bank’s opportunities for knowledge transfer through lending will also shrink. This challenge will likely become more important as a significant number of countries become eligible for graduation during the coming decade.11 In all, the Bank’s ability to influence policy and the general course of development through finance alone is also likely to become more limited. The figures help tell the story. Net funds flowing from developed to developing countries totaled about $600 billion per year 10 Sharing Global Governance: The Role of Civil Society Organizations, Bertelsmann Stiftung, Spring 2011. 11 Crossing the Threshold: An Empirical Review of the IBRD Graduation Threshold. September 2010. The World Bank. 2 in 2009.12 Official development assistance (multilateral and bilateral ODA) -– was approximately $135 billion in 2009.The World Bank accounted for less than $20 billion of this. The net flows from the World Bank were less than half that amount before the crisis and sometimes were negative over the years 2001- 2008. Given the above trends that show the Bank’s declining role in finance, knowledge has the ability to have a significant impact–greater attention to the allocation of resources can deliver the essential data, the critical analysis, the new perspective that leverages a great deal of investment. Knowledge work has the potential to achieve great impact with relatively modest expenditure. The Bank is able to leverage its small share of ODA annually through the customization of existing work, convening and collaborating with others. Spreading and democratizing technology 1.10 Information, communication, and technology have transformed both the supply and demand sides of knowledge. The Bank’s experts, other knowledge-providing firms and institutions, civil society actors, and clients all have access to the internet, to ever more powerful search engines, and to email contact with global experts. The impact of the worldwide web on knowledge work cannot be overstated. Anyone, anywhere in the world, can find geo-referenced data, careful analyses, reports, and expertise through general purpose search engines such as Google. A new class of answer engines such as Wolfram Alpha, which make it possible to access expert-level knowledge on a wide range of issues, will transform the environment still further. The supply side has been similarly transformed, with scholarly papers, blogs, and ideas in all forms being fed into social networks and online services, as well as into purpose-built development websites and portals. Online collaboration tools allow experts and practitioners around the world to work together on knowledge activities ranging from co-authoring articles to designing best- practice solutions to development challenges to formulating real-time responses to natural disasters in remote locations. Online interactions can also provide policymakers with real-time knowledge about the service delivery and governance concerns of constituents, and make it possible for them to respond quickly to those concerns. 1.11 The Bank began to use information technology to produce, disseminate, and manage knowledge in the mid-1990s, and by 2003 had put in place three major knowledge platforms—the Development Gateway, the Global Development Learning Network (GDLN), and the Global Development Network (GDN).13 With advances in information technology and connectivity and more systematic collection of information, the Bank’s knowledge continues to become easier to access, fostering greater interaction among staff and with clients and collaborative exchanges across countries. These changes in knowledge management technology have been profound and paradigm shifting. However, internal evaluations have also identified persistent shortcomings in the Bank’s knowledge management practices, including weak linkages to operations and inadequate oversight and incentives.14 12 Global Development Finance 2011 13 Sharing Knowledge: Innovations and Remaining Challenges. An OED Evaluation 2003. Operations Evaluation Department. The World Bank. 14 Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006 (Independent Evaluation Group-IEG 2008). . See also The Matrix 3 1.12 In considering its opportunities for investing in the next generation of knowledge management, the Bank could learn much from private sector experience about how to improve the efficiency and responsiveness of its knowledge services. Significant investment in both human resources and technology will be required to meet the standard that some of these firms have set. One lesson is that it is relatively easy to keep pace with new technologies as they are developed. What is more difficult—and more critical for the Bank’s evolving role as a knowledge institution—is the strategic management of information and knowledge in the context of new technological possibilities. This will involve careful consideration of options not only for knowledge systems, but also for knowledge governance and incentives for knowledge work. Changes in the nature of development knowledge 1.13 The changing nature of development work has required corresponding changes in development knowledge. When the World Bank was founded, its mission was to serve as a source of infrastructure investment capital. The focus was on providing loans for reconstruction and development in Europe. There was limited knowledge content in those early loans, beyond the fiscal discipline and engineering expertise embedded in structuring the loan and using the proceeds to build infrastructure. 1.14 As European reconstruction was completed, the Bank became increasingly involved in financing projects in the developing world. Loans expanded from infrastructure into social and human services, and the Bank’s knowledge work became more important. In the 1950s and 1960s, governments in developing countries saw their role as providing public goods (bridges, roads, ports), addressing externalities (e.g., protecting “infant industries�), and sometimes as redistributing income to the poor (e.g., by keeping food prices low). Development knowledge in this context still largely consisted of advice by engineers and financial analysts on building and designing the right infrastructure, although sectoral expertise began to grow as the Bank ventured beyond infrastructure and into other development sectors. 1.15 By the 1970s and 1980s, the Bank was hiring economists, public policy experts, sector experts, and social scientists to address increasingly complex development challenges—costly infrastructure was not being maintained; protected industries were often inefficient; low food prices led to shortages and increased poverty. Development knowledge during this period began to focus on policy solutions such as “structural adjustment� to improve efficiency, reduce fiscal imbalances , and remove market obstructions (e.g., through privatization and reduction of trade barriers), based on the belief that well-functioning markets could deliver efficient outcomes. Much of the economic and sector work during this period was concerned with estimating the costs of market failure and the anticipated benefits of recommended policies to address these failures. The policy advice emerging from this theoretical work—which was promoted not only by the Bank, but also by the International Monetary Fund (IMF) and many other development agencies—was applied globally, sometimes without careful consideration of local circumstances. Resistance by client governments was often attributed to vested interests; and even when structural adjustments were implemented, they did not always deliver the intended outcomes. System at the World Bank: An IEG Evaluation. Approach Paper December 28, 2010, for a discussion and a list of completed IEG work on knowledge management. 4 1.16 Today, after the greatest financial crisis since the Great Depression—which many developing countries weathered better than their more developed counterparts—it has become clear that consensus is not the same as wisdom; but rather, that evidence-based debate and discussion, in the context of particular challenges, offers the best hope for continued improvement in policies and institutions. It is clear from the events of the past year that open debate and discussion within very different societies can bring about real and profound change. 1.17 The Bank has an important role to play as these changes take hold, as both a producer and collaborator in the production of knowledge that is cutting edge and operationally relevant, and as an intermediary that provides access to the best development knowledge produced by other institutions. In providing this knowledge—not only to client governments but also to civil society and other actors—the Bank seeks to support active dialogue about priorities and policies in societies that are struggling toward openness. Recognizing that there are many players, and that they have access to many sources of knowledge, the Bank must consider how to contribute most effectively to development debates. 1.18 The remainder of this section will consider the ways in which the Bank has become more collaborative, responsive, and multi-faceted in its knowledge work, and what needs to be done to ensure that its knowledge work is delivering better value for client countries. The IEG 2008 study found several factors that contribute to greater impact, namely: technical quality, strong client engagement beyond one- off dissemination, and relevance of issues addressed. These elements will be discussed in the following sections with a focus on quality and relevance and how to promote needed incentives. When management or task leaders choose to do one type of knowledge work instead of another, what informs those choices, and how can the Bank ensure that this work is making the greatest contribution toward development? The World Bank’s knowledge services in this changing world 1.19 As the World Bank tries to keep pace with all of these changes (multiplying knowledge sources, increasingly sophisticated clients, technological leaps, and changes in development knowledge), it will be important to be selective. The Bank cannot be everywhere and do everything, but must think strategically about how to be most effective as one player in an increasingly large universe—how to leverage the institution’s comparative advantage through openness and effective collaboration, and how to be more effective by linking the Bank’s contributions with those other institutions. Becoming more open and collaborative 1.20 Collaboration with clients is becoming more common. The World Bank’s best-known knowledge activities, classified as either economic and sector work (ESW) or technical assistance (TA), are evolving into closer collaborations with clients. In Vietnam, for example, an influential series of Public Expenditure Reviews (PERs) was launched in 1996. For the first review, Government’s role was to provide data, while Bank staff did most of the analysis and writing. In 2002, the Government wrote some sections of the review. In 2005, the Bank and Government prepared the review jointly. A subsequent study by the World Bank’s Independent Evaluation Group (IEG) found that the 2002 and 2005 PERs had important policy impacts, helping to guide laws increasing transparency in the state budget process, leading to adoption of a medium-term expenditure framework, and to unification of the public accounts 5 system.15 Another example comes from Brazil, where in 2007 the Bank team working on early childhood development and teacher quality collaborated closely with local academics and local governments. Yet another example is the Lao PDR Development Report 2010, prepared in close collaboration and partnership with the Government and bilateral donors, which fed into the Government’s medium-term strategy. Iterative and in-depth consultations with a variety of stakeholders ensured the legitimacy and relevance of the report. 1.21 Strategic experiments in collaborative knowledge include the Bank’s Development Gateway, Global Development Learning Network (GDLN), and Global Development Network. These initiatives have helped foster greater openness and seamless knowledge flow around critical global issues. By sharing policy and research experience across countries and among development institutions, the entire development community is better positioned to address the scale and complexity of critical development challenges, including climate change, urbanization, water scarcity, the AIDS epidemic, and food shortages. The GDLN, for example, is a partnership of more than 120 recognized global institutions in 80 countries that collaborates in the design of customized learning solutions for individuals and organizations working in development. The Bank coordinates contributions and requests from affiliates as diverse as the Asian Institute of Management, the Ethiopian Civil Service College, the Islamic Development Bank, and Pontifícia Universidad Católica of Peru. GDLN provides its members with direct access to local, regional, and international development experts, as well as tailored learning, knowledge, and technical assistance programs. In major and many secondary cities, GDLN provides state-of-the-art facilities for multi-point videoconferencing and internet-based learning. 1.22 The Bank is also connecting clients with the resources of other institutions. One example is the South-South knowledge exchange being mainstreamed into the Bank’s work. The World Bank Institute (WBI) now offers a range of tools and support services, including a searchable South-South knowledge portal that includes results stories and good practice examples. Consider how Ghana's and Zambia's experiences shaped the design of the road maintenance fund in Lao PDR: In February 1999, a delegation from the Government of Lao PDR visited Ghana and Zambia to learn first-hand from their successes in using sustainable financing mechanisms for road maintenance. Through the study tour, the delegation learned that road users are willing to pay for better road services, and that users need to be represented on the road fund’s board. This knowledge exchange contributed to the Lao Government’s decision to create such a fund, which today covers the maintenance of Lao PDR's entire national road network and a portion of the local road network. 1.23 Work on global public goods also benefits from collaboration. For example, the Business Environment and Enterprise Performance Survey (BEEPS)—a knowledge collaboration between the World Bank and the European Bank for Reconstruction and Development (EBRD)—generates comparative measurements in such areas as corruption, state capture, lobbying, and the quality of the business environment. These measures can then be related to specific firm characteristics and firm performance. 15 Using Knowledge to Improve Development Effectiveness, previously cited IEG 2008 6 1.24 In addition, under its Open Data initiative, launched in April 2010, the Bank is releasing its development data free of charge to the public. As this initiative moves forward, the Bank continues to increase the amount of data available for download, and develop new applications to enable easy access to data across platforms and devices. The Bank also continues to strengthen the capacity for data collection in developing countries, to help ensure the quality and availability of the data needed to effectively address country-level issues. 1.25 This Knowledge Report itself is a significant milestone, marking the first time the Bank has attempted a comprehensive and consultative process to take stock of its knowledge activities and recommend ways to measure and increase their impact. The Bank’s three knowledge roles 1.26 The Bank’s role as a knowledge institution has evolved to include three distinct functions—(a) producing and disseminating high-quality global and country-level knowledge; (b) working with clients to customize policies and programs to meet specific challenges, based on the best available knowledge from all sources.; and (c) connecting governments, civil society, and private sector actors with others that have faced similar challenges. In the effort to find development solutions, the Bank sometimes acts more in one role, sometimes more in another:  Knowledge producer. A good example is the Bank’s Development Economics (DEC) arm, which provides intellectual leadership and analytical services to the Bank and the development community. DEC has three main business lines: development research (research and knowledge creation), development prospects (global economic monitoring and projections); and development data (international statistics, statistical capacity building, and results monitoring).  Knowledge customizer. The Bank’s task teams and Country Offices interact directly with clients, and organize and apply knowledge from a wide range of sources to help countries address their particular development challenges. Working closely with clients, these teams draw on research, data, expertise, and experience to develop policy options, inform lending, and deliver technical assistance. Customizing requires combining a global perspective with deep local knowledge to support countries in their struggle toward a better life for their people. The Regional units and Network vice- presidencies are most engaged in this time of work. In addition, the Networks customize the results of research in a way that is easily applied to operational work and engagement of client countries.  Knowledge connector. The Bank’s engagement with global partnerships and with South-South exchanges highlights its role as connector. The role of connector or convener can also involve drawing from research, data, and the expertise of different stakeholders (among client countries and development partners). This is the primary function of the World Bank Institute (WBI); but is also a role undertaken by other units of the Bank. 1.27 Because of its scale and diversity, the Bank has the ability to play any or all of these roles, as needed, to address a particular development challenge. This flexibility and scope lies at the core of the Bank’s comparative advantage. At the same time, the Bank recognizes that other institutions can also play these roles, and in some contexts, can contribute more than the Bank. It is therefore important to look 7 carefully at when the Bank should play a particular role and when another institution could make a more relevant contribution. We may make a bigger contribution by doing relatively fewer things exceptionally well. The Bank’s knowledge ecosystem 1.28 The Bank’s ability to provide effective, customized advice to clients rests on a broad foundation of complementary and interrelated knowledge activities. The Bank’s development of appropriate, integrated, and customized solutions (figure 1.1) is based on often extensive supporting knowledge activities, including data collection, which deepen understanding of a particular development challenge, the global context in which it exists, and the capacities of local counterparts. The Bank’s ability to meet a client’s need for “just-in-time� advice (which could be in the form of a PowerPoint presentation or meeting of a senior Bank staff with a Minister), for example, depends crucially on staff having done different types of in-depth knowledge work on a particular country and a particular issue, often over a number of years. What the client sees is just the tip of a complex interaction between many sources and forms of information. The Bank may rely on its various roles as producer, customizer and convener or connector in the process. Staff may draw from original analytic work, plus learning from operations, as well as knowledge derived from external sources and partnerships. Underlying this ecosystem is the interaction of experts in the related areas across the Bank –whether in the Networks, or across Regions. The team may also draw from non-Bank suppliers and partners. All of this interacts to produce the “just- in-time� advice with the whole system grounded in the information technology systems and the stock of human resources or expertise. Figure 1.1. Ecosystem of inter-linked knowledge activities Source: Adapted from Towards a Conceptual Framework for the Knowledge Bank by Laura Chioda, Augusto De la Torre, and William F. Maloney (background paper for 2011 Knowledge Report). 1.29 Although the Bank is becoming more open and collaborative and playing a variety of roles in the creation and dissemination of knowledge, the paradox is that management does not know which knowledge activities are having the most impact, and often does not make knowledge an institutional priority. On the one hand, client surveys consistently show that the Bank’s knowledge is highly valued 8 and appreciated, and is seen as a comparative advantage. In the 2010 Country Survey, for example, clients cited knowledge as the Bank’s second most valuable contribution, after its financial resources. When technical, policy, and economic advice is included, knowledge work is cited as the Bank’s most valuable contribution more than twice as often as financial resources.16 On the other hand, the Independent Evaluation Group (IEG 2008) and Quality Assurance Group (QAG) have concluded that the Bank’s knowledge work is not having the impact it should have.17 A 2008 internet-based survey of leaders of thematic groups or communities of practice in the Bank indicated that limited staff time, low management support and budget were the key organizational and cultural factors that constrained the operations of these groups and the flow of knowledge. Similar results have been reported in the course of interviews conducted as part of the background work for the design of the pilot restructuring of the Financial and Private Sector Development Network. Results from the interviews indicated that there was little incentive for knowledge work, and that this work is undervalued.18 Despite the growing importance of knowledge work, the primary institutional incentives in the Bank are still related to lending. 1.30 There are many reasons why knowledge work has not developed into an institutional priority, despite the fact that President James Wolfensohn articulated the concept of the Knowledge Bank and knowledge networks back in 1996. One reason is that knowledge work must compete for scarce financial and management resources, and there is a lack of robust and systematic evidence that resources spent on knowledge work bring adequate returns. Lending, on the other hand, has built-in metrics—lending volumes and rates of return on investment are easily understood and communicated. 1.31 Beginning with this Knowledge Report, the Bank is attempting to understand the challenges inherent in this knowledge paradox and suggest ways to move forward. Chapter 2 takes stock of the Bank’s rich variety of knowledge activities, where and when they are carried out, and how they are financed. This basic stocktaking is fundamental to any discussion about strategic management of the knowledge portfolio, which is necessary to increase the impact of the Bank’s knowledge work. Chapter 3 discusses two factors that the Bank will need to address to ensure this impact: (a) systematic measurement of the results of all knowledge work; and (b) cohesive management, so that the various knowledge elements work together to produce results that have more impact. Chapter 4 summarizes the key recommendations of this Knowledge Report and reflect on topics that might be covered in future reports. 16 Country Survey, 2010 Review, The World Bank 17 See footnote number 3 for the IEG 2008 report and Quality of Country AAA (Phases I and II): A QAG Assessment (QAG 2008). 18 Citation 9 Chapter 2. The State of the World Bank’s Knowledge Services 2.1 The aim of this chapter is to shine a light on the Bank’s knowledge business—to describe the different types of knowledge services as clearly as possible. The boundaries of what is called “knowledge� are inevitably fuzzy. For the purposes of this stocktaking, the definition includes practical and theoretical knowledge related to development issues. 2.2 As mentioned in Chapter 1, the nature of the Bank’s development knowledge has evolved. It has broadened from knowledge incidental to lending to include data on global development issues and broad engagement with societies to provide reliable knowledge in support of development debates. The Bank’s knowledge services 2.3 As described in Chapter 1, the Bank plays various roles and has evolved a complex system of interrelated knowledge activities. Knowledge activities exist not only as stand alone services; they are also integrated with the Bank’s operations and underpin its lending activities. Through the work of the Knowledge Strategy Group (KSG)19 and the subsequent Bank-wide effort in preparing of this report, the extent of the Bank’s knowledge activities is becoming clearer. Unfortunately, the Bank currently has only one lens through which to view its knowledge work—the production information system and the defined knowledge “products� it captures. The management information system the Bank uses to manage knowledge products has its roots in enterprise software developed to manage industrial processes, and is built on the notion of individual knowledge products. This Knowledge Report deals mostly with those defined knowledge products and how they are produced. But we know that this perspective presents an incomplete picture of the Bank’s knowledge activities, and that even as we speak about “knowledge products� and “product lines,� it is important to situate the discussion in a broader universe of World Bank-sponsored knowledge activities:  The World Bank has identified nine “core knowledge products,� which represent about $600 million in annual expenditure.20 This number is known with some precision, as it applies to the productization of knowledge undertaken from 2003 onwards. This includes the best-known Bank knowledge products and services, such as economic and sector work, technical assistance, and the World Development Report, as well as external client training, impact evaluations, global monitoring, and new product development. These core products are discussed in some detail below.21  Other “non-core� knowledge products represent an additional $300 million in annual expenditure. This is an estimate, and there is some debate over whether some of these products should be 19 In November 2008, President Zoellick launched a knowledge renewal effort, and the Bank’s Senior Management supported effort by setting up this small strategy group reporting to Managing Director Graeme Wheeler to guide the Bank’s knowledge reform agenda. In July 2010, the KSG was roll ed into the Internal Reform Secretariat. 20 Six hundred and six million in fiscal year 2010. 21 See Part 2 of this report for a more in-depth discussion of the Bank’s knowledge products and services. 10 reclassified as core knowledge products. This category includes, for example, evaluations by the Independent Evaluation Group (IEG), country assistance strategies, and sector strategies. These have substantial knowledge content, but some argue that their primary purpose is not the production or dissemination of knowledge.  The Bank’s partnerships with other development institutions, and multilateral arrangements of various kinds, account for about $1.2 billion per annum. About half of the work of these partnerships, or around $600 million per annum, is knowledge activity of generally the same kind as the World Bank’s “core� knowledge products. Of this expenditure, $200 to $300 million is captured by the Bank’s accounting system, and an estimated $300 to 400 million more is spent by the partnerships outside of the Bank’s accounting system.  The Middle East and North Africa region (MENA) systematically reviewed the knowledge embedded in operational documents and estimated that in 2010 the region spent about 10 percent of loan proceeds on knowledge products, some of which were very much like the Bank’s core knowledge products. If similar percentages hold for other regions—and there is no reason to believe otherwise—then knowledge spending embedded in lending was an estimated $2.5 billion in 2010 2.4 The total expenditure on explicit knowledge services in 2010—by the Bank, by partnerships in which the Bank is involved, and from clients’ loan proceeds—was an estimated $4 billion (box 2.1). No attempt has been made to put a value on the tacit knowledge of staff, who have a rich diversity of backgrounds and development experience, nor that of the Bank’s partners and clients. This is knowledge that is often transferred through direct individual interaction. For now, it is sufficient to mention that this accumulated knowledge is a resource that deserves more attention in future Knowledge Reports. There are interesting discussions within the Bank about ways to capture and codify this tacit knowledge; and about challenges in designing mechanisms to ensure that it stays alive and current, and is transmitted and developed from year to year and from staff member to staff member —e.g., through mentoring, debriefing, and peer learning events. These issues will be discussed in depth in future editions of the Knowledge Report. This first report focuses on the explicit knowledge products described above—the Bank’s core knowledge products. Box 2.1. 2010 Estimate of Spending on Knowledge Core Knowledge $606 million Other bank Knowledge $300 million Partnerships $600 million Embedded in new lending $2 500 million Total Estimate $4 billion Tacit knowledge Not quantified Source: Authors’ estimates based on The World Bank’s Involvement in Global and Regional Partnership Programs: An Independent Assessment, Independent Evaluation Group, 2011; and the Bank’s management information system (Business Warehouse). 11 . Box 2.2. An evolving focus on knowledge: 15 years of the Knowledge Bank 1996 President James Wolfensohn articulated the concept of the Knowledge Bank and knowledge networks. The Bank’s "networks" initiative began with the Human Development Network (HDN), and soon grew to include three others: Environmentally and Socially Sustainable Development (ESSD); Finance, Private Sector and Infrastructure (FPSI); and Poverty Reduction and Economic Management (PREM). The networks were tasked with addressing emerging development issues, and ensuring the flow of knowledge throughout the institution to support the work of staff. 2000 The World Bank’s Development Committee report, “Poverty Reduction and Global Public Goods: Issues for the World Bank in Supporting Global Collective Action,� cited the sharing of development knowledge as one of five key areas to focus on as the Bank increases its emphasis on the production of Global Public Goods, and recommended that the Bank’s knowledge role be extended beyond country clients. 2003 The Bank’s management information systems (MIS) began incorporating knowledge “products� with deliverables and predefined milestones, with monitoring for most knowledge products similar to that being done for lending products. 2004 The Organizational Effectiveness Task Force (OETF) noted that the Bank was not capturing the full potential of being a global knowledge-based organization, and recommended building networks to leverage global knowledge and improve the quality of client services. 2006 An external panel evaluated the Bank's research and found that while some work was outstanding and innovative, many research projects had "neither great relevance to policy nor claim to academic distinction,� and there was an excessive number of "flagship" reports. 2008 The Independent Evaluation Group published a report on “Using Knowledge to Improve Development Effectiveness: an Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000-2006.� The report found that the Bank’s reports and TA were preferred to those from other institutions, and that TA and short reports are generally preferred by country clients to longer pieces of ESW. But the results tracking system for knowledge work within the Bank was found to have serious weaknesses, and more follow-up with clients after delivery of products would help strengthen impact. 2009 The Knowledge Strategy Group (KSG) was formed and soon identified nine product lines as the core of the Bank's knowledge business. The KSG divided these product lines into three major categories: (a) knowledge for clients; (b) knowledge as a public good; and (c) knowledge for internal use. 2010 The Open Knowledge initiative and Open Data policy are launched. Under the Open Data policy, the Bank has made many of its databases publicly available free of charge, including the World Development Indicators database. These data are available under an open license for use and re-use. 2011 First World Bank Knowledge Report. This report marks the first time that the Bank has undertaken a public and systematic review of its knowledge services, setting out concrete proposals for measuring results, and considering the way the Bank’s knowledge work is managed. The preparation of this report, which involved extensive internal and external consultations, lays the foundation for a series of knowledge reports that will allow us to continue these conversations and provide information to support a debate about how development knowledge can most effectively be generated, shared, and evaluated. *Banerjee, Abhijit, Angus Deaton, Nora Lustig, and Ken Rogoff. 2006. An Evaluation of World Bank Research 1998-2005. 12 The core knowledge products 2.5 The 2009 KSG report22 classified nine types of World Bank knowledge activities as core knowledge products, and broke them down into three categories of activity (box 3):  knowledge for external clients  global public goods  internal knowledge. 2.6 When a core knowledge activity is planned, the task leader will assign the work to one of these nine product Box 2.3. Core knowledge products types. While some types reflect real differences, there is also Knowledge for external clients considerable overlap—for example, an outside observer might have a hard time distinguishing between some of the  Economic and sector work technical assistance activities and some external client  Technical assistance training activities.  External client training  Impact evaluation 2.7 This core knowledge is produced all over the Bank—within Regional Vice Presidencies (Regions) and Global public goods Network Vice Presidencies (Networks), as well as by the Development Economics Vice Presidency (DEC) and the  World Development Report World Bank Institute.  Research  Global monitoring 2.8 There is significant specialization in the types of Internal knowledge activities undertaken by these different areas of the Bank (figure 2.1). The Regions, which collectively account for  Internal knowledge products more than half of core knowledge spending, work mainly on  New product development knowledge for clients. The Networks, which together spend just under a quarter of the Bank’s core knowledge budget, divide their spending about equally between knowledge for clients and knowledge for internal use. WBI, which accounts for 9 percent of core knowledge spending (see Annex A), does mostly training for external clients. Finally, DEC spends about 10 percent of the total, and works mostly on global public goods, research, and monitoring. Other central units account for the remainder, such as Operational Policies and Country Services (OPCS), Legal, and Concessional Finance. 22 Transforming the Bank’s Knowledge Agenda: A Framework for Action. February 2010. The World Bank. 13 Figure 2.1. Distribution of core knowledge activities in fiscal year 2010 (Bank budget and trust funds) 24 25 Knowledge for 1 Clients $444 1 3 1 6 Knowledge as 5 Publ Good $32 1 1 8 1 Knowledge for Internal Use $99 1 43%* 7 5 2 1 0 Total $606 Mill. * Includes 2% of core knowledge from other VPUs Note: WBI data includes scholarships as part of trust funds. Source: Business Warehouse. Category 1: Knowledge for external clients 2.9 Knowledge services for external clients accounted for some $444 million in fiscal 2010, or almost three quarters of the Bank’s core knowledge spending. Knowledge for clients can include work done for specific countries and also work done at a regional or even global level. The mix of products varies only slightly across the Bank’s operating regions, but there are notable differences from region to region in the percentage of spending represented by trust funds. 2.10 Product categories and spending by category are discussed in detail in the Background Report that accompanies this report. This synthesis report focuses on characterizing the work, and reflecting on factors that tend to make knowledge services for clients especially effective—length of engagement, involvement and ownership of country counterparts, soundness of technical work, and in some cases, the additionality of lending. Economic and sector work 2.11 ESW is analytical work meant to inform and influence policy choices. It can include studies on agriculture, energy, institutions, poverty, or climate change, to name but a few recent examples. During the Bank’s 2010 fiscal year, there were 1,626 active ESW tasks. Most of this work is done by country or sector teams within the Bank’s Regions. 2.12 For many years, economic and sector work accounted for the biggest share of the Bank’s core knowledge services, but in 2010, the Bank for the first time spent more on technical assistance (box 2.4). 14 There is some concern that this trend toward less economic and sector work will impair our clients ability to rely on solid evidence to inform their policy choices. In many cases, the Bank’s ability to offer high- quality technical assistance depends on a sound analytical foundation. Without accurate diagnosis, it can be difficult to offer credible advice. 2.13 The number of ESW deliveries has also dropped significantly, from nearly 700 in fiscal 2005 to fewer than 500 in fiscal 2010. Most of the Bank’s ESW is country focused. From fiscal years 2002 through 2010, the Bank’s top five ESW clients were India, with 260 tasks totaling $60 million; China, with 192 tasks totaling $49 million; Indonesia, with 149 tasks totaling $36 million; Pakistan, with 96 tasks totaling $30 million; and Brazil, with 138 tasks totaling $25 million. Overall, however, the share of country-focused ESW is declining relative to regional and global ESW. Box 2.4. Trends in ESW Analytical work at the Bank has been declining relative to practical support for policy implementation —i.e., technical assistance. Part of this trend reflects a real change in client demand—many Country Directors report an increasing need for quick practical notes and implementation support, and decreasing interest in scholarly, thoroughly documented analytics. But it is not clear how reliable the data are, both because the dividing line between ESW and TA can be fuzzy, and because operational team leaders, responding to client demand, often deliver different kinds of knowledge support under a single product code. The fact is that the Bank’s work with clients is evolving and often includes hybrid activities that do not match existing product types. Another trend is that spending on essential ESW diagnostic work, such as Country Economic Memoranda, Country Financial Accountability Assessments, Country Procurement Assessment Reports, Public Expenditure Reviews, Poverty Assessments, has been declining since 2006, in both absolute and relative terms. The main cause of this arises from a decision then taken to ensure that countries are fully covered by these five core economic and sector diagnostic work.* A Country Director determines the frequency and programming of this core diagnostic work. This determination is tied to the type and level of the Bank’s engagement and the country’s priorities and circumstances, including consideration of relevant knowledge work from partners (which may obviate the need for Bank work). This declining trend in ESW therefore reflects closer collaboration with partners, drawing on their work. * This core diagnostic work is a subset of ESW. It should not be confused with core knowledge products, one type of which is ESW. 2.14 The 2008 report by the Bank’s Independent Evaluation Group, (see footnote 3), which reports to the Board of Executive Directors, pointed to Public Expenditure Reviews as one type of ESW that has been influential both in client countries and in shaping Bank strategies and lending. Effective collaboration between Bank and client can improve the quality and increase the impact of economic and sector work. An example of influential ESW is the Africa Infrastructure Country Diagnostic exercise, which was an unprecedented effort to document public expenditure and sector performance in each of the main infrastructure sectors, culminating with the 2010 publication of the flagship report “Africa’s 15 Infrastructure: A Time for Transformation� along with a series of comp anion pieces for various audiences. This activity has now become the reference point on Africa’s infrastructure challenges, providing an empirical basis for prioritizing investments and designing policy reforms. 2.15 Long-term, broad-based knowledge engagements can be especially fruitful in supporting policy change. Consider the Bank’s work with Indonesia on poverty analysis. This was a three -year effort (2003-2006), costing nearly a million dollars of Bank funds, which leveraged more than twice that amount in donor (DFID) trust funds. The objective was to consider ways to help the Bank and Government to collaboratively attack the problems of poverty as the central development challenge. This required analysis, dialogue about the issues, and the development of a comprehensive poverty strategy that focused on making growth work for the poor, making public expenditure work for the poor, making services work for the poor—in essence, making government work for the poor. The program’s success was due in part to its cross-sectoral mandate, which involved counterparts from seven different ministries, as well as from local government and other agencies. The entire activity was characterized as a single piece of economic and sector work. 2.16 Quality assurance for economic and sector work is provided through peer reviews and sector management, with mixed results. Peer reviews are conducted at the concept stage, when approval for a new or soon to be launched knowledge service is sought from management, and at the delivery stage, when a draft report has been completed. Peer reviewers are proposed by the task team, usually the team leader with approval of the relevant manager. A task team is also asked to assess the likely impact of its own economic and sector work as part of an activity completion summary, which is required but not always done. There is no systematic client feedback recorded at the Bank23, and no third party assessment of impact, either at completion or subsequently. In addition to assuming responsibility for the product, management ensures that there has been compliance with the requirements for the key production control points or processes. 2.17 In 2008, the Independent Evaluation Group, which reports to the Board of Executive Directors, did a retrospective study which recommended, inter alia, greater selectivity in economic and sector work, and more serious attention to tracking results.24 2.18 In 2009, the Quality Assurance Group (QAG), found that in the overwhelming majority of cases, ESW tasks were appropriately focused on issues of key importance to clients.25 However, there was less attention on other key factors (i.e. client participation, dialogue, and dissemination) that determine impact. In addition, the QAG report noted that managerial attention to quality is concentrated on the front end, at the conceptualization or design stage, and tends to drop off during implementation. QAG also observed that incentives for Bank staff emphasize the production and delivery of economic and sector work, as opposed to its impact. Technical assistance 23 One exception is the Financial Sector Assessment Program, jointly conducted by the Bank and the IMF, which conducts follow-up surveys with all client countries; see IEG 2008. 24 Using Knowledge to Improve Development Effectiveness, IEG 2008. 25 Synthesis of Past Assessments of Analytical and Advisory Activities (AAA): a QAG Review , QAG, 2009 16 2.19 Technical assistance supports clients’ efforts to implement reforms and strengthen institutions. A good example is the work done in Mauritius on “Aid for Trade� in 2006. The Government of Mauritius asked for help to define its trade reform program and to mobilize funds as part of a multi-donor initiative. Within two months of the request, a Bank team provided analysis of possible reforms, including estimates of adjustment costs of lowering trade barriers, tax reforms, and strategies to lower the cost of key services. At the end of a two-week Bank mission, the team presented the Government with a summary of the main reform options. Among the important factors underlying the effectiveness of the TA were the timely nature of the support and the ability to build on existing economic and sector work. After extensive internal debate, the Government adopted many of the team’s recommendations in its reform p rogram, which was eventually supported by a Bank loan. More recently, a programmatic technical assistance activity is helping Turkey to modernize its agri-food supply chains in preparation for EU accession. 2.20 Some technical assistance tasks are subject to peer review, generally by Bank staff, at initiation. As with ESW, the task team is asked to assess the likely impact of the task as part of an activity completion summary, but this is not always done. The 2008 Independent Evaluation Group study included technical assistance and found that clients preferred technical assistance to ESW in both IBRD and IDA countries.26 The 2009 Quality Assurance Group report also rated technical assistance tasks higher than ESW on quality, inter alia because objectives were more clearly stated, because of greater client involvement and coordination with bilateral donors, and because of better dissemination arrangements. 2.21 Surveys show that the Bank’s clients place a high value on technical assistance, and because the nature of the work requires client engagement, collaboration is a natural part of TA tasks. A good example of such collaborative work is the ECA Region’s PEMPAL (Public Expenditure Management Peer Assisted Learning) program, designed to enhance domestic capacity in public expenditure and financial management, to play a catalytic role in scaling up aid, and to strengthen institutions and policies by establishing a forum for peer-to-peer exchange. One key to its success was the involvement of many local researchers and thought leaders. External client training 2.22 External client training is a category of knowledge product that is used mostly by the World Bank Institute (WBI). In practice, this category encompasses a broad range of activities, from events that focus on awareness raising or skills enhancement to design of curricula and learning materials. As WBI has evolved and refined its mission, this category of work has also evolved. In 2008, WBI began implementing a programmatic approach to external training, bundling individual tasks into a smaller number of larger initiatives. This has resulted in a sharp drop in the number of tasks, from 1,677 in fiscal 2005 to just 292 tasks in FY10. Total expenditure declined at a slower rate, from almost $72 million to about $55 million over the same period. WBI has also expanded beyond the traditional focus on technical training of individuals from client countries, to new approaches such as leadership and coalition building to help leaders navigate the political economy of reforms. WBI also works to build the capacity of both 26 Using Knowledge to Improve Development Effectiveness, An Evaluation of World Bank Economic Assistance, Independent Evaluation Group, World Bank, 2008. 17 government and non-governmental actors (parliaments, civil society, private sector, youth), and helps convene platforms for this diverse set of actors to coalesce around policy and institutional reforms. 2.23 WBI has devoted significant attention to measuring results, and has developed a Capacity Development and Results Framework that facilitates this approach. This results framework was launched in the April 2009 WBI strategy, which included the CDRF as its results framework. The framework links capacity development initiatives to achievement of the sustainable institutional changes required for development results. It allows for measurement of achievements at the level of intermediate capacity outcomes (awareness, skills, coalitions, networks, know-how), while promoting flexibility and innovation in the change process. WBI is making significant investments to apply this strategic results-focused approach in its programs, and to support learning about what works, and why, in capacity development. Neither the Quality Assurance Group nor the Independent Evaluation group have done a retrospective evaluation of work in the external client training category. Impact evaluation 2.24 Impact evaluation is the newest category of knowledge product. It was established in 2006 by the Chief Economist’s Office, as part of the Development Impact Evaluation (DIME) initiative. An impact evaluation is empirical work to measure the effect of a development intervention on outcomes. By definition, this category of knowledge work measures outcomes relative to a well-specified counterfactual. It is aimed at informing an external client, and leads to a freestanding report even when it is linked to the supervision of a Bank-financed project or program, or to the preparation of a new one. 2.25 One example of this kind of work comes from Madagascar, where the Government is working to improve school management. With support from a Bank team, a group of educators developed a new set of management tools that was validated and owned by practitioners. The Government wanted to test the efficacy of the new tools before scaling them up countrywide, and so decided to undertake a rigorous impact evaluation. The Bank team facilitated the Malagasy team’s participation in training workshops and engagement with external experts to conceptualize the design of the impact evaluation, which the country team then implemented. When the evaluation found that the new tools were effective, Government officials made the decision to disseminate them more widely. More generally, the impact evaluation gave Malagasy policymakers an appreciation of the benefits of impact evaluation, and the same team subsequently launched another impact evaluation focused on school feeding. 2.26 In 2010, the Bank spent about $14 million on impact evaluation work, of which $11.5 million was specifically associated with the DIME initiative. There have been 162 impact evaluation tasks between 2006 and 2010, the unit cost of which varied from less than $50,000 to almost $500,000. Impact evaluation, more than other knowledge activities, depends on trust funds.27 There has not yet been an evaluation of the Bank’s impact evaluation work, but the Independent Evaluation Group is planning to do one in the near future. 27 Development Impact Evaluation Initiative: A World Bank-Wide Strategic Approach to Enhance Developmental Effectiveness. June 2010. Discussed at informal CODE meeting on July 14, 2010. 18 Category 2: Global public goods 2.27 The goal of the Bank’s global public goods work is to support debate, discussion, and policymaking on issues of global significance. While all of the Bank’s work contributes to the public good, the term, as used here, refers to work that has no specified external or internal client. This is knowledge that would not necessarily be demanded by countries and Country Directors, but which ultimately benefits the cause of development worldwide. Three types of knowledge services are explicitly categorized as global public goods:  the annual World Development Report (WDR), which consolidates development knowledge on a particular topic, to stimulate debate on new directions for policy;  research, which is original analysis using rigorous methods to advance knowledge or stimulate debate; and  global monitoring, which provides data to enable policymakers and advocacy groups to make better-informed decisions. 2.28 Taken together, these three types of knowledge activity accounted for approximately $62 million (or 10 percent) of core knowledge expenditure in fiscal 2010. World Development Report 2.29 The World Development Report is a substantial annual effort which provides an in-depth discussion and analysis of a specific aspect of development issues based on existing research, and represents state-of-the-art knowledge in that area. The 2010 WDR focused on Development and Climate Change. The 2011 WDR focuses on Conflict, Security and Development. WDRs bring together research, case studies, and the results of extensive consultations with leaders and development practitioners throughout the world. The publication of each year’s WDR is a major event in the development community, and has a significant influence on the debate going forward. The WDR is one of the few pieces of knowledge work scheduled for discussion by the Bank’s Board of Executive Directors. Research 2.30 Research is carried out primarily by the Development Economics (DEC) arm of the Bank. DEC is the source of about three fourths of the Bank’s research, and research i s the greatest share of what DEC does. Unlike work for clients, research products do not necessarily represent Bank views (work is owned by the authors, not the unit), and a disclaimer to this effect is included in research publications. The DEC Research Group focuses on four main activities:  applied, policy-oriented, research on pressing development issues;  influencing policy debates on development, both globally and in specific countries;  producing data geared to addressing the most important development issues, and connecting data producers with data users; and  creating robust operational tools to facilitate analysis by practitioners in developing countries. 19 2.31 Most research products undergo a more rigorous peer review than is the practice for economic and sector work. For example, DEC requires that the task team leader not be involved in the selection of reviewers or in the flow of documentation. Some research tasks are selected competitively, and citation in external peer-reviewed journals is encouraged as evidence of development impact. Global monitoring 2.32 Global monitoring products are often open ended, unlike other knowledge products that are expected to have defined starting and closing points. For example, the Living Standards Measurement (LSM) survey series has been in use since the 1980s. The World Development Indicators (WDI), the Purchasing Power Index database, the International Household Survey Network, and the Bank’s poverty mapping databases are all part of this category of global public goods. Some knowledge work that the Bank treats as economic and sector work is conceptually similar to global monitoring products. For example the Bank’s Doing Business 2011 report is the most downloaded report from World Bank websites so far this year; it has had 2.4 million visits, 6.6 million page views, and 309,000 total files downloaded. 2.33 As a part of its global monitoring, the Bank combines expertise in regional issues, market trends (e.g., world trade, financial conditions) and forecasting methods to construct a consistent view of global economic prospects and the implications for macroeconomic trends in developing countries. These opinions are issued as the Bank's outlook on the global economy through the Bank's Global Economic Prospects website. The Bank's Global Development Horizons 2011 addresses the trend toward multi- polarity in the global economy and its associated structural changes in growth dynamics, corporate investment, and international monetary and trade arrangements. The annual Global Monitoring Report is a joint World Bank and International Monetary Fund effort to monitor progress toward achieving the Millennium Development Goals (MDGs) by 2015, and includes priorities for policy responses by both developing countries and the international community. 2.34 The Bank’s new Open Data policy has greatly expanded the number of users downloading development data from the Bank’s databases. At the time of the Open Data announcement in April 2010, there were about 140,000 paid subscribers to the databases. Providing free and easier access to the databases has had an immediate impact on data use. Since April 2010, the new Open Data site has recorded well over 20 million page views, and the number of visitors using the data site has more than tripled. Over the past year, the number of entries in the data catalog has increased dramatically; and of the 7,000 available indicators, more than 1,200 can be accessed in Chinese, Spanish, French, and Arabic as well as English. Recently the Bank added a Central Microdata Catalog, which operates as a portal for Bank and non-Bank household survey data which meet international standards for data quality. As of July 2011, the catalog contains more than 600 surveys with some 360,000 searchable variables. Category 3: Knowledge for internal use 2.35 This category of knowledge work accounted for some $99 million (16 percent) of knowledge expenditures in 2010. Two types of knowledge services fall into this category:  internal knowledge products, and  new product development. 20 Internal knowledge products 2.36 Internal knowledge products are the Bank’s internal activities associated with the production and delivery of sectoral reports, policy papers, tools, and databases developed primarily for Bank staff. The terms “knowledge management� and “internal knowledge products� are both used for this category of activity—a mix in terminology that reveals and conceals some ambiguity in whether these activities are viewed as concrete products, analogous to economic and sector work intended for Bank staff; or as processes to ensure connectivity. The stated objective of this category of activities is “to ensure that staff have the knowledge and skills required to achieve results for/with clients.�28 2.37 The internal knowledge product line accounted for expenditures of $81 million in FY10—the third largest expenditure among the nine core knowledge product lines, after technical assistance ($195 million) and economic and sector work ($184 million), and ahead of training for external clients ($54.5 million). 2.38 Internal knowledge products are aimed at an important goal—ensuring that sector staff are well connected across the globe, and have access to the ideas, experiences, research, and global knowledge needed to be as effective as possible in their work with clients. The investments currently being made in this category, like most of the Bank’s knowledge work, are decided on a highly decentralized basis, sector by sector. A substantial amount of the Bank’s core knowledge expenditure—nearly 14 percent—goes to internal knowledge products, but accountability, measurement, and evaluation are not strong. 2.39 From a product perspective, the 2009 QAG report (see footnote 16) found that the Bank’s internal knowledge products appear to be of significantly lower quality than its knowledge products for clients. The QAG report confirmed concerns that these products were supply driven and undermanaged. The report pointed to poor quality of data on product costs and delivery dates, and weak document retention within responsible units; in some cases, key documents could not be located due to the transfer or retirement of task team leaders, even though the delivery had been less than two years earlier. 2.40 About 80 percent of these internal knowledge products are produced by the Bank’ s Networks, with the Sustainable Development Network consistently the largest producer. In part as a result of the QAG study, SDN has proactively worked to improve the quality of its internal knowledge products, and has started to subject them to the same governance processes as economic and sector work done for external clients. A good example is SDN’s new system for climate change knowledge management. 2.41 Recent internal knowledge studies include Policy Briefs on the Financial Crisis, and Costing for Nutrition Interventions; the Pharmaceutical Policy Book is an operational guide; and an example of an internally-focused database is Logistics Perception Indicators. 28 Source: Human Resources Leadership and Organizational Effectiveness website: http://go.worldbank.org/YNUY52WVF0 21 New products 2.42 New products refers to knowledge work carried out to develop new Bank business products, such as pilots and prototypes of products, including new computer systems and software, for potential use by external clients. New products are developed by many units in the Bank. Examples include:  development and pilot testing of the Prototype Carbon Fund;  development of new subnational finance products;  development of a proposal for IDA partial credit guarantees;  development of weather risk management tools; and  expansion of IDA/IBRD credit buy-downs. 2.43 Initial analysis suggests that the knowledge content of many of these products is probably less than in other categories. There is debate within the Bank about whether some or all of this work should be included in the Bank’s core knowledge products. There has been no independent evaluation of work in this category. Key trends 2.44 For this report a survey was conducted of a broad range of end users of knowledge activities completed between FY08 and FY10. The survey was sent to 1,220 end-users, who could provide their responses in English, French, Spanish, Portuguese, Russian, or Chinese. The survey was designed to be representative of ESW and TA work conducted during that period. The responses received do not form a representative sample29. Despite this, it gives an indication of client perceptions, which are largely consistent with the Independent Evaluation Group’s 2008 findings. In general, respondents considered the Bank's economic and sector work and TA products to be effective in supporting policy decisions and implementation. The survey also showed an almost inverse relationship between cost and impact on policy—smaller reports (costing less than $50,000) were considered to have had the most impact, and larger reports (costing more than $500,000) the least impact. Clients also reported significant preference for TA over economic and sector work in terms of providing support for policy decisions, timeliness, and relevance; and as a source of relevant information on global good practices30. These findings indicate that the traditional roles of economic and sector work and TA may be blurred – TA may inform client policies as well as ESW. 29 The survey was planned and conducted year or two after the ESW and TA was completed. Often task teams had difficulty locating the contact information of their clients. One of the recommendations of this report is to make it mandatory to identify and agree with management the list of clients at the beginning of any activity. 30 Non-lending Technical Assistance (NLTA) captures services provided by the Bank to clients as a way to assist them in policy / program implementation, developing / strengthening institutions, and facilitating knowledge exchange; it has no Bank financing attached to it. In contrast, a Technical Assistance Loan (TAL) is used to build institutional capacity in a borrower country. It may focus on organizational arrangements, staffing methods, and technical, physical, or financial resources in key agencies. Since TALs are tied to Bank financing, they typically require agreement on specific action programs to strengthen organizations, and on terms of reference for the appointment of consultants and local counterparts. 22 Trust funds are increasing resources and shaping priorities 2.45 The availability and use of trust funds for knowledge work varies tremendously, especially across regions, as shown in figure 3. In fiscal 2010, trust funds accounted for about 40 percent of core knowledge spending, up from about 30 percent in fiscal 2005. Figure 2.2. Spending on Knowledge Services, by Source of Funds and By Unit (US$ millions) Notes: WBI data includes scholarships as part of trust funds. Bank budget (BB) includes reimbursables. Source: Business Warehouse. 2.46 Trust funds, which provide additional resources beyond what is available in the Bank’s operational budget, make it possible for the Bank to deliver more knowledge services and to respond to a wider range of client needs. Two examples of areas where trust funds have been instrumental in raising awareness and profile of critical issues are on the topic of Community Driven Development programs and climate change. However, the increasing role of trust funds, which must be used for specific purposes, has raised questions about the extent to which trust-funded knowledge activities impose externalities and divert staff and management resources from mandated activities. Trust fund priorities affect both the sectors that benefit from those funds (e.g., prioritizing climate change over other environmental issues) and the type of work that is done (e.g., technical assistance rather than analytical work). While some in the Bank question the rising influence of trust funds, some outside the Bank see this as a healthy development, and as a benign avenue for development funders to influence the knowledge agenda. This report acknowledges the important role of trust funds in enabling some work in areas that would otherwise not be financed through the Bank budget, but recommends further alignment, simplification and consolidation and for a more strategic approach to their management. The “umbrella� approach recommended by the Independent Evaluation Group could be used to ensure more results orientation. 23 Shifts within the knowledge portfolio 2.47 Technical assistance appears to have overtaken economic and sector work. In 2002 expenditure on economic and sector work was nearly twice that on technical assistance. By 2010, spending on technical assistance services had more than tripled and accounted for more spending than economic and sector work (see Annex A). Within the technical assistance category, “how to� guidance has increased even more rapidly, and now accounts for more than half of all technical assistance tasks. Since clients value technical assistance more highly, this is generally seen as a positive trend. 2.48 The Bank is doing less core diagnostic work. The number of core diagnostic reports31 has dropped by 40 percent over the last five years. This shift is seen by some as compromising the foundations of the Bank’s knowledge – the argument is that the ecosystem illustrated in Figure 1.1 depends on a healthy mix of interrelated knowledge activities. Those holding this view argue that integrated, customized solutions cannot emerge from technical assistance alone. Others in the Bank see this shift in a more positive light, pointing out that much of the Bank’s core diagnostic work has an extended useful life, and that the drop in core diagnostic work represents a deliberate policy choice, with the Bank intentionally relying more on data and diagnostics produced by, or in collaboration with, other institutions (governments, think tanks, and bilateral and multilateral development partners). Five years into this decline, it is appropriate to investigate whether this is affecting the quality of technical assistance – perhaps through independent evaluation, or through a survey of the degree to which country teams are able to partner with other institutions to jointly undertake diagnostic work. 2.49 Fragmentation in knowledge work is being reduced. An increasing number of tasks suggests a degree of fragmentation that is not conducive to results and client orientation. Between FY02 and FY10 the total number of knowledge products increased from 3,672 to 5,192 (an increase of 41 percent). There is concern that the proliferation of tasks may be a symptom of working in silos, multiplying tasks and duplicating effort. It is difficult for the institution to effectively manage so many tasks and to measure their impact—particularly when a focus on fewer, more strategically selected tasks is likely to have great impact. One approach to addressing fragmentation is to bundle tasks and adopt a programmatic approach with a focus on results and economies of scale. Various units across the Bank are moving toward such an approach. Consider the experience of the World Bank Institute, whose programmatic approach to external training involved bundling individual tasks into a smaller number of initiatives. This led to a drop in number of external training activities from 1,677 to 292 between FY05 and FY10, more than an 80% reduction in the number of separate tasks. 2.50 More knowledge work is being done in partnership with others. Although the Bank does not currently track the outputs, it is clear that an increasing amount of knowledge work is being conducted in collaboration with others. For example, the Water and Sanitation Program (WSP), a partnership involving 14 bilateral donors, the Bill and Melinda Gates Foundation, and The World Bank, generates extensive data and analytical work. Much of the work on inclusive finance is done through the Consultative Group to Assist the Poor (CGAP), a collaboration involving more than a dozen bilateral and 31 This sub-category includes Country Economic Memoranda, Development Policy Reviews, Poverty Assessments, Public Expenditure Reviews, Country Financial Accountability Assessments, and Country Procurement Assessment Reviews, and Integrated Fiduciary Assessments. 24 multilateral agencies. Other partnerships accounting for significant knowledge production and dissemination include Public Private Infrastructure Advisory Facility (PPIAF), Energy Sector Management Assistance Program (ESMAP), Cities Alliance, the Program on Forests (PROFOR) and the Global Program on Sustainable Fisheries (PROFISH). Overall, the World Bank is involved in more than 200 such collaborations, most of which are described as being involved in knowledge brokering. Knowledge is becoming more open 2.51 The Bank spends more on development knowledge than many other institutions, and its contributions are an important part of a broad stream of development knowledge. More openness will only increase the importance of the Bank’s knowledge work. In his speech at Georgetown University in September 2010, World Bank President Robert Zoellick articulated a need to democratize development economics: "We need to throw open the doors, recognizing that others can find and create their own solutions. And this open research revolution is underway. We need to recognize that development knowledge is no longer the sole province of the researcher, the scholar or the ivory tower." Our aim, President Zoellick said, is to “open the treasure chest of the World Bank's data and knowledge to every village health care worker, every researcher, everyone.�32 2.52 Open data. In April 2010, the Bank took a critical step toward openness by making its development data available for download free of charge. The Open Data initiative is moving forward rapidly—continually expanding the amount of data available for download; developing new applications to enable easy access to data across platforms and devices; and strengthening capacity in developing countries to collect and disseminate high-quality and relevant data on the ground. 2.53 Open tools. The Bank’s traditional research model is product oriented. Researchers investigate a specific issue over a period of time and produce a paper on their findings, which is then disseminated to the public, other researchers, and policymakers. The Bank is a world leader in producing such products on a wide range of development issues.33 The Bank also remains a leader in development economics research, although the vast majority of development economics researchers are affiliated with other institutions. New development thinking and experience in the wider academic and development community are increasingly found in developing countries themselves. To support this work, the Bank’s Development Economics arm (DEC) is beginning to produce research tools for others to use, and is providing open access to those tools. The Open Tools initiative has three objectives —(a) to empower researchers in developing countries to do better research to inform development policy and practice; (b) over the longer term, expand opportunities to collaborate more closely with colleagues in developing countries as full peers, to the benefit of both; and (c) help ensure more open and transparent policy analysis. One of DEC’s new tools is the innovative software program ADePT (Automated DEC Poverty Tables), which simplifies and speeds up the production of standardized tables and graphs in many areas of economic analysis, with a particular focus on country- and regional-level analytic work. ADePT extracts indicators from micro-level surveys and presents them in a print-ready form, making it possible for analytical reports that used to take months to be automatically generated in minutes. ADePT is a free, 32 “Democratizing Development Economics� - http://go.worldbank.org/5VEUBEBHY0. 33 See “The World Bank’s Publication Record,� by Martin Ravallion and Adam Wagstaff, Policy Research Working Paper 5374, World Bank, Washington DC, 2010. 25 stand-alone program, available for download by anyone in the world. It currently includes modules on poverty, labor, health, education, inequality, social protection, and gender, and the Bank is adding new modules, translating the interface into several languages, and developing ADePT training courses. This tool already has several thousand users in the Bank and around the world, and the user base is growing fast. In March 2010 the first localized version of ADePT was launched by the Bank and Bappenas, the Indonesian Ministry of Planning, translated into Bahasa Indonesia and including new tables customized for analysis at the decentralized Kabupaten level. The PREM Network lead economist for Indonesia observed “This project is the poster child for how the Bank will work in the future - building institutions and putting the analytical tools for evidence based decision making in the hands of high level policy makers and district level planners alike.� (ENP Newswire, March 15, 2010) 2.54 Another example of technology being used to create a more open research environment is DEC’s ongoing global poverty monitoring work, which aims at producing the most accurate data possible on poverty reduction.34 One problem with this poverty monitoring has been that the Bank’s “$1 a day� poverty calculations could not be easily replicated, and that the model would not allow for the use of different assumptions, such as purchasing power parity rates or poverty lines. To address this problem, DEC created PovcalNet, an online analytical tool for global poverty and inequality analysis. Users can replicate the Bank’s global poverty measures using primary data drawn from 700 household surveys, and can choose their own poverty line, purchasing power parity exchange rates, and aggregation group. PovcalNet is now widely used for research and as an educational tool for students of development economics. 2.55 Open and collaborative knowledge generation. The Bank is experimenting with a few promising tools to support external research, economic and sector work, and technical assistance: a. Knowledge platforms. The Bank is investing about $3 million a year in six knowledge platforms as an experiment in greater openness. The platforms will promote connectivity inside the Bank and across the international development community in addressing critical, strategic, cross-cutting, and transformational development issues. The first three platforms, competitively selected in January 2011, promote open knowledge in the areas of urbanization, green development, and “eTransform�—ICT for accountability and development. The second set of platforms has just been selected. b. E-collaboration IT platform. The Bank is investing in technology that will enable people inside and outside the Bank to connect in e-space. This will allow worldwide communities of practice inside and outside to share, store, and retrieve knowledge products. The Bank’s work in the future, and especially its knowledge work, needs to involve worldwide communities of development agencies, civil society organizations, governments, and practitioners. The Bank is moving away from working exclusively among ourselves, and towards greater integration with the larger development community. 34 For the latest update, see Shaohua Chen and Martin Ravallion, “The Developing World is Poorer than We Thought, but no Less Successful in the Fight Against Poverty,� Quarterly Journal of Economics 125(4): 1577–162, 2010. 26 2.56 Open publishing. The Bank's Office of the Publisher is making the transition from a traditional book publishing model to a new model for the dissemination of the Bank's formal research outputs, whether published by the Bank or authored by Bank staff and published externally. The new publishing model entails immediate free access to the Bank's peer-reviewed literature and data, with no restrictions on the use and reuse of the Bank's knowledge; adoption of a Creative Commons-like attribution license; and creation of an Open Knowledge Repository for all Bank books, journal articles, working papers, relevant ESW, and their associated datasets. 2.57 Open debate. The Bank’s legacy model of development work rests on bilateral dialogue with government officials. Over the past 10 years, this model has become more inclusive, expanding to involve civil society, NGOs, and the private sector in evidence-based debate on policy and investment choices. This broader involvement—the democratization of development, the empowerment of local change agents—can be difficult and contentious, but it has the potential to bring about broad-based and sustainable progress. One example of empowering local change agents comes from the World Bank Institute, which has been convening, facilitating, and supporting a multi-stakeholder coalition involving public, private, and civil society actors pursuing better governance of the Ghanaian oil sector. In addition, Ghana is part of a regional initiative sponsored by WBI, the Africa Regional Vice Presidency, and a network of global and local partners, to promote greater accountability and transparency in the extractive sectors. By supporting the process of multi-stakeholder engagement, the program strengthens both the technical capacities and collaborative skills of those involved. The Bank’s knowledge work going forward 2.58 The Bank’s knowledge services are evolving. As the Bank adapts its knowledge services to current demands, there are changes in all three of the Bank’s knowledge roles. As a knowledge producer, the Bank is doing less economic and sector work and more technical assistance. The Bank’s research activities are becoming more open and collaborative, and independent researchers worldwide are being empowered by access to the Bank’s open data and robust tools. As a knowledge customizer, the Bank is fostering more collaboration between and among Bank teams, government teams, civil society, and academia to apply global knowledge to particular challenges. As a knowledge connector, the Bank is linking up practitioners and development professionals across the globe. Increasingly, the focus is on empowering non-state actors to demand greater accountability through their engagement in public debate and the political process (compared to the past when the focus was primarily state actors). 2.59 There are two vital challenges. As shown in figure 1.1, the Bank’s knowledge ecosystem comprises a diverse range of activities. These various activities, combined with its global reach and experience, gives the Bank a significant comparative advantage in connecting development knowledge to those who need it. The Bank has flexibility and scope that other institutions may not. At the same time, the Bank cannot and should not try to do it all, but should focus on the areas in which it can make the greatest contribution. And in those limited areas, it is more important than ever that the Bank deliver work of unquestionable quality and relevance. This perspective leaves the Bank with two vital challenges: First, the Bank must begin to systematically measure the results of all of its knowledge services. Only by knowing the results of its various kinds of knowledge work can the Bank improve its impact, and ensure the best choice of activities in which to invest. 27 Second, the Bank must begin to manage its knowledge work with an eye towards connectivity and synergy—i.e., towards linking up different knowledge activities to produce better outcomes. The next chapter focuses on these two challenges. 28 Chapter 3. Managing for Results 3.1 Measuring results. The first two chapters discussed the ways in which the Bank’s knowledge activities have been shifting. Some shifts have been in response to changing demands—e.g., the growing preference for technical assistance over economic and sector work. Other changes have been purposeful initiatives—e.g., moving from a closed shop toward more transparency and collaboration, as with the Bank’s Open Data Initiative. Many of the most exciting changes have come from experiments and management effort within networks, regions, or other units. The Bank is now at the threshold of a still bigger effort, one that reaches across all of the Bank’s units, and that holds the promise of helping resolve the knowledge paradox. This is an effort to systematically measure the results of all of the Bank’s core knowledge services. In measuring the impact of this knowledge work, the Bank will begin to have a common language and currency with which to discuss knowledge as an institutional priority. 3.2 Managing the knowledge portfolio. From one perspective, it is too early to speak about managing the Bank’s knowledge services as a portfolio, because of the lack of a common results framework. But even while the Bank works to develop such measures, much can still be done to improve the way knowledge work is managed in terms of quality assurance and governance. Some knowledge services have relatively clear rules and strong management; others less so. But few have the benefit of regular and independent evaluation, to provide the feedback essential for improving quality, relevance, and impact. Within some product lines, there is significant progress being made, while others lack the fundamental tools needed for effective management. Measuring results and ensuring quality 3.3 Across the Bank, is has become more important to focus on measurable results. This year, a new annual Results Report and a Corporate Scorecard will be introduced, as part of a comprehensive program of internal reforms to make the Bank more client- and results-oriented and accountable. And in 2012, the Board will consider a new lending instrument designed to disburse against clients’ results. 3.4 Measuring the results of knowledge services. There are many reasons for the Bank to be more disciplined and systematic about measuring the results of its knowledge work:  First, because core knowledge services represent a commitment of nearly $600 million per year, the Bank has an obligation to speak coherently about the results of its work. The Bank must move beyond counting the number of reports delivered, and begin the difficult process of measuring what can be measured, with due deference to the complex and sometimes unmeasurable factors that contribute to development.  Second, because of the need to have conversations about value for money regarding the whole knowledge services portfolio—difficult but important conversations about how to prioritize scarce resources. The Bank currently has no common framework for speaking about the relative or combined contributions of its different kinds of knowledge work. Can the Bank begin to rely on external research to support its technical assistance and policy 29 implementation work? Conversely, should the Bank focus more on global public good knowledge work, and leave more of the development and implementation of policies to other? What is the right mix of activities for the amount the Bank has to spend, in the context of a world of increasingly open knowledge?  Third, because the Bank needs to understand its relative strengths and maximize its contribution to development by focusing on its comparative advantages. By better understanding how each of its knowledge roles—producer, customizer, and connector— contributes to development globally and in particular countries, the Bank can focus its resources where they make the greatest contribution. 3.5 Measuring the impact of knowledge services is inherently difficult. If this were not so, the Bank would already have systems in place. There are challenges associated with attribution, with the lag time that can be associated with the impact of knowledge work, and with the selection of appropriate and measurable indicators. Despite these challenges, the Bank can no longer afford the luxury of measuring knowledge work primarily in terms of money spent and reports written. As measurement systems are put into place, there will be disagreements about many of the details (e.g., how standardized and aggregable should impact measurements be? At what level should impacts be assessed?). This will be an evolving effort, and the best should not be the enemy of the good. As an institution, the Bank needs to be able to measure and speak about impact if it is to mobilize the resources needed for knowledge work, allocate the available resources effectively, and appropriately support and reward knowledge workers. 3.6 Objectives, results, and impact. The central challenge for any knowledge activity is to be clear about what is to be accomplished, and then to think through how to measure progress toward that objective. A systematic attempt to measure results begins with clarity of intended outcomes on the ground. If the intended outcomes are clearly stated, it becomes possible to think about what intermediate results can be tracked to assess progress toward those outcomes, and how intermediate results can be measured. 3.7 Different kinds of results. The term “results� covers a range of indicators from outputs, through outcomes, to impact, i.e., the final effect on beneficiaries. The ways that results are measured will vary from one type of knowledge activity to another. The results of a piece of economic and sector work might be measured by the extent to which it contributed to domestic debate in a client country, which could ultimately lead to improved policies. Such an index of knowledge impact could be composed of surveys of opinion makers, references to World Bank studies in the media and other public discussions, and independent assessments of the likely impact of the study. The results of research work might be measured in terms of intermediate indicators of impact, such as the number of times research is accessed, or data sets are downloaded. There are various ways to measure the results of knowledge work, and new approaches are being developed. 3.8 Different parts of the knowledge portfolio are at different stages in the transition towards a greater focus on impact. The Bank is in a period of evolution and experimentation, and there is much to learn. The following sections describe the progress towards results measurement across the three categories of knowledge work. 30 Knowledge work for external clients 3.9 Knowledge work for external clients is the biggest category of the Bank’s knowledge work, representing nearly three fourths of core knowledge spending. Significant attention has already been paid, therefore, to governance and results measurement in this category. For economic and sector work, technical assistance, and external training, task teams are required to specify the outcome or change objectives of the work. At completion of the activity, teams are asked to assess results based on the degree to which these outcomes and objectives were achieved. With the exception of WBI’s external training services, however, there is no routine mechanism for gathering feedback on knowledge work for external clients, either from the clients themselves or from stakeholders or observers, 3.10 Quality assurance. The Bank’s systems and incentives have tended to focus on ensuring the quality and relevance of the knowledge product, even though Bank staff repeatedly find that it is the quality of engagement that makes the real difference. For much of this work, the Bank has no systems to track, let alone measure, the extent and quality of engagement with clients. Little systematic attention is paid to what happens after the product is delivered. There is a need for better systems to track and measure the extent, quality, and results of client engagement. 3.11 For client-facing work, the rule set and management attention appear to be strongest for economic and sector work and TA. In the case of the regional (multi-country) studies programs, Chief Economists consistently play a strong and direct role, including in financing, strategic selection, quality control, and dissemination. With regard to economic and sector work TA dealing with individual countries, there is general and consistent understanding across all Regions that accountability for technical quality and strategic relevance lie with the Sector Manager and the Country Director, respectively. These accountabilities are clearly established in the Bank’s business systems. Across the Regions, there is some variance in arrangements for quality monitoring, support and control for country economic and sector work and TA. Regional models range from proactive and hands-on support and advice from the Chief Economist, as is the case in Africa (with special focus on weaker country programs) to a much more selective and arms-length role, as is the case in Latin America and the Caribbean. While there is room for such variation, a more consistent approach would ensure clarity as to accountability for technical quality, client engagement, relevance and results. At present, there is not internal Bank-wide quality assurance function that may promote such consistency. 3.12 An essential element of quality assurance for economic and sector work is the peer review process at both the concept stage and the delivery stage. In most units, task team leaders propose their own peer reviewers to the sector manager. By contrast, in the DEC research unit, management chooses peer reviewers and it is common practice to include external reviewers. There are no guidelines or criteria to assist reviewers or ensure consistency across the Bank in assessing quality, relevance and results. The peer review process could be significantly strengthened and made more consistent across the Bank, by including external reviewers, where appropriate; and by specifically assessing the articulation of a development objective, the identification of indicators used to measure achievement of the objective, and the quality of the engagement. 31 3.13 Results Measurement. The Background Report accompanying this Knowledge Report provides the details of plans now being put into place for more rigorous assessments of knowledge results. In terms of knowledge work for external clients, these changes have come about through two main efforts:  a FY 2011 review of the Bank’s analytical and advisory activities (the AAA Review), focused particularly on economic and sector work and technical assistance, being led by the Bank’s Operational Policies and Country Services Vice Presidency; and  the 2009 adoption of a Capacity Development Results Framework by the World Bank Institute, which relates to the external training product line, and has the potential for broader applicability. Both of these efforts call for greater attention to the question of why a given piece of work is being done, what development contribution is intended, and how to determine whether the work is achieving its intended objectives. 3.14 ESW and technical assistance. Starting in fiscal 2012, task leaders for technical assistance and economic and sector work will be required to clearly articulate, at the concept stage, the change objective and related indicators, including how they will be measured to demonstrate their contribution to results. Defining a set of intermediate outcomes for ESW and TA will help to shift the focus from outputs to outcomes, and will also help to capture the Bank’s convener or connector role in many knowledge activities. The AAA Review aimed to improve staff and management understanding of accountabilities for economic and sector work and TA, and to put in place systems to strengthen the outcome orientation of these products. The design work has been completed, and new tools are being incorporated into the Bank’s business systems, to be available in the coming months. The changes include the addition of a programmatic approach and flexible “just-in-time� products; an enhanced pilot Country Portfolio Reporting Tool (CPRT); simplification of control points; development of staff guidelines; revision of the existing self-assessment results tool; development of a Concept Note template to improve coherence and outcome orientation; and the adoption of an enhanced storage and retrieval system— i.e. WBDocs, as part of the Operational Knowledge and Support Platform (OKSP). 3.15 External training. Beginning in fiscal 2010, WBI task teams have been using a core set of intermediate and final outcome indicators to inform program design and results measurement. The intermediate indicators for external training are focused on the outcomes of engagement, with the aim of strengthening the capacity of local (non-state and state) actors to participate in the change process. The integration of monitoring and evaluation (M&E) at all stages promotes systematic learning to determine what works and does not work, and enables timely corrective action. 3.16 Feedback from within the Bank. Improved indicators are only part of the results story. Systematic feedback on the perceived impact of the Bank’s knowledge work is also needed from outside the task team. Within the institution, this means that the relevant Country Director and Country Management Unit should provide feedback on the degree to which a piece of knowledge work has contributed, or is likely to contribute, to a particular development objective. Feedback also provides an opportunity for the country team to consider what additional support may be needed after a knowledge product has been delivered to the client. 32 3.17 Feedback from outside the Bank. Taking into account the views of both government and nongovernmental informants is also important, and there is an emerging consensus that their views need to be incorporated into how the Bank measures results. There is debate within the Bank about the degree to which this should be done on a task-by-task basis as opposed to a portfolio basis, e.g. once a year. And there are practical problems in attempting to get that feedback. In conducting a pilot survey for this Knowledge Report, the Knowledge Group team found that one major cause of delays and gaps in receiving client feedback was the time it took to obtain client contact information. Going forward, the Bank could require staff to identify clients and provide contact information in the management information system, at the time of initiation of the task, to facilitate later follow-up. 3.18 Retrospective evaluations. Because knowledge activities can take some time to generate results, it is also important to look back on the work after it is completed. In 2008, IEG did the only retrospective analysis of economic and sector work and technical assistance.35 This study made five recommendations: First, reinvigorate the mandate… to maintain a strong knowledge base on countries and sectors where the Bank is providing or planning to provide funds. Second, ensure that ESW in IDA countries is adequately resourced (even if it means fewer ESW tasks), since cost matters for quality, and quality matters for effectiveness. Third, enhance institutional arrangements for ESW and TA by ensuring substantive task team presence in country offices—particularly in countries with low institutional capacity—to facilitate closer client collaboration. Fourth, recognize, receive, and build on client feedback to counter-balance current Bank incentives for lending over nonlending and ESW over TA. Fifth, take the results tracking framework for ESW and TA more seriously, including systematizing client feedback. 3.19 Progress has been most noticeable on the third, fourth, and fifth recommendations. Decentralization of Bank staff has ensured a substantive presence in client countries. The Bank is moving toward increasing client feedback on non-lending activities, both through sample surveys in regions (and for this report), and by reconfiguring the main client survey instrument. And, as described in this report, the need for a results tracking framework for all knowledge services is being taken more seriously. 3.20 The Development Impact Evaluation initiative. Within the category of knowledge work for clients, the Bank’s impact evaluation work is somewhat unusual. Evaluating the development impact of impact evaluation has not yet been given much attention within the Bank, although the DIME Secretariat is planning an independent evaluation of the DIME initiative, and IEG is planning to evaluate the role of impact evaluation in Bank operations. Although impact evaluation is an important and fast-growing element in the Bank’s knowledge work, it is also the newest element, and accounts for less than 2 percent of the Bank’s core knowledge. The DIME and IEG evaluations will provide important information on the direction this work should take. Global public goods 3.21 Quality assurance for the World Development Report. The preparation of the Bank’s annual World Development Report is a broadly consultative process, with extensive internal and external review 35 Using Knowledge to Improve Development Effectiveness, previously cited. 33 to ensure quality and relevance. An independent evaluation pointed to the "lasting importance and influence" of several WDR reports.36 For example: "The 1984 WDR on population argued that population growth was indeed a problem for development; the 1990 WDR on poverty introduced the $1- a-day poverty measure, which marked the Bank's commitment to poverty reduction; and the 1993 WDR on health introduced Disability Adjusted Life Years (DALYs) and the Global Burden of Disease,� which influenced the creation of the Bill and Melinda Gates Foundation. 3.22 Quality assurance for research work. More than most other knowledge areas, the World Bank’s research work has well-developed quality control mechanisms and quality and relevance standards for its outputs. The Bank’s research is most effective when it involves a close collaboration between researchers and operational parts of the Bank. Research can draw lessons from lending operations and from development policy operations. It can also help policymakers explore the likely outcomes of different policy options by applying historically derived knowledge to current development issues. To strengthen this linkage with operations, the Research Group (DEC) aims to dedicate one third of its staff time to cross support; the other two thirds is spent on more open-ended research. 3.23 Results measurement. DEC makes every effort to have its research, data, and analytic work available online, and routinely tracks statistics on web page views, unique visitors, file downloads, and dataset access. The strong point of these types of measurements is that they use objective criteria to gauge the extent of the Bank’s interaction with the outside world. These measures are also related to staff incentives—the number of times a researcher’s work is cited in the academic literature is tied to salary increases. 3.24 These web statistics are an important piece of the results chain. While downloading a research report does not in itself result in policy change, but it is an indispensable first step toward that change. An appropriate follow-up might be to survey the users of that information to see whether they found it useful. Permission to contact the user with a follow-up survey could be solicited as part of the download process, with follow-up a week or two later. Internal Knowledge 3.25 Quality assurance and results measurement. The $97 million spent on internal products in fiscal 2010 (16 percent of the Bank’s core knowledge expenditure) represents a significant resource for improving the global connectivity of the Bank’s internal knowledge. These efforts must be appropriately managed and their results must be measured. While peer reviews are now required for internal knowledge products costing over $30,000, there is no such requirement for new product development. During its lifetime, the Quality Assurance Group reviewed internal knowledge products once, but never reviewed new product development. 3.26 The need to clarify objectives and specify indicators. As with any other knowledge product, the Bank should require clarity of objectives for internal knowledge products and new product development. Network and sector management should require task teams to clearly articulate the results 36 Banerjee, Deaton, Lustig and Rogoff 2006, pp. 76-77; see also Ravallion, Martin and Adam Wagstaff. The World Bank’s Publication Record. Policy Research Working Paper 5374. 2010, p. 12. 34 they expect, with a particular focus on connectivity across the Bank, and to measure progress toward those results. This will give management the tools it needs to make strategic choices, including choices about what weight to give each of the Bank’s three knowledge roles (knowledge producer, knowledge customizer, and knowledge connector) in each sector. 3.27 In the spirit of open knowledge, it may also be time to consider that the Bank should dispense with the distinction between knowledge for external clients and knowledge for internal use. In practice, both types of knowledge are often shared: external products with internal audiences, and internal products with external audiences. At a minimum, there is an opportunity to bring more external collaborators into this mix, to ensure that Bank staff have access to world-class best practices. 3.28 A two-pronged approach is needed. Addressing the limitations and challenges of a knowledge results framework will require a two-pronged approach. First, there is a need to ensure that existing knowledge products have results frameworks that provide information to assess quality, relevance, and results. But a focus on individual product lines still promotes fragmentation. Therefore, the Bank ultimately needs to move beyond separate results frameworks and towards greater interconnectivity in knowledge activities, with a focus on intended intermediate outcomes rather than individual activities. The Background Report accompanying this synthesis report outlines the technical details of how this approach would work. It builds on the Bank’s defined results chain for economic and sector work, technical assistance, and external training to define a typology of intermediate indicators Of outcomes to which the Bank’s knowledge services contribute. Managing knowledge as a portfolio 3.29 While it is too early to speak about managing knowledge services as a results-oriented portfolio, it is past time to bring clarity and accountability to the basic production function. 3.30 Management arrangements vary by product line. The production function for knowledge services is driven by the nine core knowledge product lines. Some elements are missing, including output review, and compliance monitoring (see table 3.1 below). There are also gaps in assigned responsibility for setting guidelines about internal knowledge products, which gives individual managers in the Bank tremendous latitude to initiate tasks with little or no systematic review (though some units do require reviews). The lack of retrospective evaluation means that the Bank is not systematically capturing lessons to use in strengthening its knowledge activities. The lack of systematic archiving means that a significant portion of our written knowledge products cannot be reliably found by human or machine searches – knowledge work sometimes loses relevance and connectivity when the responsible task leader moves on. Most of the Bank’s knowledge spending is subject to generally clear governance systems, but there are governance gaps within many product lines. In the case of internal knowledge, there are profound gaps—a consistent framework is needed to ensure rules are standardized and followed, compliance is monitored, and results are captured and used to inform staff learning. 3.31 Within product lines, management varies. For the ESW and technical assistance product lines, a Bank-wide unit monitors compliance with processing guidelines that are intended to ensure quality control and eventually to make results measurement feasible. However, for the other product lines, monitoring is up to the implementing unit, and this can leave significant gaps, even where clear rules are 35 in place. For example, although DEC monitors all research products financed through its budget, it does not monitor compliance for research products financed from other units’ budgets. Similarly, WBI monitors compliance for its own external training products, but not for those executed elsewhere in the Bank. Table 3.1. Governance arrangements for core knowledge products Connectivity 3.32 An internal quasi-market helps prioritize regional knowledge work. Consider the right side of the pie chart in figure 2.1, which illustrates the knowledge work done in regional units, mostly at the country level, and mostly for the benefit of external clients. On this right side of the pie chart, the Bank’s matrix management system ensures that there is a quasi-market interaction between a country team and a sector team. Together, these teams decide which knowledge tasks are funded, at least out of the Bank’s own budget. The Country Director controls the budget and “buys� services, which can include knowledge services, from the Bank’s sectors. This interaction brings at least two perspectives to the consideration of a particular knowledge task. The Country Director is typically based in the region, and is in close touch with Government officials about what they need from the Bank. The Sector Director is typically based at headquarters, but brings to bear global experience in the particular sector or technical field. When both agree, this quasi-market interaction offers some assurance of the knowledge task’s relevance and quality. 3.33 Setting priorities for Bank-wide units. This quasi-market dynamic does not exist on the left side of the pie chart, nor for knowledge work funded by trust funds outside the control of the matrix. Networks and sector boards set priorities with varying degrees of input from sector staff across the Bank. Where sector staff in regions have a strong and effective voice in setting priorities for the anchors, this helps ensure the relevance of sector work. But trust funds come with their own priorities, and Bank staff can tap into these funds only for specified types of knowledge work. The quasi-market is also weak with regard to WBI and DEC: there are no buyers and sellers to create a system of checks and balances, or to 36 ensure operational relevance. The absence of a quasi-market on the left side of the pie chart means that a significant share of the Bank’s knowledge work is not automatically connected to client demand, or even to internal Bank demand. 3.34 Helping knowledge flow across the Bank. Unless networks and sector boards take a strong role in ensuring connectivity, the Bank’s decentralized management tends to immobilize knowledge. The low level of cross support typically available to senior technical staff is one indicator of this immobilization. This impairs the Bank’s ability to effectively move lessons learned to and from its 120 country offices and around the world. Countries facing challenges for the first time can often learn a great deal from the experiences of other countries that have recently faced similar challenges. One example of what might be done to help knowledge flow comes from the Finance and Private Sector Development network, which will radically restructure its regional and sectoral management in July 2012 to break down barriers to knowledge flow and create incentives to think globally. 3.35 Helping knowledge flow within regions. Some Regions are charging their regional Chief Economists with managing a portion of the resources allocated for knowledge work. In the Bank’s Latin America and Caribbean Region, for example, the Chief Economist is accorded a budget similar to those of the region’s six Country Directors. The Chief Economist’s office develops regional programs in conjunction with Country Directors to ensure that its knowledge services support operational needs. At the same time, the Chief Economist’s office maintains significant links with academia and performs a quality assurance function for regional knowledge services. 3.36 Spending outcomes. Annex A shows the breakdown of core knowledge spending from fiscal 2002 to 2010. Of the $606 million spent in 2010, 74 percent was allocated to knowledge for external clients, 10 percent to global public knowledge, and 16 percent to internal knowledge. This spending pattern has no particular correlation to any stated priorities—it is an outcome of budget allocation dynamics across Bank units in the first instance, and within each unit in the second instance. 3.37 Conscious prioritization. To optimize the results and impact of its collection of knowledge activities, the Bank could take a portfolio approach to managing knowledge investments. This could begin with identifying the objectives of each knowledge activity. Consideration of the strategic and relative priority of each activity’s intended results would involve asking questions which, by and large, are not being asked and for which there is no established Bank-wide forum. A process of prioritization could consider, for example, what the Bank aims to accomplish with its spending on global knowledge, and whether global knowledge should be only 10 percent of total knowledge spending. For another example, a Bank-wide forum could consider the concrete objectives of internal knowledge activities, and whether its 16 percent of core knowledge spending is appropriate. 3.38 Incentives for knowledge production and connectivity. The experience of many staff members is that management values lending over knowledge, so that large and high-profile lending operations hold more potential for career advancement than does knowledge work. And those involved in knowledge work find that with management systems oriented to discrete knowledge products, there are stronger incentives for knowledge production than for knowledge flow—it is easier to find funding and management support for producing knowledge than for sharing the work with colleagues and counterparts around the world. Changing staff behaviors and attitudes will require altered incentives—a mix of 37 signaling, rewards, and consequences. Task leaders who do interesting knowledge work and share it with colleagues or country counterparts could be celebrated and rewarded. Supplemental resources and recognition of all team members could encourage multi-sector teams. Performance evaluations could take into account more knowledge related behaviors, including client engagement, multi-sector and multi- region collaboration, and follow-on dialogue and dissemination. When resources are tied to connectivity, behaviors change: the Knowledge Platforms, a competitive initiative of the Bank’s Knowledge and Learning Council, are judged in part on their multi-sectoralism. As a result, the teams submitting proposals reach beyond sectoral and regional boundaries, make new connections, and find new synergies. 3.39 Financing arrangements contribute to fragmentation. Current financing arrangements for knowledge work contribute to its fragmentation. The highly decentralized structure of the Bank’s regions and networks means that the Bank’s budget resources for knowledge work are also decentralized. There is also a separate pattern of allocating trust funds for knowledge work outside of the Bank budget, which further contributes to fragmentation— an enterprising task leader or manager who is unsuccessful in securing budget for a knowledge activity can often find trust funds to finance the work. Trust-funded knowledge work has grown dramatically, and is having a profound effect on the kind of work that is done, and where and by whom. Local and sector unit outcomes may be optimized by trust-funded work, but taken as a whole, as a portfolio, it is not clear that this collection of activities —a result of thousands of separate decisions—is optimized for results and impact. 3.40 Cohesive financing for knowledge work. The demand for knowledge is increasing, while the demand for investment financing, which supports much of the Bank’s knowledge work, may be circumscribed. One response to these shifts must be to become more selective and efficient. For example, the Bank needs to be more aware of work done by other institutions, and avoid duplicating that work. In the event that the demand for knowledge services were to expand significantly, or if funding for knowledge services from the Bank’s budget should shrink (e.g., because low interest rates reduce the earnings on the Bank’s capital), then it would be wise to have experiments underway for other means of financing the Bank’s knowledge work. These could include: (a) more pro-active recruitment of trust funds, with institutional clarity about development objectives, strategies, and results measurement; (b) continued expansion of fee-based services—in the last decade, the Bank has undertaken more than 300 such activities in 30 countries; (c) creation of a small number of multi-donor and multi-recipient umbrella facilities, as recommended in the 2008 IEG report,37 to support one or more strategic priorities agreed upon by the Bank and other donors. 37 Using Knowledge to Improve Development Effectiveness, Independent Evaluation Group (IEG) 2008. 38 Chapter 4. Conclusion 4.2. The World Bank’s contribution to development is changing in response to expanding sources of knowledge, the demands of increasingly sophisticated clients, changing technology, and the evolving nature of development itself. 4.3. This Knowledge Report marks an important step toward the Bank’s recognition that its legacy systems for managing its knowledge work need attention. There are gaps in governance; little attention is paid to measuring the results of knowledge work; and conversations about how to allocate limited resources are only just beginning. There is broad agreement within the Bank on the need to be more selective in knowledge work, but no process to use to discuss priorities. 4.4. The Bank also needs to find the right balance between responding to client demands and continuing to engage in the rigorous analytical work that underpins the technical assistance and external training that clients value so highly. Recommendations 4.5. Results measurement. Results measurement needs to be strengthened. Many steps are already being taken, and systems for measuring the results of economic and sector work and technical assistance are being put in place. These systems could be expanded to assess the results of all core knowledge work. This requires a few basic elements:  a clear objective when the knowledge activity is initiated;  agreement on intermediate outcomes and measurable indicators;  budgeted plans for assessing outcomes; and  mechanisms to ensure lessons are captured and fed back into the design and implementation of future activities. 4.6. External and independent feedback. The Bank can no longer rely on task leaders self- assessing the likely impact of their knowledge work. The Bank must assess results through multiple evaluation methods (or triangulation), complementing self-assessments with client and stakeholder surveys, and with periodic independent and retrospective evaluations. 4.7. Quality Assurance. For economic and sector work and technical assistance funded by the Regions, the role of Sector management in ensuring technical quality and of Country management in ensuring relevance is clear. As importantly, these two different perspectives tend to produce work that is relevant for effective client engagement. Quality assurance, support and monitoring at other levels could be enhanced as follows:  Peer review practices should be strengthened and systematized, with external experts and client perspectives included in peer review panels. The role of sector boards may be enhanced in ensuring that global experience is taken into account in the peer review process. 39  Management arrangements and responsibilities should be made consistent across all product lines over time. With respect to figure 2.1, it is essential that management have appropriate tools to ensure quality, results, and relevance. A clearer “sponsorship� of each product line responsible for setting guidelines (as the Bank’s Operational Policies and Country Services Vice Presidency does for economic and sector work, and technical assistance), regardless of the unit that generates the knowledge product (and which is responsible for the quality of the work) may be considered. This sponsoring vice presidency would be responsible for monitoring the portfolio of individual product lines, and for arranging post-review of quality, relevance, and results within the product line. 4.8. Managing knowledge as a portfolio and strengthening connectivity. The Bank’s primary building block for delivering knowledge services is the country level, even though just about half of funding spent on core knowledge is for clients. In order to ensure that country clients get the maximum impact from Bank knowledge services, the institution needs a forum or a process to ensure that knowledge from the rest of the Bank (via Networks), new knowledge from networks, knowledge from DEC and from WBI is made readily available and supports the country programs. Likewise, the Bank’s engagement in global public goods, driven by the networks, DEC and WBI, would have greater impact if it could be linked to country programs, especially given the local presence in 120 countries. However, Bank resources are scarce, and the Bank cannot afford to be heading in multiple directions at once, even if trust funds play a significant role in alleviating resource constraints. As such, this report argues that knowledge also requires a corporate perspective, as well as regional and sectoral perspectives. The Bank’s decentralized management system produces locally optimized outcomes with regard to both priorities and quality assurance, but this systems limits connectivity and opportunities for synergies and economies of scale. Moving toward conscious priority setting will require debate and discussion. The Bank needs to create explicit mechanisms for the discussion of work programs and priorities among networks and regions. This has the potential to strengthen quality, relevance, and results; prevent avoidable overlaps and duplication; and achieve genuine synergies. Quality and results can also be improved through external discussion and review of how the Bank prioritizes its knowledge activities to address urgent development challenges. 4.9. Connectivity. When the Bank’s performance evaluation system is next redesigned, it would be appropriate to include knowledge production and flow as critical Bank behaviors. Signaling by senior management about the importance of knowledge services is critical. All knowledge outputs should be captured in the Bank’s document systems, in a form that is easily retrievable and searchable. These systems should be consistent with systems and processes in the private sector to ensure the free flow of information. Looking ahead 4.10. This Knowledge Report is the first in a series of Knowledge Reports. The research and consultations on which it is based have helped to illuminate the Bank’s knowledge landscape, and to map a portion of that vast territory. Within the territory covered by this report, specific actions are recommended to bring more coherence, oversight, and accountability to the Bank’s knowledge work. Other issues, as they continue to evolve through further research and consultations, will be covered in 40 future reports. The implementation and evolution of this year’s recommendations will be discussed in the 2012 Knowledge Report. 4.11. The next Knowledge Report could focus on knowledge flow. The discussion in this first Knowledge Report has largely been grounded in the reality that the Bank manages knowledge activities much the same as it manages lending projects. The management information system we use to manage knowledge actually has its roots in enterprise software developed to manage industrial processes. This makes it convenient to speak about the creation and completion of discrete knowledge products. The grid shown below (table 4.1) is one helpful way of thinking about the challenges of managing knowledge in an institution with a mandate as broad and complex as that of the World Bank. This report has mostly dealt with explicit, codified knowledge products—those in the upper left quadrant of the grid. The other three quadrants—having to do with the Bank’s stock of tacit knowledge, the transfer of tacit knowledge, and the transfer of explicitly knowledge—will be considered in future reports. Table 4.1. Dimensions of knowledge Source: Developed by Finance and Private Sector Development Network. 4.12. Future Knowledge Reports could focus on knowledge partnerships and collaboration; on embedded knowledge and knowledge flow; or on other emerging areas of interest. Knowledge collaborations are an increasing part of the Bank’s work. They are often open-ended and involve multiple funding sources and shared governance arrangements, resulting in decisions and priorities that reflect a larger consensus. The knowledge work involved in these collaborations is growing much faster than the Bank’s own knowledge work, and will soon exceed the Bank’s core knowledge production. Alternatively, future Knowledge Reports could focus on tacit and embedded knowledge. Embedded 41 knowledge is knowledge work funded by borrowers for the purpose of project preparation or supervision. The Middle East and North Africa region (MENA) estimates that in 2010, the region spent about 10 percent of loan proceeds on knowledge products, some of which were generally of the same type as the Bank’s core knowledge products. If this percentage holds for other regions, and there is no reason to believe otherwise, then knowledge spending embedded in lending totaled some $2.5 billion for 2010. Tacit knowledge is the knowledge our staff and clients have gained through experience, which could be translated into explicit knowledge by capturing it in documentary form, or transferred to others through mentoring and collaboration. Other possible areas of focus include investigation of human and technological systems to capture and retrieve applied knowledge; institutional priorities as between knowledge production and knowledge retrieval; and outsourcing options for producing more knowledge by contract with external partners, rather than with our own staff. A broad array of knowledge management challenges awaits discussion. 42 Annex A. Spending on Core Knowledge Products Notes: WBI data includes scholarships as part of trust funds. Source: Business Warehouse. 43