100236 Approach Paper Systematic Country Diagnostic and Country Partnership Framework Process Evaluation June 26, 2015 Introduction 1. The new World Bank Group Strategy1 aims to focus country programs on accelerating progress toward sustainably reducing poverty and building shared prosperity (known as the “twin goals”). It recognizes that while today’s country programs are broadly aligned with the Bank Group’s mission, they need to provide countries with more coordinated and focused programs: activities and interventions need to be better prioritized according to their expected impacts on the twin goals. The new strategy incorporates two instruments that will be used to enhance country engagement: A new Systematic Country Diagnostic (SCD) will use data and analytic methods to support country clients and Bank Group teams in identifying the most critical constraints to, and opportunities for, reducing poverty and building shared prosperity sustainably, while considering the voices of the poor and the views of the private sector. The Country Partnership Framework (CPF) will discuss focus areas for Bank Group support, aligned with the country’s own development agenda and selected primarily to address the key constraints. 2. The Bank Group’s Board of Executive Directors and senior management have high expectations for this new process. Hence, the Board requested that IEG evaluate the new country engagement model as it is being implemented. This process evaluation will have the following objectives:  To explore whether the guidelines, directives, and other resource materials related to the new country engagement process have thus far helped World Bank Group teams produce SCDs and CPFs that are focused on helping countries to tackle key constraints to poverty reduction and shared prosperity in accordance with the Bank Group’s comparative advantage, as well as whether they have fostered coordination among the Bank Group institutions.  To provide the Board and management with early information about the functioning of the SCD/CPF process and the extent to which the shortcomings related to the previous processes have been addressed, so as to enable changes to improve the process.  To identify good practice examples and lessons learned thus far. 3. This is a real-time process evaluation to provide early, evaluative input into the operationalization and roll-out of the Bank Group’s new country engagement approach. The evaluation will assess the first group of SCDs and CPFs that will have been presented to the Board by the end of December 2015. 1 World Bank Group Strategy, World Bank Group, Washington DC, October 2013. 1 Background and Context 4. Country strategies at the World Bank Group have evolved from internal management tools to public documents that strive to be country-owned, results-driven, and flexible.2 For most of its early history, the Bank engaged with countries mainly though lending operations without articulating a broader country strategy. Internal planning of the lending programs at the country level began in the early 1970s with the Country Program Papers. When the Board of Executive Directors began to seek information on the strategy for individual countries in the late 1980s, individual lending operations began to be put in the context of the Bank’s program of country activities. Then, in 1990, IDA deputies asked that strategies be prepared for all IDA countries and discussed with the Board as part of Bank operations. These early country assistance strategies (CASs) were 10- to 15-page notes appended to loan documents. They were Bank- centric and rarely involved consultation with country governments or other stakeholders. 5. The Board extended the preparation of CASs to all active Bank borrowers, IBRD as well as IDA, in 1992. At that point, CASs were prepared in parallel with internal country strategy papers prepared for management. In 1994, the CAS became a free-standing document and the sole strategy document for both management and Board review. The change to a free-standing document effectively delinked CAS discussions from lending operations in 1996. With the advent of public disclosure in the late 1990s, the CAS process began to include consultations with key stakeholders and ultimately became the center of Bank-government interaction. 6. The starting point for these CASs was the country’s own development strategy. To ground the CAS program in the country context, a good strategy started with a careful diagnosis of country conditions and a clear description of the government’s own priorities and development strategy. Over time, the coverage of the government’s agenda and development priorities improved and by late 1997 the government’s development agenda and priorities were covered substantively in a separate section of the document. By then, more than 90 percent of CASs discussed the goals and implementation of the government’s ongoing policy and reform programs, and roughly two-thirds did so in some detail and relatively well. However, in fiscal 1998 only about 40 percent of CASs put ongoing reforms in perspective by examining the effectiveness of past reforms, and only 20 percent did so systematically. 7. The early CASs focused mostly on Bank products and activities, but as they evolved, they came to reflect more strategic engagement based on country development results. As CASs aimed to address an expanding range of issues, however, they became more complex, less strategic, and longer. They were being overloaded with goals and agendas. Addressing this concern, the first CAS retrospective in 1998 devoted considerable attention to strategic selectivity, providing 2This section is based on four country assistance strategy retrospectives (1998, 2000, 2003, and 2009) and on Results Focus in Country Assistance Strategies: A Stocktaking of Results-Based CASs, Operations Policy and Country Services, World Bank, Washington DC, July 2005. 2 quantitative analysis of sectoral selectivity, defining strategic selectivity as a matter of qualitative judgment based on three criteria,3 and recommending actions to strengthen selectivity and focus. 8. As experience grew, the focus of the CASs shifted from inputs—a series of products and activities—to development results. With this evolution, CASs started to be designed to attain certain development outcomes. Results-based CASs were piloted between 2003 and 2005, and were mainstreamed after that. The results-based CAS was meant to help make country programs more strategic as well as more selective. The central innovation was the results framework, which specified links between the Bank’s interventions and long-term development goals. Measurable indicators of progress would be tracked through CAS implementation, encouraging active management of CAS programs and allowing both self and independent evaluation. The new approach was expected to create a better strategy, with greater chances of success and of development effectiveness.4 9. In fiscal 2005, management established a four-year CAS cycle with a midpoint Progress Report. It also introduced the Interim Strategy Note for use when a full CAS could not be prepared, as typically happened in fragile or conflict-affected states, and it initiated the use of the CAS Completion Report. Around this time, the Bank also adopted a new title for the documents. The Country Partnership Strategy (CPS), as it was now called, was thought to better reflect a relationship with the country client based on mutual learning and responsibility. 10. As it evolved, the CAS also became more flexible, allowing Bank teams to customize Bank services to the country’s own needs and priorities as well as to changing circumstances. Changes in country demand during the CAS period meant that the Bank needed to remain flexible during the CAS implementation to ensure continued alignment with country priorities. In higher-income countries with access to alternative financing and knowledge services, as well as in fragile and conflict-affected states—where needs can change quickly—the Bank needed to change and adapt its services during implementation. 11. As the importance of the CAS increased, it became clear that IBRD and IDA-funded activities would have greater development impact if they were designed and implemented as part of an integrated Bank Group–wide approach. CAS guidelines called for joint Bank-IFC CASs in all countries where there was substantial scope for collaboration and synergies in private sector development activities. Synergies ultimately would be a matter of collaboration at the operational level, but Bank staff believed that joint CASs could set the stage by articulating the private sector role—and thus IFC’s role—in addressing key development challenges. 12. CASs are often implemented in uncertain environments and are thus subject to risks. All CASs now have a section that discusses the main risks and what the Bank intends to do in case a 3Potential magnitude of impact, likelihood of country action, and additionality of Bank contribution. 4Results Focus in Country Assistance Strategies: A Stocktaking of Results-Based CASs, Operations Policy and Country Services, World Bank, Washington DC, July 2005. 3 given risk materializes. However, the 2009 retrospective of CASs5 noted that even when the risks were well articulated and comprehensive, it was difficult to distinguish how the perceived risk had been addressed in the development of the strategy, the instruments, and the choice of interventions. 13. The 2009 CAS Retrospective also pointed out that a CAS was expected to reflect the existing state of country knowledge; the country program; dialogue with government, donors, and others; and relationships with stakeholders. While the CAS was not meant to be analytic, upstream analysis, including on the sources of growth, macroeconomic sustainability, determinants of poverty, and public expenditure and financial management systems, was expected to be carried out regularly, in advance of CASs, and to explicitly inform the CAS diagnosis and the design of Bank support. 14. Despite the evolution of CASs, Bank staff have identified four important shortcomings:  The Bank’s sector and corporate objectives (such as for infrastructure, finance, and social security) have often been in tension with the country’s vision and priorities. The government may or may not prioritize the strategic objectives that the Bank espouses. Moreover, the Bank has not always appeared to invest resources in a way that would maximize the chances of achieving both country and Bank objectives. Weak results frameworks were partly to blame, as a list of activities spread across many sectors failed to constitute a critical mass of interventions needed for maximum development impact.  Despite the emphasis on results, some CASs have still taken a transaction-focused approach instead of concentrating on more strategic objectives and expected development outcomes. While Bank interventions have broadly aligned with the mission of poverty reduction, in some cases they have lacked focus and have not concentrated on issues likely to have the greatest impact on countries’ development. A need remained for more selectivity and focus.  The Bank and IFC have not operated in unison in countries. While the formality of joint CASs after fiscal 2009 enhanced World Bank–IFC cooperation during strategy formulation, cooperation during program implementation has remained erratic.  The discussion of risks has been weak in many CASs, without explicit links between identified risks and the CAS program or without sufficient discussion of the ways in which risks would be managed. A more pointed identification of risks and measures was required to deal with them in CASs. A New Bank Group Strategy and Its Instruments 15. Against this background and the weaknesses identified, the new World Bank Group Strategy (October 2013)6 aims to focus country programs on accelerating progress toward reducing poverty and building shared prosperity (known as the “twin goals”) in a sustainable manner. It recognizes that while today’s country strategies are broadly aligned with the Bank 5Country Assistance Strategies: Retrospective and Future Directions, Operational Policy and Country Services, World Bank, Washington DC, September 1, 2009. 6 World Bank Group Strategy, World Bank Group, Washington DC, October 2013. 4 Group’s mission, they need to provide countries with more coordinated and focused programs, and activities and interventions need to be better prioritized according to their expected impact on the twin goals. It also recognizes that the Bank Group needs to establish an effective mechanism to reconcile country demands with the twin goals and move away from a supply- driven approach (where that existed). 16. The new strategy sets two ambitious goals: end extreme poverty by reducing the percentage of people living on less than $1.25 a day to 3 percent by 2030; and promote shared prosperity by fostering income growth of the bottom 40 percent of the population in every country in a sustainable manner. To accomplish these goals the Bank Group would: align all of its activities and resources to the twin goals; use its strengths as a global institution by supporting clients in delivering development solutions; and work as “one World Bank Group” by leveraging the strength of all constituent agencies (World Bank, IFC, and MIGA). The Bank Group would strengthen the focus of its country programs by developing a more evidence-based and selective country engagement model in the context of country ownership and national priorities, and in coordination with other development partners. 17. Implementation of the strategy started in 2014 with a reorganization of the Bank Group into global practices intended to help accelerate progress toward the twin goals through a new form of problem-solving engagement with countries. In parallel, the Bank Group’s engagement with country members is being revamped7 as outlined below to make country programs more strategic and strengthen analytical aspects of the engagements, ensuring that the twin goals are at the center of the engagements while also concentrating efforts and resources in areas that are key for the country development strategy. 18. Two instruments will underpin the enhanced country engagement model (Figure 1). A new Systematic Country Diagnostic (SCD) will use data and analytic methods to support country clients and Bank Group teams in identifying the most critical constraints to, and opportunities for, reducing poverty and building shared prosperity sustainably, while considering the voices of the poor and the views of the private sector. The Country Partnership Framework (CPF), evolving from the CAS, will discuss focus areas for Bank Group support, aligned with the country’s own development agenda and selected primarily to address the key constraints and opportunities identified in the SCD. Figure 1. The New Approach to Country Engagement Systematic  Country  Performance &  Completion &  Country  Partnership  Learning  Learning  Diagnostic Framework Review Review Source: World Bank Group: A New Approach to Country Engagement, March 2014. 7World Bank Group: A New Approach to Country Engagement, World Bank Group, Washington DC, April 29, 2014. 5 19. The SCD will be prepared upstream of the CPF and will identify priority areas for the country to accelerate progress toward achieving the twin goals in ways that take into account the sustainability of the program. The SCD will be prepared in consultation with national authorities, the private sector, and other stakeholders, but it will reflect the evidence and analysis of the Bank Group. It will become the reference point for client consultations and help focus the country, the Bank Group, and other development partners on goals with the most potential for impact on achieving the twin goals. 20. The CPF will replace the CAS as the main tool guiding the Bank Group’s support for member countries’ development programs. The CPF will maintain the Bank Group’s country- driven model and will start from the country’s vision of its development goals as set out in its own strategy. The Bank and the country will further draw upon the SCD and together develop the CPF objectives. The objectives will be derived from those country development goals that reflect the Bank Group’s comparative advantage as well as align with the twin goals, as identified in the SCD. The keys to selecting CPF objectives require that each objective: (a) focuses on areas identified by the SCD or other analysis as critical for achieving the twin goals in a sustainable manner, (b) is consistent with the World Bank Group’s comparative advantage and capacity to deliver based on past experience, and (c) is congruent with the country’s own development goals and capacity to deliver (see Figure 2). In the end, while selectivity is multidimensional and complex, IEG’s evidence from Country Program Evaluations and country strategies suggests that selectivity can be enhanced by paying careful attention to client country capacities, grounding all operations in solid analytical work, taking a long-term view and sequencing interventions accordingly, properly factoring in design and implementation risks to results delivery, and taking account of the Bank’s comparative advantage.8 Figure 2. CPF Selectivity Filters Country  CPF Objectives Twin  Development  Goals Goals Bank Group  Comparative  Advantage Source: World Bank Group: A New Approach to Country Engagement, March 2014. 8 Selectivity in Country Strategies: The Evidence, IEG, 2014. 6 21. The new approach is expected to emphasize learning at all stages. During implementation, the country and the Bank Group are to engage in a continuous process of monitoring and learning. Every two years, this will lead to a Performance and Learning Review (PLR, the third block in Figure 1), which will be used to adjust the program and to update the Board on progress. The PLR will be used to flesh out details on development objectives and the necessary activities to achieve those objectives that were not clear when the CPF was originally prepared or last updated. At the end of the CPF period, a PLR may be used to extend the CPF by up to two years. 22. At the end of the CPF period, the Bank Group will engage with the country to complete a self-assessment and issue a Completion and Learning Review (CLR, the fourth block in Figure 1). The CLR will be both a means to enhance the Bank Group’s knowledge and to increase development effectiveness and a tool for accountability. As part of the CLR, the country team will complete a self-assessment of the program’s performance, Bank Group performance, and how well aligned the strategy was with the twin goals. The self-assessment will be based upon the results framework of the most recent PLR, and IEG will validate the Bank Group’s self- assessments and verify its findings. 23. Working as one Bank Group—intra-agency cooperation—is central to the new strategy. The CPF is expected to mainstream joint business planning as the backbone for strengthening operational collaboration. The Bank Group is expected to increase the number of joint projects and review its portfolio of products and services to improve synergies and eliminate overlap. These initiatives will be reinforced by stepped-up efforts to align policies and practices and promote changes in the operational culture of each agency. A new Regional Coordination Mechanism is to formalize country- and regional-level coordination among the Bank, IFC, and MIGA, and help the Bank Group with its regional engagements. 24. The CPF is to improve the identification and managing of risks to the Bank Group program. The risk section will take a systematic approach to risk based on the Systematic Operations Risk- Taking Tool (Figure 3),9 to ensure that teams consider all major risks to achieving CPF objectives, as well as any unintended adverse consequence that may be associated with the program. The risk section is designed with the intention to increase transparency by better informing stakeholders of program risks. Teams are expected to identify risks and propose measures to manage or mitigate them, and may also discuss the risks of not engaging in the proposed program, thereby highlighting the program’s risk-reward trade-offs. 25. The risk assessment will take into account the likelihood of the risks materializing and the severity of their impacts on the achievement of the intended results. The assessment will be based on current residual risk, that is, after taking into account the impact of mitigation measures that have already been implemented but not presuming any future additional mitigation measures (Figure 3). The risk categories have been designed with the intention to capture the key risks that could affect programs in CPFs. The categories are: political and governance, macroeconomic, sector strategies 9Framework for Management of Risk in Operations: Proactive Risk Management for Better Results, OPCS, World Bank, Washington DC, August 21, 2014. 7 and policies, technical design of program, institutional capacity for implementation and sustainability, fiduciary, environment and social, stakeholders, and other. Figure 3. Likelihood of Risk and Impact High impact  High impact  Low likelihood High likelihood Impact on development results Low impact  Low impact  Low likelihood High likelihood Likelihood of risk materializing Source: Framework for Management of Risks in Operations—Proactive Risk Management for Better Results (2014). How Will the New Approach to Country Engagement Address Problems Identified in the CASs? 26. The new approach to country engagement intends to address the identified weaknesses of the previous CAS process as follows:  Tensions between Bank objectives and the need to align with the country’s own visions and priorities. The SCD is intended to identify country priorities based on the twin goals. The CPF would then identify a common set of priorities with the country’s development agenda that would underpin the Bank Group’s program. The Bank Group program would focus on those priorities where it has a comparative advantage.  Lack of selectivity and focus. The main objective of the SCD is to identify key development priorities in the country relying on existing and ongoing analytical work. The CPF will sharpen and order those priorities in consultation with countries and take into account the Bank’s comparative advantage.  Weakness in Bank Group coordination. Protocols for cooperation are expected to be strengthened and reinforced by stepped-up efforts to align policies and practices, and promote changes in the operational culture of each agency.  Lack of systematic risk analysis. In CPFs the risk analysis is to be strengthened by using a much more systematic approach to risk based on the Systematic Operations Risk-Taking Tool. 8 27. The Bank Group has issued guidelines for producing SCDs and CPFs.10 Based on those guidelines, SCDs are owned by the Country Management Unit, which is to nominate a task team leader with integrative and analytical skills. Task teams are expected to draw on expertise across global practices and across the Bank Group. All SCDs will be joint Bank Group products, with IFC participating more extensively in countries where it has a deeper presence and country knowledge; MIGA is represented by the relevant regional economist. The guidelines foresee a robust and contestable review process of SCDs, chaired by regional vice presidents (for IFC and the Bank). All SCDs are to be approved by Bank, IFC, and MIGA managements. Finally, disclosure of SCDs will follow the Access to Information Policies of three institutions. As of June 25, 2015, concept reviews were completed for 42 SCDs and decision drafts were completed for 24 SCDs. Ongoing SCDs span all regions and country types. Concept reviews were completed for 15 CPFs, of which six (Albania, Costa Rica, El Salvador, Myanmar, Panama and Serbia) had been presented to the Board.11 28. Based on the Board Update on implementation of the new strategy,12 there is a strong pipeline of SCDs aligned with twin goals and sustainability, and underlying a first group of CPFs scheduled for Board discussion in CY15. The Board Update also states the need to identify the biggest country challenges and be more selective, improving quality and periodicity of data, especially to better benchmark shared prosperity and poverty, and continue to make knowledge more accessible, cutting edge, and evidence-based. Conceptual Aspects of the Evaluation 29. This will be a real-time process evaluation that aims to provide early, evaluative input into the operationalization and roll-out of the Bank Group strategy through the new country engagement approach. The evaluation will assess the first group of SCDs and CPFs produced by the Bank Group. 30. As SCDs and CPFs are new products and the new country engagement model is being tested with the authorities, it is quite possible that the production of SCDs and CPFs and the engagement model itself will evolve through experiential learning. IEG expects to contribute to this evolution by assessing the preparation process and quality of SCDs and CPFs. Given that the new CPFs are only being approved now, IEG will not have country-level implementation results from the new country engagement model and will rely instead on evidence on steps taken to produce SCDs and CPFs, both from desk reviews and field visits. The evaluation will assess SCDs and CPFs against the key issues the new country engagement model seeks to overcome (see para. 26), and will validate its assessment through stakeholder interviews and field visits that 10Interim Guidelines for Systematic Country Diagnostics, February 21, 2014, and World Bank Group Guidance for Country Partnership Framework Products, December 24, 2014. Both were posted on the internal Bank Group website. 11Systematic Country Diagnostic—An Update, March 20, 2015 and OPCS Country Partnership Framework Pipeline Database. 12World Bank Implementation Update, SecM2015-0029, January 20, 2015 (presentation for informal Board meeting on Thursday, January 29, 2015). 9 will look into country ownership, realism, and inclusiveness (in the country and with other partners). In doing these assessments, the evaluation team will take appropriate note of any modifications that may take place in the SCD and CPF processes, and will also seek to benefit from the results that might become available from ongoing internal stocktaking exercises by Bank Group management. The early evaluation findings will also help shed light on whether the guidelines and directives on the SCD and CPF need additional revisions. Links to Previous Evaluations 31. The evaluation will substantially benefit from earlier IEG products, including country and corporate evaluations. IEG’s country-level evaluations, such as Country Program Evaluations (CPEs) and CAS Completion Report (CASCR) Reviews, provide in-depth prior evaluative evidence for an exhaustive list of countries, including those that will be the subject of country case studies. These evaluations will provide insights into the extent of selectivity, focus, effectiveness, and Bank Group collaboration at the country level under the previous CAS model. Also, IEG’s ongoing assessment of Bank Group assistance in a cluster of resource-rich countries will provide an analytical framework on how to compare and contrast the Bank Group approach in addressing similar development challenges across countries both in terms of diagnostics and country program design.13 32. The evaluation will also benefit from findings and lessons learned related to political economy analysis and its integration in and impact on country strategies from IEG’s recent evaluation of Bank Group assistance to fragile and conflict-affected countries.14 IEG’s learning product on Maximizing Development Impact in IDA, which is being prepared in parallel, may also provide findings relevant to the SCD/CPF process evaluation. Finally, IEG’s annual review of Bank Group results and performance in 2013,15 which assessed performance of country strategies and projects including by the World Bank, IFC, and MIGA, will serve as an important source for learning as it provides in-depth analysis of country and project performances and determines the contributing factors. 33. IEG’s other thematic evaluations and learning products on poverty, results frameworks, selectivity, and Bank-IFC cooperation will help guide both SCD and CPF assessments. IEG’s recent poverty evaluation16 will be an input for both SCD and CPF assessments as it analyzes the poverty focus of the World Bank interventions and support for poverty data and poverty diagnostics. It also evaluates the poverty focus of country strategies and studies the consistency between formulated country strategies and poverty diagnostics. Finally, the evaluation will use 13Clustered Country Program Evaluation for Resource Rich Developing Countries: Cases of Bolivia, Kazakhstan, Mongolia, and Zambia, IEG, 2015. 14 World Bank Group Assistance to Low-Income Fragile and Conflict-Affected Countries, IEG, 2014. 15 Results and Performance of the World Bank Group, IEG, 2014. 16 The Poverty Focus of Country Programs: Lessons from World Bank Experience, IEG, 2015. 10 lessons and findings from IEG’s three learning products on results frameworks,17 selectivity in country strategy,18 and Bank-IFC cooperation.19 Purpose, Objectives, and Audience 34. Purpose: This evaluation is being undertaken at the request of the Board of Executive Directors, and will:  Provide information about how the SCD/CPF process is working.  Identify areas that can be strengthened to enhance country engagement throughout the production of SCDs and CPFs. 35. The findings and lessons of the evaluation will give particular attention to alignment with the twin goals, ownership, realism, selectivity, quality of results frameworks, coordination of Bank Group institutions, and inclusiveness (in the country and with other partners). 36. The assessment will focus on four dimensions for each SCD and CPF (see Attachment 3). For the SCD, these will be: evidence basis and framing of the issues; growth, poverty, shared prosperity, and sustainability; identification of priorities for achieving the twin goals; and process and one World Bank Group integration. For the CPF, the dimensions will be: alignment with country development priorities and the twin goals, and sustainability; country ownership, selectivity, realism, and systematic accounting of risks; quality of results frameworks; and one World Bank Group integration. 37. Objectives: The evaluation has three objectives:  To explore whether the guidelines, directives, other resource materials related to the new country engagement process have thus far helped Bank Group teams produce SCDs and CPFs that are focused on helping countries to tackle key constraints to poverty reduction and shared prosperity in accordance with the Bank Group’s comparative advantage and whether they have fostered coordination among the Bank Group institutions.  To provide the Board and management with early information about the functioning of the SCD/CPF process and the extent to which the shortcomings related to the previous processes have been addressed, so as to enable changes to improve the process.  To identify good practice examples and lessons learned thus far. 38. Audience: The main audiences are the Board and senior management of the Bank Group, as both have high hopes for the new country engagement model and are eager to see if the initial batch of SCDs/CPFs justifies those hopes. Moreover, the Board, management, and staff would like to have feedback about the process of SCD/CPF production across regions, which could contribute to improving the engagement with countries and the quality of SCDs and CPFs. Some 17 Results Frameworks in Country Strategies—Lessons from Evaluations, IEG, 2014. 18 Selectivity in Country Strategies: The Evidence, IEG, 2014. 19 Past and Future—Bank-IFC Cooperation at the Country Strategy Level, IEG, 2014. 11 regions may be interested in learning how others tackled selected issues involved in SCD/CPF preparation, with the intention of applying effective practices to their own regions. Task teams also could learn useful lessons from the experience of teams in other regions facing similar issues with implementation of the new country engagement model. Evaluation Questions and Coverage/Scope 39. The key evaluation questions are designed to explore the extent to which the new country engagement model enhances selectivity and Bank Group coordination in establishing evidence-based country priorities that are aligned with the twin goals. Three main questions are supported by multiple sub-questions (also see Attachment 3). 40. Question 1: Has the SCD identified priorities in a convincing analytical way that helped design a program of selective interventions under the CPF? 41. The new approach is expected to help ensure that country development objectives in the CPF are aligned with the Bank Group’s corporate strategy and comparative advantage. It begins with a new country diagnostic that is expected to provide a rigorous and timely analysis of a country’s development challenges and opportunities, particularly in relation to the twin goals, to underpin the subsequent partnership document. This first step is crucial for the new engagement model to work well and will therefore be the focus of the first evaluation question. Specifically:  Did the SCD use high quality analytics (including political economy analysis), and identify key challenges and opportunities related to accelerating the country’s progress toward development goals with primary impact on poverty and shared prosperity?  Did the SCD identify a limited number of areas on the critical path to achieving the twin goals, and thus inform strategic discussions with clients about priority areas for Bank Group interventions? Was the prioritization in the SCD specific enough to help guide selectivity in the CPF?  Did preparation of the SCD reflect the “one World Bank Group” objective? That is, was IFC an integral part of the team and did the SCDs reflect IFC and MIGA participation? Furthermore, were appropriate internal reviews conducted and were the comments and suggestions from those reviews addressed?  How was the engagement with Global Practices (GPs)? 42. Question 2: Has the CPF designed a program of lending and knowledge activities that aligns (not necessarily one-to-one) with country priorities and opportunities identified in the SCD and that makes the best use of Bank Group resources and comparative advantage, including joint World Bank–IFC-MIGA projects where appropriate? 43. Country Management Units are expected to produce a CPF focusing on key areas of intervention to achieve the twin goals. Per the guidelines, the CPF will start from the country’s vision of its development goals as defined in its strategic documents. The Bank Group and country counterparts will further draw upon the SCD to develop the CPF objectives together, deriving them from those country development goals that reflect the Bank Group’s comparative advantage as well as alignment with the twin goals, and taking into account opportunities for leveraging the private sector to provide sustainable solutions to development problems. The CPF 12 will then outline a selective and flexible program that will help the country achieve CPF objectives. This step in the new country engagement model will be the focus of the second question of the evaluation. Specifically:  Does the CPF present an assessment of the government’s program, identify divergent views between the country and the Bank Group, highlight country poverty and prosperity goals, and discuss how they align with the twin goals? Does it explain clearly and concisely the reasons why some key priorities from the SCD may not have been included in the CPF?  Is there evidence—taking into account the limited available sample of countries—of appropriate collaboration and division of labor between the Bank, IFC, and MIGA in working toward CPF objectives, and are IFC-MIGA interventions included in the results framework with appropriate, measurable indicators?  Were the criteria for selecting the portfolio under preparation discussed in the CPF, was the overall program contribution to CPF objectives explained, and was the framework based on outcomes (rather than outputs/inputs/processes) that are measurable and measured (with a system set up for measuring them)? Was the comparative advantage of the Bank in the selected activities explained?  Does the CPF articulate in its results framework—measured with relevant indicators— the links of Bank Group activities to poverty, shared prosperity, and sustainability?  Does the CPF propose analytic work that is consistent with the CPF objectives and SCD recommendations?  To what extent does the CPF consider major risks to program implementation— including national, international and Bank-related risks—and identify options to manage the most serious ones? 44. Question 3. Has the new country engagement model fostered country ownership, realism, and selectivity? 45. The CPF process is expected to be built around broad consultation and engagement with stakeholders. The voice of primary stakeholders, including government, citizen groups, and the private sector, are to be taken into account. In keeping with this emphasis, the new approach is to reflect extensive internal and external consultations, both to formulate the Bank Group strategy and to improve the effectiveness of country programs. The CPF process is expected to integrate learning more systematically than the previous CAS process, allowing country teams to adjust more readily to country needs and delivery experience. It is also expected to integrate systematic risk assessment, which was previously done in a perfunctory way in a number of CASs. Therefore, the third question of the evaluation will focus on aspects related to country ownership, realism, selectivity, and risk assessment, and also reflect on the Board’s engagement with the strategy process. Specifically:  Were there consultations with stakeholders, and were those consultations reflected in broad commitment (particularly from the government) and in the design of the strategy and interventions, with identification of any important differences of view between a government and the Bank? 13  Does the Bank Group country program match government capacity for implementation and is government commitment accurately assessed?  Taking into account the Bank’s financing constraints, do CPF objectives concentrate resources on areas that intersect with country development goals, contribute toward the attainment of twin goals identified in the SCD, and are consistent with the Bank Group’s comparative advantage?  Has the SCD identified the key knowledge and data gaps, and are these addressed in the knowledge program outlined in the CPF?  Does the Bank Group program explore synergies with other multilaterals and bilaterals and reflect the Bank Group’s comparative advantage? 46. Scope. The evaluation will cover SCDs and CPFs presented to the Board by the end of December 2015 (but if the current delivery program should prove optimistic, the evaluation might apply a later cut-off in order to ensure a reasonable universe of available documents). It will assess analytic quality and other attributes prescribed in the Bank guidelines for SCDs and CPFs. The evaluation will not discuss country-level development results—it is too early for that—but rather, focus on the process used to put together SCDs and CPFs. It will explore what lessons can be learned from the initial batch of SCDs/CPFs. Given the newness of the approach it will also be particularly important to assess the relevance of SCDs in the design of CPFs, in particular issues such as:  Whether the guidelines, directives, and technical resources present a good basis for Bank Group teams to prepare SCDs and CPFs. Whether the guidelines, directives, and current practices provide reasonable assurance that the SCD is an independent and objective piece of analysis, including in its prioritization of objectives.  What has it cost to prepare SCD/CPFs thus far, and how does the cost compare to that of the earlier country assistance strategy preparation? Was the incremental effort of producing two separate documents reflected in the quality of the CPFs? Was this incremental effort at the expense of other products? Has the internal World Bank Group coordination been productive? What is the source of any visible inefficiency? Based on the early information that may be available for the evaluation, do the documents give appropriate weight to private sector issues?  Was the new approach helpful in integrating the work of the three institutions (World Bank, IFC, and MIGA), their strategic approaches, and interactions with government authorities and private sector, taking into account the differences between the three institutions in their ability to plan their work programs? Evaluation Design 47. Analytical Framework: The evaluation will build on the Bank Group’s preparation guidelines and directives to assess the SCD and CPF preparation process and content. The evaluation team already arranged two workshops with the Bank Group and IEG staff to understand the new processes to help develop the methodology for reviewing SCDs and CPFs. Attachment 3 provides a more detailed outline of the assessment methodology. 48. The evaluation will use the following building blocks: 14  Systematic review of SCDs and CPFs. IEG will undertake a systematic desk review of SCDs and CPFs completed and submitted to the Board through December 2015. IEG will strive to include at least two countries in each region, and a total of about 20, subject to SCDs/CPFs being completed in each region by end December 2015 (or later if the Bank Group’s delivery program should be delayed). The desk reviews will evaluate SCDs and CPFs based on a specifically designed assessment template (see Attachment 3) and supplement the SCD/CPF review with other available evaluative evidence to form a comprehensive view on the content of SCDs and CPFs and their preparation process. Document reviews will be supplemented by key informant interviews to help gain insights into the SCD and CPF preparation process.  Field visits will be carried out, tentatively to 12 countries with ideally two countries in each region, with completed SCD and CPF. The consultations with country authorities and other stakeholders through these visits are very important to validate and expand the findings of the desk reviews and gain insights from country counterparts and Bank Group country team members.  Consultations within the Bank Group, including interviews with the Bank regional chief economists, global practices, IFC and MIGA chief economists, country directors and country representatives, and SCD and CPF team leaders.  Literature survey of relevant materials concerning the new country engagement model. 49. Sampling Strategy. The Bank Group expects to present SCDs and CPFs for 23 countries to the Board by the end of December 2015.20 Given this relatively small universe of products, the evaluation will strive to include all countries where both documents have been completed by the cut-off date. The evaluation will also note any limitations to the findings from the small initial universe of SCD-CPF countries. Countries for field-based case studies will be selected, to the extent possible, to ensure a mix of IBRD and IDA countries, balanced regional representation, and Bank Group engagement. Subject to completion of SCD/CPFs by the cut-off date, two countries per region will be selected for field visits. 50. The strength of the evaluation resides in its timing and ability to help Bank Group management take corrective action in the application of the new country engagement model if warranted or receive confirmation that the process used is in line with the Bank Group’s stated objectives and expectations. Since this is a real-time formative evaluation, the evaluation team will be able to examine Bank Group staff and country counterparts’ recent experience and capture the advantages and challenges encountered while they are still fresh in participants’ minds. The limitation of such an evaluation is that it will not be able to establish the extent to which the new approach is delivering high-impact development results in the pursuit of the twin goals. Therefore, this will by necessity be a partial assessment of the new engagement model. A 20The SCD/CPF preparation schedule suggests that SCDs and CPFs for 23 countries should be discussed by the Board through the end of December 2015, including 6 in Sub-Saharan Africa, 2 in East Asia and Pacific, 5 in Europe and Central Asia, 7 in Latin America and the Caribbean, 2 in Middle East and North Africa, and 1 in South Asia. 15 more definite evaluation on its development effectiveness will need to be undertaken once the initial and early repeat crop of CPFs have been completed and implemented. Quality Assurance Process 51. The evaluation will be overseen by Nick York, Director, and Monika Huppi, Acting Manager, of IEGCC. Peer reviewers will be James Adams (former World Bank Vice President for Operations Policy and Country Services [OPCS] and East Asia and Pacific Region), David Dollar (former World Bank Country Director for China), and Mwange Kimenyi (Director of Africa Growth Initiative, Brookings Institution). The critical skills required for this evaluation include a deep understanding of the process of Bank Group engagement with countries, how country partnership strategies are put together and implemented, including the macroeconomic analysis, growth, and poverty knowledge, substantial prior operational and evaluation experience, and in-depth knowledge on sustainability and country risk assessments. The highly experienced evaluation team possesses all these skills. 52. The evaluation team will be led by Ismail Arslan (extensive country, operational, and evaluation experience). The core team includes Monika Huppi (focus on evaluation design; extensive operational and evaluation experience with a broad range of Bank country engagements and lending operations across sectors and income groups); Juan-Jose Fernandez- Ansola (focus on evaluation design and SCD/CPF reviews; extensive knowledge on implementation of Bank country partnership strategies); Nils Fostvedt (focus on SCD/CPF assessments; extensive operational and evaluation experience on country engagement and economic and sector work); Xue Li and Kendra White (focus on economic and portfolio analysis; advanced skills in statistical methods and in data research and country portfolio analysis); Basil Kavalsky (focus on SCD/CPF assessments; former country director with broad experience in macroeconomic analysis and country engagement); Paul Levy (focus on SCD/CPF assessments; extensive operational and evaluation experience on country engagement and economic and sector work); Neeta Sirur (focus on poverty and human development; wide- ranging experience in country strategies in Latin American and Western Balkan countries); and Aurora Medina Siy (focus on evaluation design; extensive operational and evaluation experience on IFC /MIGA). The team will be supplemented with sector specialists as needed. Expected Outcomes and Dissemination 53. The output will be a process evaluation report for the Committee on Development Effectiveness (CODE) and the Board. Separate reports on the SCD and CPF assessments will be shared with the relevant country teams for their information and comments. The final evaluation report will be disclosed in the normal manner. The demand for special internal or external dissemination will be assessed at a later stage. 54. In light of the potential value of the results of this real-time process evaluation to the operationalization and roll-out of the Bank Group strategy through the new country engagement approach, the evaluation team will engage early with IEG’s communication and learning specialists with a view to developing an outreach strategy that will involve innovative means and methods through which key messages and learning can be effectively communicated and disseminated among all relevant Bank Group stakeholders and actors. 16 Resources 55. Timeline and budget. Work on SCD and CPF assessments is ongoing via pilot testing of the assessment templates. Scoping missions to Myanmar and Panama were completed in June 2015. The remaining field visits will take place between October 2015 and March 2016, with the exact schedule determined by the schedule of CPF approvals. The draft report will be shared with IEGCC management in June 2016 with a one-stop review meeting in July 2016 and finalization August–September 2016. However, if there should be delays in the completion of SCDs and CPFs, this timeline could slip modestly in order to ensure a reasonable universe of evaluable documents. 56. The budget needed for this evaluation is $1.250 million over two fiscal years (FY15-16). The desk reviews of SCDs and CPFs and field visits will constitute the core of the work and a large share of the total cost. The cost of field visits may be controlled to some extent through combining visits to more than one country on the same trip, and ensuring that relevant desk assessments have been completed prior to the visits. The project team will also review within IEG any possibilities for combining visits for different purposes. 17 Attachment 1: Bibliography IEG. 2011 World Bank Progress in Harmonization and Alignment in Low-Income Countries. ————. 2014a. Results Frameworks in Country Strategies—Lessons from Evaluations. ————. 2014b. Selectivity in Country Strategies: The Evidence. ————. 2014c. Past and Future—Bank-IFC Cooperation at the Country Strategy Level. ————. 2014d. World Bank Group Assistance to Low-Income Fragile and Conflict- Affected Countries. ————. 2014e. Results and Performance of the World Bank Group. ————. 2014f. Results Frameworks in Country Strategies – Lessons from Evaluations. ————. 2015a. Clustered Country Program Evaluation for Resource Rich Developing Countries: Cases of Bolivia, Kazakhstan, Mongolia, and Zambia. ————. 2015b. The Poverty Focus of Country Programs: Lessons from World Bank Experience. OPCS. 1998. Country Assistance Strategies: Retrospective and Outlook. ————. 2000. Country Assistance Strategies: Retrospective and Implications. ————. 2003. Country Assistance Strategies: Retrospective and Future Directions. ————. 2005. Results Focus in Country Assistance Strategies: A Stocktaking of Results-Based CASs. ————. 2009. Country Assistance Strategies: Retrospective and Future Directions. ————. 2014. Framework for Management of Risk in Operations: Proactive Risk Management for Better Results. World Bank Group. 2013. World Bank Group Strategy. ————. 2014a. Interim Guidelines for Systematic Country Diagnostics. ————. 2014b. A New Approach to Country Engagement. ————. 2014c. World Bank Group Guidance for Country Partnership Framework Products. ————. 2015a. World Bank Implementation Update. ————. 2015b. Systematic Country Diagnostic – An Update. 18 Attachment 2: Evaluation Design Matrix SPECIFIC EVALUATION QUESTIONS, SUB-QUESTIONS, AND DATA SOURCES Question Data sources Q1: Has the Systematic Country Diagnostic (SCD) identified priorities in a convincing analytical way that helped design a program of selective interventions under the Country Partnership Framework (CPF)? - Did the SCD use high quality analytics (including Interviews, document reviews, political economy analysis), and identify key case studies challenges and opportunities related to accelerating the country’s progress toward development goals with primary impact on poverty and shared prosperity? - Did the SCD identify a limited number of areas on the Interviews, document reviews, critical path to achieving the twin goals, and thus case studies inform strategic discussions with clients about priority areas for Bank Group interventions? Was the prioritization specific enough to help guide selectivity in the CPF? - Did preparation of the SCD reflect the “one World Interviews, document reviews, Bank Group” objective? That is, was IFC an integral case studies part of the team and did the SCDs reflect IFC and MIGA participation? Furthermore, were appropriate internal reviews conducted and were the comments and suggestions from those reviews addressed? - How was the engagement with Global Practices Interviews, document reviews, (GPs)? case studies Q2: Has the CPF designed a program of lending and knowledge activities that aligns (not necessarily one-to- one) with country priorities and opportunities identified in the SCD and that makes the best use of Bank Group resources and comparative advantage, including joint World Bank–IFC-MIGA operations where appropriate? - Does the CPF present an assessment of the Interviews, document reviews, government’s program, identify divergent views case studies between the country and the Bank Group, highlight country poverty and prosperity goals, and discuss how they align with the twin goals? Does it explain clearly and concisely the reasons why some key priorities from the SCD may not have been included in the CPF? - Is there evidence – taking into account the limited Interviews, document reviews, available sample of countries- of appropriate case studies collaboration and division of labor between the Bank, IFC, and MIGA in working toward CPF objectives, 19 and are IFC-MIGA interventions included in the results framework with appropriate, measurable indicators? - Were the criteria for selecting the portfolio under Interviews, document reviews, preparation discussed in the CPF, was the overall case studies program contribution to CPF objectives explained, and was the framework based on outcomes (rather than outputs/inputs/processes) that are measurable and measured (with a system set up for measuring them)? Was the comparative advantage of the Bank in the selected activities explained? - Does the CPF articulate in its results framework— Interviews, document reviews, measured with relevant indicators—the links of Bank case studies Group activities to poverty, shared prosperity, and sustainability? - Does the CPF propose analytic work that is Interviews, document reviews, consistent with the CPF objectives and SCD case studies recommendations? - To what extent does the CPF consider major risks to Interviews, document reviews, program implementation—including national, case studies international, and Bank-related risks—and identify options to manage the most serious ones? Q3: Has the new country engagement model fostered country ownership, realism, and selectivity? - Were there consultations with stakeholders, and were Interviews, document reviews, those consultations reflected in broad commitment case studies (particularly from the government) and in the design of the strategy and interventions, with identification of any important differences of view between a government and the Bank? - Does the Bank Group country program match Interviews, document reviews, government capacity for implementation and is case studies government commitment accurately assessed? - Taking into account the Bank’s financing constraints, Interviews, document reviews, do CPF objectives concentrate resources on areas that case studies intersect with country development goals, contribute toward the attainment of the twin goals identified in the SCD, and are consistent with the Bank Group’s comparative advantage? - Has the SCD identified the key knowledge and data Interviews, document reviews, gaps, are these addressed in the knowledge program case studies outlined in the CPF? - Does the Bank Group program explore synergies Interviews, document reviews, with other multilaterals and bilateral and reflect the case studies Bank Group’s comparative advantage? 20 Attachment: 3 Methodology Note for Review of SCDs and CPFs QUESTIONNAIRE GUIDANCE 1. The guidance to the attached Basic Template Questionnaire is based on the guidelines for SCD and CPF preparation, the Bank Group strategy of October 2013, and a note on the new approach to Bank Group country engagement (see references in footnotes 10, 11, and 12). It takes into account the application of the questionnaire to a few pilot cases, lessons learned from those pilots, as well as lessons learned by IEG and presented in learning notes on results frameworks, selectivity, and World Bank–IFC cooperation (see footnotes 17, 18, and 19). The questionnaire will enable panelists to assess SCDs and CPFs systematically. SYSTEMATIC COUNTRY DIAGNOSTICS 2. Dimensions to be assessed in the SCD review:  Evidence basis and framing of the issues  Growth, poverty and shared prosperity, and sustainability  Identification of priorities for achieving the twin goals  Process and one World Bank Group 3. The attached template provides the questions to be answered for each dimension. The evaluators will rate each dimension using a 6-point scale based on criteria set out in the attached Rating Scale Table. 4. Evidence basis and framing of the issues will assess the quality of the background analytical work and data the SCD used to frame the main issues and challenges facing the country in question. It also will assess how the SCD identified and addressed data gaps and weaknesses found during its production. 5. Growth, poverty and shared prosperity, and sustainability will assess whether the SCD convincingly shows the key opportunities and constraints the country faces. It will assess whether sources of growth are properly identified, their impact on poverty and inclusiveness, and prospects of economic growth (fiscal and debt), social, and environmental sustainability. These issues will be analyzed against the background of the twin goals. 6. Identification of priorities for achieving the twin goals will assess whether the SCD identified some key areas (sufficiently narrow in scope to reflect an effective selectivity process) on the critical path to reducing poverty and enhancing shared prosperity, including progress achieved so far. The review will assess whether priorities are aligned to government development priorities, while spelling out quantitative aspirations for poverty reduction and shared prosperity. The report will assess whether future priorities are in the context of growth, inclusiveness, and sustainability. This dimension also will assess whether the SCD had an adequate discussion of trade-offs between short- and long-term goals. 7. Process and one World Bank Group will assess whether the production of the SCD, and any consultation, reflected a unified approach by the World Bank, IFC, and MIGA to underpin 21 the new country engagement model. It also will assess the quality of corporate review, and the adequacy of the Bank Group’s interactions with the government and civil society stakeholders during SCD preparation. COUNTRY PARTNERSHIP FRAMEWORKS 8. Dimensions to be assessed in the CPF review:  Alignment with country development priorities and the Bank Group’s opportunities to achieve the twin goals identified in the SCD, and sustainability  Country ownership, realism, and selectivity  Quality of results framework and systematic risk assessment  One World Bank Group 9. As with the SCD, for each dimension of the assessment the evaluators will provide a rating on the 6-point scale based on criteria set out in the attached Rating Scale Table. 10. Alignment with country development priorities and the twin goals, and sustainability will assess whether the CPF (following its guidance note) found the proper intersection between country development goals and opportunities to achieve the twin goals identified in the SCD when putting together the Bank Group interventions and the results framework of the program. Clarity of the country’s development strategy and definition of its priorities in the CPF will be key in that assessment. The sustainability of policies also will be assessed in this section. 11. Country ownership, realism, and selectivity will assess the extent to which the CPF took advantage of consultations with stakeholders—and particularly with the government—to develop a program whose success is enhanced by an adequate assessment of country capacity and government commitment to the policies. This section will also assess selectivity by evaluating whether resources were concentrated in areas representing an adequate intersection of country development goals, the twin goals, and the Bank Group’s comparative advantage. This section will also assess whether proposed analytic work is relevant and consistent with CPF priorities and SCD findings. Moreover, it will assess the extent to which the CPF adequately considers major risks to the implementation of the Bank Group program and options to manage the most serious ones. 12. Quality of results framework and systematic risk assessment will assess the appropriateness and realism of the results framework, focusing on the results chain running from Bank Group interventions to specified outcomes. It will also assess whether objectives are achievable, outcome indicators are appropriate to measure progress toward the objectives, and whether they are measurable and tracked through a system set up for measuring them. 13. One World Bank Group will assess evidence of appropriate collaboration between the World Bank, IFC, and MIGA in working toward CPF objectives, and whether the results framework offers evidence of joint interventions by these institutions 22 Documentation 14. The IEG reviewers will have access to relevant documentation for the SCD and CPF review. The main documents are the following:  Concept Notes for SCD and CPF  Systematic Country Diagnostic Decision Draft  Systematic Country Diagnostic Final Report  Country Partnership Framework Decision Draft  Country Partnership Framework Board Version  Minutes of the Operations Committee (OC) and Regional Operations Committee (ROC) meetings for the SCD and CPF  Comments provided in the context of the OC and ROC meetings  Latest IMF Article IV Consultation  Other relevant recent reports INTERVIEWS WITH MANAGEMENT, STAFF, BORROWER, STAKEHOLDERS, AND PARTNERS 15. After the desk review and initial assessment of the SCD and CPF, the reviewers will identify information gaps and determine whether interviewing specific people is necessary, specifying the proposed agenda and questions. The interview can take the form of e-mails, phone calls, face-to-face meetings, or short questionnaires. During field visits, the IEG team will meet stakeholders in the country (government, disenfranchised groups, academia, private sector, NGOs, etc.) and multilateral and bilateral partners, to validate consistency of views reflected in the SCD and CPF documents. The task manager(s) for the SCD/CPF evaluation will sign off on all the interviews. QUALITY ASSURANCE OF THE SCD/CPF ASSESSMENTS 16. The IEG staff or consultant conducting the SCD/CPF review will be responsible for the quality of the assessments. [Some assessments may be peer reviewed.] Basic Template for Systematic Country Diagnostic and Country Partnership Process Evaluation I. SYSTEMATIC COUNTRY DIAGNOSTIC (SCD) A. Evidence basis and framing of the issues  To what extent is the SCD providing a basis for outlining the main issues and challenges for the country?  Are gaps in evidence and areas where additional work is required identified during the SCD preparation period or during the CPF?  Are the quality of the data discussion and the steps that have been taken to address data gaps and weaknesses, including the proxies used for missing data, appropriate?  Are there synergies on the priorities and opportunities identified across the themes of growth, inclusion, and sustainability? 23 B. Growth, poverty and shared prosperity, and sustainability  To what extent is the SCD identifying key opportunities for overall economic growth?  To what extent is the SCD analyzing adequate and equal opportunities, especially the distribution of the gains from growth among the poor and the bottom 40 percent, and basic elements of social well-being and shared prosperity?  Does the SCD look sufficiently into sustainability issues (fiscal and debt, social, and environmental)?  Are significant threats to social and political stability adequately analyzed? C. Identification of priorities for achieving the twin goals  Does the SCD identify limited areas that are on the critical path to achieving the goals?  Are the priorities for poverty reduction and shared growth clearly spelled out in quantitative terms and are they realistic?  Is the results chain needed to achieve the goals clearly specified and consistent with the objectives?  Is there a discussion of possible trade-offs between short- and long-term goals?  To what extent is governance (institutional arrangements and capacity, transparency, and accountability) addressed in the SCD?  Does the SCD distinguish clearly between the most important priorities and other, less critical constraints?  Does the SCD effectively address cross-cutting issues? D. Process and one World Bank Group  How generally did the processes go, including whether there were any issues from the relative timing of the SCD and the CPF?  Was the corporate review process of adequate quality?  Was the preparation of the SCD reflective of the one Bank Group objective, with inputs sought from IFC and MIGA?  Were the SCD findings and priorities informed by stakeholder consultation?  Were stakeholder comments presented in the report?  Were those who participated in consultation informed as to how the Bank has dealt with their comments and suggestions?  Were time constraints important for the quality of the report? II. COUNTRY PARTNERSHIP FRAMEWORK (CPF) A. Alignment with country development priorities and the twin goals and sustainability  Does the CPF present an assessment of the government’s program and make clear any divergence of views between the country and the Bank Group?  Does it highlight any country poverty or prosperity goals and discuss how they align with the twin goals? 24  Does the CPF provide a concise description of the country’s poverty and income distribution patterns and trends, including at least poverty trends using national and international poverty lines, inequality measures, and income/consumption growth of the bottom 40 percent of the population?  Does the CPF present information on the causes of poverty, including the major obstacles to poverty reduction and the set of structural and social elements that are essential to poverty reduction?  Is there a discussion of any progress in poverty reduction since the last strategy?  Is there an analysis of the linkages between poverty reduction and the level and pattern of growth in the country?  Is there an assessment of the country’s capacity to monitor poverty indicators?  Are links to poverty, shared prosperity, and sustainability (fiscal, social, environmental) articulated well in the CPF results framework? B. Country ownership, realism, and selectivity To what extent:  Were there consultations with stakeholders, and were those consultations reflected in broad commitment (particularly from the government) and in the design of the strategy?  Is the Bank Group’s country program matched to country capacity and appropriately estimated government commitment?  Are the program objectives consistent with the Bank Group’s comparative advantage?  CPF objectives concentrate resources into critical areas where they will have the most development impact (intersection of country development goals, twin goals, and Bank Group comparative advantage)? C. Quality of results framework and systematic risk assessment  Does the CPF articulate in its results framework—measured with relevant indicators— the links of Bank Group activities to poverty, shared prosperity, and sustainability?  Does the framework emphasize outcomes over outputs/inputs/processes?  Are all outcomes measurable and measured (with a system set up for measuring them), including through indicators?  Do indicators have quantified baselines and targets?  Do the chosen indicators adequately measure final outcomes (say, quantity or quality) and have the proper scope (say, same geographic coverage as the final outcome)?  Are the indicators time-bound and consistent with time-dimension of the CPF’s targeted outcomes?  Are results chains from interventions to outcomes explained sufficiently, including any scaling-up of interventions expected to achieve the targeted outcomes? Are obstacles to be overcome, logical causality, assumptions, and risks highlighted? 25  Were the criteria for selecting the ongoing portfolio and the portfolio under preparation discussed explicitly in the CPF, and was the overall program contribution to CPF objectives explained?  Is proposed analytic work relevant and consistent with CPF priorities and SCD findings?  To what extent does the CPF consider major risks (domestic, international, and Bank- related) and options to manage the most serious ones? D. One World Bank Group  Is there evidence of appropriate collaboration and appropriate division of labor between the Bank, IFC, and MIGA in working toward CPF objectives?  Is there evidence of smooth working of the Regional Coordination Mechanism, with joint implementation work (World Bank, IFC, MIGA) where relevant, and timely inclusion of IFC and MIGA in joint work?  How meaningful was donor coordination in the preparation of the CPF?  Are there program complementarities (World Bank, IFC, MIGA) in achieving common goals, and are those complementarities expressed in the results matrix?  Is IFC working on poverty issues? Are IFC and MIGA interventions included in the results framework with appropriate, measurable indicators?  Is there evidence of joint work World Bank, IFC, MIGA in specific interventions? 26 Rating Scale for SCD/CPF Process Questionnaire For internal evaluation purposes, the responses to the questions will be rated on a six-point scale using the criteria in the table below. This rating system is a tool for the aggregation and reporting of specific evaluation findings. The ratings on individual questions will then be aggregated to arrive at a rating for each block of issues under the SCD process and the CPF process. The rating for each block will be a simple average of the ratings for each question under the block, giving equal weight for all the questions in the block; the possibility of using unequal weights may be considered during the evaluation if warranted by special issues. SCD/CPF Process Questionnaire Rating Scale Question/ Criteria for the rating Numerical block rating rating Highly Exceptional attainment on the issue raised by the question or 6 satisfactory dimension. No shortcomings identified. Satisfactory Good attainment on the issue raised or dimension. No major 5 shortcomings identified. Moderately The issue or dimension was addressed but not 4 satisfactory comprehensively, or was addressed in a way not fully helpful for the development of the SCD or CPF. Some shortcomings identified. Moderately The issue or dimension was addressed in a way that was not 3 unsatisfactory helpful for the SCD or CPF. Several shortcomings identified. Unsatisfactory The issue or dimension was addressed perfunctorily, 2 representing a detriment to the quality of the SCD and CPF. Many shortcomings identified. Highly The issue or dimension was not addressed by the SCD or CPF. 1 unsatisfactory 27 Attachment 4: IEG Learning Notes on Selectivity; Bank-IFC Cooperation; and Results Frameworks SELECTIVITY Introduction 1. To operationalize the twin goals, the Bank Group has developed a more evidence-based and selective country engagement model, the Country Partnership Framework (CPF). While the CPF is sharpening the Bank Group’s focus on strategic selectivity in its country programs, the issue is longstanding. For example, OPCS’s 1998 Country Assistance Strategies: Retrospective and Outlook (CAS Retro I) devoted considerable attention to the issue of strategic selectivity, providing a quantitative analysis of sectoral selectivity, defining strategic selectivity as a matter of qualitative judgment based on three criteria, and recommending specific actions by OPCS, the Networks, and country teams in order to strengthen strategic selectivity and focus in the CASs. The two subsequent CAS Retrospectives (2000 and 2003) reported an improvement in the treatment of selectivity in CASs. OPCS noted that although most CASs considered one or more selectivity dimensions outlined in CAS Retro I, many did not fully distill their implications for the design of Bank Group programs and that the selectivity improvement was mostly in discussing the rationale for choosing one instrument over another. With the adoption of results-based country strategies, selectivity is viewed as “a natural by-product of the careful construction of a results framework” (CAS Retro IV, 2005), and the CASs are judged as having become “more selective and focused” and “broadly satisfactory in prioritizing areas for Bank support” (CAS Retro V, 2009). Key Findings  The Bank Group is paying significant and increasing attention to selectivity. Many country strategies explicitly address selectivity, articulate specific criteria for exercising selectivity, and draw lessons from the past for improving selectivity.  Selectivity has improved over time. The Bank Group’s country strategies have become more selective over time.  Selectivity is positively correlated with country program outcomes based on empirical analysis from CASCR Reviews. In addition to country ownership and good results frameworks, selectivity is correlated with better country outcomes. Moreover, selectivity is more important in countries with higher levels of extreme poverty, which emphasizes the importance of selectivity in achieving the twin goals.  Selectivity is likely to be contextual. There are no simple criteria to determine the size and composition of a program; that is, the number of objectives and number of activities cannot be predetermined. What matters is the fit between design and context.  While selectivity is multidimensional and complex, evidence from CPEs and country strategies suggests that selectivity can be enhanced by paying attention to client country capacities, grounding all operations in solid analytical work, taking a long-term view, and sequencing interventions, properly factoring in design and implementation risks to results delivery, and using the Bank’s comparative advantage. 28 BANK-IFC COOPERATION AT THE COUNTRY STRATEGY LEVEL Introduction 2. The needs of Bank Group clients have been changing as the private sector increasingly becomes the engine of growth, and government attention is shifting from public projects to dealing with the growing private sector by regulating it, by establishing partnerships, or by transferring certain economic activities to it. In this new landscape, the best way to optimize the Bank Group’s development impact, promote its overarching goal of eliminating extreme poverty and boosting shared prosperity in a sustainable manner is to put the array of private sector instruments to full use, bringing enhanced cooperation between the Bank and IFC at the country level to the forefront. As such, the two are expected to work more closely together from diagnosis to strategy formulation, solutions design, execution, evaluation, and learning. This would help ensure that the private sector’s role in addressing the challenges of development is fully embedded and the Bank Group’s assistance is integrated by capturing the benefits of knowledge exchanges and joint operations across the Bank and IFC. Experience has shown that a realistic and selective approach to Bank-IFC cooperation, based on appropriate resource allocation and staff incentives, may yield significantly better outcomes. Thus, the challenges of the new CPF process are to identify where and when cooperation is likely to improve efficiency and development outcome; redefine job descriptions of various administrative units and reassign existing staff resources; and provide staff incentives for joint work. Key Findings  Despite the increase in the number of “joint” country strategies, cooperation between the Bank and IFC has varied significantly across countries, with the majority of country strategies failing to include specific proposals to implement Bank Group cooperation.  There are structural reasons for the low levels of cooperation at the country strategy level: IFC’s business is determined by the market demand for its products, which is inherently difficult to plan for; there are concerns about conflicts of interest; IFC’s strategist/economist resources are extremely limited; and staff incentives may need to be tailored to encourage cooperation.  Selective Bank-IFC cooperation has the potential to improve the effectiveness and efficiency of Bank Group operations, and improve its development impact in client countries, while lack of cooperation can hinder or reduce potential benefits to clients, lead to duplication of activities, and ultimately raise operating costs.  Truly joint country strategy teams have led to better coordination and helped define the respective roles of the two institutions. Professional relationships between the staff members of the Bank and IFC have facilitated knowledge exchange and readiness to work together. However, Bank-IFC cooperation has been ad hoc.  Cooperation between the Bank and IFC is not always necessary or productive for every sector in a country. Elevating cooperation, which is an instrument, to the level of a goal on its own, may generate unnecessary processes, hence inefficiency. Benefits of cooperation depend on the sector and the stage of its development in a country. Since the cost of cooperation may sometimes outweigh the benefits, careful cost-benefit analysis of cooperation is warranted at the early stage of new Bank Group country strategy formation. 29 RESULTS FRAMEWORKS IN COUNTRY STRATEGIES Introduction 3. In 2005, the results framework was introduced in the CAS as a tool for improving the quality of the strategy in borrowing member countries, maximizing the development effectiveness of Bank Group assistance, and demonstrating measurable results of international aid in fostering growth and reducing poverty. Since then, progress has been achieved in developing a results framework in the Bank Group’s country strategies. Today, all country strategies discussed at the Board have a results framework, usually in the form of a matrix in the annex. The results-based approach has brought several benefits, including the focus on results, better alignment between the Bank Group country engagement and national priorities, and flexibility in instruments to accommodate programming for the increasingly diverse set of Bank clients, including IDA, IBRD, and fragile countries. Despite this progress, IEG’s CPEs and CASCR Reviews have highlighted a few pitfalls in the results frameworks. Common deficiencies include a focus on outputs instead of outcomes, a weak link between designed interventions and outcomes, and the absence of monitoring indicators to track outcomes. Several recent IEG evaluations and reports have demonstrated that a weak CAS results framework is a key determinant of unsatisfactory outcome performance at the country program level. Key Findings  As a management tool, a good results framework has two interlinked but distinct components that serve different purposes: a results chain that explains how the Bank Group program would achieve each objective and contribute to the country goals that the Bank Group chooses to support; and a results matrix that provides the metrics for assessing the achievement of the objectives.  A strong results chain should be explicit about the underlying assumptions that the Bank Group is making with regard to the actions by other development partners and external factors. Clarity about the assumptions, and thus the associated risks, gives strength to a results chain. In selecting engagement areas and setting objectives, considerations should be given not just to the country needs and priorities, or diagnosis of potential solutions, which are clearly important, but also to the time needed to produce results, the resource constraints of all partners, and other factors that may affect the delivery of results.  Bank Group teams should ensure that “the whole is more than the sum of the parts;” that individual sector and objective decisions are related to the overall shape of the program; that there is a dialogue with the country on policy and strategic direction; that the country context and its own objectives are reflected; that the capacity of the country and the Bank Group team are considered; and that proper selectivity and sequencing are applied. The role of other stakeholders—the government, development partners, civil society, the private sector, and beneficiaries—must also be factored in as their involvement and agreement are critical to the exercise and ultimately determine the success of the country assistance program.  For a results framework to be an effective management tool, it should be current. All parts of the results framework should be subject to regular review and update. 30 Attachment 5: Report to the Board from the Committee on Development Effectiveness Sub-Committee Report Meeting of September 14, 2015 The Sub-Committee (SC) of the Committee on Development Effectiveness (CODE) considered IEG’s approach paper on Systematic Country Diagnostic and Country Partnership Framework Process Evaluation (CODE2015-0047). The Sub-Committee welcomed the approach paper on the Systematic Country Diagnostic (SCD) and Country Partnership Framework (CPF) Process Evaluation. They appreciated the real time assessment, noting it will provide early signals to enable timely course corrections. Members acknowledged the limitations of this early assessment and emphasized the importance of maintaining flexibility in the timing to allow for a sufficient number of SCD and CPFs to be considered in the exercise to provide a reasonable evidence base. They requested IEG to include in the evaluation the client perspective and how the WBG takes government priorities into account. Members expressed interest in learning how priorities are determined when there is a mismatch between the government’s interests and the most pressing constraints to achieving the corporate goals, which could potentially affect ownership at implementation. Members cautioned about drawing specific conclusions from this analysis. They noted that county and regional sensitivities should be kept in mind and that conclusions should be linked to regional realities. Members emphasized the need for an appropriate set of skills in the evaluation team, including staff who have an understanding of the country realities. They appreciated IEG’s explanation that the team would include IFC and MIGA experts, strong economists, and experts who have worked across a number of areas and countries; IEG added that attention will be given to getting local country specific expertise, within the budget available. The Committee was pleased to learn that the cross-cutting dimension of sustainability will be a key part of the evaluation and that the linkages between and the roles that regional teams, CCSAs and Global Practices play in the development of SCDs and CPFs will be addressed. They asked IEG to identify obstacles that prevent coordination within the WBG, and circumstances when the SCD/CPF required a variation in the approach. Members encouraged IEG to categorize and draw lessons, if possible, of the different approaches that have been taken. Acknowledging that SCDs priorities go beyond what the WBG can do, members encouraged IEG to contact as many stakeholders as possible to verify that the diagnostics have been shared with client countries, stakeholders and partners who could potentially carry out the activities that the WBG will not undertake. They also asked IEG to assess whether SCDs have been translated into local languages and if a dissemination strategy has been put in place to share these documents with a wider audience. Members encouraged IEG to look at the quality of the consultations and the extent to which civil society has had the opportunity to provide input. Members noted that the Approach Paper focuses on countries subject to SCD and CPF and not on countries subject to Country Engagement Notes. IEG was asked to work with OPCS to find a way to include these countries, as this would provide valuable information and knowledge.