83328 Program-for-Results Two-Year Review: Concept Note November 11, 2013 Operational Policy and Country Services ABBREVIATIONS AND ACRONYMS ACGs Anticorruption Guidelines BP Bank Procedure CODE Committee on Development Effectiveness CPS Country Partnership Strategy DLI Disbursement-linked Indicator DPL Development Policy Financing ECR External and Corporate Relations ESSA Environmental and Social Systems Assessment HDN Human Development Network IBRD International Bank for Reconstruction and Development IDA International Development Association IEG Independent Evaluation Group INT Integrity Vice Presidency IPF Investment Project Financing ISR Implementation Status and Results Report MDG Millennium Development Goals OP Operational Policy OPCS Operations Policies and Country Services PforR Program-for-Results PORG World Bank Public Opinion Research Group PREM Poverty Reduction and Economic Management RBA Results-based approach SDN Sustainable Development Network WBG World Bank Group PROGRAM-FOR-RESULTS TWO-YEAR REVIEW: DRAFT CONCEPT NOTE CONTENTS I. Introduction ......................................................................................................................... 1 II. Initial Experience with the PforR Instrument ..................................................................... 2 III. Proposed Structure and Content of Report ......................................................................... 8 A. Chapter III. Overview of Progress ............................................................................ 8 B. Chapter IV. Experience with Specific Features of PforR Operations ....................... 9 C. Chapter V. Internal Organization and Management ............................................... 14 D. Chapter VI. Emerging Lessons and Recommendations.......................................... 15 IV. Methodology and Process ................................................................................................. 16 V. Issues and Next Steps........................................................................................................ 18 Annex A. Management Proposal to the Board on Initial Implementation Review for Program- for-Results Operations ...................................................................................................... 19 Annex B. Suggested Outline of the Review ................................................................................. 21 PROGRAM-FOR-RESULTS TWO-YEAR REVIEW: DRAFT CONCEPT NOTE I. Introduction 1. Features of PforR. In early 2012, the World Bank introduced a new lending instrument, Program-for Results (PforR), with several innovative features: it would (a) finance and support borrowers’ programs of expenditures and activities, (b) disburse against achievement of program results rather than against inputs, (c) focus on strengthening institutional capacity to implement the program, and (d) provide assurance that the Bank’s financing is used appropriately and that the environmental and social impacts of the program are adequately addressed. In addition, the instrument was intended to promote working in partnerships.1 2. PforR in the Context of the WBG Strategy. At the time of its approval, PforR was envisaged as an instrument for the future. Its focus on results, and support of government programs and capacity, represented a new approach to financing development. Since its approval, the World Bank Group (WBG) Governors endorsed two goals to guide the institution and a new WBG Strategy which maps out broad directions for the way forward. The PforR instrument is fully in line with the WBG Strategy and can be a critical element of Strategy implementation. By emphasizing delivery of government programs, the PforR epitomizes problem solving engagement and "moving from a project mentality to a development solutions culture" which is a cornerstone of the Strategy. The PforR also has an explicit objective to foster partnerships in support of borrower programs--also central to the Strategy--recognizing that the Bank and Governments must align all partners toward common goals, deliverables and results. With its ability to measure results, the PforR represents an approach to accountability that is key to the WBG Strategy and critical to effective development support. 3. Initial Review. The Board paper recommending approval of the new instrument stated, “After about two years of implementation, Management will carry out an initial review of experience with the new instrument.” The Board paper recognized that it would take time for programs to be prepared, approved, and launched, and that implementation experience in the first two years would be limited. Nonetheless it was expected that some valuable lessons could be learned from an initial review of the preparation of operations and of early implementation experience; an annex to the Board paper listed some of the questions the review might address and outlined some of the methodologies that might be used (see Annex A). 4. Results-based Approaches. The PforR review will be informed by and will contribute to the ongoing dialogue on results-based approaches (RBAs) to development support. The design of the instrument drew on a broad range of experience with results-based aid and financing. Since Board approval of the PforR, the Asian Development Bank has introduced a new results-based lending instrument, several bilateral development partners are piloting RBAs, and other multilateral and bilateral agencies are considering introducing such 1 A New Instrument to Advance Development Effectiveness: Program-for-Results financing”, December 29, 2011, R2011-0282, IDA/R2011-0350. 1 approaches. Additional work on the merits of different RBAs has also been carried out by the Center for Global Development and other research institutions. 5. Objectives of Review. The two major objectives of the review are to: (a) assess the early experience with the design and implementation of PforR operations and the challenges faced by borrowers, development partners, and Bank staff; and (b) identify emerging early lessons and recommend any proposed changes to the PforR framework that would strengthen the instrument. These objectives reflect the discussions with the Board at the time the instrument was approved. The review will also be an opportunity to make any changes needed in light of the recent changes across the institution. 6. Approach to Review. To help ensure rigor, specificity, and credibility, the review will draw on a detailed review of available documentation and on feedback from governments, development partners, other stakeholders, and Bank managers and staff (elicited through surveys, interviews, and consultations). The review takes place while clients, development partners, and Bank staff are still learning how to use the PforR instrument. In this early phase, it is important to allow room for experimentation and sharing of diverging viewpoints. It is proposed that the review be conducted in this spirit, not focusing on making pass/fail judgments, but rather stressing emerging good practices, identifying systemic weaknesses, and proposing any improvements needed to help institutionalize the instrument. The review will rely on a range of methodologies including literature and desk review, surveys, structured, in- depth interviews, and consultations. The process will also involve the engagement of an Independent Expert Panel (see section IV for more details). 7. Coverage and Timing. The review will cover all PforR operations approved and under preparation at the end of October 2013: eight approved operations (six in IDA countries and two in IBRD countries) and 16 under preparation. 8. Outline of the Concept Note. Following this introduction, Section II discusses the initial experience with the PforR instrument since its approval by the Board in January 2012. Section III provides the proposed content and structure of the review. Section IV discusses the proposed methodology and process. Section V concludes with issues and next steps. Annex A shows how the review was expected to be conducted at the time of Board approval of the instrument and Annex B outlines the review. II. Initial Experience with the PforR Instrument 9. How has the instrument been used so far? The eight PforR operations that have been approved by the Board provide $1.4 billion of Bank financing to support a total of $4.1 billion in government programs (see Table 1). The 16 operations under preparation represent over $2.4 billion of Bank financing. About 63 percent of the approved and proposed financing is in 12 IDA countries (including blend countries), and the rest in six IBRD countries. The total volume of approved operations by the end of the initial two years is expected to be about $2.1 billion, below the cap of five percent of total IBRD/IDA commitments that the Board 2 approved.2 Since the instrument’s inception, PforR -related commitments have been increasing significantly (see Table 2). Table 1: Summary of Approved PforR Operations (US$, million) Board IDA/ PforR Country Program approval Sector IBRD Program Bank financing Ethiopia Health Millennium 2/28/2013 Health IDA 676 120 Development Goals Kenya National Integrated 7/23/2013 Social IDA 953 250 Safety Net Protection Morocco National Initiative for 6/28/2012 Social IBRD 1,115 300 Human Development Development (INDH) Phase II Nepal Results-Based Bridges 6/28/2012 Transport IDA 148 60 Improvement and Maintenance Tanzania Urban Local Government 10/23/2012 Urban IDA 255 255 Strengthening Uganda Support to Municipal 3/28/2013 Urban IDA 160 150 Infrastructure Development Uruguay Road Rehabilitation and 11/13/2012 Transport IBRD 510 66 Maintenance Vietnam Results-Based Rural 11/1/2012 Water & IDA 260 200 Water Supply and Sanitation Sanitation Table 2: IBRD/IDA Lending, Total and for PforR Operations, FY12 to FY14 (US$, billion) Commitments FY12 FY13 FY14* IBRD lending under PforR operations 0.30 0.07 0.99 IDA lending under PforR operations 0.06 0.71 1.35 Total IBRD/IDA lending under PforR operations 0.36 0.77 2.35 Total IBRD lending 20.37 14.79 16.70 Total IDA lending 14.31 16.24 17.60 Total IBRD/IDA lending 34.67 31.03 34.30 *: Total expected IBRD/IDA lending data for FY14 to be updated upon release of the Q1FY14 Quarterly Lending and Disbursement Projections Report. 2 At the time of the Board’s approval of the new PforR instrument, the Board approved Management’s proposal to limit IBRD/IDA commitments for PforR operations to five percent of total IBRD/IDA commitments for the first two years. It was also agreed that Management would propose lifting this cap if justified by a review of implementation experience. 3 10. All Regions have PforR operations approved Figure 1: Pipeline and approved Operations or under preparation; the Africa and South Asia by Region Region leads the way, with seven and five operations (financing amount) respectively. PforR operations cover most of the key sectors in which the Bank traditionally provides financing: the Sustainable Development Network (SDN) has 13 operations and about 60 percent of proposed financing, the Human Development Network (HDN) has seven operations and about 23 percent of proposed financing, and the Poverty Reduction and Economic Management network (PREM) has four operations and about 17 percent of proposed financing (see Figures 1 and 2). Figure 2: Pipeline and approved Operations by Sector 11. All eight approved operations emphasize the (financing amount) importance of partnerships with other development partners. Three are being co-financed with other donors, and another three were prepared in close collaboration with partners. Under the Ethiopia operation, for example, World Bank support for the Millennium Development Goals (MDG) fund for the health sector is being complemented by support from nine other multilateral and bilateral partners. 12. Beyond these basic data, the review will (a) endeavor to gauge how efficiently the instrument has been used so far (in terms of processing times, costs etc.); a key issue in this respect will be to define suitable comparators; and (b) study the dynamics involved in the choice of instrument, including by examining operations that were initially expected to use the PforR instrument that ended up using the IPF or DPF instruments, and those that were initially slated to use one of the other instruments and ended up as PforR operations. It will be particularly interesting to analyze the trade-offs made by client countries and the Bank in choosing one type of operation over another. 13. Do client countries know about the instrument, and do country strategies reflect that knowledge? Eighteen countries are now using or actively preparing a PforR operation. Several other countries are also considering it. For instance, of the 54 Country Partnership Strategies (CPSs) discussed by the Board since the approval of the PforR instrument, 18 (33 percent) refer explicitly to the use of PforR lending.3 Many of these strategies emphasize the potential value of the PforR instrument in focusing more directly on enhancing the quality (rather than the quantity) of government services, helping strengthen government institutions and systems and build capacity, focusing more directly on results, and improving coordination among development partners. Some strategies envision “piloting” the PforR instrument in particular 3 This is below the 65 percent of strategies approved during 2008-10 that proposed some form of programmatic lending/sectorwide approach. It will be important to analyze whether there has been an overall reduction in proposals for programmatic lending or whether a distinction is being made between PforR operations and other forms of programmatic lending that will continue to be processed using the investment lending instrument. 4 sectors and operations before determining its potential broader value-added. The CPSs also consider a range of scenarios for the use of PforR relative to other Bank instruments: some intend to use all three of the Bank’s lending instruments, others intend to move away from development policy lending toward greater use of PforR operations (as part of a shift in focus from policy to institution building and system strengthening), and others see PforR as building on work already underway in the context of sectorwide approaches and results-focused investment lending. The review will help understand these dynamics. The review will also survey client countries that are not yet using the PforR instrument to ask about their potential interest in the instrument. 14. What has been the experience with implementing PforR operations? Implementation experience is still very limited. Of the eight approved operations, six are now effective. As of October 31, 2013, $122 million has been disbursed as advances and against prior results. In addition, $19 million have been disbursed against achieved results. Table 3 provides a summary of the original disbursement estimates for approved operations. It is noteworthy that few operations are using the option of disbursing against prior results, an option that clients considered to be quite attractive during the consultations on the design of the instrument. It would be useful to learn why this option is not being used—whether because of lack of knowledge, challenges in verifying past results, or for other reasons. For all operations, the Implementation Status and Results Reports (ISRs) record generally satisfactory Implementation Progress and progress in achieving the Development Objectives. The team proposes to gather more structured feedback and qualitative data on implementation, with special emphasis on (a) the nature of the dialogue between clients and the Bank during implementation; (b) the actual scope of Bank implementation support, in terms of both compliance due diligence and capacity building support to borrowers; and (c) potential systemic issues emerging from initial implementation experience. 15. What is the initial client feedback? Anecdotal feedback from clients who have prepared a PforR operation has been appreciative of the reduced Bank transaction processes. Clients also welcome the use of their own country systems, the shift in dialogue and focus from details to results, and the shift away from no-objection letters. Client representatives, especially those from central ministries, also find that the increased choice in Bank financing instruments allows for better matching to needs on the ground and “more intellectual engagement” in implementation support. However, clients have expressed concerns about certain features including limits in contract values, the Bank’s right to investigate, and the role of the Bank’s Integrity Vice Presidency (INT) as described in the PforR Anticorruption Guidelines (ACGs). More specifically, clients expressed concerns about the perceived intrusive nature of the ACGs, and the fact that the approach under the ACGs is not consistent with the overall philosophy of PforR. The instrument also seems to generate an interesting dynamic across line ministries, as the focus on results and increased accountability is received differently by different parts of the government and different levels of management within the government systems. Finally, like Bank staff, clients are still learning about the instrument and how best to use it, and there have been some early challenges. What is encouraging is that of the initial batch of countries that have had PforR operations, four are now preparing a second one (Ethiopia, Morocco, Tanzania, and Vietnam). The review will analyze in more detail both the concerns expressed by clients and the differing perspectives of different parts of government. The review will provide an opportunity to seek more comprehensive feedback from clients to inform the report. 5 Table 3: Expected Disbursements under Approved PforR Operations (US$ million) of which of which prior Country Program FY13 FY14 FY15 FY16 FY17 FY18 TOTAL advance results Ethiopia Health Millennium 36 8 36 15 13 12 120 26 10 Development Goals Kenya National Integrated 20 33 71 67 59 250 63 Safety Net Morocco National Initiative for Human Development 60 60 90 90 300 75 (INDH) Phase II Nepal Results-Based Bridges Improvement 12 12 12 12 12 60 15 and Maintenance Tanzania Urban Local Government 18 44 71 61 61 255 Strengthening Uganda Support to Municipal Infrastructure 1 26 25 34 34 30 150 5 Development Uruguay Road Rehabilitation 5 18 26 17 66 14 6 and Maintenance Results-Based Rural Vietnam 50 11 26 44 55 15 200 50 Water and Sanitation TOTAL 164 173 291 354 242 177 1,401 248 16 16. What is the initial feedback from development partners? Early informal feedback seems to indicate that development partners welcome the instrument, and have developed or are preparing a similar instrument. For instance, the Asian Development Bank has just approved an instrument that is very similar to PforR, the African Development Bank has expressed interest in learning about the instrument, and the Inter-American Development Bank is conducting a review of experience with its Performance Driven Loans, a results-focused type of investment loan. Feedback from bilaterals—for example, Germany and the United Kingdom—has also been supportive, and the European Community has expressed interest in co-financing a number of PforR operations. The review will give attention to this area; for example, consultations with key development partners will provide insights about their experiences with the instrument to date. The team also plans to explore any constraints to partnership under the instrument and ways to address such constraints. It will also review recent experiences with and good practices in results-based financing in general. 17. What is the initial feedback from other stakeholders? At the time the instrument was approved, some representatives of civil society organizations and business groups were concerned that the Bank was lowering its fiduciary, environmental and social standards. However and more recently, other civil society organizations have expressed interest in the development of the instrument and in learning how they can be directly involved in specific 6 operations. They recognize the potential benefits of the instrument and want to learn more about the implementation challenges. They are particularly interested in the results frameworks and disbursement-linked indicators (DLIs), including how the DLIs are determined, how they will be verified, and what role civil society groups can play in the verification process. They also want to know about the assessment process (particularly the environmental and social assessment) and how it is working in practice, and what broader lessons can be learned from the assessments that have been undertaken. They would also like to better understand the internal process within the Bank for undertaking assessments and determining the Bank’s position on assessment findings. Another area of particular interest to civil society organizations concerns the definition of PforR programs and how “exclusions” and the results of different assessments are affecting the scope of PforR programs. During the review, the team will obtain more feedback from a broad range of civil society, private sector and other stakeholders on their experiences with and perspective on the new instrument. 18. What is the feedback from Bank staff? Informal feedback from staff through team interactions and learning events point to both positive experiences and areas of concern. The need to do things differently (the culture change) and staff risk aversion are proving to be challenges. Some of the early messages are as follows:  The PforR instrument works and can be used for a wide range of countries and sectors.  PforR changes the dialogue with government counterparts from inputs to the results that the client wants to achieve and the measurement of those results.  PforR affects the dialogue and interactions among government agencies (including finance, budget, planning, and sector departments) about how results will be achieved, who is accountable, who will validate the results, and how disbursements will be made.  It shifts the relationship between the client and the Bank to one of working in partnership.  The focus on results and capacity/institution building requires different skills from the Bank.  The learning curve is steep; this is really a different way to conduct business.  Ensuring that staff and managers understand PforR and working in an integrated manner are necessary to get the full benefits of the PforR instrument.  Some of the assessments have been overambitious in scope. Assessments need to be appropriate to the context so as not to undermine the PforR’s focus on the client’s own program systems.  Clients and teams have not always worked in a sufficiently integrated fashion, which has led to some inefficiencies in preparing PforR operations. 7  The PforR exclusions and risk aversion have led to artificial definitions of programs in a number of cases.  The broad scope for Bank involvement in investigations of fraud and corruption allegations (as outlined in the ACGs) has proved to be a contentious point in the preparation of PforR operations, even deterring clients from using the PforR or pushing them to ring-fence the Bank’s contribution to their programs, which is contrary to the PforR spirit. 19. The review will gather more information on all these areas, giving special emphasis to (a) collecting and analyzing staff and management experience with the preparation and implementation of PforR operations; (b) better understanding the perspective of managers and staff who considered using the PforR instrument and then decided against it; (c) understanding the role of internal communications and of training and guidance efforts; and (d) better understanding the impact of the cap and the exclusions on the decisions of staff and management in shaping the PforR pipeline, structuring individual operations, and so on. It is also important to emphasize that the initial feedback from staff as well as from clients, development partners and other stakeholders will be subject to further analysis as part of the review. This initial feedback is helpful in indicating issues and concerns but will need to be assessed by the deeper analysis to be undertaken as part of the review. III. Proposed Structure and Content of Report 20. The review will be structured in six chapters. The first chapter will provide a brief introduction to the objectives, coverage, timing, and structure of the paper, and the second will summarize the rationale for the new instrument and its main features. This section describes the expected content of Chapters III to VI. A. Chapter III. Overview of Progress 21. This chapter will provide an overview of the main trends in the provision of PforR operations in the first two years since the new instrument was approved and will examine how PforR lending is being integrated into the Bank Group’s overall CPSs. 1. Preparation, Approval, and Implementation of Operations 22. This section will provide an overview of the preparation, approval, and implementation of operations to date. It will begin with a profile of operations already approved: the number of operations, balance between IBRD and IDA countries, volume of Bank and other financing, regional and sectoral characteristics, number and nature of DLIs, risk ratings, etc. To the extent data permit, the review will compare the profile of approved operations with the profile of operations now under preparation and endeavor to better understand the evolving dynamics around the distribution of operations across countries and sectors. It will also endeavor to better understand how decisions on instrument selection are being made. It will then examine the processing of PforR operations in terms of preparation activities, times and costs, the extent and number of reviews, the consistency of approved operations with the design of PforR approved by the Board, and the commitment to transparency. In this context, it will also review the merits of 8 hybrid operations (PforR plus IPF operations). Finally, the review will examine initial implementation experience: the time taken from approval to signing and effectiveness, initial advances and disbursements, overall implementation progress (including implementation of action plans and/or contractual agreements set out in the legal documents), changes in ratings, costs of implementation support and so on. Preparation of this section will draw on the desk review of available documentation and data (including feedback from the Board and management responses) as well as on structured interviews with Bank managers and staff and their counterparts in government and among other stakeholders. 2. Dialogue on New PforR Operations 23. The review will aim to arrive at a good understanding of the extent and nature of the dialogue with Bank member countries on the possible use and value-added of the new instrument. It will analyze the nature and outcomes of this dialogue, examining: (a) who is engaged in the dialogue and how it is taking place; (b) how governments, development partners, other stakeholders, and the Bank itself conceive using the new instrument as a complement to or substitute for other Bank instruments; (c) what different stakeholders perceive as the advantages and disadvantages of the new instrument; and (d) how the final decisions are being made on whether to make use of the new instrument and in what specific circumstances. It will also analyze how all these dimensions of the dialogue around the PforR instrument vary across Regions and types of countries, sectors, and so on. Some of this information is contained in recently approved CPSs. The review will also seek additional information and feedback externally from a survey of representatives of concerned governments and internally from a survey of country directors/country managers (Section IV discusses the methodology in greater detail).These surveys would be followed up by some in-depth interviews. B. Chapter IV. Experience with Specific Features of PforR Operations 24. The review will look in some detail at the specific features of the instrument to inform a judgment on whether adjustments or additional guidance are needed. This analysis will draw on the results of a desk review of documentation and data, supplemented by a survey of the Bank teams working on PforR operations and their country counterparts and by follow-up interviews with concerned Bank managers and staff, government officials, and representatives of other stakeholders. 1. Program Definition 25. This section will review the range of government programs being supported and under consideration for future support and will analyze how PforR Programs are being defined in relation to governments’ overall programs. Operations approved so far have focused on a broad range of government programs, including sectoral programs in transport, rural water supply and sanitation, and health and social protection as well as urban and local government-strengthening programs of various kinds. In most cases, the PforR Programs supporting these overall government programs have been more narrowly defined, including with respect to the range of themes, activities, or subprograms being supported, the geographical scope of the program, and the “windows” of the government program that will benefit from PforR support. Anecdotal evidence suggests that key determinants of the scope of the PforR Program compared with the 9 government program may include: (a) the need to take account of the PforR exclusions; (b) clients’ and Bank staffs’ perceptions of the risks of involving the Bank in specific activities; (c) the broad scope of Bank’s potential involvement in investigations of fraud and corruption allegations; and (d) a habit of determining the scope of the Bank support more along the lines of IPF/DPF operations, focusing on a set of selected activities/reforms packaged as a project or a program of reforms. 26. Clearly defining both the government program and the PforR Program is key to confirming the links between the two programs, defining a robust results framework and associated DLIs, and establishing the boundaries within which assessments will be undertaken. It is also important in defining the scope of the legal obligations and liability of clients and the Bank. But the more narrowly Programs are defined, the greater the complexity and challenge governments will face in managing their overall programs and the greater the risk that PforR operations themselves may look too much like more traditional investment project operations. This section will examine the rationale behind the determination of specific Programs, including the extent to which the assessments or other aspects of the PforR framework may be encouraging excessive risk aversion and an undue narrowing in the scope of the Programs themselves. It will also examine whether the OP/BP and guidance note provide the appropriate degree of flexibility with respect to the definition of the PforR Program and how best to support staff on issues of Program concept and design, including through more detailed guidance, more examples and practical solutions, etc. 2. Results Frameworks and DLIs 27. This section will examine the experience with results frameworks and DLIs. Each PforR operation includes a set of results indicators selected to measure success in achieving the PforR Program’s overall development objectives and a set of DLIs that reflect areas of improvement to be made under the Program. It is still too early to review the verification of DLIs and the achievement of development results, but several aspects of the process for developing the results frameworks and DLIs can be analyzed:  The challenge of: establishing results chains that lead to the expected key results of the operation; selecting DLIs that, as much as possible, provide incentives to the clients to achieve results and overcome key obstacles along the results chains; and balancing results and DLIs so that they are ambitious but not so ambitious that they substantially increase disbursement risks and the reputational risks for government staff if they are not achieved.  The nature and quality of the results and DLIs, including the availability of baseline data, the extent to which DLIs are focused on outcomes or process or other intermediate indicators, the merits of using physical as opposed to institutional results, and the types of results and DLIs that have been selected.  The linkages between and among the results, DLIs, and action plans; and the iterative process of developing a PforR Program’s development objectives, results framework, DLIs, and action plan. 10  Monitoring/evaluation and verification, including how it is done, who pays for it, the nature and extent of third-party verification, and how beneficiary feedback can be further encouraged.  The importance of ensuring that the dialogue around results and DLIs focuses not just on disbursement of World Bank financing but also on the results focus of the government’s overall program and the shift to a more results focused culture.  The value added of the guidance and whether any changes would be helpful. 3. Program Assessments, Fraud and Corruption, and Action Plans 28. This section will analyze experience in preparing the various assessments required for a PforR operation, discussing those assessments with the government and other stakeholders, and incorporating some of the key recommendations of the assessments into the action plan and DLIs for a PforR operation. Early feedback suggests that, overall the assessments are contributing to more substantive understanding of the systems issues and a positive dialogue about program systems, their performance, and how best to improve that performance. However, some assessments appear to have been excessive and not well focused on the key questions that they are meant to answer; thus a deeper understanding about the drivers of the heaviness of some of the assessments is clearly needed. To this end, the team will assess whether the assessments carried out for approved PforR operations had the appropriate scope and specificity and whether the guidance notes themselves need adjustment or the interpretation of the guidance notes needs modification. In this context, the review will seek feedback from relevant Bank staff (technical, procurement, financial management, governance, and environmental and social specialists, as well as members of task teams and management) about their experiences and lessons learned in doing these assessments as well as feedback from government officials and other stakeholders on their perspectives on the value of the assessments. 29. Technical Assessment and Economic Evaluation. It appears that the preparation and discussion of the technical assessment and the associated economic evaluation are proceeding with relatively few issues or concerns. The assessment is being used to show the technical soundness of the proposed Program, and the evaluation elaborates on the Program’s economic impact and provides the rationale for public provision. Both the assessment and the evaluation are discussed with government and provide a basis for agreeing on improvements to be made to the Program and included in the program action plan. 30. Fiduciary Systems Assessment. The fiduciary systems assessments of PforR operations often draw on the government’s ongoing dialogue and work with the Bank on issues of public financial management, providing the basis both for determining the overall fiduciary risk of the operation and for agreeing with the government on improvements in fiduciary arrangements that can help mitigate that risk. Finding reliable and sufficient data on fiduciary performance for such an assessment is not always easy, and the necessary integration among financial management, procurement, and governance staff within the Bank remains a challenge. This section will examine in more detail the overall process for preparing and discussing the fiduciary systems assessments and whether they have drawn sufficiently on work already done by the Bank or other partners. It will also look into the process for agreeing on necessary system improvements 11 and how best to assess fiduciary performance during implementation. In addition, this section will review how procurement exclusions have been handled and how they have affected the overall design of the PforR program and the dialogue with clients. Finally, it will reflect, as appropriate, implications of the management response to the recent IEG recommendation to adopt a risk-based approach to handling the highest risk/value contracts in Bank procurement. 31. Fraud and Corruption. All fiduciary systems assessments address the issue of fraud and corruption and define the specific arrangements under each PforR operation to be in conformity with the ACGs. This is a particularly sensitive issue because, as has been noted, the ACGs are a source of discomfort for many clients, who feel that they are intrusive with respect to the role of the Bank and do not take proper account of countries’ legal frameworks. Thus it is clearly very important to discuss the ACGs with government counterparts in a timely fashion and to agree on a practical way forward. This appears to have happened for some of the approved PforR operations, and in most cases the program document explains how the government handles allegations of fraud and corruption (including areas of improvements, where appropriate) and sets out the specific arrangements for handling the ACGs requirements. But the agreed role of INT appears to vary somewhat from one operation to another, with some program documents referring very explicitly to the right of INT to conduct its own independent investigation and others referring only to INT cooperating with the national anticorruption agency in a joint investigation. The review will analyze how the issues of transparency, fraud and corruption, and handling of allegations as outlined by the ACGs are addressed in practice, including the role of MOU’s between INT and countries’ anti-corruption agencies, how potential inconsistencies between the ACGs and national legislation are being addressed, and the nature of the discussions around ACGs between Bank teams and their counterparts in country. This will help determine whether additional guidance is required. The review will also assess whether and how the ACGs are influencing the definition of the overall PforR Programs (and even countries’ interest in the PforR instrument itself). 32. Environmental and Social Assessment. A systematic review of an initial cluster of seven environmental and social systems assessments (ESSAs) and a discussion of the results of the review highlighted some important issues for consideration. While there was no evidence that high-risk activities were being included with the introduction of the new instrument, none of the ESSAs reviewed made explicit use of the screening tool to address the five risk criteria, the environmental and social aspects of the assessments were often not well integrated, the links between the analysis in the assessment and the proposed actions were not always clear, disproportionate attention was sometimes given to relatively minor issues, and the sheer magnitude of the assessments was a challenge for the consultation process. The discussion also brought out the importance of early engagement with the government on the objectives of the ESSA and the process for preparing it and consulting on it, as well as on the assessment’s potential value-added for the government’s overall program. This review will assess all the ESSAs done to date with a view to gauging their overall quality as well as the lessons learned from the consultation process, how grievance redress mechanisms are being handled, and the impact of the exclusion of high-risk activities. This will help determine whether any modifications to the guidance note are required or other changes are needed in how the findings of ESSAs are being integrated into the dialogue around the design of a Program. 12 33. Action Plans. Based on the outcome of the various assessments, the Bank and the other development partners involved agree with government counterparts on action plans to improve the technical dimensions of the Program, the systems to be used to implement the Program and the capacity and performance of the agencies involved. The review will examine how the action plans are being formulated, the type of system enhancement and capacity strengthening measures that are being included, and the measures to support the implementation of the plans. To the extent possible, it will also examine initial experience with implementation of action plans. 4. Risk Assessment and Management 34. For the eight approved operations, the overall program risk rating varies: two operations received moderate risk ratings, five substantial, and one high. Fiduciary risks were generally rated the highest—moderate for one operation, substantial for five and high in the remaining two. The review will analyze how those ratings were determined, their realism, and how they have informed the overall design of PforR operations and the actions plans for systems improvements. It will also check on whether all activities governed by exclusion provisions have been excluded from the supported programs and what the implications of those exclusions have been. Of particular importance is whether task teams have been able to work iteratively with regard to the scope of the program, its results, the DLIs, the action plan, and the risks. Depending on the risks, for example, the program scope may need to change. The instrument allows for close linkages between assessments, risks, and results which should help guide the definition of the Program and its scope. That in turn can help enhance the development impact of the Programs supported by PforR. 5. Capacity Building, Institution Strengthening, and Implementation Support 35. This section will focus on capacity building and institutional strengthening, which are expected to be key features of PforR operations. An initial review of approved PforR operations suggests that capacity-building activities are being informed by the program’s technical, fiduciary, and environmental and social systems assessments, and that priority capacity-building measures are being included in Programs and linked to specific DLIs and/or made part of the Program action plan. The implementation support plans also emphasize the Bank teams’ role in providing capacity-building support in addition to monitoring implementation progress. However, clients and task teams are still figuring out how to use the PforR instrument appropriately to support institution building, using a combination of Bank implementation support, DLIs, actions under the Program action plan, and, as necessary, legal covenants. Staff have expressed a concern that achievement of the PforR’s capacity-building and institution- strengthening objective may be constrained by lack of access to traditional Bank financing for technical assistance (i.e., through an IPF operation) and insufficient Bank resources. PforR implementation support plans also show substantial differences in the expected amount of time and resources to be allocated for implementation support in different operations. The review will examine the capacity-building and institution-strengthening measures being included in PforR operations. It will also examine the different models for providing and financing capacity building support as part of a PforR operation or through a separate technical assistance operation and the level and nature of implementation support required from the Bank itself. 13 6. Partnerships and Financing 36. This section will focus on the partnership aspects of PforR operations, examining how effectively the Bank is working in partnership with governments, development partners, and other stakeholders. It will also consider the financing aspects of these partnerships and the extent to which Bank resources are effectively leveraging funding from other sources. In PforR operations it is expected that the Bank will partner with governments in assessing programs and the potential for improvements, and will reach agreement with government counterparts on specific improvements and the Bank’s role in supporting those improvements; and Bank staff have observed that PforR is shifting the relationship toward working more in partnership. The review will aim to develop a better understanding of how this is happening in practice and how those partnerships can be strengthened. It will also assess whether, as anticipated, PforR is helping the Bank work better with development partners at all phases of the development and implementation of PforR operations, including conducting the various assessments, appraising operations, and supporting the system-strengthening and capacity-building aspects of a PforR Program, and whether it is possible to improve the effectiveness and efficiency of these partnership arrangements. PforR was also expected to offer opportunities for the Bank to partner with other stakeholders—the private sector, civil society, and others—in helping governments improve the design of programs, implement them more effectively, and monitor and evaluate their results, and the review will assess whether and how this is happening in practice. Finally, with respect to financing, in the approved PforR operations World Bank financing is part of a broader program being supported financially by the government. But the Bank’s percentage contribution and the extent of financing from other partners vary widely in the different operations. In some the involvement of other partners is small and limited to technical support; by contrast, in the Ethiopia PforR operation there are nine partners in addition to the World Bank, and Bank financing is projected to account for 18 percent of the Health Millennium Development Goals financing and less than three percent of total financing of the overall Health Sector Development Program. Accordingly the review will also examine the financing dimensions of partnerships, including the extent to which the current PforR framework is conducive to co-financing with partners, the feasibility of using trust funds to support PforR operations, and so on. C. Chapter V. Internal Organization and Management 37. This chapter will draw on the desk review of documentation and interviews with managers and staff to analyze how effectively the PforR instrument has been rolled out, how well the Bank has adjusted internally to the introduction of the new lending instrument, and what more is needed to promote the instrument and familiarize staff with its use. It is clear that for PforR the quality of the task teams and the support they receive is critical, particularly for the initial operations. Regions need to assign experienced task team leaders to PforR operations and support them in putting together teams that have the range of expertise and experience required for such operations and that can work together in an integrated manner. Country directors, country managers, and other staff also need to understand the PforR approach well enough to effectively discuss its potential value with government counterparts. Those who review prospective PforR operations need to understand how such operations are distinct from both DPF and IPF operations if they are to provide the appropriate advice and recommendations. In this regard the advice and guidance of the Bank’s corporate oversight units are of great importance, particularly at this early stage in the use of the PforR 14 instrument when best practice is still being determined. Finally, senior management needs to be well- versed in the rationale for and features of the new instrument to facilitate its institutionalization and make the appropriate interventions and decisions on the processing of particular operations; and the Board needs to be thoroughly familiar with the instrument so that it can provide valuable feedback and guidance as it discusses and approves individual operations. This section will focus on all of these dimensions of the Bank’s internal organization and management, examine the resource requirements of preparation and implementation support and the level of management and administration effort, and consider what further changes or improvements should be made, including on-going changes in the organization of the Bank. It will look specifically at the issue of exclusions (PforR excludes activities with potential high negative environment and social impacts, as well as contracts above specific monetary thresholds) and the lending cap (five percent of total Bank lending for the first two years of implementation), assessing how these issues have been managed and monitored and what impact they have had on the design of operations and the overall rollout of the new instrument. D. Chapter VI. Emerging Lessons and Recommendations 38. Finally, the review will identify lessons of experience with the dialogue around and the use of the PforR instrument and will identify areas that need to be adjusted or strengthened, while taken into account limited implementation experience to draw on. Possible areas to be addressed include the following:  Internal and external communications around the PforR instrument, to better enhance the understanding of its potential value-added within member countries, among development partners and other stakeholders, and within the Bank itself.  Guidance notes—is additional or changed guidance (including simplification of guidelines) needed to ensure that Bank staff, government counterparts, and development partners are in a good position to prepare, appraise, and implement PforR operations in a manner consistent with the OP/BP.  Training and other mechanisms for improving understanding of the new instrument, building up a good knowledge base, and collecting, sharing and learning from good practices—for example, through e-learning modules, a PforR academy, and consolidating the community of practice.  Internal organization and management arrangements to further the institutionalization of the PforR instrument and its appropriate use in IBRD and IDA countries.  Possible revisions to OP/BP 9.00, and ways to address the five percent lending cap. 39. It is expected that the review will draw on the lessons of experience to recommend adjustments that can enhance and expand the benefits of the PforR instrument. 15 IV. Methodology and Process 40. The review will rely on a four-pronged methodological approach:  Literature and desk review. A desk review of CPSs, PforR Program documents, ISRs, comments from and results of review meetings, and other relevant documents will inform the overview of progress and the sections on experience with specific features of PforR operations and internal organization and management issues.  Surveys. Two surveys are proposed. The first would survey country directors/country managers and their counterparts in countries where there are no PforR operations under implementation or preparation, to assess the extent and nature of the knowledge and dialogue around the PforR instrument in all the countries that have lending programs with the Bank. This survey could also be used to better understand the overall perspective of development partners on the potential value of and challenges associated with the instrument. The second survey would focus on the perspectives of Bank managers and task teams, their counterparts in government, and representatives of development partners and other stakeholders concerning approved PforR operations and those under preparation. This survey would endeavor to enrich the understanding of both what is going well with respect to PforR operations and what the concerns and challenges are from the perspective of those who have been deeply involved in the design and implementation of PforR operations. The surveys’ scope will be informed by the topics discussed in Section II above. For both types of survey, an effort would be made to ensure that many of the questions are the same for the different groups being surveyed (to facilitate direct comparisons, etc.), although some questions will need to be unique to particular stakeholder groups.  Interviews. Within the Bank, in-depth, structured interviews will be conducted with country directors/country managers and sector directors/sector managers as well as with team leaders and members for PforR operations under preparation or implementation and with those involved in the review of those operations. In-depth interviews will also be conducted with government officials and representatives of other stakeholder groups that have been directly involved in the PforR dialogue, as well as with the in-country teams (at both national and sub-national/local levels) that have been working on the preparation and implementation of specific PforR operations. These interviews will enable the review to go beyond the information available in the various documents and from the surveys to better understand: (a) how the broader dialogue around the PforR instrument is evolving, and (b) what are the dynamics around the preparation and initial implementation of specific PforR operations. Who would be interviewed and about what would be determined from the analysis of the results of the surveys.  Consultations. Following the completion of the surveys and the interviews, the team will carry out consultations with key stakeholders on the dialogue around and the use of the PforR instrument to support government programs, strengthen capacity, and deliver results. These consultations will include a discussion of the preliminary findings emerging from the analysis of the results of the surveys and the interviews. They will probably take place in some of the countries where the PforR instrument is 16 already in use, as well as with key stakeholder groups, including multilateral and bilateral development partners, civil society organizations, and the private sector. A dedicated website for the PforR Review will keep interested stakeholders informed and allow the team to receive feedback as the preparation of the review progresses. 41. Process. OPCS will enlist staff in the Networks, Regions, and Central Departments to collaborate with OPCS in reviewing the Program Appraisal Documents and ISRs for PforR operations in their areas of expertise and in interviewing managers and staff. OPCS and outside experts will work with the Bank’s Public Opinion Research Group (PORG) (which is responsible for a range of surveys carried out by the Bank, including the country client surveys) to design the survey instruments; and PORG will manage the survey process and prepare reports on the results of the surveys. A team of Bank staff and outside experts will analyze the results of the surveys and the literature/desk review; and conduct structured follow-up interviews within the Bank and with government officials and other stakeholders. The involvement of outside experts will enhance the rigor, transparency, and credibility of the process. External and Corporate Relations (ECR) staff will support OPCS in conducting the consultations. Finally, a core team within OPCS will prepare the final draft of the review, drawing on these various inputs and guided by an internal Bank Advisory Group (of experienced managers and others with direct experience in using the PforR instrument) and an Independent External Panel of Experts (representing governments, development partners, civil society, academia and other stakeholders). In addition to monitoring the process and providing guidance to the Bank team working on the Management Review, the Independent External Panel of Experts will have opportunities to comment and will provide a statement to accompany the staff report. 42. Role of IEG. OPCS also intends to engage with IEG to the extent possible throughout the process. In this context, OPCS and IEG have been exchanging views about the appropriate modality of IEG’s contributions to this assessment, while maintaining its ability to provide independent views on PforR at various stages. Some of the ideas being pursued include IEG to provide suggestions on the types of information to collect in the planned surveys and structured interviews as well as on possible candidates for the Independent External Panel of Experts. With respect to the statement in the Board documentation for PforR that “Management welcomes an early evaluation of implementation by the Bank’s Independent Evaluation Group which would provide a useful independent input for ongoing improvements to the instrument”, IEG plans to carry out its evaluation of PforR in FY16 in accordance with the current work program. This will allow for at least several of the current projects to have been completed and assessed. 43. Team Composition. An intersectoral team will carry out the review, with members from OPCS and other Bank units, including the Regions and Networks, ECR (including PORG), Legal, and Controller’s as well as external experts. The team will include technical/sectoral staff as well as fiduciary staff and environmental and social specialists, with some team members having direct experience of working on specific PforR operations. As noted above, the team will be guided by an internal Bank Advisory Group and an Independent External Panel of Experts. 17 V. Issues and Next Steps 44. Management looks forward to guidance from the Committee on Development Effectiveness (CODE) on the concept note for the two year review of the PforR instrument, including: a. Is the scope of the review adequate to explore and learn from the issues and challenges that have emerged from the dialogue around PforR and initial experience in designing and implementing specific PforR operations? b. Are Executive Directors comfortable with the proposed approach, methodology and process for the review? 45. Next Steps. Drawing on the discussions at CODE, Management would finalize the concept note and make it publicly available. The Independent External Panel of Experts as well as the internal Bank Advisory Group would guide the review team in analyzing documentation, conducting the surveys and structured interviews, carrying out consultations and preparing the review document. A draft of the paper is expected to be submitted to CODE before the end of the FY. 18 Annex A. Management Proposal to the Board on Initial Implementation Review for Program-for-Results Operations4 1. Management proposes to rollout the Program-for-Results instrument cautiously in the early years, including a limit on overall lending during that period. Management will propose to the Board to lift the lending limit following an initial review expected after approximately after 2years of implementation. This note outlines how such a review will be approached. 2. Implementation experience will be limited in the first two years by the proposed lending cap and the time taken for programs to be prepared, approved, and launched. While evaluation of the full impact of the first Program-for-Results programs will likely require more time, it is expected that some valuable lessons can be learned. Many of these will focus on how each operation was prepared and early implementation experience. The initial review will involve basic descriptive statistics supplemented by qualitative evaluation techniques. Among the questions to be answered are:  Are we doing things right?5 Is the instrument being implemented according to the design approved by the Board? What are the challenges faced by borrowers, development partners, and staff?  Where is the instrument being used? What countries and sectors are using the instrument? Is it working across the board or are there specific country or sectoral issues that need to be taken into account?  What is the experience with systems assessments? Are there specific issues related to the assessments—technical, fiduciary (including fraud and corruption), environmental and social, and risk? Are the assessments too intrusive? Are there gaps? What is the early experience with the action plans to enhance capacity and manage risks?  What is the early experience in measuring results? Is the results framework aligned with program objectives? What is the link between program results and DLIs? Is baseline data in place? Are credible verification protocols in place and how are they working in the early stages? Is M&E capacity adequate? What is the status of capacity building measures? Are disbursements taking place as expected? How much are Bank resources being leveraged?  What is the early experience in risk management? Have all activities governed by exclusion provisions been excluded from the supported programs? What is the assessed risk profile of the key dimensions of the risk framework? 4 Source: Annex 1 of the Board paper “A New Instrument to Advance Development Effectiveness: P rogram-for- Results Financing”, December 29, 2011. 5 Evaluations try to answer two main questions: (a) are we doing the right things and (b) are we doing things right? The first question will require more time to assess and evaluate and will be addressed in subsequent reviews. 19  What are the views of others? How do clients, development partners, and other stakeholders view the early implementation experience? Has the Bank’s commitment to transparency been manifest in each program?  What is the level of management and administration effort? What are the resource requirements of preparation and implementation support?  What are emerging early lessons? 3. The preparation of the review will adopt a number of methodologies to help answer the above questions:  Solicit feedback from countries, development partners, and staff. This will be done through a combination of a survey as well as qualitative evaluations  Establish mechanisms for data monitoring.  Conduct in-depth interviews with all stakeholders.  Explore options for cross-regional learning, including with other development partners.  Encourage an on-going dialogue. Maintain an active external website that will allow dissemination and provide an official forum for an on-going discussion. 20 Annex B. Suggested Outline of the Review I. Introduction II. Background III. Overview of Progress A. Preparation, Approval, and Implementation of Operations B. Dialogue on New PforR Operations IV. Experience with Specific Features of PforR Operations A. Program Definition B. Results Frameworks and Disbursement-Linked Indicators C. Program Assessments, Fraud and Corruption, and Action Plans D. Risk Assessment and Management E. Capacity Building, Institution Strengthening, and Implementation Support F. Partnerships and Financing V. Internal Organization and Management VI. Emerging Lessons and Recommendations Annexes, including results of surveys 21