Approach Paper Rwanda Country Program Evaluation FY09-17 December 13, 2017 Background and Context Introduction to the evaluation 1. The primary goal of the evaluation is to assess the Bank Group’s development effectiveness in Rwanda during the period FY09-17 and to inform the design and implementation of future WBG activities in Rwanda. The Country Program Evaluation (CPE) will place strong emphasis on assessing World Bank Group (WBG) strategic positioning and program delivery to help Rwanda achieve its development goals, notably to become a lower middle income country by 2020. The selection of Rwanda for this CPE – one of two undertaken by IEG in FY18 – is motivated by the country’s imp ortant development achievements during the period of analysis as well as by the major challenges the country faces in order to sustain those gains going forward. The CPE seeks to provide inputs for the next Country Partnership Framework (CPF) ). While the CPE is primarily aimed at informing future WBG support to Rwanda, the evaluation findings are expected to provide lessons for WBG programs in countries that share similar characteristics and aspirations—for example, other small size, low-income, post-conflict countries in Sub-Saharan Africa seeking to achieve high rates of growth and poverty reduction, or countries facing the challenge of employment creation for a young population in an economy that is highly dependent on the service sector.1 2. Since the turn of the century, Rwanda has become one of the world’s ten fastest growing economies—significantly reducing poverty in the process—and is now striving to quickly achieve middle income status. Rwanda’s GNI per capita increased from $250 in 2006 to $710 in 2015. Poverty rates 2 fell from 56.7% in 2005/06 to 39.1% in 2013/14. Impressive progress was also observed in various non-monetary measures of wellbeing, as evidenced in the achievement, by the country, of most Millennium Development Goals (MDGs) by 2015. Over the years, the country has developed a strong and consistent track record of reforms and economic management. However, further improvements in growth and poverty reduction—from already impressive achievements—will require significant structural change in the economy. Rwanda Vision 2020, which the country prepared in 2000 and revised in 2011, foresees the country as a lower-middle income by the end of this decade.3 The vision is based on the country developing into a knowledge-based regional service hub supported by: (i) good governance and a capable state; (ii) human resource development and a knowledge-based economy; (iii) a private sector-led economy; (iv) infrastructure development; (v) productive and market-oriented 1 agriculture; and (vi) regional and international economic integration. It also emphasizes the importance of progress on four cross-cutting issues: (i) gender equality; (ii) natural resources; (iii) the environment; and (iv) science, technology and ICT. Rwanda’s longer-term aspirations are to reach upper middle income status by 2035 and high income status by 2050. Country Context, Development Gains, and Ongoing Challenges 3. Geopolitical challenges notwithstanding, Rwanda’s location offers potential. As a small landlocked country located in Eastern Africa’s Great Lakes region, Rwanda faces fundamental development constraints. Although blessed with fertile, mountainous terrain and ample rainfall, the high population density (483 people per per square km of land area in 2016) strains the country’s limited land and natural resources. However, there are multiple sources of insecurity and tension in the region. Enhanced regional peace and security would facilitate closer regional integration. Rwanda’s location affords membership of t he East African Community, with a combined GDP 20 times its own, and closer integration would offer access to more affordable energy, lower transport costs, and expanded markets. 4. Rwanda has seen remarkable political stability and continuity of leadership, albeit at the expense of a comparatively low standing in “voice and accountability.” The peace and political stability re-established in the wake of the 1994 genocide have been sustained. At the head of the dominant Rwandan Patriotic Front, Paul Kagame was overwhelmingly re-elected to a third term as President in August 2017.4 Rwanda’s success in rebuilding democratic institutions and processes and resolving the fallout from the genocide owes in part to an inclusive form of governance based on national identity and rooted in Rwandan culture and tradition.5,6 The country’s top political leadership has maintained a zero-tolerance approach to corruption, and has effectually used legal and administrative sanctions to ensure performance and discipline. The formal justice system has also been considerably strengthened since 1994. Despite these successes, there is much room for improving Rwanda’s international rankings in voice and accountability. Although it improved slightly from 2010 to 2015, Rwanda’s World Governance Indicators ranking in voice and accountability (17th percentile) remains well under the Sub- Saharan average (33rd percentile) and that of neighboring countries such as Kenya (42nd) and Uganda (29th). As indicated in the FY14-18 CPS (para. 34, p12 of CPS), international advocacy groups have decried restrictions on media and civil liberties.7 5. Rwanda has articulated a clear vision of its development path, and put in place implementation modalities. In December 1994, the government endorsed an economic agenda for a “New Rwanda” that embodied a commitment to a market economy with a strong private sector presence. A series of measures subsequently liberalized the economy.8 In 2000, the government decided to embark on a multi-phased decentralization initiative in an effort to foster 2 reconciliation of the Rwandan people, engage citizens in participatory planning and decision- making, promote accountability, and enhance service delivery.9 The long-term vision for the country’s development articulated in Rwanda Vision 2020 has been operationalized through a series of national and sectoral medium-term strategies, district development plans, and annual performance contracts. The Economic Development and Poverty Reduction Strategies EDPRS1 (2008-12) and EDPRS2 (2013-18) were the two national plans that span the evaluation period.10As indicated earlier, Rwanda’s practical approach to forcefully setting a vision, aligning resources with it, and establishing specific performance contracts to hold teams accountable for delivery, has been associated with remarkable results.11 6. Rwanda’s progress on poverty and shared prosperity is largely attributable to its strong growth performance. Growth was high throughout the decade of the 2000s, but it is mostly over 2006-11 that higher, pro-poor growth, concentrated notably in rural areas, brought a sharp reduction in the poverty headcount from 56.9 to 44.9 percent.12 Accompanying this was a modest decline in inequality—the Gini coefficient decreased from 0.52 in 2006 to 0.49 in 2011. Government sources indicate that more recent 2013/14 household survey showed a further decline in poverty to 39.1 percent. Government estimates indicate a further drop of the Gini coefficient to 0.45 in 2014, noting that the reduction in inequality has accounted for more than 40 percent of the recent reduction in poverty.13 Nevertheless, despite this positive trend, inequality remains high for Rwanda’s income level.14 7. Improvements in non-monetary indicators of well-being were also remarkable, pushing Rwanda beyond the lower-middle-income country average in several areas, although challenges remain. Rwanda met most of the MDGs by the end of 2015. It reached near universal primary school enrolment, although net attendance in secondary schools and quality remain low (Table 1). Child mortality dropped outstandingly—by two-thirds—and immunization coverage increased. However, chronic malnutrition remains widespread with persistently high rates of stunting, which jeopardizes children’s learning abilities. Large-scale public investments have contributed to improved access to water sources and sanitation, road transport, electricity, and ICT, as well as housing conditions. But unpaved rural and feeder roads remain in poor condition, impeding farmers’ connections to input and output markets, while the high cost and low reliability of energy is a persistent hurdle to enterprise development. 8. Despite its impressive achievements, Rwanda faces multiple challenges to its quest for upper-middle-income status by 2035. The structural transformation process in Rwanda is still at an early stage, and despite improvement in the investment climate, the private sector remains underdeveloped with a narrow manufacturing base, a small financial sector, and an underdeveloped services sector. The Government recently requested Bank support in identifying future drivers of growth to help underpin a long-term vision for the country to 2050, and the 3 work has been launched. According to the World Bank’s initial assessment, “Several emerging areas for policy concern include: (i) low domestic saving rates with too much reliance on external assistance; (ii) a slowdown in productivity growth since 2010; (iii) a disproportionately large role for public sector investment; (iv) insufficient outward orientation of the economy; (v) lagging urbanization of the economy, which prevents Rwanda from reaping the benefits of agglomeration economies; (vi) concerns regarding the relevance and quality of education; and (v) slower progress in transparency and civic and media participation.”15 Table 1. Rwanda: Selected Social and Economic Indicators LIC LMIC average average Indicators 1994 2000/01 2005/06 2010/11 2013/14 2013/14 2013/14 Health Immunization, measles (% of children 25 69 95 95 97 76.7 79.1 ages 12-23 months) Births attended by skilled health staffs .. 92 96 98 99 55.9 71.5 Maternal Mortality ratio (per 100,000 live .. 1,071 750 476 210 513 260 births) Mortality rate under 5 (per 1,000 live .. 196 152 76 50 78.9 54.5 births) Life expectancy 28.6 48.2 56.4 62.2 64.0 61.3 67.2 Education Net attendance ratio- secondary (%) 10.4 17.8 23 Infrastructure (% population with access) Improved sanitation .. 58.5 74.5 83.4 27.9 51.6 Improved water source .. 70.3 74.2 84.8 65.1 88.5 Electricity as main source of lighting .. 4.3 10.8 19.8 28.3 79.5 Road .. .. 85.8 .. .. .. .. Sources: Integrated Household Living Conditions Survey EICV1, EICV2, EICV3, EICV4 and WDI. World Bank Group Objectives 9. Two successive strategy documents aligned with Rwanda Vision 2020 steered WBG support to Rwanda during the evaluation period. Over the period FY09-17 that the CPE will cover, two strategic frameworks (both prepared prior to the new country engagement model) guided the WBG program. The first, the FY09-12 Country Assistance Strategy (CAS), underwent amendment in a CAS Progress Report (CASPR, FY11). The second, the FY14-18 Country Partnership Strategy (CPS), remains under implementation and has been revised and extended in time (to FY20) by a recent (April 2017) CPS Performance and Learning Review (PLR). Both strategies were jointly prepared by IDA, the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA), with the goal of drawing 4 public-private synergies and catalyzing private sector resources in support of Rwanda’s development. 10. The FY09-12 CAS was anchored on Rwanda’s second-generation PRSP, the first Economic Development and Poverty Reduction Strategy (EDPRS1). While the first PRSP, adopted in 2002, focused on managing a transition period of rehabilitation and reconstruction, EDPRS1 (2008-12) focused squarely on growth, articulating its priorities through three flagship programs:16 1) Sustainable Growth for Jobs and Exports, which supported interventions to develop skills, improve economic infrastructure, promote Science, Technology, and Innovation (STI) and strengthen the financial sector; 2) Vision 2020 Umurenge,17 which provided public works, credit packages, and direct support to create opportunities for the poorest Rwandans and foster shared growth; 3) Governance, to strengthen both political and economic governance and build institutions and the capacity of the state, including further decentralization. 11. The CAS sought primarily to contribute to the EDPRS1 objective of significantly raising growth and creating jobs. Its secondary objective was to consolidate specific elements of Rwanda’s social progress by tackling selected aspects of vulnerability. There was a one-year extension of the CAS implementation period to align with an extension of the EDPRS1. 12. Following adoption of the CAS, the government and development partners established a common performance assessment framework (CPAF) as a basis for budget support and tracking performance in EDPRS implementation. Development of the CPAF began in April 2008 and was based on the EDPRS Results and Policy Matrix for national-level monitoring. In parallel, a Joint Governance Assessment (JGA) was launched in March 2008 to assess government performance in implementing governance reforms in areas relating to justice, voice and empowerment, and accountability. The CPAF was derived as a subset of the national-level EDPRS monitoring framework supplemented with selected governance-related indicators based on the JGA and the government’s public financial management (PFM) action plan. Overall, the CPAF provided a relatively lean and harmonized monitoring and accountability framework. Nevertheless, in some sectors, indicators tended to be excessively process- rather than policy- and outcome-based. 13. The government also formalized a division of labor (DOL) among development partners in 2010, assigning each to three sectors to reduce transaction costs and improve aid effectiveness (Attachment 4). Consolidation notwithstanding, the DOL did allow the use of delegated cooperation and “silent partnership” agreements.18 This provided for a phased 5 approach through which a given partner could continue to support several sectors over the medium term, while at the same time streamlining policy dialogue. However, the DOL excludes: (i) coverage of a sector through budget support; (ii) basket funds; (iii) support for exceptional expenditures such as demobilization, regional projects, and emergency assistance; and (iv) support to NGOs and private sector entities (which implies that the DOL does not cover IFC and MIGA activity). 14. The FY11 CASPR amended the Bank’s program and the CAS results framework to align these with the CPAF and DOL. In light of the DOL, which noted the Bank’s priority engagement in three sectors (agriculture, energy, and transport, including ICT), the Bank decided to “focus the rest of the CAS period on promoting economic transformation and increasing growth, especially of exports.” However, the CASPR also noted that “[w]hile the Bank under the DOL will no longer provide direct financing in some sectors—health and education—it will nevertheless continue to be a silent partner in social protection and support cross-cutting areas including skills, employment and demobilization.” Other adjustments involved stepping up efforts to deepen intra-WBG collaboration on the proposed focus areas, and providing for new Bank support to the government’s strategy for strengthening capacity. The CASPR also noted the need to re-examine WBG support for regional projects in light of government questioning of their development effectiveness, given their complexity and the need for timely alignment of the objectives of several countries. Certain elements of the CAS results matrix were modified to ensure alignment with the CPAF and reflect changes in program delivery. 15. The FY14-18 CPS sought to support selected objectives of EDPRS2. The latter set an extremely ambitious inclusive growth agenda for Rwanda to achieve lower middle-income status by 2020. It focused on: (i) economic transformation to foster a shift towards industry and services; (ii) rural development to reduce the national poverty rate below 30 percent; (iii) productivity and youth employment; and (iv) accountable governance to improve service delivery as well as citizen participation and satisfaction. In line with the DOL, the government asked the WBG to remain the lead development partner in energy, though not in transport, but requested the Bank to take the lead on a new sector where it had no longer been engaged under the previous CAS, namely urban development. The GoR also asked the Bank to remain actively engaged, but no longer lead, in the rural sector, and to operate through low transaction cost modalities in social protection19 (silent partner providing budget support). In addition, the GoR asked the Bank to support progress on PFM and decentralization together with DfID, KfW, and the EU. The FY14-18 CPS was well aligned with the government requests, and the program was articulated around three objectives: accelerating economic growth, notably through support in energy and urban development; improving the productivity and incomes of the poor through rural development and social protection; and supporting accountable governance through public financial management and decentralization. 6 16. The 2017 PLR proposed adjustments in the WBG engagement approach to sharpen focus on the “how” of enhancing private sector participation in a range of sectors. The CPS was extended to FY20 to ensure that the next Country Partnership Framework (CPF) would be precisely anchored onto the National Transformation Strategy (the planned successor to EDPRS2), as well as to enable the upcoming Systematic Country Diagnostic (SCD) to be informed by the results of the 2018 Integrated Households Living Conditions survey. The PLR noted a need for the WBG to do more to address structural reforms in the energy sector and support the emergence of a new agricultural sector. Some of the CPS results matrix indicators were amended in an effort to ensure stronger measurability of outcomes and linkages to the WBG program. WBG Program 17. The WBG provided significant financing covering key sectors (using a mix of instruments) as well as knowledge work over the evaluation period. IDA approvals totaled $1.75 billion for 31 World Bank financing operations during the period FY09-17 together with associated policy dialogue and an active program of Advisory Services and Analytics (ASA) and Trust Fund use. Prior to 2013, the share of Bank commitments in the form of development policy financing (DPF) was around two-thirds; after 2013, DPF and Program for Results (P4R) financing taken together continued to account for a similar share. In terms of sector focus by leading Global Practice20, social protection accounted for the largest share of commitments (25 percent), followed by macroeconomic and fiscal management (24 percent), and agriculture (17 percent). The shares of social protection and agriculture in commitments increased between FY09-13 and FY14-17, even though the Bank became a silent partner in social protection. The dominance of social protection in the Bank’s portfolio reflects perceived need and government demand. The share of energy & mining and transport & ICT taken together remained stable at around 16 percent throughout the evaluation period. Rwanda also participated in several regional projects approved during the period. Energy & mining and transport & ICT taken together accounted for 60 percent of the total regional lending. IFC investments were equally active, totaling some $251 million in commitments (albeit with a decline in the latter part of the evaluation period) alongside 21 advisory service (AS) projects during FY09-17. The largest sector was Finance & Insurance, which accounted for 64 percent of total IFC investment commitments (9 projects). The largest business line for IFC advisory services was Cross- Industry (5 projects). MIGA issued six guarantees. 18. The performance of the WBG program in Rwanda has been above the Bank-wide average portfolio performance during FY09-17. IDA portfolio performance in Rwanda was notably good, with the share of commitments at risk a low 5.7 percent during FY09-17, compared with Sub-Saharan Africa and Bank-wide averages of 29 and 20 percent, respectively. 7 The percentage of operations exiting the portfolio with IEG outcome ratings of moderately satisfactory or better stood at 88 percent over FY09-17 (100 percent after FY11), compared to 64.9 percent for Sub-Saharan Africa and 74.2 percent globally. IFC investments and AS performance was more mixed. 19. After political developments saw a sharp but temporary dropoff in ODA in 2012,21 Bank engagement continued to feature budget support, and saw the introduction of the P4R instrument. In the early part of the evaluation period (and even prior to it), the Bank had relied heavily on budget support—delivered notably in the form of poverty reduction support credits (PRSCs) or grants—to provide financing to a reform-minded government. However, in 2012, controversy surrounding alleged Rwandan involvement in fomenting conflict in neighboring DRC led several development partners to suspend or significantly reduce their support (notably funds previously delivered using the PRSC platform). Although it discontinued use of the PRSC instrument, the Bank was able through discussions with partners to secure a consensus that allowed it to continue with budget support in 201322 to help safeguard Rwanda’s growth and poverty reduction gains. Following the 2012 decline, overall ODA to the country rebounded, although ODA as a share of GNI has trended downward slightly. At the same time, some increases in foreign direct and equity investment have been observed. However, the 2012 episode sharpened government attention to aid dependency and the predictability of ODA flows, and the Bank was asked to find ways to deliver more predictable financing. Since then, DPF has been used mainly in support of decentralized service delivery and social protection, while P4R financing has begun supporting programs in agriculture, public sector governance, and education and skills development. Previous Evaluations 20. While the last comprehensive in-depth assessment of the Rwanda program dates to well over a decade ago, numerous sector and thematic evaluations have looked at specific aspects of WBG support. Attachment 2 provides details on the last (2004) Rwanda Country Assistance Evaluation and recent IEG thematic evaluations that have encompassed a Rwanda case study. The large amount of evaluation material available on Rwanda (both from IEG and other partner institutions) will be used as the essential foundation of the CPE and will help focus its evaluative efforts. At the same time, the CPE will endeavor independently to reconfirm—or qualify— previous evaluation findings. Purpose, Objectives, and Audience 21. The CPE’s primary goal is to assess the development effectiveness of the WBG in Rwanda during the period FY09-17 against the backdrop of Rwanda’s aspirations to reach lower middle income status by 2020, and to inform the design and implementation of future 8 WBG activities in the country. The value added of this report will be to blend existing evaluation findings with new evaluative efforts where there are gaps in order to address the overarching question: Did the WBG strategy and program amount to a coherent “whole” (with suitable complementarity and sequencing of instruments) that helped the country tackle issues critical to attaining the MIC—and later HIC—status to which it aspires? In this regard, the CPE work is closely related to the two strategic objectives in IEG’s results framework: determining “what works and why” and facilitating “real-time learning.” In terms of what works, the CPE will seek to deepen evidence about the results achieved by the WBG program in Rwanda and how it contributed to WBG strategic priorities – ultimately to the Twin Goals. The CPE will also seek to facilitate real-time learning on the effectiveness of WBG operational choices in Rwanda. In so doing, the CPE can help inform the next CPF as well as the design and implementation of WBG strategies in countries that share similar development challenges. For example, useful lessons may emerge for post conflict countries seeking to build social cohesion or manage geo-political drivers of donor support, or for countries facing challenges in gender equity or demographic trends and youth employment. 22. The two specific objectives of the Rwanda CPE are to assess, over the FY09-17 period: (i) WBG strategic positioning and program delivery relative to the country’s higher-order goals; and (ii) the relevance and effectiveness of WBG support under each of the specific areas where it was involved. While ratings of the extent to which the WBG achieved its relevant objectives will be provided in line with the traditional accountability mandate of CPEs, the emphasis in the document’s organization and design will be on lesson-learning and recommendations apt to help enhance program effectiveness. 23. The CPE is expected to be of potential use to the Government of Rwanda, the WBG, and other development practitioners and stakeholders. The intended audience for the Rwanda CPE consists primarily of the WBG Board, WBG Management and Staff, government counterparts, and other stakeholders interested in Rwanda’s development achievements and associated WBG contributions (e.g., civil society and private sector organizations, and other development partners present in Rwanda). Given Rwanda’s track record as one of the strongest reformers in Africa, findings would also likely be of substantial relevance to WBG development effectiveness across a broad spectrum of client countries sharing similar development challenges and aspirations. Moreover, given the large share of budget support in the Rwanda portfolio, CPE findings would likely be of interest to OPCS in the context of the 2018 DPF retrospective. 9 Evaluation Questions and Coverage/Scope Scope of the Evaluation 24. The scope of the CPE will be determined by the key areas in which the WBG aimed to make significant contributions, which changed little over the evaluation period. Despite some variation in strategic objectives over time, the broad areas of WBG intervention have not changed significantly over the evaluation period; the most marked change in direction has been the post-2014 focus on urbanization. Table 2 presents the strategic objectives or “pillars” in the two WBG documents that framed the institution’s engagement in Rwanda. Table 2. Strategic Pillars in the FY09-12 CAS and FY14-18 CPS WBG strategy documents Pillar 1 Pillar 2 Pillar 3 FY09-12 CAS Promote Rwanda’s economic Decrease Social vulnerability (Extended to transformation for sustained (targeted interventions linked to FY09-13) growth (agriculture, infrastructure, health, social protection and environment for PSD, demobilization/integration of soldiers) management of public resources at central and local level) Cross cutting theme: Mainstreaming Support to Capacity Building FY14-18 CPS Accelerate economic growth that is Improving the productivity and Supporting accountable (Extended to private-sector-driven and job- incomes of the poor through rural governance through public FY14-20) creating (energy, urban development and social protection financial management and development, improved (agriculture production, decentralization (improved environment for PSD, integration commercialization and rural road, delivery of decentralized into the EAC regional market) targeted SP interventions) services, strengthened accountability). Sources: FY09-12 CAS and FY14-18 CPS. 25. The CPE will use three organizing—but interlinked—strategic objectives or “pillars” to cover the entire evaluation period (Table 3). The three pillars—promoting economic transformation for sustained growth, reducing social vulnerability and improving the productivity and income of the poor, and supporting accountable governance—adequately consolidate WBG objectives over the CAS and CPS periods. Under the framework’s first pillar, the CPE will assess the relevance and efficacy of the WBG program in encouraging sustainable private sector development through its support in four areas: (i) improving access and quality of key economic infrastructure services; (ii) improving the business environment, notably by fostering skills development; (iii) deepening the financial sector; and (iv) fostering urban development. Under the second pillar, the evaluation will examine the relevance and efficacy of WBG support in decreasing vulnerability and fostering more inclusive growth by helping to: (i) promote agriculture; (ii) reducing vulnerability to health risks; (iii) strengthen and expand the coverage of the social protection system (including support for demobilization and reintegration of ex-combatants). Under the third pillar, the CPE will assess WBG support in areas related to 10 public financial management, decentralization and citizen accountability. Table 3 also highlights in bold the areas where the Bank was asked to lead or strongly engage under the DOL. There are inter-linkages and overlaps across the three pillars: for example, rural electrification can help transform the economy for sustained growth, but is also key to improving rural incomes. Table 3. Evaluation Pillars for the FY09-18 CPE Pillar 2: Decreasing Social Vulnerability and Pillar 1: Promoting Rwanda Economic Improving Productivity and Income of the Pillar 3: Supporting Transformation for Sustained Growth Poor Accountable Governance 1. Infrastructure – Energy, Transport, ICT 1. Agriculture 1. Public Financial Management 2. Business Environment – Skills, 2. Health 2. Decentralization Investment Climate, PPPs 3. Financial Sector – Banking, Rural 3. Social Protection (incl. the Vision Umerenge 3. Citizen Accountability Finance, Capital Markets Program and demobilization and reintegration programs) 4. Urban Sector – Urban Planning, Housing, Urban Infrastructure Sources: IEG. Evaluation Questions 26. To assess to what extent WBG support met its relevant objectives, the CPE will examine two sets of questions mapped to its two specific objectives (para 21). 27. Under the first objective, evaluation questions will assess the overall strategic coherence of WBG support, including the extent to which it takes account of Rwanda’s specific context. The CPE will address some of the following questions depending on the quality of data and evidence collected during the implementation of the evaluation. Was overall packaging and sequencing of WBG activities and products well aligned with the country’s quest to become a LMIC by 2020? How well did the WBG exercise intellectual leadership in policy dialogue in sectors where it took the lead? To what extent did the program seek to foster synergies—both intra-WBG and with other development partners (DPs)—to help Rwanda achieve its development goals? In particular, how did the WBG program accommodate the government’s active leadership in managing development partner support? Did the WBG work effectively with other DPs under the DOL arrangement, notably in its leadership and delegated cooperation agreement/silent partnership areas, to help Rwanda advance its structural transformation? How disruptive was the DOL, specifically in areas where cooperation was discontinued, and can lessons be drawn on how to strengthen initiatives of this kind? Among the key items to examine—and to draw lessons from—in assessing overall WBG strategic positioning are how it has taken into account context-specific issues. To what extent was the WBG program tailored to the country’s aspirations for rapid development through specialization as a service-oriented regional logistics hub? How did the WBG program approach political economy factors, 11 including historical drivers of conflict in the country? What were the pros and cons of the operating environment of rapidly changing government reform goals, and how did the WBG program adapt? How customized to Rwanda’s needs was WBG support for entrepreneurship development, given the challenges of employing a very young labor force? Did the WBG program account for the need to balance rapid growth with reduced inequality? And to what extent did WBG support the development of client capacity to make more efficient, equitable, and sustainable use of its resources? Specific evaluation questions that will guide the CPE’s assessment of the WBG’s overall strategy and program in supporting Rwanda’s development aspirations are set out in greater detail in Attachment 1. 28. Under the second objective, questions will focus on assessing development effectiveness under each (sub-)pillar in Table 3. Questions regarding relevance and effectiveness of WBG support will be structured around the results frameworks (RFs) that the WBG put forward in its strategies. While taking account of RF quality (e.g., soundness of the underlying logic and quality of indicators used), the results chain will help trace through links between WBG strategic objectives, WBG activities and associated outputs, and the higher-order outcomes that it sought to influence, in assessing the likelihood of a WBG contribution. Does the combination of WBG lending and non-lending activities deployed in search of a particular outcome make sense, or are obvious gaps left unfilled? What influence, if any, did the DOL arrangement have? Did the WBG activities help realize the policy changes, investments, knowledge, or capacity development sought in the time frame envisioned? And what happened at the level of intermediate outcomes (increased generation of and access to electricity, say) and higher-order outcomes (private-sector-driven, job creating growth)? The CPE will also assess the sustainability or resilience to risk of the development benefits to which the WBG program may have contributed under each (sub-) pillar. More specifically worded questions under each (sub-) pillar are detailed in Attachment 1. Evaluation Design and Evaluability Assessment 29. Inherent limitations on the evaluability of a country program notwithstanding, the CPE will examine the likelihood of WBG program contribution to Rwanda’s development results. The broad stability in the development objectives underlying the WBG program and the use of results frameworks to relate the outputs of WBG interventions—policy changes, investments, increased knowledge and capacity, and harmonization, among others —to the intermediate and final outcomes sought will help structure and facilitate the evaluation process. However, the fundamental problem of establishing a deterministic causal relationship between WBG interventions—structured as they are into an overall program, not simply as individual lending and non-lending products—and higher-order country outcomes persists. While in many cases the problem is ultimately insurmountable, the CPE will endeavor to assess the likelihood (or otherwise) of WBG contribution. Only by clarifying and juxtaposing the various links in the 12 results chain (para. 27), and by assessing the “whole” to which the “sum of the parts” of the WBG program amount, can the likelihood of WBG additionality be assessed. The picture is of course further complicated by the presence of an unusually large number of active DPs in Rwanda. Analytical Methods and Data Requirements 30. The CPE will draw on a broad range of informational sources and analytical methods.23 The evaluation design matrix (Attachment 2) presents greater detail on the various informational/data needs and sources, as well as the data collection and analytical methods envisaged, to answer the main evaluation questions. As described further below, the CPE will use a combination of quantitative and qualitative data/information and methods, document reviews, and stakeholder interviews to address the evaluation questions, and the use of diverse sources of information will provide for some “triangulation” to test robustness of findings. Source of Information 31. Data and documentation. No primary collection of quantitative data is envisioned for the Rwanda CPE. Collecting the required information will therefore involve drawing on WBG, country, or other partner—for example, IMF, UN, or OECD—quantitative databases, as well as gathering all available WBG, IEG, and relevant outside documentation.24 32. Desk reviews of operational and evaluation documents. The CPE team will undertake a thorough desk review of country-level documents as well as specific project and program documents—including (for lending products) ISRs, ICRs, and XPSRs, and (for non-lending products) economic and sector work and advisory services documents, technical notes, and policy briefs. In terms of prior evaluation work, the CPE team will draw upon the IEG review of the FY09-12 CAS. In addition, during the evaluation review period, IEG has reviewed 100 percent of the completed ICRs for Rwanda Projects, and 100 percent of ICRs for regional projects involving Rwanda, although ICRs are still lacking on 4 closed projects. A large part of the portfolio includes lending projects that are still active, so ICRs/ICRRs are available for just 52% of lending operations and 29% of regional projects. During FY18, IEG is undertaking PPARs for the PRSG IV-VII series delivered over FY08-11 and the 2013 DPO on decentralized services. Given the evaluation’s limited budget, the large amount of already available evaluation material on Rwanda, both from IEG and other partner institutions (Attachment 3 Boxes 1 and 2), will provide a foundation for the CPE and help clarify where fresh evaluative efforts are needed. 33. IEG “deep dive week.” The CPE team will organize a series of semi-structured interviews——with current and former Country Directors, Country Managers, and Task Team Leaders, including IFC and MIGA staff, to get their views on WBG strategy and operations in 13 Rwanda. Since the IEG deep dive week will take place immediately following the initial desk review, the CPE team also plans to use the discussions to begin testing some of its preliminary findings. 34. Stakeholder views and opinions. Information will also be gathered through semi-structured interviews with representatives of the various stakeholder constituencies. During the mission, the CPE team will collect views from: (i) country officials, on overall and specific aspects of WBG support; (ii) other country stakeholders, such as beneficiaries and civil society representatives, on specific aspects of WBG support and the quality of consultation; and (iii) development partners, on the quality of WBG coordination and collaboration with other partners. Analytical Methods 35. The CPE will use multiple methods and data for the analysis of the above information to respond to the evaluation questions. Analysis of available macro- and socio-economic data will help gauge evolution of, and variations in, conditions over time. Analysis of WBG portfolio quality will involve tracing the evolution of key summary quality indicators (for example, the extent to which development objectives have been—or are likely to be—met, implementation status, and proactivity indices) over time and effecting comparisons with the same measures in comparator countries and in WBG programs in general. 25 Desk-based reviews—including content analysis, where relevant—of the available WBG, IEG, and outside documentation, as well as of notes from interviews with stakeholder representatives, would be mainly qualitative in nature. Regarding the WBG country strategy, retrospective reviews would be undertaken to compare outturns with outputs and outcomes targeted ex ante. 36. In support of the overall assessment of the WBG strategy and program, the evaluation will also focus on partnerships, gender, and capacity development issues. In collecting evidence to assess the treatment of these aspects, the evaluation team will: (i) ensure that attention to partnerships, gender, and capacity are mainstreamed in interviews with WBG staff, partners, and client representatives; (ii) review the treatment of these aspects in relevant documents and strategies, including results frameworks; (iii) undertake portfolio analysis with special reference to these aspects; and (iv) conduct in-depth interviews with staff, donors, and partnership representatives specifically focused on these aspects. With respect to capacity development, the team will take guidance from the 2006 OECD-DAC report on the challenge of capacity development and the 2008 ECDPM capacity, change and performance report. With respect to partnerships, social network analysis could provide a useful approach to assessing WBG positioning within the crowded DP landscape in Rwanda. While it is unlikely that time and resources will allow the CPE team to conduct original analysis, potential synergies in this regard with the regional integration evaluation will be pursued and capitalized upon. IEG’s methods 14 adviser will be consulted as needed to help respond to the evaluative questions described in the approach paper. Quality Assurance Process 37. The quality assurance process will be ensured through appropriate peer review arrangements and deployment of the full range of IEG knowledge resources. Aloysius Ordu (former Country Director, African Development Bank; and former Operations Director, World Bank OPCS) and Monika Huppi (IDB) have agreed to serve as external peer reviewers. Together, the peer reviewers offer expertise in a range of thematic and operational matters pertinent to Rwanda. The report will be prepared under the direct supervision and guidance of IEGEC manager and the IEGHE Director, and the review process will engage all the IEG units and associated expertise. Expected Outputs, Outreach, and Tracking 38. Outputs. The main output for this task will be a CPE evaluation report (the CPE) that presents relevant findings, lessons and recommendations. Once discussed by the CODE sub- committee of the Board, the report will be disclosed publicly in accordance with IEG’s access to information policy. Depending on the findings, shorter briefs (for example, IEG insights), blogs, or podcasts may also be considered subsequently. 39. Outreach. Appropriate opportunities for disseminating the report to internal WBG audiences will be explored, including specific events organized jointly with the Rwanda country team26. Suggestions for in-country events to engage stakeholders on CPE findings and recommendations will be developed in consultation with the WBG country team, Rwandan counterparts, and development partner representatives. A complete dissemination and outreach strategy will be prepared prior to the one stop review meeting for the full report. Timeline and Team 40. Timeline. Tasks will be sequenced, starting with desk reviews of the documentation relating to WBG strategy, IDA lending and AAA tasks, IFC investments and advisory services, and MIGA guarantees. This will be followed by interviews with WBG during the deep dive week, planned for early December. A country visit is planned for end January-early February to complete the assessment and triangulate findings. The evaluation report will be submitted to CODE by end May 2018. 41. Team and skills mix. The IEGEC evaluation team will comprise Florence Charlier (Task Team Leader, Senior Economist), Yumeka Hirano (Economist), Takatoshi Kamezawa (Senior 15 Evaluation Officer), Lourdes Pagaran (Senior Evaluation Officer), and Ali Khadr (Consultant). Team composition reflects broad thematic and evaluative expertise. Corky De Asis and Carla F. Coles will provide research assistance and administrative support to the task team. 16 References IEG (Independent Evaluation Group). 2013. Youth Employment Programs. An Evaluation of World Bank and IFC Support. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/ye_eval.pdf ———. 2014a. The World Bank Group and the Global Food Crisis: An Evaluation of the World Bank Group Response. Washington DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/food_crisis_eval_1.pdf ———. 2014b. World Bank Group Support for Innovation and Entrepreneurship: An Independent evaluation. Washington, DC: World Bank http://ieg.worldbankgroup.org/sites/default/files/Data/reports/innovation_eval.pdf ———. 2014c. Investment Climate Reform: An Independent Evaluation of World Bank Group Support to Reforms of business regulations. Washington, DC: World Bank https://openknowledge.worldbank.org/handle/10986/22724 ———. 2014d. World Bank Group Support to Health Financing. Washington, DC: World Bank https://ieg.worldbankgroup.org/sites/default/files/Data/reports/chapters/health_finance_ev aluation_w_appendix_updated_0.pdf ———. 2014e. Rwanda - Country Assistance Strategy Completion Report (CASCR) review for the period FY09-13. Washington, DC: World Bank http://ieg.worldbankgroup.org/sites/default/files/Data/reports/Rwanda_CASPRR_Rev_8827 60CASCR0P1010Box385212B00OUO090_0.pdf ———. 2015. World Bank Group Support to Electricity Access, FY2000-2014: An Independent Evaluation. Washington, DC: World Bank http://ieg.worldbankgroup.org/sites/default/files/Data/reports/Electricity_Access.pdf ———. 2016a. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Washington, DC: World Bank http://documents.worldbank.org/curated/en/171501476992732097/pdf/109333-WP-REVISED- PUBLIC.pdf ———. 2016b. Program-for-Results: An Early-Stage Assessment of The Process and Effects of A New Lending Instrument. Washington, DC: World Bank http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/program-for-results- full.pdf ———. 2017. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Washington, D.C.: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/datafordevelopment.pdf 17 World Bank. 2008. Rwanda - Country Assistance Strategy for The Period FY09-FY12. Report No.44938- RW. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/754501468336542352/pdf/449380CAS0P1001ON LY10IDA1R200810257.pdf ———. 2011. Rwanda - Country Assistance Strategy Progress Report. Report No. 58075-RW. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/975751468106734590/pdf/580750CASP0P111OFF ICIAL0USE0ONLY161.pdf ———. 2014a. Rwanda Country Partnership Strategy FY2014-2018. Executive Summary. Report No. 88941 V1. Washington, DC: World Bank http://documents.worldbank.org/curated/en/596091468305661843/pdf/889410CAS0P1300e0su mmary000Volume01.pdf ———. 2014b. FY201-2018 Country Partnership Strategy for Rwanda. Report No.88941 V2. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/735901468336703252/pdf/889410CAS0P1300wan da0CPS000Volume02.pdf ———. 2014c. Rwanda Country partnership strategy FY2014-2018. Report No. 88941 V4. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/608101468306898768/pdf/889410v30Box380FY20 14020180Annexes.pdf ———. 2017. Rwanda. Performance and Learning Review of The Country Partnership Strategy for the period of FY14 - FY18. Report No. 106731-RW. Washington, DC: World Bank http://documents.worldbank.org/curated/en/857581492048851343/pdf/Final-PLR-March-20- 2017-Final-for-SECPO-03202017.pdf 18 Attachment 1 Detailed Evaluation Questions A. Specific Queries under the first objective: Assessing the “overall strategic coherence” of the WBG’s approach in relation to Rwanda’s goal of reaching lower middle income status by 2020 1. The CPE will address some of the following questions depending on the quality of data and evidence collected during the implementation of the evaluation. 2. Relevance of Objectives: Were WBG strategic objectives well aligned with the government’s own development objectives as spelled out in its strategic documents (EDPRS1 and EDPRS2), in turn guided by Rwanda vision 2020 (as updated in 2012)? Were overall development objectives appropriate and realistic given initial conditions and forecast challenges? Did the sector focus of WBG strategic objectives reflect the institution’s comparative advantage? Was the full set of priorities needed to ensure progress towards Rwanda’s aspirational goals adequately covered by the WBG or other DPs? To what extent did the WBG assess and mitigate risks of inadequate coverage of priorities in sectors where it was not present? 3. Relevance and effectiveness of the overall package of WBG strategies and interventions in support of Rwanda’s goals: Were WBG interventions adequately customized to Rwanda’s needs and backed by an adequate understanding of the political economy, including its ethnicity- rooted history of conflict and genocide? Did the WBG strategies and program align with the country’s aspirations for rapid Singapore-style development and specialization? Did the WBG help confront demographic challenges, notably that of employing a very young labor force, through relevant education and entrepreneurship programs? Did WBG strategies and interventions help Rwanda balance rapid growth with reduced inequality? Were WBG interventions satisfactorily tailored to the capacity of counterpart organizations? How effective was the widespread use of budget support, and more recently P4R, relative to traditional investment operations, and what (if any) was its catalytic effect? Did the program of knowledge activities help identify key constraints and opportunities in the countr y’s quest to reach LMIC status? Was ASA work complementary to other forms of support provided for in the WBG strategies and program? In particular, were DPF operations underpinned by sound ASA? 4. Flexibility in WBG support: Did the WBG’s strategic approach maintain flexibility to respond to shocks? How did the WBG adapt to the 2012 drop-off in external financing as several partners pulled back? Did the adjustments in WBG support over the evaluation period help address potential gaps in areas deemed critical to meeting the country’s goals? 19 5. Synergies within the WBG and with DPs: The WBG aimed to contribute to Rwanda’s objectives of accelerating structural transformation and reducing social vulnerability through a program of lending and non-lending services—in sectors determined through a highly proactive government approach to aid coordination. o Within the WBG: Did the WBG create synergies among IDA/IFC/MIGA activities to help boost the effects of its support? Were there synergies across WBG intervention areas/sectors? o With other DPs: How was the DOL approach implemented in practice? Did the WBG cooperate efficiently with other donors to optimize aid effectiveness, contain transaction costs, and lessen the drain on Rwanda limited capacity? Specific questions will review the following aspects of partnerships: Were the sector working groups and the Development Partners’ Consultative Group effective platforms to engage with the government on policy dialogue and reporting? Was the WBG able to work closely with DPs to align technical assistance (TA) and capacity building activities? How was the sequencing of activities impacted as the sectors where the WBG was asked to lead changed over the evaluation period? To what extent did this influence progress in the implementation of the reform agenda? When the WBG was absent from policy dialogue in sectors in which it was expected to contribute in a “silent” way, there is a risk of the WBG not having been able to influence the policy environment in which its activities took place. Did this risk materialize and limit the contribution of its activities towards Rwanda’s higher-level development objectives or was the WBG able to work efficiently under silent partnership or delegated cooperation agreement (to ensure views were adequately shared and conveyed to the GoR by the lead donor)? In sectors where it was completely absent, was the WBG still able to use partnerships to make its global development experience available to Rwanda? 6. Monitoring and Evaluation: Were WBG results frameworks (including associated indicators) of good quality and well aligned with the common performance assessment framework (CPAF), established by the Government and DPs as the basis for budget support and tracking performance of EDPRS implementation? Was the CPAF an effective framework to monitor national-level progress towards higher-order goals? What, if any, were its drawbacks? Was the WBG program flexible enough to adapt to annual CPAF revisions? Were WBG results frameworks effectively used to conduct M&E? 20 B. Specific queries under the second objective: Assessing the achievement of relevant WBG objectives by (sub-) pillar, with reference to the results frameworks 7. For each WBG strategic (sub-) objective or pillar, the CPE will evaluate relevant results achieved by the country and assess the likelihood of WBG contribution. The discussion will include an assessment of whether the choice of financing instruments used were appropriate, as well as of knowledge work, notably whether it helped identify risks and provided relevant recommendations to guide support and help enhance its impact. The CPE will also assess to what extent the WBG helped address institutional capacity constraints to development initiatives. To what extent were achievements impacted where the Bank was working under delegated cooperation or silent partnership arrangements? 8. The paragraphs below set out specific questions under each (sub-) pillar to guide assessments of the extent to which WBG strategic objectives and targeted outcomes were achieved. (i) Pillar 1 9. The over-arching question concerns the relevance and effectiveness of WBG programs in promoting Rwanda’s economic transformation for sustained growth. The Rwanda EDPRS2 looks to the private sector to drive growth and job creation over the medium term. The evaluation will assess to what extent WBG support contributed to: (i) driving down the cost of doing business by removing key bottlenecks in infrastructure, the financial sector, and the business environment; and (ii) spurring urbanization to develop the potential of secondary cities and reap the benefits of increased agglomeration. 10. Infrastructure: The high cost and low reliability of energy is one of the primary bottlenecks to enterprise development. Given that the WBG is the lead donor in energy, specific attention will be devoted to this sub-sector to answer the following questions: To what extent did the WBG help the government develop power generation and rehabilitate and expand Rwanda’s national transmission and distribution network to improve access to and reliability of electricity? How effective was WBG support in connecting Rwanda to the regional transmission network within the East African Power Pool to reduce overall generation investment costs? Did the WBG contribute to strengthening the financial and technical efficiency of the Electricity Regulatory Agency? To what extent did the WBG contribute to promoting private sector participation in renewable energy? The CPE will also seek to assess to what extent the WBG support in: (i) transport contributed to improve regional connectivity and access to priority urban infrastructure in Kigali and secondary cities; and (ii) ICT helped improve domestic and international connectivity. 21 11. Urban Development : The Bank was asked in preparing the FY14 CPS to take the lead in the urban sector. At 15 percent, Rwanda is among the least urbanized countries in SSA. Given land constraints, industry is expected to sustain 8 percent annual growth and create non-farm jobs, which will require developing the potential of secondary cities beyond Kigali. The evaluation will assess the relevance of early Bank engagement and to what extent WBG support is contributing to improving urban planning and developing funding mechanisms for affordable housing. 12. Investment Climate: How effective was WBG in helping to shape a business environment with reduced entry costs, operational risks, and distortions to a level playing field? To what extent did the WBG support enterprise development and trade expansion through sustainable development of new products and markets? Did the WBG contribute to building skills and capacity in the private sector?27 Was the WBG able to promote entrepreneurial skills and investments through catalytic interventions? 13. Financial Sector: Businesses rate insufficient access to affordable credit as one of the biggest constraints. Bank penetration is very low compared to regional peers. Domestic capital markets, which could provide a source of lower-cost and longer-term local currency financing for the private sector, remain under-developed. The evaluation will assess to what extent the WBG contributed to strengthening the legal and regulatory framework and supervisory capacity in the financial sector? Did the WBG contribute to greater financial inclusion for individuals as well as micro-, small-, and medium-sized enterprises (MSMEs), including in the rural sector? To which extent did the WBG contribute to developing capital markets? (ii) Pillar 2 14. The over-arching question concerns the relevance and effectiveness of WBG programs in decreasing social vulnerability and promoting the productivity and income of the poor. This pillar will assess how effective Bank support was in promoting growth of the rural economy and improving rural livelihood and building effective national social protection programs. 15. Rural Development : Given that 69 percent of the Rwandan labor force depends on agriculture28, increasing income from agriculture is the most effective way to reduce poverty and increase shared prosperity. Under the 2010 DOL, the Government asked the Bank to lead in agriculture and to remain engaged in this sector in the FY14 CPS. Therefore, the CPE will take an in-depth look at the extent to which Bank support has achieved its objectives and the sustainability of results over time. Specific questions include to what extent the WBG has contributed to helping Rwanda: (i) intensify agricultural productivity and sustainability, notably through irrigation and sustainable land management; (ii) facilitate transition from subsistence to more commercial practices, including through the provision of financing and market-oriented 22 infrastructure (such as feeder roads and storage to facilitate market access); and (iii) develop new agribusiness projects. 16. Health: Prior to the evaluation period, the WBG supported major health sector reforms that contributed to greater efficiency and equity in healthcare service delivery, notably through performance financing. The FY09 CAS sought to build on this agenda with a specific focus on maternal and child health. However, the CASPR noted that in line with the 2010 DOL, the Bank would phase out lending activities in health, which would increasingly come from other development partners. In light of this, the CPE will review Bank achievements as reported in the CASCR and relevant ICRs and the associated IEG Reviews as well as the IEG Health Financing evaluation. It will not, however, conduct an in-depth primary assessment of Bank involvement in health during the evaluation period. 17. Social Protection: While the Bank has been a silent partner in social protection under DOL, it supported this sector with significant lending, and social protection accounted for 25 percent of the total portfolio during the evaluation period. The CPE will assess to what extent WBG support contributed to enhancing effectiveness and expanding coverage of social protection mechanisms to serve those most in need. The CPE will also assess the implications of silent partnership under the DOL on the Bank’s own processes, notably its ability to mobilize resources for investment and technical assistance in response to government needs. Did Bank assistance contribute to harmonizing the main social protection programs,29 increasing their transparency, and strengthening government implementation capacity? How sustainable is the Vision Umurenge Program (VUP) and what can the Bank do to help strengthen the prospects for its sustainability? How effective was the Bank in supporting demobilization and reintegration of former combatants and their families? (iii) Pillar 3 18. The over-arching question concerns the relevance and effectiveness of the WBG programs in supporting accountable governance. 19. Strengthening public financial management : To what extent did the WBG contribute to increasing the effectiveness of public resource use at the central level through improved public financial management—including budgeting, accounting and auditing, procurement, reporting, and external oversight? Did WBG support for the government’s strategic approach to capacity strengthening help address the issues of poor retention, high turnover, and skills gaps that afflict the civil service? 20. Decentralization: To what extent did the WBG work with the national and local governments on the decentralization framework, as well as to enhance revenue generation and 23 administration to underpin local governments’ broadened service delivery responsibilities? Did the WBG help strengthen decentralized service delivery through improved transparency, value for money, and accountability in the use of public funds? Did WBG support contribute to building capacity at subnational level? Overall, what lessons can be drawn from Rwanda’s decentralization experience and Bank support for it? 21. Strengthening social accountability: One stated priority of EDPRS2 is to mobilize citizen participation to help improve service delivery quality. Did Bank-supported activities contribute to expanding the role of civil society organizations and encouraging fiduciary accountability on their part? (iv) Cross-cutting issues 22. Whenever relevant, the assessment under each (sub-) pillar will discuss the following dimensions: 23. Gender: Did the WBG recognize gender issues and impacts in country strategy documents and track gender-specific outcomes in its results frameworks? Did the Bank undertake any gender-focused analytical work? Have WBG programs been effective in helping to identify and make progress towards gender-based outcomes related to human development, economic empowerment, and voice and participation? 24. Regional Integration: Given Rwanda’s size and location, stronger regional integration will be crucial to sustain long-term growth. To what extent was the WBG able to initiate projects involving multiple Great Lakes region countries (with benefits spilling over national boundaries), notably in trade and infrastructure crucial to Rwanda’s connectivity, using regional IDA funds? To what extent was the WBG able to promote shared or harmonized solutions implemented through the East African Community (EAC)? Did the WBG use regional or multicountry knowledge vehicles to encourage Rwanda’s integration into the EAC regional markets? 25. Environment: Rwanda’s size, topography and high population density present risks to environmental sustainability. Did the Bank help raise awareness on the potential trade-offs between rapid growth and environmental sustainability? To what extent did the Bank seek to mainstream climate resilience and environmental sustainability through its sector operations? 24 Attachment 2 Evaluation Design Matrix Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods A. Evaluation questions regarding the “overall strategic coherence” of the WBG’s approach in relation to Rwanda’s goal of reaching lower middle income status by 2020 Relevance of Objectives: Development objectives The WBG country Downloading relevant Content analysis, to Limitation on data availability, Were WBG strategic objectives in development strategies documents from WBG assess the extent to especially on initial conditions well aligned with the strategies databases and the which objectives during/after civil war government’s own development The government’s website (WBG, IMF, UN, reflected priority issues objectives as spelled out in its Country socioeconomic strategies AfDB, governments, etc.) and took into account Limitation on the assessment strategic documents, in turn context and major country context. on relevance and guided by Rwanda vision 2020? development challenges Major Interviews with key effectiveness of the WBG Were overall development macroeconomic stakeholders: bank group Time-series and cross- support under the FY14-18, Regional and country objectives appropriate and databases and socio staff, researchers, and country analyses which was extended to FY20 risk profiles realistic given initial conditions economic indicators experts, government and forecast challenges? Did counterparts and civil Qualitative analysis of Interviewee sample the sector focus of WBG Findings from society interviews limitations strategic objectives reflect the interviews institution’s comparative advantage? Was the full set of priorities needed to ensure progress towards Rwanda’s aspirational goals adequately covered by the WBG or other DPs? To what extent did the WBG assess and mitigate risks of inadequate coverage of priorities in sectors where it was not present? Relevance and effectiveness Evidence on policy Country strategies, Downloading documents Desk review of country of the overall package of dialogues in supporting project documents, and data from the WBG strategies and projects WBG strategies and and IEG evaluations website. documents 25 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods interventions in support of the achievement the Rwanda’s goals: Were WBG development goals. Aid memoire Interviews with key Qualitative analysis of Staff turnover in partner interventions adequately Evidence on influence stakeholders, especially interviews agencies can limit feedback customized to Rwanda’s needs and development impact Findings from with stake holders who and backed by an adequate of the WBG support interviews have institutional understanding of the political memories in long-term Project Difficulties and sensitivities economy? Did the WBG Project portfolios WBG project portfolio/statistical involved in assessing political strategies and program align databases Interviews with key analyses economy with the country’s aspirations? Lessons on political stakeholders Did the WBG help confront economy from previous IEG evaluation on Limited evidence on demographic challenges operations P4R effectiveness of P4R. Four through relevant education and Downloading relevant P4R projects were approved entrepreneurship programs? Levels and changes in IEG evaluation on data from WBG during FY15-18 and they are Did WBG strategies and quality of trust fund and data for databases still active. interventions help Rwanda policy/institutions development balance rapid growth with reduced inequality? Were WBG Lessons on institutional Governance and interventions satisfactorily capacity from previous institutional indicators tailored to the capacity of operations from WBG, think- counterpart organizations? How tank, etc. effective was the widespread Analytical underpinnings use of budget support and P4R, of country and regional Key stakeholders; relative to traditional investment strategies and project development partners operations, and what (if any) design and area experts was its catalytic effect? Did the inside and outside the program of knowledge activities WBG Number of AAA and relevant Usage and impact of help identify key constraints and literature are high and knowledge work opportunities in the country’s household data is rich quest to reach LMIC status? (frequency and quality), as Was ASA work complementary compared with other SSA to other forms of support? In countries. particular, were DPF operations underpinned by sound ASA? 26 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods Flexibility in WBG support: Evidence on flexibility in Key stakeholders; Interviews with key Qualitative analysis of Did the WBG’s strategic WBG support development partners stakeholders interviews approach maintain flexibility to and area experts respond to shocks? How did the Data on financial flow inside and outside the WBG adapt to the 2012 drop-off and budget WBG Downloading relevant Project in external financing as several data from WBG portfolio/statistical partners pulled back? Did the WBG project databases analyses adjustments in WBG support databases and over the evaluation period help economic databases address potential gaps in areas deemed critical to meeting the country’s goals? Synergies within the WBG Evidence on synergies Country strategies, Downloading from the Desk review of country and with DPs: in the WBG engagement project documents, WBG and government strategies and projects Within the WBG: Did the WBG IEG evaluations and websites documents create synergies among Evidence on donor other relevant IDA/IFC/MIGA activities to help coordination, documents (e.g., Qualitative analysis of boost the effects of its support? harmonization with other CPAF-related Interviews with key interviews Were there synergies across development partners, material, joint budget stakeholders, especially WBG intervention and effective platforms support reviews, joint with IMF, EU, AfDB and areas/sectors? for policy dialogue sector reviews) other development With other DPs: How was the partners DOL approach implemented in Evaluations by DPs practice? Did the WBG cooperate efficiently with other Findings from donors to optimize aid interviews effectiveness, contain transaction costs, and lessen the drain on Rwanda limited capacity? 27 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods Monitoring and Evaluation: Monitoring indicators Projects documents Downloading from the Retrospective review of Were WBG results and the project results (progress/completion WBG website the results frameworks frameworks (including reports) associated indicators) of good Assessment of the IEG reports, including quality and well aligned with design of results ICRRs and PPARs the common performance frameworks assessment framework (CPAF), established by the Government and DPs as the basis for budget support and tracking performance of EDPRS implementation? Was the CPAF an effective framework to monitor national- level progress towards higher- order goals? What, if any, were its drawbacks? Was the WBG program flexible enough to adapt to annual CPAF revisions? Were WBG results frameworks effectively used to conduct M&E? B. Evaluation questions on the achievement of relevant WBG objectives by (sub-) pillar, with reference to the results frameworks To what extent did the WBG contribute to the achievement of objectives and related sub-objectives under each of the three pillars and cross-cutting issues? In addition to discussing the pertinence of each sub-objective (and the indicators used to measure progress) under the respective pillars, the evaluative questions under each pillar will assess, inter alia: (i) Were the instruments used consistent with the objective? (ii) What were the actual results achieved by Rwanda in the respective area? (iii) What was the contribution of the WBG to the results achieved? (iv) To what extent did the WBG help address institutional capacity constraints to development initiatives? Overarching Questions – Pillar 1: 28 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods How relevant and effective were the WBG programs in promoting Rwanda’s economic transformation for sustained growth by (i) driving down the cost of doing business by removing key bottlenecks in infrastructure, the financial sector, and the business environment; and (ii) spurring urbanization to develop the potential of secondary cities and reap the benefits of increased agglomeration? Infrastructure: To what Evidence on Country strategies, Downloading documents Desk review of country Potential weakness of links of extent did the WBG help the effectiveness of relevant project documents, and data from the WBG strategies and projects infrastructure to GoR to (i) develop power interventions for and IEG evaluations website documents economic/business outcomes generation, rehabilitate and infrastructure due to other binding expand Rwanda’s national development IEG evaluation on Qualitative analysis of constraints transmission and distribution electricity access Interviews with key interviews network to improve access to stakeholders and reliability of electricity? (ii) IEG PPARs on improve regional connectivity regional power in transport and access to projects and highway priority urban infrastructure in projects Kigali and other secondary cities? (iii) improve domestic Findings from and international connectivity interviews in ICT? Urban Development: How Evidence on relevance Project documents, Interviews with key Limitations on availability of relevant was the Bank’s early of WBG support for including supervision stakeholders evidence on urban engagement and to what Rwanda’s urban reports development projects were extent is WBG support development initiated in recent years. This contributing to improving evaluation will focus on urban planning and relevance. developing funding mechanisms for affordable housing? Investment Climate: How Evidence on WBG Doing Business Downloading documents Desk review of country Attribution problems, effective was WBG support in effectiveness of relevant Reports, Enterprise and data from the WBG strategies, projects especially given the various helping to shape a business interventions in Survey 2011 and the government documents, WBG factors affecting investment environment with reduced improving business website. reports. climate. There may be entry costs, operational risks, environment IEG evaluations on mitigating or contributing and distortions to a level investment climate factors that are difficult to 29 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods playing field? To what extent reforms, youth Interviews with key Qualitative analysis of identify such as contributions did the WBG support employment, and stakeholders, including interviews of local actors and other enterprise development and industrial business leaders/owners, donors. trade expansion through competitiveness and business associations, sustainable development of jobs international investors new products and markets? etc. Did the WBG contribute to National Manpower building skills and capacity in survey 2011-12 the private sector? Was the WBG able to promote IEG PPAR on IFC entrepreneurial skills and projects investments through catalytic interventions? Data and statistics on investment Financial Sector: Evidence on Documents on legal Stakeholder interviews Qualitative analysis of Impact of global financial To which extent did the WBG effectiveness of relevant and regulatory interviews crisis and other regional and contribute to strengthening the interventions in framework of the global events may overwhelm legal and regulatory strengthening financial financial sector Statistical data that of reform efforts, leading framework and the sector, promoting analyses to challenges in attribution of supervisory capacity of the financial inclusion, and GoR Finscope Survey effects to WBG interventions financial sector? Did the WBG developing capital 2012 on financial contribute to greater financial market. inclusion inclusion for individuals as well as micro-, small-, and Data and statistics on medium-sized enterprises enterprises (MSMEs), including in the (enterprise surveys) rural sector? To which extent and capital markets did the WBG contribute to developing capital markets? Data from the Central Bank and private sector financial institutions Specific Questions – Pillar 2: How relevant and effective were the WBG programs in decreasing social vulnerability and promoting the productivity and income of the poor? 30 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods Rural Development: to what Evidence on Country strategies, Downloading documents Desk review of country Attribution problem in extent has the WBG helped effectiveness of relevant project documents, and data from the WBG strategies, projects promoting productivity Rwanda: (i) intensify interventions for rural and IEG evaluations and the government documents, WBG through WBG support. agricultural productivity and development focusing website. reports. sustainability notably through on agriculture IEG evaluation on irrigation and sustainable land productivity rural non-farm Interviews with key Qualitative analysis of management? (ii) facilitate economy and IEG stakeholders, including interviews transition from subsistence to Projects data on PPAR on Mutara beneficiaries, farmers, more commercial practices agribusiness agriculture projects agricultural association, including through the provision relevant ministries, and of agriculture financing and National surveys and business owners. market oriented infrastructure studies, including (such as feeder roads and Seasonal Agricultural storage to facilitate market Survey Report 2015 access)? (iii) develop new and Comprehensive agribusiness projects? Food Security and Vulnerability Analysis Published literature Findings from interviews IFC reports and data Health: To what extent did the Evidence on CASCR, ICRs and The CASPR noted that in line WBG contribute to the effectiveness of relevant IEG health financing with the 2010 DOL, the Bank achievements regarding interventions in health study would phase out lending greater efficiency and equity of sector activities in health, which health services? Rwanda Population would increasingly come from and Housing Census other development partners. 2012, Demographic In light of this, the CPE will Health Survey review Bank achievements as reported in the CASCR and relevant ICRs and the 31 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods associated IEG Reviews as well as the IEG Health Financing evaluation. It will not, however, conduct an in- depth primary assessment of Bank involvement in health during the evaluation period. Social Protection: To which Evidence on IEG evaluations on Downloading documents Desk-based review and There is rich household data extent did WBG support effectiveness of relevant food crisis and social and data from the WBG analysis of quantitative (as compared with other SSA enhance the effectiveness and interventions in protection DPL and the government and qualitative data countries, in terms of expand the coverage of social promoting social series. website. frequency and quality) to protection mechanisms to serve protection Time-series and cross- assess income of the poor those most in need? How Household surveys country data analyses and social protection. sustainable is the scale-up of Data and analysis (EICV I-IV) the flagship Vision Umurenge relating to the coverage Difficulty of meaningfully Program (VUP)? Did Bank of social protection, Socio-economic assessing effectiveness in assistance contribute to social protection policy, indicators supporting demobilization, harmonizing the main social socio-economic reinsertion and reintegration protection programs, increasing inclusion, and of former combatants and their transparency, and demobilization their families. strengthening government implementation capacity? How effective was the Bank in supporting demobilization and reintegration of former combatants and their families? Specific Questions – Pillar 3: How relevant and effective were the WBG programs in supporting accountable governance? 32 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods Public Financial Evidence on Country strategies, Downloading documents Desk-based review and Attribution problem given the Management: To what extent effectiveness of relevant project documents, and data from the WBG analysis of quantitative important role played by did the WBG contribute to interventions in PFM and IEG evaluations and the government and qualitative data and many donors. increasing the effectiveness of website. relevant reports public resource use at the IEG PPAR on TA for Interviews with key central level through improved PFM stakeholders, including Qualitative analysis of public financial management— relevant ministries, local interviews including budgeting, accounting Relevant PFM reports governments, and civil and auditing, procurement, (WBG, IMF, etc.) society. reporting, and external oversight? Did WBG support for Findings from the government’s strategic interviews approach to capacity strengthening help address the issues of poor retention, high turnover, and skills gaps that afflict the civil service? Decentralization: To what Evidence on IEG evaluations, Collecting data during the extent did the WBG help local effectiveness of relevant including two PPARs mission governments enhance revenue interventions in on decentralization generation and administration to promoting DPLs. underpin broadened service decentralization delivery responsibilities? Did the Data from local WBG help strengthen government decentralized service delivery through improved transparency, value for money, and accountability in the use of public funds? Did WBG support contribute to building capacity at the subnational level? Social Accountability: Did Limited information on use of Bank-supported activities results data. 33 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods contribute to expanding the role of civil society organizations and encouraging fiduciary accountability on their part? Overarching Questions – Cross-cutting Issues To what extent has Bank support been mainstreaming cross-cutting issues in its strategies and its lending and non-lending products, and to what extent has it been contributing to the achievement of targeted outcomes? Gender: Did the WBG Evidence on Country strategies, Downloading from the Desk review of country This evaluation will focus on recognize gender issues and incorporating gender project documents, WBG website strategies and projects relevance and effectiveness impacts in country strategy perspectives in and IEG evaluations documents of incorporating gender documents and track gender- strategies and projects. (Gender sections in perspectives in strategies specific outcomes in its results RAP, ICRR on Interviews with key Qualitative analysis of and design of projects. frameworks? Did the Bank gender issues, etc.) stakeholders interviews Limited evidence on undertake any gender-focused effectiveness of gender- analytical work? Have WBG Findings from related work. programs been effective in interviews helping to identify and make progress towards gender-based outcomes related to human development, economic empowerment, and voice and participation? Regional Evidence on initiating IEG PPARs on Downloading from the Desk review of country Limitations on availability of Integration/Cooperation: To and implementing regional power WBG website strategies and projects evidence on regional projects what extent was the WBG able strategies and projects projects documents for Eastern Africa. The to initiate projects involving for regional regional projects were multiple Great Lakes region integration/cooperation. Qualitative analysis of initiated in recent years and countries (with benefits spilling Interviews with key interviews many projects are still active. over national boundaries), Data on the progress of stakeholders, including notably in trade and regional stakeholders from EAC Statistical analysis on Challenges in attribution of infrastructure crucial to integration/cooperation and the regional office the progress on effects to WBG interventions. Rwanda’s connectivity, using Potential weakness of links of 34 Data collection Data analysis Key Questions Information required Information sources Strengths and limitations methods methods regional IDA funds? To what regional interventions to sectoral extent was the WBG able to integration/cooperation performance due to other promote shared or harmonized binding constraints solutions implemented through the East African Community (EAC)? Did the WBG use regional or multicountry knowledge vehicles to encourage Rwanda’s integration into the EAC regional markets? Environment: Did the Bank Evidence concerning Country strategies, Downloading from the Desk review of country help raise awareness on the environmental project documents, WBG website strategies and projects trade-offs between rapid growth sustainability in and IEG evaluations documents and environmental strategies and projects. sustainability? To what extent Findings from Interviews with key Qualitative analysis of did the Bank seek to interviews stakeholders interviews mainstream climate resilience and environmental sustainability through its sector operations? 35 Attachment 3 A. Literature Review of IEG Evaluations 1. The 2004 OED Country Assistance Evaluation (CAE), which examined the support provided since 1989, rated the outcome of the Bank’s program unsatisfactory during the pre-genocide period and moderately satisfactory in the post-genocide period.30 After 1994, IDA assistance successfully provided emergency relief and helped with stabilization and budget management under extremely challenging circumstances. Yet, OED noted continuing difficulties in the investment project portfolio and low provision of AAA. 2. In addition to the 2004 CAE, over the past several years, IEG has built up a substantial evidentiary base concerning specific areas of WBG intervention in Rwanda in thematic evaluations and PPARs, which will provide insights on the nature and effectiveness of WBG support under several priority areas covered by this CPE (Box 1). The Two Project Performance Assessment Reports (PPARs) have covered the Bank decentralization and community development projects (2015) and the IFC Bakhresa project (2016). Box 1: Rwanda Case Studies in IEG Thematic Evaluations, FY2011-17 2011 Trust Fund Support for Development: An Evaluation of the World Bank's Trust Fund Portfolio 2012 Youth Employment Programs: An Evaluation of World Bank and IFC Support 2013 The World Bank Group and the Global Food Crisis: An Evaluation of the World Bank Group Response 2013 World Bank Group Support for Innovation and Entrepreneurship 2015 Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations 2014 World Bank Group Support to Health Financing 2015 World Bank Group Support to Electricity Access, FY2000-2014 2016 Industrial Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industrial Competitiveness and Its Implications for Jobs 2016 Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument 2017 Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity 2017 Growing the Rural Non-Farm Economy to Alleviate Poverty: An IEG Evaluation of World Bank Group Support, 2004- 2014 Sources: IEG. 3. The IEG evaluation on the WBG response to the 2008 food crisis found that the Bank was well prepared to provide assistance to Rwanda. Its program rightly focused on supporting the government’s fertilizer program in the shorter term, and working through agriculture-supporting investments in land, water, and transport in the longer term to increase agricultural production in order to build resilience. 4. The IEG evaluation on youth employment (2012) found that in Rwanda the WBG has built knowledge (through the global partnership for youth employment) to better understand the constraints youth are facing in their transition to work, and on “what works” to increase their 36 employability. The Bank and IFC support the investment climate to encourage employers to hire young workers. A TVET project rightly targeted sectors where there is high demand for skills building (e.g., horticulture, agro-processing, tourism, arts and crafts, technical servicing, solar technology, information and communications technology, and secretarial services). The government of Rwanda also subsidizes internships in the public and private sector for tertiary students, but the subsidy is regressive as these internships mainly cater to the upper income groups. 5. The IEG WBG support for innovation and entrepreneurship evaluation (2013) noted that some R&D capability is necessary to follow and obtain knowledge from abroad and adapt it to local conditions and that domestic capability is necessary to benefit from the knowledge spillovers from FDI. The evaluation found that Rwanda has made some progress, for instance in the use of mobile phones to provide up-to-date market pricing information to farmers, consumers, and traders. But Rwanda’s technology and innovation capabilities are nascent, which make it more difficult to benefit from the global system. 6. The IEG evaluation on health financing (2014) found that results-based financing mainly increased the use of services with higher unit payments, which were easier to control by providers, but that service use for other rewarded services did not change. The report also noted that results based financing without measures to address demand side barriers has a limited effect on equity in service use. 7. The IEG investment climate reform evaluation (2015) concluded that improvement in regulatory indicators in Rwanda proved insufficient to guarantee an impact on investment, employment, and growth. Rwanda has been a champion of Doing Business reforms since 2005 and has sustained the momentum of investment climate reforms that it implemented, thanks to strong political commitment. However, even though some increase in FDI was recorded after the initial reforms, the actual value has been far short of expectations. The GoR is broadening the scope of reforms to address other binding constraints to private businesses by focusing notably on SEZs, trade logistics and regional integration, support to key sectors such as tourism and agribusiness, and greater long-term efforts focused on power generation and transport. 8. IEG findings relating to WBG support for electricity access (2015) found that the first sectorwide programs in the electricity sector, in Rwanda and Kenya, are showing better results than what can be achieved using a project-by-project approach. The Bank supported Rwanda in developing a national geospatial access rollout plan that combined geographic, demographic, and technical parameters to scale up access in a least-cost combination of grid and off-grid electrification and in a time-bound manner. The plan addressed equity and shared prosperity considerations through policies for keeping connection charges affordable for the poor. A substantial off-grid program gave priority to connecting public facilities (schools, clinics, primary 37 health centers, and administrative centers) so that developmental impacts could attained even ahead of the progress on household connections. The plan also forms the basis for structured engagement by the government with multilateral banks, donors, and private sector partners, and has led to significant financing commitments from development partners. The private sector has also made commitments it may not have made without the sectorwide programs. 9. IEG findings relating to WBG support for industry competitiveness and jobs (2016) concluded that successful interventions required a breadth of engagement, proper sequencing, and implementation within a long-term vision. These key success factors prevailed in the privatization of tea factories in Rwanda as ownership transfer was accompanied by several parallel measures, including implementation of a tea leaf price reform and the creation of farmer cooperatives which effectively strengthened the relationship between farmers and factories. Conversely, the evaluation found that the lack of success in horticulture in Rwanda was due to one critical element missing in the support, namely cold storage infrastructure, suggesting that interventions must pay attention to the full set of binding constraints. Similarly, the attempt to attract foreign direct investment in Rwanda was less effective because of poor timing, since it started before all relevant risks could be mitigated. 10. The IEG learning note on “supporting transformational change for poverty reduction and shared prosperity” (2016) concluded that Rwanda has succeeded in sustaining economic growth and social progress over an extended period and in initiating its transformation into a less fragile and more dynamic economy compared with the fragile and conflict-affected states as a group. Consistent government commitment and effective management of the reform agenda were key factors underpinning the positive role that the WBG and donors were able to play in supporting Rwanda’s transformation from a conflict affected to a developing country. The Bank’s program was characterized by (i) a clear understanding of the most binding constraints based on careful economic and sector work, (ii) close coordination among development partners, with each focusing on its area of comparative advantage, and (iii) a more selective program in recent years focusing on high- priority sectors. These lessons can be applied in other fragile and conflict- affected states, even if Rwanda’s dependence on aid creates a sense of uncertainty about the sustainability of its impressive progress. 11. The IEG early state assessment of the process and effects of the performance for results instrument (2016) indicated that Rwanda was positively disposed to the PforR instrument. Short preparation times were feasible for the first two PforR operations in Rwanda since established government programs and the government itself focused on results. The programs cover both partnerships and ownership well. In the case of the Rwanda Agriculture PforR, to enhance ownership by all stakeholders, the government signed a memorandum of understanding with the private sector, civil society, and development partners supporting the principles and objectives of the program, building on previous work. This program also includes cofinancing by the UK 38 Department for International Development (DfID) through a multidonor trust fund, with common results and disbursement linked indicators (DLIs). However, the report noted an underrated technical risk in the Rwanda Agriculture program, namely the weak capacity of the private sector to implement a critical component and the lack of clarity concerning its response. 12. The IEG evaluation on growing the rural non-farm economy (RNFE) concluded that overall Bank support for promoting RNFE in Rwanda, while perhaps not conceptualized as such in the various strategies, is highly relevant. The Bank program supported all necessary components of a sound RNFE program: (i) developing agriculture through increased productivity and diversification into higher value, marketable crops, and thus increase farm incomes; (ii) improving farm-to-market, regional, and international connectivity; (iii) increasing access to electricity; (iv) promoting the private sector, domestic and international; (v) developing relevant skills for the changing economy; and (vi) developing secondary cities that can become economic hubs for RNFE. In practice the strong and sustained emphasis given to agriculture over the last 15 years has been critical in developing RNFE. Farm incomes improved as a result of higher production or diversification or improved market linkages. Higher agricultural incomes are translated into more disposable income that families begin to spend on non-farm activities such as construction and trading that, in turn, create non-farm employment. In addition, the labor-intensive methods used in the rural sector support project (RSSP) and Land Husbandry, Water Harvesting and Hillside Irrigation (LWH) program, with payment for labor deposited in a financial institution, proved a very creative way to inject cash in the rural economy that also created demand for non-farm products and services. The development of agriculture needs to be complemented with supporting value chains that help farmers market their produce and move into higher value products, and also create non-farm employment. The evaluation also found that Rwanda has followed a very pragmatic approach to skills development, emphasizing the very basic skills (e.g., in construction, hospitality, and trading) that were linked closely to the type of non-farm economy that was being created. However, there were some deficiencies in the other components of the RNFE program. The private sector agenda remains fragmented and Rwanda has yet to develop a coherent approach to municipal management, resource mobilization, and access to land. 13. The PPAR for the decentralization and community development project points to several relevant lessons: (i) in a post-conflict environment with a history of highly centralized structures and a dominant central government, decentralization can be fostered by well-designed participatory processes to promote demand-side governance and empower communities, along with intensive institutional and capacity building at central and local levels; (ii) decentralization and capacity building are a long term process and need sustained government and Bank engagement; and (iii) flexibility to adapt to changing circumstances is critical to the continued relevance and success of the operation. The ability of the project to respond quickly after a major territorial administration 39 reform and rapidly evolving decentralization priorities was central to its success in achieving development objectives. B. Literature Review of Key Evaluations and Studies by Other Development Partners 14. The CPE will also draw upon evaluation reports and studies conducted by other development partners in recent years (See Box 2). While evaluations by other development partners primarily focus on the assessment of their contributions and interventions, the preliminary literature review suggests that they also provide useful evaluative evidence on Rwanda-specific issues and aspects as well as cross-cutting issues such as partnerships and aid coordination, capacity development, effectiveness of instruments, and impact of external assistance in supporting the Government to achieve its development goals. 15. Leadership and partnership. The country program evaluations by other development partners affirm the government’s strong leadership and effective partnerships among development partners. UNDP’s country-level evaluation (2008) mentions that the GoR plays an effective role in aid coordination, despite the capacity problems of the public sector, exercising clear ownership and leadership of external assistance. The overall structure of development assistance has been designed in consensus with the development partners and they all participate actively. IFAD’s country program evaluation (2011) notes that there have been regular contacts and some project-level collaboration with the WB, while there was no formal co-financing agreement. The study on aid effectiveness by Institute of Policy Analysis and Research (IPAR- Rwanda) suggests that a major driver of the remarkable progress towards achieving the MDGs is a committed Government and a strong partnership for development between the Government and donors. 16. Capacity development. The World Bank plays important roles to support capacity development in Rwanda. UNDP’s evaluation (2008) noted the WB role in supporting national capacity development strategy (UNDP, 2008, Annex F). The IMF’s evaluation on technical assistance (2009) notes the WB’s assistance in supporting National Strategy for the Development of Statistics. The IMF evaluation points to some success in improving statistical capacity; nevertheless, the evaluation concludes that much remains to be done, as would be expected with long-term efforts to undertake institutional capacity building in a post-conflict country. 17. Instrument - budget support. There is a need to assess effectiveness of instruments selected by the WBG. Some evaluations during the 2000s mention that GoR has a strong preference for sector budget support, inter alia because it reduces the burden and transaction costs of managing many projects in the face of limited human resources (AfDB, 2011; IFAD, 2011). 40 18. Policy dialogue and knowledge work. The WB may have comparative advantages in leading policy and knowledge work in Rwanda. The evaluation of policy-based lending operations in the African Development Bank (AfDB) (2011) suggests that the AfDB does very little analytical work, which weakens its capacity to undertake work that would strengthen policy dialogue or contribute to DP activities such as economic and sector work. Box 2: Key Select Evaluations and Studies by Other Development Partners in Recent Years Dev. Partners Year Title Focus / Type UNDP 2008 Assessment of Development Results: Rwanda Country Program IFAD 2011 Republic of Rwanda: Country Program Evaluation Country Program Evaluation of Policy Based Operations in the African AfDB 2011 Development Bank, 1999-2009, Country Case Study: Instrument - DPL Rwanda Evaluation of Technical Assistance to Post-Conflict Countries: IMF 2009 Instrument - TA Mozambique and Rwanda AfDB and IFAD 2010 Towards purposeful partnerships in African agriculture Theme - Partnerships Millennium Development Goals Progress Report: UNDP 2010 Theme - MDGs Rwanda Country Report 2010 Theme - Aid effectiveness IPAR-Rwanda 2012 Aid Effectiveness in Rwanda: Who Benefits? and partnerships DIE, ECDPM, and Theme - Development and 2015 European Report Rwanda Case Study ODI partnerships IPAR-Rwanda 2011 The Impact of Tax Incentive: Rwanda Case Study Report Theme - Economic Raising Productivity and Reducing the Risk of Household IPAR-Rwanda 2015 Theme - Economic Enterprises: Rwanda Country Study Evaluation of FAO’s contribution to the reduction of rural Theme - Gender and rural FAO 2017 poverty through Strategic Program 3 (Annex 6: Analysis of poverty Contributions to Gender Equality) Access to Energy in Rwanda: Impact evaluation of activities MOFA, Project/Program-level - 2014 supported by the Dutch Promoting Renewable Energy the Netherlands Energy (impact evaluation) Program Performance Evaluation of the Integrated Improved Project/Program-level - Rural USAID 2015 Livelihoods Program (IILP) poverty (mixed -methods) Evaluation of the Impact of Malaria Control Interventions on Project/Program-level -Health USAID 2015 All-Cause Mortality in Children under Five Years of Age in (impact evaluation) Rwanda, 2000–2010 Rwanda: Comprehensive Evaluation of the Community Project/Program-level - UNICEF 2016 Health Program in Rwanda Health (mixed-method) Project/Program-level - UNICEF 2016 Rwanda: Rwanda Rapid SMS Impact Evaluation Health (impact evaluation) Mid-Term Evaluation of the National Employment Program, Project/Program-level -Jobs Sida 2017 Rwanda (mixed-method) Sources: IEG. 41 19. DOL and DPs’ support by sector. Several project/program-level evaluations assess bilateral and multilateral donor contributions in specific areas or sectors, consistent with assignments under the DOL. The impact evaluation by USAID (2015) provides evidence that malaria control interventions substantially contributed to the observed decline in under-five mortality in Rwanda from 2000-2010. The UNICEF evaluation (2016) concludes that the institution’s program has been a contributor to improved maternal, newborn child health and nutrition in Rwanda. 42 Attachment 4 Donor Mapping Table 1. Donor Division of Labour in 2010 Source: Ministry of Finance and Economic Planning, Rwanda. 43 Table 2. Donor Division of Labour in 2013 Source: Ministry of Finance and Economic Planning, Rwanda. 44 Table 3: Donor Division of Labour in 2014 (revised) Source: Ministry of Finance and Economic Planning, Rwanda. 45 Endnotes 1 The median age of the population was very low after the genocide and remains around 19 years today. Programs focusing on the employment of young people could thus provide interesting lessons for neighboring and MENA countries. 2 The precise estimates of poverty have reportedly been a source of contention between the government and the Bank in the past. The CPE will document the issue, and its disposition, as needed. 3 The World Bank’s classification of countries into the lower middle income category currently requires a GNI per capita between $1,026 and $4,035 in 2015, calculated using the World Bank Atlas method. In 2015, Rwanda had a GNI per capita (Atlas Method) of $710. 4 In December 2015, the Rwandan constitution was amended to allow the president to run for a third in 2017. Presidential terms were also reduced from seven years to five. 5 For example, the president cannot appoint more than 50 percent of cabinet members from his party and the speaker of parliament is required to be from the minority party. The constitution requires that women hold at least 30 percent of positions in all higher level public institutions. 6 Rwanda’s resolution of the fallout from the genocide saw the establishment of a National Unity and Reconciliation Commission in 1999, and eventually of the “gacaca” courts, modeled on a traditional approach to settling disputes. 7 The Press Freedom Index of Reporters without Borders ranks Rwanda 156 out of 178 countries for 2012. Freedom House ranks Rwanda 178 out of 197 countries in its 2012 World Press Freedom Index. The IREX Media Sustainability index indicates a downward trend since 2008. 8 Most price controls were eliminated, the exchange rate was liberalized in 1995, import tariffs were lowered, the monetary and financial sectors were liberalized, state-owned banks were privatized and private commercial banks created, current account operations were liberalized, and restrictions on capital flows eased (from Ggombe and Newfarmer, 2017). 9 The decentralization process led initially to major territorial administration reforms, including the consolidation of districts and the establishment of local (district) governments as key units for service delivery, and later sought to extend the range of services devolved to local governments while addressing outstanding issues in their capacity and fiscal and financial autonomy. 10 EDPRS 2 aims to achieve the following goals by 2018: raise the gross domestic product (GDP) per capita to $1,000; have less than 30% of the population below the poverty line; and have less than 9% of the population living in extreme poverty. 11 While the system of cascading performance contracts (imihigo) has indisputable advantages, there are still important limitations, including the limited space given for independent validation (see Rwanda case study for the Data for Development Evaluation). 12 Household consumption growth was in fact pro-rich during 2001-06, leading to an increase in inequality (the Gini coefficient increased from 0.507 to 0.522). Poverty dropped by only two percentage points—from 58.9 to 56.9 percent—over this period. 13 PLR, para 10. 46 14 The Gini coefficients in Burkina Faso (GDP per capita PPP constant 2011 international: $1,536) and Madagascar (US$1,371), whose income levels were similar to Rwanda’s (US$ 1,539), were 35.3 in 2014 and 0.43 in 2012, respectively. 15 Source: Draft Concept Note for “Rwanda: Future Drivers of Growth,” World Bank, 2017. 16 The EDPRS also sought to consolidate gains under the 2002 PRSP in health, water, and basic education. 17 Interventions supported under Vision 2020 Umurenge seek to enhance agricultural productivity, provide labor intensive employment for landless Rwandans without job opportunities, provide social assistance for people incapable of work and improve credit access for input purchases or micro enterprise development. 18 A silent partnership or delegated cooperation agreement refers to a formal agreement in which authority for the management of an ODA contribution (or part thereof) is delegated to another donor, thus facilitating harmonization and reducing the transaction costs faced by the GoR in managing multiple two-way arrangements. A delegated cooperation arrangement is formalized by a written agreement under which one donor represents another, undertaking at least two of the following tasks on its behalf: (i) disbursement of funds; (ii) policy dialogue at the sector/project/program level; and (iii) monitoring, review, or reporting activities. In cases where the strict application of a formal delegated cooperation arrangement is considered infeasible, other silent partnership options can be negotiated on case-by-case basis through consultations between the GoR and the DPs. Such silent partnerships should lead to the extent possible to reduced transaction costs for the Government. 19 The UK is taking the lead on social protection. 20 For multi-sector budget support operations, we have allocated funding to the "leading" GP — that is, the GP in which the operations were housed, namely MFM (or more correctly, the predecessor network, namely PREM). We will undertake a finer analysis of the GP composition of the PRSC operations in the CPE report. 21 Strictly speaking, the sharp drop off in ODA was recorded in Rwanda’s 2012/2013 fiscal year (July 1 – June 30), so both the 2012 and 2013 calendar years were affected. 22 This was done through the 2013 DPO on decentralization. 23 In broad terms, the analytical approach will be guided by the 2005 retrospective of CAEs done by OED (IEG’s predecessor) and the Evaluation Cooperation Group’s 2008 Good Practice Standards on the Evaluation of Country Strategies and Programs , which IEG was instrumental in preparing. 24 Data and related documents, including WBG and IEG documentation relating to all lending and non- lending products in the Rwanda program during the evaluation period, has already been compiled and is shared among team members under a one drive folder. 25 The portfolio analysis will include a focus on portfolio quality indicators (covering both the now-closed and still-active portfolios) in order to help form judgments on the efficiency of WBG support. 26 The preparation of the SCD/CPF as well as the sources of growth work in support of Rwanda’s Vision 2050 may also provide relevant platforms for IEG to present key CPE findings. 27 Under the 2010 DOL, education is no longer a Bank sector. This evaluation will not assess but only list results achieved under the FY00 Human Resource development project. However, recognizing the issue of skill constraints, the Government has requested the Bank to address skill shortages as a cross cutting area. This evaluation will therefore review outcomes under the skill development project approved in FY11 as well as policy measures supported by the PRSG/C series. 47 28 Source : EICV4 (2013/14) 29 The main social protection programs include (i) the VUP, (ii) the Genocide Survivors Support Fund (FARG), (iii) the Rwanda Demobilization and Reintegration program and the Decentralization Funding Program. 30 In the pre-genocide period 1990-94, IDA assistance emphasized stabilization and growth but had few positive results. Adjustment lending was largely unsuccessful because of the civil war and investment lending displayed deficient project design and supervision. (CAE, 2004). 48