WPS7157 Policy Research Working Paper 7157 Impact Evaluation Helps Deliver Development Projects Arianna Legovini Vincenzo Di Maro Caio Piza Development Research Group Impact Evaluation Team January 2015 Policy Research Working Paper 7157 Abstract Does research add value to aid? Specifically, does impact actual disbursements is reduced by half. Evidence-based evaluation research help or hinder the delivery of devel- mid-course corrections, a clearer implementation road opment projects? This paper analyzes the question by map, strengthened capacity on the ground, and observer constructing a new data set of 100 impact evaluations and effects are possible channels to explain the results. Hope- 1,135 projects approved by the World Bank between 2005 fully, this analysis will stimulate discussion over the optimal and 2011. The analysis finds that the delivery of projects balance between project financing and the impact evalu- with impact evaluation is significantly timelier: common ation research needed to deliver development outcomes. delays are avoided and the gap between planned and This paper is a product of the Impact Evaluation Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at alegovini@worldbank.org, vdimaro@worldbank.org and ctpiza@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team IMPACT EVALUATION HELPS DELIVER DEVELOPMENT PROJECTS Arianna Legovini, Vincenzo Di Maro, and Caio Piza * JEL No. O01, O12, O19, O02, D04, H43, C93, F61 Keywords: Impact Evaluation, Development Effectiveness, Development Policy, Experiments, Projects *Arianna Legovini is the Manager of Development Impact Evaluation in the Research Group of the World Bank, and Vincenzo Di Maro is a senior economist and Caio Piza is an economist in the same group. We are especially grateful to Aart Kraay, Guadalupe Bedoya, Florence Kondylis, Daniel Rogger, and Daniel Stein for their detailed comments on an earlier draft. We thank Pavel Luengas and Susumu Yoshida for their assistance in collating project-level data from the World Bank’s databases and providing research assistance. 1. INTRODUCTION Development effectiveness has been at the center of the economic debate since at least the establishment of the Bretton-Woods institutions. With $16 trillion in aid delivered since then, this is by all means an important issue. 1 Despite all efforts, however, the knowledge gap on what really works in development remains large. The literature investigating the relationship between country growth and aid flows has for the most part failed to provide satisfactory answers to the question of whether aid is effective and under what conditions. This is in part because aid measured in dollars and cents implicitly assumes that aid is a transfer of resources, a noisy proxy for the bundle of technical and financial assistance through which aid is delivered. Far from being a direct transfer, aid is delivered through development projects and general budget support, each representing historically 86 and 14 percent of the total. 2 Both of these instruments bundle technical assistance with financial flows, the former affecting the effectiveness of the latter. Recent evidence (Denizer et al. 2013) finds that most of the variation in aid performance is explained by project characteristics (and not by country conditions). This suggests that the relevant question may not be whether aid is effective but what quality of projects and project characteristics make aid effective. With $3.5 trillion committed to developing countries since 1970, 3 the World Bank is a large development actor, and one that invests much effort in the quality of projects, from peer reviews to quality enhancing processes. Still, about a quarter of projects are assessed to have been of unsatisfactory quality at entry 4 and a quarter rated unsatisfactory at exit. Projects also experience implementation delays with lags of six to seven quarters between project approval and first disbursement and continued gaps between actual and planned disbursements thereafter (World Bank 2014a, 2014b). In 2005, the Bank started an institutional effort to do impact evaluation (IE) to generate knowledge in the context of national policies and programs in developing countries (Figure 1). This is also when the Bank started tracking the link between IE and its projects. 5 Over the next few years, and starting 1 Net official development assistance and official aid received (current U.S. dollars). Source: data.worldbank.org. 2 http://aiddata.org. 3 IBRD loans and IDA credits (DOD, current U.S. dollars). http://data.worldbank.org. 4 Quality at entry is assessed by the Quality Assurance Group (QAG) on the basis of a sample of projects approved by the Board in the previous fiscal year. Reports are shared with management for remedial action. 5 Before 2005, there were one to five studies a year evaluating World Bank projects. After 2005, the Bank started a concerted effort to evaluate Bank projects and track the link between projects and IEs. The number of IEs related to projects jumped to about 20 a year after 2005. 2 from a very low baseline, IE came to cover about 10 percent of the Bank’s projects and establish, through counterfactual analysis, the causal pathways of what policies financed by these projects work and why. 6 The Bank’s impact evaluations involve research and field staff working with the project and government teams to introduce structured and iterative learning in project operations. They aim to inform policy design, identify policy-relevant research questions, run experiments, and collect and analyze data, to feed evidence back into policy decisions during implementation. At their best, impact evaluations are a framework of collaboration for the generation and real-time use of policy-relevant evidence. This study investigates the role of these impact evaluations on the quality of projects. For this purpose, we assume that we can summarize the quality of projects by three conditions: (i) the problem/goal is correctly identified, (ii) an appropriate course of action and financing is agreed to, and (iii) project activities are implemented as planned. While impact evaluation can potentially influence (i-iii), it is hard either to measure or find good proxies for (i) and (ii). We thus restrict our analysis to (iii), which is measurable and a necessary condition for projects to achieve their impact. While measuring the link between IE and development outcomes of projects is of great interest to the development community, it is at best challenging. There is plenty of evidence established on a case-by-case basis, 7 but estimating the impact of IEs in general requires a valid counterfactual and a common impact metric across the policies being evaluated. Constructing such a counterfactual seems unfeasible. Our second-best, but tractable, approach is to investigate the link between IE and project implementation. To do this, we construct a panel using administrative data on the whole portfolio of World Bank impact evaluations and projects approved during 2005–11. Linking information on projects’ disbursements with information on impact evaluation allows us to explore the relationship between IE and project implementation using 100 impact evaluations and 1,135 projects implemented in 114 countries in 11 sectors during 28 quarters. We use project disbursements as a measure of implementation. This is because disbursements are made against evidence that the agreed activities are being implemented by the relevant government and supervised by the Bank project staff. We use three specific quarterly measures: (i) cumulative disbursements relative to loan size, (ii) the gap between planned and actual disbursements relative to loan size, and (iii) the ratio of project to average disbursements for all projects in the relevant sector. To our knowledge, this is the first quantitative study aiming at identifying the effect of IE on the implementation of development projects. 6 In addition to IE, all Bank projects must complete an implementation completion report. About a third of them are also subject to project evaluation by IEG. These assessments are descriptive in nature, not causal. 7 Examples from World Bank IE work are Bruhn et al. (2013), Brune et al. (2011), Friedman et al. (2010), Kondylis, Jones, and Stein (2014). 3 To control for project selection into IE, we include both institutional and idiosyncratic factors, assuming the confounders are observed or time-invariant. These factors range from time, sector and country effects, project and project leader fixed effects, and project leader changes. If project leaders of potentially high-disbursing projects are more likely to select into IE, then projects with IE may disburse faster partially due to the project leader decision to adopt an IE. If, on the other hand, project leaders select into IE because their projects need additional help and support, then projects with IE will disburse at a slower pace. Our data provide some support for the latter case. We find that impact evaluations have a large and significant effect on the rate of disbursement of operations, reducing the gap between planned and actual disbursements and increasing disbursements relative to average disbursements in the typical project in the same sector. By averting common delays in the early stages of the project cycle, impact evaluations might make less relevant the need for teams to push disbursements out in the later stage of project implementation. We conclude that projects with IE are more likely to implement their activities as planned and, in so doing, may be more likely to achieve their objectives. 8 Furthermore, our results support the idea that project financing and IE research are complementary outputs of the World Bank. This might have implications for the use of IE as an operational tool to improve delivery of projects, and for weighing the costs of IE research against the benefits on the Bank’s project portfolio. For development organizations like the World Bank, an optimal balance between project financing and IE research may help them reach their efficient frontier. More generally, we hope our results may contribute to shifting the discussion away from just questioning the effectiveness of aid to a discussion on constructive ways to make aid more effective. Finally, we consider the potential channels through which World Bank impact evaluations can affect project implementation. We speculate that these channels include: 9 (a) better planning and evidence- base in project design, (b) greater implementation capacity due to training and support by research team and field staff, (c) better data for policy decisions, and (d) observer effects and motivation. These channels are not necessarily specific to IE research and learning more about them could inform development policy more widely. Finally, we discuss the limitations of our analysis and outline an initial agenda for future research. The remainder of our paper is organized as follows. Section 2 summarizes the literature. Section 3 describes the institutional background and the data. Section 4 explains the analytical strategy. Section 5 summarizes the results. Section 6 concludes. 2. LITERATURE REVIEW 8 Here, we are assuming that assumptions (i) and (ii) above are met; that is, (i) the problem/goal is correctly identified and (ii) an appropriate course of action and financing agreed to. 9 See also the discussion in Legovini (2010). 4 Most of the evaluation literature has focused on assessing the impact of interventions. This paper aims to evaluate the impact of using evaluation itself on the implementation of projects. While a vibrant literature exists on the question of whether aid is effective (see Temple 2010 for a review), we could find very little on the question of how to make aid effective. On the question of whether IE research helps deliver development aid, the contributions in the literature are all theoretical or descriptive. The Independent Evaluation Group report on IE bases its findings on a beneficiary assessment of completed IE studies; the results are self-reported and no comparison group is provided (World Bank 2012b). The 2011 DIME Progress Report provides a qualitative assessment of the transmission mechanisms through which IE affects the quality of operations and examples of how this has worked in practice (Legovini 2011). Dhaliwal and Tulloch (2012) argue, based on the experience within J-PAL, that evidence can influence policy making, especially if it is independent, methodologically rigorous, substantive, actionable, easy to understand, cumulative, and easy to explain to constituents. On this last point, Masset et al. (2013) run an experiment in research dissemination to find that policy briefs on the impact of agricultural interventions did not have any effect in changing readers’ prior beliefs. Ruprah and Marcano (2009) show that technical assistance (that shares evidence with policy makers) has a positive impact on project outcomes and that this impact goes beyond the impact of financing alone. Overall, this literature is still nascent and inconclusive over the effect of IE on project implementation. A complementary literature on the factors that explain the success of individual projects is also relatively thin. Some studies have investigated the country-level determinants of World Bank project outcomes. Explanatory factors include democracy, civil liberties, and conflict and different macroeconomic measures ranging from the quality of macroeconomic policy and market-oriented reforms to volatility and growth (Denizer et al. 2013). Other studies have looked at the link between project-level variations in outcomes and project-level factors, such as staff time devoted to preparation and supervision. Deininger, Squire, and Basu (1998) find that more analytical economic and sector work (as opposed to specific project preparation work) is correlated with the quality of World Bank loans and their level of disbursement. Kilby (2000, 2011, 2013a and 2013b) finds that longer project preparation is associated with better project ratings. Ika, Diallo, and Thuillier (2012) show a statistically significant and positive relationship between a set of five critical success factors (in particular, they focus on monitoring, coordination, design, training, and institutional environment) and project success. Denizer et al. (2013) find that 80 percent of total variation in project performance comes from within-country variation, and correlates with project size, sector, quality of the project leader, and the amount of resources devoted to project preparation and supervision. We take this last study as a point of departure to motivate the importance of investigating whether IE research can affect the quality of projects. 5 3. INSTITUTIONAL BACKGROUND AND DATA 3.A. INSTITUTIONAL BACKGROUND World Bank development financing is delivered in the form of projects, which are implemented by governments. Each project, led by a project leader, 10 follows the cycle of preparation, approval, effectiveness, implementation, and closing. During preparation and appraisal, the team designs the operation with a government agency, including planned activities and schedule of disbursements. During implementation the recipient government undertakes activities and submits disbursement requests to the Bank. The Bank project team supervises activities and clears disbursements for expenditures made in conformity with Bank guidelines (World Bank 2012a). These disbursements are recorded in the Bank’s systems. Because disbursements are made against agreed activities and supervised by the Bank, we use disbursements as a credible signal for project implementation. This is in line with common management practice: project disbursements are used as the main indicator of project success during implementation, and they are the main focus of weekly portfolio review meetings and fiscal year target exercises. Furthermore, about two-thirds into the planned duration of a project, management conducts a midterm review to assess the progress of each project. Delays are common before midterm; this implies that schools, health facilities, or roads may not be built or staffed as planned. Furthermore, delayed projects may rush to make up for lost time when midterm approaches and in the relatively short time between midterm and project closing, when the remaining balance of the loan must be disbursed or is lost. We assume that implementing a project as planned and avoiding the rush to push out disbursements during the later stages of the project life cycle is bound to have beneficial effects on project outcomes. It might also have, on aggregate, important benefits for the reputation of an institution like the Bank, which receives donor funding and is periodically assessed against commitment and disbursement targets. World Bank impact evaluations are conducted by the Bank’s internal researchers, often in collaboration with external researchers. These researchers use counterfactual analysis to evaluate policy interventions that are either Bank-financed or financed by other governments and institutions. This analysis only includes IEs of Bank-financed interventions, because they can be matched with project information from the Bank’s information systems. Most of these IEs are still ongoing, as are the projects they evaluate. Notwithstanding the potential value of the knowledge generated for future projects, these IEs do operational research and have the potential for affecting current projects in real time through both the research and capacity-building channels. Research teams work with governments and project teams to affect policy design, agree on research design, adapt implementation modalities, and use 10 In the World Bank, project leaders are called team task leaders, or TTLs. 6 data and evidence generated to affect midcourse corrections in policy implementation as well as to influence scale-up or -down decisions. These IEs are bundled with capacity-building activities and monitoring. Government and project teams attend workshops and trainings, are exposed to international evidence and the use of analytical tools for decision making, and participate in IE discussions and analysis. The IE research and field teams accompany the projects through their life cycle, monitoring IE implementation and data collection, and often supporting the implementation of project activities (World Bank 2006). Because of the bundled nature of Bank IEs, the analysis cannot separate the effects of IE research from that of capacity building and monitoring. Furthermore, because the Bank IEs have been subject to internal quality assurance processes since 2007 (which means they require managerial approval, peer review, and discussion), they are less likely to be purely research led and more likely to have stronger links to policy. For these reasons, the estimates may not be applicable to more research-led IEs. 3.B. DATA Our panel covers the period since 2005 when the Bank formally started doing IEs (Figure 1) and includes the 1,135 projects approved during 2005–11 that are in sectors that have at least one IE. Of these, 100 have an IE. Project data are extracted from the World Bank’s Business Warehouse. Time-invariant characteristics include project id, region, country, sector, date of approval, and closing date. Time- varying characteristics are planned disbursements (initial and possible revisions), actual disbursements, and identity of the project leader. The data include 1,252 entries that are the universe of the projects approved by the Board between the first quarter of 2005 and the last quarter of 2011, across 28 quarters. As sectors with no IEs are excluded from the analysis, the sample drops to 1,135 projects, with up to those 28 quarterly observations, corresponding to 17,784 total quarterly observations. 11 Impact evaluation data comes from the IE database maintained by the Development Impact Evaluation Unit (DIME) in the Research Department of the World Bank. The database includes 377 entries in the period 2005–11. Of these, 100 are for Bank projects approved in the same period, or 9 percent of the project in our sample. 12 Time-invariant characteristics include IE id, region, country, sector, start date, and project id. Time-varying characteristics include IE status and identity of IE leader. We use project id to merge the data. When the variable “project id” is missing, it indicates that 11The number of observations for a project depends on its vintage or year of approval. 12The remaining is for policy interventions not financed by the World Bank, for which implementation data is not available. 7 the IE is evaluating interventions that are not financed by a Bank project. Clearly, when project and IE data are merged, IEs with missing project id are automatically dropped. Data is organized according to the quarter of implementation of the project. As such, quarter 1 is the first quarter of project implementation after effectiveness for all projects in the sample, and it can correspond to different calendar quarters depending on when the project started. By construction, there are 1,135 observations in quarter 1 with observations falling monotonically to 32 projects in quarter 28 (Figure 2). This represents an unbalanced panel of projects in different quarters, with an average project duration of 15 quarters across the 2005:Q1 to 2011:Q4 period. This sample has a substantial amount of variation in project vintage and duration across quarters. We use the period before midterm as the institutional benchmark. As argued above, the period before midterm is of great importance as this is when projects are substantially off track. These delays may force teams to “push out” disbursements during the later phase of the project to avoid losing funding. We define the subset of projects that have reached midterm and the subset of quarters up to that midterm (an institutional benchmark) as the set that allows us to reduce composition effects. As Figure 3 illustrates, this approach reduces the heterogeneity in project duration and vintage, making sector composition more stable. 3.C. DESCRIPTIVE STATISTICS Projects in our panel span 11 sectors and all six regions of the World Bank. The largest sectors in number of projects are agriculture and rural development, and transport, while the largest region is Africa. IEs are represented in regions and sectors to varying degrees (see Figure 4 and Table 1). Non- infrastructure sectors are well represented, while infrastructure sectors are less so. The public sector governance sector is excluded, as it did not have any IEs in this period of time. 13 In Table 2, we present descriptive statistics for projects with and without IE. The next few paragraphs discuss the statistics in this table. Eighty percent of the projects are still active (that is, under implementation) and the rest have closed, with different vintages (or start years) being evenly represented in the sample. About half of all projects (52 percent) have reached midterm, typically at two-thirds of their planned life. 14 The median midterm falls on quarter 13, with more than 50 percent of projects reaching midterm between Q11 and Q16. On average, projects are in their fourth year of implementation (15 quarters) out of the five years (20 quarters) originally planned. In terms of duration and other project characteristics, no difference between project with and without IE is found to be statistically significant, with the exception of loan size. The average loan size for projects without IE is $109 million versus $149 million for projects with IE. Regarding the outcomes of interest, total disbursements are higher for projects with IE. This is in line with the larger loan size for projects with IE. Moreover, projects with IE disburse more relative to 13 This has changed since then. 14 Midterm should happen about halfway in project implementation. In practice, this is scheduled at two-thirds into project implementation due to delays. 8 loan size than projects without IE. Projects with IE that have actually reached midterm have disbursed 45 percent of the loan in total versus only 35 percent for projects without IE; and by closing date, they have disbursed 86 percent of the loan versus only 77 percent for projects without IE. Similarly, the average quarterly cumulative disbursement gap (the difference between planned and actual disbursements relative to total loan size) is lower for projects with IE (17 percent in the quarters up to midterm) than for projects without IE (27 percent). In other words, projects with IE are closer to their planned timeline by a large margin. Finally, relative to other projects in their sector, projects with IE disburse around 10 percent more by midterm as well as on average. 15 Because IEs start at different points in the project cycle, observing outcomes conditional on IE start date is instructive. A third of the IEs start by the first quarter, a third in the following year, and the rest thereafter (Figure 5). We plot the disbursements of (i) projects that do not have an IE (D=0), (ii) projects that will have an IE for quarters before the IE started (D=1 & T=0), and (iii) projects that have an IE and the IE has started (D=1 & T=1) to understand whether “IE assignment” (selection) or “doing IE” (treatment) is associated with increases in disbursements. Over the project life cycle, data is consistent with the possibility that “doing IE” has a positive effect on disbursements. This is because, first, before starting an IE, projects without IE and projects that will have an IE in future quarters, (i) and (ii), have the same rate of disbursement (Figure 6). We interpret this as an indication of a good degree of comparability between projects with and without IE before actual selection into IE. Second, the rate of disbursement for projects that have started an IE, (iii), is consistently above (i) and (ii) for all quarters in the graph. This might suggest that disbursements increase after the IE starts and thus more likely to be due to “doing IE” than to “IE assignment.” This is true for the whole sample as well as for the sample up to midterm (0). We discuss the selection issue in the next section. In Figure 8 and 0 we look at the disbursement gap. We find that the disbursement gap is smaller for projects with IE, and especially before the IE starts. If we couple this with the facts above about actual disbursement (higher rate of disbursement for projects with IE), the observed behavior in disbursement gap means that projects that will start an IE tend to have more pessimistic implementation plans. For example, this could mean that project leaders were expecting these projects to be slow or difficult to implement, and may have selected into IE to get additional help. As such, data is suggestive that the measured effect could be a lower bound of the actual effect due to possible “adverse selection” of projects expected to be slower into IE. Indeed, adverse selection of potential slower projects into IE would tend to bias downward our estimates of the effect of IE on disbursements. 15Often projects pay for the costs of data collection. These costs are in the order of $0.5 million or less than 1 percent of the loan. This can affect, but only marginally, project outlays. The costs of IE research, field coordination, and travel are borne by the Bank and are not part of this analysis. 9 When we then compare project disbursements relative to their sector average: 16 projects that will select into IE perform below their sector average before the IE starts and above the sector average after the IE starts (Figure 10 and Figure 11). This again points to the possibility of adverse project selection into IE that might bias our estimates downward. Together, these observations suggest that implementation speeds up once projects start an IE, irrespective of the measure used. Further and by various measures, these projects do as well or worse before the IE starts than projects without IE. These findings suggest that our estimates may be a lower bound of the effect of IE on implementation. 4. ANALYTICAL STRATEGY 4.A. CONCEPTUAL FRAMEWORK This section describes a simple conceptual framework used to estimate the effect of IE on project implementation. We model three dependent implementation variables: (y1) rate of disbursement (cumulative actual disbursement divided by total loan size); (y2) gap in the rate of disbursement (difference between cumulative planned and actual disbursement divided by total loan size); and (y3) deviation from sector-quarter average rate of disbursement (rate of cumulative disbursement divided by the sector average rate of cumulative disbursement in that quarter for all projects in the sample). The variable y1 is used to measure the effect of the IE on the rate of implementation. Variable y2 is used to understand the effect of IE on implementation relative to the project team’s expectations. Finally, y3 is used to benchmark the effect of IE on implementation relative to the standard in the relevant sector. We model implementation along the project life cycle as a function of time-specific factors, such as economic and political conditions; country-specific factors, such as institutional capacity; sector- specific factors that slow down or speed up implementation, such as the construction of infrastructure; and project-specific factors, such as the quality of project management or supervision. We model the effect of the IE as depending on (i) whether the project will have an IE, even if it has not started yet (project effect), and (ii) whether the IE started by that particular quarter (quarter effect). We will interpret (i) as the value of assignment to IE and (ii) as the effect of IE on implementation. 16Sector averages are defined as the quarter averages of disbursement rates in each sector (that is, we have 28 different averages for the 11 sectors in our sample). 10 Because projects should eventually close their disbursement to commitment gap, the effect of the IE may not be linear. It might be stronger early on, as, perhaps, the IE provides for a larger team and more defined implementation plan, and taper off later. 4.B. PROJECT SELECTION INTO IE Here we discuss project selection into IE, the main source of bias that may affect our empirical analysis. Project selection into IE is due to both institutional factors and project characteristics. Institutional factors determine bulk uptake into IE for different sectors at different points in time. This is because the World Bank made discrete efforts to increase IE in certain sectors and regions in certain years: education, health, and social protection since 2005; HIV/AIDS and malaria since 2007; agriculture and gender since 2008; finance and private sector since 2009; fragile states and financial literacy since 2010; infrastructure since 2011; business climate since 2012; and governance since 2013. Furthermore, commitments to donors in 2009 made IE compulsory for a sample of 2011–13 projects. These efforts resulted in discrete expansions of the IE portfolio in different sectors and regions. In addition to these institutional factors, project-specific characteristics, such as the identity of the project leader, have been at play. If project leaders of potentially high-disbursing projects are more likely to select into IE, then projects with IE may disburse faster partially due to the project leader decision to adopt an IE. If, on the other hand, project leaders select into IE because their projects need additional help and support, then projects with IE will disburse at a slower pace. Descriptive statistics above provide some support for the latter case. Our strategy to control for selection into IE relies on the assumption that the confounders are observed or time-invariant. Observable characteristics include portfolio-wide and project-specific factors. Portfolio factors include sector and country effects due to institutional efforts to introduce IE in those areas and related availability of funding for IE, and project leader fixed effects. In addition, we control for fiscal year and a third-order polynomial of average cumulative quarterly disbursement per sector. This function is added to capture potential composition effects that could confound the effect of IE. Project factors are accounted for via project fixed effects, and the identity of the project leader, or, alternatively, the recorded switches in project leader identity. The last variable is added because project leader turnover might slow down projects disbursements, as the new project leader will have to learn about the project and build relationships with the government counterpart. Unobserved variables that we can control for include all characteristics related to projects’ quality that are time-invariant. There may be other unobservable characteristics that change over time that we cannot control for. For example, a project leader who selects into IE may change his/her attitudes toward IE as he/she learns the costs and benefits of doing IE. However, it is not clear how this may affect selection into IE prior to doing IE. 4.C. ESTIMATION STRATEGY The estimation strategy approximates a difference-in-difference (DD) approach. The DD estimator identifies the average treatment effect of IE on the treated when the following assumptions are 11 satisfied: (i) in the absence of the treatment, the disbursements paths of projects with and without IE would evolve in parallel; and (ii) the unobserved characteristics that might affect selection into IE are additive and time-invariant. From the descriptive findings we discussed above, we have some indication that assumption (i) would be satisfied in our context. Indeed, we found that the disbursement trends of projects with IE before the IE starts are similar (at least for the disbursement rate measure) to those of projects without IE. Assumption (ii) is consistent with project team leaders having different attitudes toward IE, as long as those attitudes do not change over time. We use different methods and specifications to check robustness of the results. Let D be a dummy coded 0 for projects without IE and 1 for projects with IE, whether or not the IE started, and T a dummy coded 0 in quarters before the IE starts and 1 thereafter. The DD estimate can be identified through the following pooled OLS model: Yit = α + δ 0 Di + δ1Tit + X it β + Z i λ + γ t + ε it , (1) where Y is the outcome variable; X a vector that includes dummies for number of times the project leader switches, the project leader’s Unique Personnel Number (UPI), and a third-order polynomial of average cumulative quarterly disbursement per sector; and Z a vector that includes time-invariant variables such as the fiscal year, country, and sector. The vector of parameters γ captures the quarter effects. The composite error term ε it is given by ci + uit , a time-invariant random component and an idiosyncratic component. The composite error term is assumed uncorrelated with observed variables and treatment status. 17 We can then interpret the parameter δ0 as measuring the effect of IE “assignment.” The parameter of interest is δ1 , which measures the DD estimated effect of IE on outcome Y. The reason why our strategy “approximates” a DD approach is because some IEs start at the same time as the project; therefore, we do not have any before data for this specific sample. Our DD estimator is therefore not the classic before-and-after framework. In particular, 3 percent of projects do not have a before comparison. 18 Note then that, unlike the standard DD model, where the coefficient of the dummy D provides the difference in level of the outcome between projects with and without IE at baseline, here the coefficient δ0 gives the difference in level between the subset of projects that do not have an IE yet and the projects that will never have an IE. 19 This allows us to interpret the coefficient of D as the 17 If the error term is uncorrelated with X, D, and T, then there is no need for the assumption (ii) above and the estimates will be consistent and efficient. 18 When we re-estimate our model excluding these observations, results do not change. Estimates available upon request. 19 As showed with the simple algebra, the effect of the 3 percent will be captured by the DD coefficient, whereas the coefficient of D will capture the difference in level between projects that do not have an active IE yet and projects without IE in that quarter. 12 pre-IE treatment differences in outcome levels after the IE is already assigned. If δ0 were to be insignificantly different from zero, we would conclude that projects with and without IE have the same rate of disbursement before the IE starts, and that this would indicate comparability between IE and non-IE projects at the outset. Our preferred estimation is a fixed effects (FE) model, because it allows for correlation between the error term and the covariates through the fixed effect ci , the time-invariant component of the error term. This term captures characteristics of the projects that are correlated with the presence of an IE. With the FE transformation, the model takes the following form: ̈ = 1 ̈ + ̈ + ̈ + + ̈ , (2) The transformation automatically drops all the variables that are fixed in time, such as dummies for sector, fiscal year, and country and the D dummy. The FE model generates consistent estimates when the unobserved fixed effects are correlated with the covariates. Given that projects approved within the same sector in a given time are more likely to share similar characteristics, standard errors are clustered at sector-time level. To check for robustness, we also estimate a pooled OLS, and a random effects (RE) model (estimated through a feasible generalized least squares model). These models require fewer restrictions because they do not control for project fixed effects. The reason for reestimating (1) using RE is to improve efficiency and to understand how important project fixed effects are (through the comparison of RE and FE). Given the relatively long horizon in our panel, the error term will be serially correlated across time. We try to control for this clustering standard errors at the sector and quarter levels. 13 5. RESULTS We find that doing impact evaluation has large, positive, and significant effects on project implementation, especially when measured against plans (see Table 3 and the snippet of the main results here within the text). Moving from project design to implementation is an arduous process. During the period before midterm, or about two-thirds of the way into projects’ implementation, projects are delayed, in the average quarter having cumulatively disbursed at a rate that is only around half of what they had planned (15 instead of 29 percent of loan size). The estimates for the effect of the IE, in this context, have high economic significance. To midterm, the effect size accounts for two-fifths (40.8 percent) increase in average cumulative disbursements, significant at 1 percent level. The planned to actual disbursements gap is reduced by one-half (54 percent, significant at 1 percent level), an 8 percentage point effect that helps realign projects to their planned implementation path. These effects should be not affected by sector composition, because we are using the sample of projects that reached midterm and for quarters up to midterm. Indeed, when compared to the disbursement path of other projects in the same sector, the IE increases project disbursements by 40 percent, significant at 1 percent level. In short, doing IE seems to be a powerful tool to move projects from design to implementation. Even when the whole sample is used (that is, when we do not restrict the sample at the midterm milestone), results carry through for disbursements relative to plan (45 percent, significant at 1 14 percent level) and relative to sector disbursements (49 percent, significant at 1 percent level), although not for average cumulative disbursements. Smaller point estimates are due to the partial convergence of disbursements to total loan amount as projects approach their closing date. This is only partial because (i) most projects in our sample have not yet closed (74 percent), and (ii) projects on average do not disburse their whole loan amount (78 percent in our sample). Average cumulative disbursements estimates are also affected by the substantial amount of heterogeneity in duration that is introduced when the whole sample is used, while the heterogeneity is controlled when the same disbursements are benchmarked against project plans or sector averages. The effects are economically large even using these smaller point estimates. The summary estimates above are based on an FE model that includes as controls dummies for the number of times the project leader changes and a third-order polynomial of average cumulative quarterly disbursement per sector. The estimates are robust to different specifications and the inclusion of different sets of controls. We briefly discuss these additional estimates below. 5.A. QUARTERS UP TO MIDTERM The effect of IE on disbursements for the quarters up to midterm is large, positive, and significant. It is also robust to different specifications (Table 4). In the FE model, the estimates range between 3.5 and 6 percentage points, corresponding to 24 to 41 percent of cumulative disbursements, significant at 1–5 percent level (with the exceptions of the specification in the last column of Table 4). The average project in the sample has only disbursed 14.7 percent of its loan amount in the average quarter up to midterm, or only about half of what was originally planned (29 percent). The effect size is thus large in absolute terms, contributing substantially to project implementation. Furthermore, the estimates are stable across different models, ranging between 5.6 and 6.4 percentage points (pps) in the RE and 5.6 and 8.2 pps in the OLS (all significant at 1–10 percent level). The latter models also provide an estimate for the coefficient on the IE assignment. Point estimates of the effect of the dummy D are negative, suggesting a degree of adverse selection into IE. We conclude that the IE seems to have a very considerable effect on the implementation speed up to midterm and that this effect is not due to IE assignment (selection) but to doing IE (treatment). Even more striking is the result on the effect of doing IE on the gap between planned and actual cumulative disbursements during the first two-thirds of project implementation. This result incorporates the acceleration of disbursements described above, but now takes into account the expectations that the project team originally had on the speed of implementation of their specific project. As can be seen in Table 5, the FE estimated effect of IE on the planned to actual gap ranges between -5 and -8 pps (significant at 1 percent level), which, relative to the mean gap of 14.4 percent of the loan amount, implies a 35 to 54 percent reduction in the disbursement gap. The effect is economically important: projects with IE reduce delays by half. Results carry through using alternative models, and the effect of IE assignment is not significantly different from zero when project leader fixed effects are included (with the exception of few cases in the OLS specification). Because projects in different sectors may follow different implementation paths (for example, infrastructure projects are slower to implement than non-infrastructure projects), we worry about 15 controlling for sector effects along the project life cycle. We thus estimate the average disbursement paths in each sector and normalize specific project disbursements to it. 20 When we then re-estimate the model, we find that the IE pushes disbursements way above the norm achieved by other projects in that same sector. Table 6 shows the estimates for the average effect of the IE on the rate of disbursement adjusted by the sector quarter average. By midterm, the FE estimates of the effect of IE range between 40 and 42 percent increase above the norm. The effects are highly significant in both statistical (1–5 percent level) and economic terms. Once OLS and RE models are compared, point estimates are consistent, and, if anything, larger. In those models, some of the variation is captured by the D dummy, with a zero or negative coefficient, giving limited support to the possibility that there might be adverse selection of slower projects into IE as we argued above. 5.B. ALL QUARTERS Around and after midterm, the projects that are delayed might rush to close their disbursement gap in view of the approaching review and closing date. Therefore, we would expect the effect of the IE over the whole implementation period to be smaller than for the period before midterm. Indeed, the FE point estimates for cumulative disbursements when using quarters up to closing date are only about a third of the effect to midterm and are not statistically significant (Table 7). It is hard, however, to interpret the results for the whole sample straightforwardly because a substantial amount of heterogeneity in duration is introduced. Specifically, there are very few projects with active IE after quarter 20. Therefore, using the whole sample of data assigns too much weight to quarters where the effect of IE is expected to be zero and IEs are almost absent. A different story arises, however, when we benchmark disbursements against project plans or sector averages. Here the IE continues to have strong and statistically significant effects, with FE estimates ranging between 3.4 and 6.6 pps, or 23 to 45 percent of total gap. The effect of the IE on disbursements relative to average sector disbursements is also large and significant (40 to 42 percent). The results are relatively stable across models and specifications and statistically significant at 1–10 percent level (Table 8 and Table 9). The difference between the results on disbursements and disbursement gap is instructive. The analysis on disbursements makes no presumption on the desired path of disbursements and finds little IE effects. When, however, the analysis incorporates project-specific expectations on their disbursement path, the IE has large and significant estimated effects. An interpretation of this is that planned disbursements might capture project’s implementation path in more subtle ways than project fixed effects. This seems to be especially important for the analysis across all quarters, as there is a lot of heterogeneity in project duration. We conclude that, given the unbalanced panel, planned expenditure helps control for the composition effect across quarters and helps identify the IE effects. The same argument carries through in the case of sector-specific disbursement paths. 20 By construction, for the average project, the ratio between rate of disbursement and average disbursement is 1. 16 Overall, we find positive and significant effects of IE on project disbursements, especially when measured against plans and sector averages. The effects are stable across models and specifications, especially up to midterm, where comparability is greater and issues of composition are less important. For the quarters up to midterm, the effect is so large as to reduce the breach between project plans and actual implementation by half. 5.C. POTENTIAL MECHANISMS The potential channels through which World Bank IE research can affect projects are described through anecdotal evidence in Legovini (2010, 2011). We also build on Legovini and Bedoya (2014), who develop a set of policy influence indicators for IEs that cover policy design, capacity building and data, policy implementation, and evidence-based decisions. They collect data from IE teams for a set of 116 IEs covering both World Bank projects and other investments. We report here some findings from that study with the aim of presenting the channels that could be more relevant to explain the effect of IE. During the early stages of engagement between research and project teams, the issue of rigor in policy design is made more salient. Teams report that existing evidence is incorporated in policy design (42 percent of the cases) and a structured learning strategy is introduced. The latter is measured by counting whether projects introduce experimental treatment arms to compare different implementation models or mechanisms (53 percent of the cases versus none for projects without IE). As part of this process, teams develop detailed implementation plans and become more aware of the potential contribution of their work to the development objectives of their countries and their role in securing results. Team motivation may be affected by the process of making plans more actionable and the link with results more explicit. During preparation and implementation, several efforts are geared toward increasing teams’ capacity. We monitor client attendance to one-week IE workshops 21 (88 percent) or data training (36 percent), and incidence of discussions on baseline results (74 percent) and IE results (75 percent). Capacity is measured by participant learning and the technical quality of the planned IEs. As measured by the difference in IE workshop pre- and post-test results, participants increase their impact evaluation knowledge by one standard deviation on average. Further, as measured by double- blind external reviews, the average technical rating for proposals submitted by teams who participated in an IE workshop exceeds by half a standard deviation that of proposals submitted by other teams. 22 In addition to training, capacity is affected by supplementing the operations teams with a research and by placing a full-time field coordinator (100 percent). 21 See events at dime.worldbank.org 22 Authors’ calculations. 17 Further effort is expanded on collection of high-quality data, often the first set of sector data an agency will have access to. IE-generated data provide a client with at least two rounds of survey data (84 percent) and support to help improve administrative data systems (48 percent). Survey data are generally not available for projects without IE. Evidence-based decisions are taken at multiple stages of IE implementation. Policy changes are recorded following baseline results discussions (36 percent), and when different mechanisms are tested (23 percent). Policy scale-up and -down decisions are recorded following discussions on IE results in 30 percent of the cases. The relative importance of these different mechanisms will be the subject of future research as we expand the IE coverage and frequency of data collection of these monitoring indicators. 6. CONCLUSIONS This is the first study that investigates the effect of IE research on contemporaneous measures of aid- financed project implementation. It builds on the notion that the effectiveness of aid greatly depends on the quality of projects through which aid is delivered, and builds a case for a fruitful complementarity between aid financing and IE research. The study presents evidence that impact evaluation may help improve project implementation. Since project implementation is a necessary condition (although not sufficient) for projects to reach their development outcomes, then impact evaluation might be an enabling mechanism to secure developmental effectiveness. We find that impact evaluation has large and significant effects on the disbursements of World Bank projects, increasing the average level of quarterly disbursements by 40 percent by midterm, helping projects reduce delays relative to plan by half, and increasing by 40 percent project disbursements relative to the average in the typical project in the same sector. We conclude that projects with IE are more likely to implement their activities as planned and, in so doing, are more likely to achieve their objectives. This is of significance for both the Bank and development policy. For the Bank, the introduction of IE might help avert the need to push disbursements out toward the latter half of project duration, or to restructure and cancel nonperforming projects. It can also help manage the potential reputational risk of committing funding without being able to secure their disbursements. More important, for developing countries, avoiding delays may mean that schools, health facilities, or roads are built now instead of next year or not at all, securing the potential for educating one more generation, saving more lives, or connecting to market more people and goods. Furthermore, our results support the idea that project financing and IE research are complementary outputs of the World Bank. This has implications for the use of IE as an operational tool to improve 18 delivery of projects, and for weighing the costs of IE research 23 against the benefits on the Bank’s project portfolio. For development organizations like the World Bank, understanding the degree of substitutability or complementarity between these outputs can help strike an optimal balance between project financing and IE research to help us reach the efficient frontier. A more general implication of this work is that the development community overall may be underinvesting in the type of research that has positive externalities on development investments, thus forsaking the opportunity to increase developmental effectiveness. Presently, IE research coverage is about 10 percent of the World Bank portfolio. This is likely to be lower in other development banks and institutions that, on average, have less research capacity. Whether this level strikes the optimal balance remains to be assessed. Finally, our results question the notion that impact evaluation is solely a knowledge-generating public good, and puts a value to the use of impact evaluation as a private good, that is, one that helps individual projects deliver on their promises. If IE has public and private good features, then the case for what constitutes knowledge generation value and who should contribute to IE financing could be investigated. Future research can help better inform this debate. First, research should focus on understanding the underlying mechanisms through which IE research affects the path of project implementation and tracing the pathways through which these effects can be strengthened. Second, research should focus on quantifying the benefits of timely project implementation on development outcomes and identifying whom the benefits accrue to. This could help inform allocative choices in institutions like the World Bank and other development agencies. REFERENCES Bruhn, M., L. Leao, A. Legovini, R. Marchetti, and B. Zia. 2013. “The Impact of High School Financial Education: Experimental Evidence from Brazil.” Policy Research Working Paper 6723, World Bank, Washington, DC. Brune, L., X. Giné, J. Goldberg, and D. Yang. 2011. “Commitments to Save: A Field Experiment in Rural Malawi.” Policy Research Working Paper 5748, World Bank, Washington, DC. Denizer, C., D. Kaufmann, and A. Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105:288–302. 23 The cost to the Bank of a completed IE product was, on average, $279,493 in fiscal 2007–14. This represents less than 0.2 percent of loan size. 19 Deininger, K., L. Squire, and S. Basu. 1998. “Does Economic Analysis Improve the Quality of Foreign Assistance?” World Bank Economic Review 12 (3): 385–418. Dhaliwal, I., and C. Tulloch. 2012. “From Research to Policy: Using Evidence from Impact Evaluations to Inform Development Policy.” Journal of Development Effectiveness 4 (4): 515–36. Friedman, J. M., M. Vledder, M. Sjoblom, and P. Yadav. Forthcoming. “Improving Patient Access to Malaria and Other Essential Medicines in Zambia: Results of a Pilot Project.” Presented at the Transport Learning Event, Washington, DC, 2010. http://siteresources.worldbank.org/INTTRANSPORT/Resources/336291- 1297096897336/7715763-1297096948414/Presentation-Friedman.pdf. Ika, L. A., A. Diallo, and D. Thuillier. 2012. “Critical Success Factors for World Bank Projects: An Empirical Investigation.” International Journal of Project Management 30 (1): 105–16. Kilby, C. 2000. “Supervision and Performance: The Case of World Bank Projects.” Journal of Development Economics 62 (1): 233–59. Kilby, C. 2011. “Assessing the Contribution of Donor Agencies to Aid Effectiveness: The Impact of World Bank Preparation on Project Outcomes.” Villanova School of Business Economics Working Paper 20, Villanova University, Pennsylvania. Kilby, C. 2013a. “The Political Economy of Project Preparation: An Empirical Analysis of World Bank Projects.” Journal of Development Economics 105:211–25. Kilby, C. 2013b. “An Empirical Assessment of Informal Influence in the World Bank.” Economic Development and Cultural Change 61 (2): 431–64. Kondylis, F., M. Jones, and D. Stein. 2014. “Commitment, Mental Accounting, & Agricultural Investment: Evidence from Rwanda.” Mimeo. World Bank Legovini, A. 2010. “Development Impact Evaluation Initiative: A World Bank–Wide Strategic Approach to Enhance Developmental Effectiveness.” World Bank, Washington, DC. http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/3998199- 1286546178578/7465778-1291306572028/Legovini_dime_paper_ext.pdf. Legovini, A. 2011. “DIME Progress Report.”, World Bank, Washington, DC. http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/3998199- 1372170088287/DIME_progress_report_2012.pdf. Legovini, A., and G. Bedoya. 2014. “Dime’s Current Portfolio and Approach to Impact Evaluation.” Technical note, World Bank, Washington, DC. 20 Masset, E., M. Gaarder, P. Beynon, and C. Chapoy. 2013. “What Is the Impact of a Policy Brief? Results of an Experiment in Research Dissemination.” Journal of Development Effectiveness 5 (1): 50–63. Millennium Challenge Corporation (MCC). 2012. “MCC’s Independent Evaluations.” Fact sheet, MCC, Washington, DC. http://www.mcc.gov/documents/press/factsheet-2012002116501-independent- evaluations.pdf. Neilson, S. 2001. “Knowledge Utilization and Public Policy Processes: A Literature Review.” Ottawa: IDRC Evaluation Unit. Ruprah, I., and L. Marcano. 2009. “Does Technical Assistance Matter? An Impact Evaluation Approach to Estimate Its Value Added.” Journal of Development Effectiveness 1 (4): 507–28. Temple, J. R. 2010. “Aid and Conditionality.” In Handbook of Development Economics, edited by D. Rodrik and M. Rosenzweig, 5:4415–523. Amsterdam: North Holland. Thomas, V., and J. Tominaga. 2010. “Evaluation for Better Development Results.” Journal of Development Effectiveness 2 (3): 371–86. World Bank. 2006. “Impact Evaluation and the Project Cycle: Doing Impact Evaluation No. 1.” http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and- Materials/ie_and_projectcycle.pdf. World Bank. 2009. Annual Review of Development Effectiveness, 2009: Achieving Sustainable Development. Washington, DC: IEG, World Bank. http://siteresources.worldbank.org/EXT2009ARDE/Resources/arde09_web.pdf. World Bank. 2012a. The World Bank Annual Report 2012: Volume 3. Annexes. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/11846. World Bank. 2012b World Bank Group Impact Evaluations. Relevance and Effectiveness.. Washington, DC: IEG, World Bank. http://ieg.worldbank.org/Data/reports/impact_eval_report.pdf World Bank. 2014a. “World Bank Corporate Scorecard, April 2014.” Brochure, World Bank, Washington, DC. http://siteresources.worldbank.org/CSCARDEXT/Resources/2014_WBG_corporate_scorecard_e- version.pdf. World Bank. 2014b. “Review of the World Bank FY13 Portfolio and Update on Actions to Assure Quality.” World Bank, forthcoming FIGURES AND TABLES 21 Figure 1. WORLD BANK IES BY START DATE 80 70 60 50 40 IE started 30 IE started of WB projects 20 10 0 Figure 2. DISTRIBUTION OF PROJECTS WITH AND WITHOUT IE BY QUARTER Figure 3. PROJECTS AND QUARTERS AND MIDTERM MILESTONE 22 Figure 4. NUMBER OF PROJECTS PER SECTOR AND REGION Figure 5. WHEN DOES THE IE KICK IN? 23 Figure 6. RATE OF DISBURSEMENT AND IE Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 & T=0) line is truncated because all IEs start eventually after Q14. Figure 7. RATE OF DISBURSEMENT AND IE (UP TO MIDTERM REVIEW) 24 Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 & T=0) line is truncated because all IEs start eventually after Q14. Figure 8. GAP (PLANNED – ACTUAL) IN RATE OF DISBURSEMENT AND IE Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 & T=0) line is truncated because all IEs start eventually after Q14. 25 Figure 9. GAP (PLANNED – ACTUAL) IN RATE OF DISBURSEMENT AND IE (UP TO MIDTERM) Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 & T=0) line is truncated because all IEs start eventually after Q14. Figure 10. RATE OF DISBURSEMENT AS RATIO TO QUARTER SECTOR AVERAGE RATIO AND IE Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 &T=0) line is truncated because all IEs start eventually after Q14. 26 Figure 11. RATE OF DISBURSEMENT AS RATIO TO QUARTER SECTOR AVERAGE RATIO AND IE (UP TO MIDTERM) Note: Data is plotted only when the number of projects in each sample at each quarter was greater or equal to 5. For sample, (D=1 & T=0) line is truncated because all IEs start eventually after Q14. 27 Table 1. PROJECTS WITH AND WITHOUT IE BY SECTOR AND FISCAL YEAR Fiscal year 2005 2006 2007 2008 2009 2010 2011 TOTAL IE/TOTAL Sector IE No IE IE No IE IE No IE IE No IE IE No IE IE No IE IE No IE IE No IE (%) Agriculture and Rural Development 0 32 4 42 4 35 3 30 7 28 5 25 1 9 24 201 12 Education 3 22 3 17 5 15 1 16 3 15 2 10 0 2 17 97 18 Energy and Mining 0 21 2 25 0 12 0 26 0 20 0 14 0 6 2 124 2 Environment 0 9 1 9 0 2 1 4 0 4 0 3 0 1 2 32 6 Financial and Private Sector Development 2 17 0 14 2 9 2 6 2 10 2 12 1 3 11 71 15 Health, Nutrition and Population 1 20 4 16 3 13 2 18 1 13 1 13 0 2 12 95 13 Social Development 2 4 0 4 0 3 1 0 1 3 0 3 0 1 4 18 22 Social Protection 1 10 4 3 1 2 3 9 6 8 0 16 1 1 16 49 33 Transport 0 17 1 22 1 29 1 24 0 20 0 30 0 2 3 144 2 Urban Development 1 18 2 13 0 18 0 21 1 10 0 21 1 7 5 108 5 Water 1 17 0 12 1 22 0 16 1 15 1 14 0 0 4 96 4 Total 11 187 21 177 17 160 14 170 22 146 11 161 4 34 100 1035 10 28 Table 2. DESCRIPTIVE STATISTICS No IE IE Total Sample size P value from T test Project Characteristics Number of projects 1035 100 1135 Number of quarters 16540 1145 17685 Fiscal Year of Effectiveness is 2005 187 11 198 … 2006 177 21 198 … 2007 160 17 177 … 2008 170 14 184 … 2009 146 22 168 … 2010 161 11 172 … 2011 34 4 38 Active projects 760 78 838 Closed projects 275 22 297 Total loan size committed (USD million) 109 149 112 0.07 Midterm review completed 537 55 592 Number of TTLs per project 2.1 2.11 2.1 0.92 Duration Actual duration in quarters (all) 15.69 15.39 15.67 1135 0.68 Planned duration in quarters (all) 20.02 19.64 19.99 1135 0.43 Actual duration in quarters (active projects) 13.87 14.30 13.91 838 0.59 Planned duration in quarters (active projects) 21.01 20.65 20.97 838 0.47 Actual duration in quarters (closed projects) 20.72 19.22 20.61 297 0.20 Planned duration in quarters (closed projects) 17.29 16.04 17.19 297 0.24 Actual duration (projects with midterm review) 18.11 18.52 18.15 592 0.60 Actual Duration up to midterm review 13.38 13.25 13.37 592 0.79 Disbursement Planned disbursement Average (USD million) 61.13 86.45 63.36 1135 0.04 Average at midterm (USD million) 52.86 76.66 55.07 592 0.05 Rate (total planned disb/loan committed) 0.64 0.60 0.63 1135 0.34 Rate at midterm (if midterm reached) 0.62 0.62 0.62 592 0.9 Actual disbursement Average (USD million) 44.23 79.20 47.31 1135 0.00 Average at midterm (USD million) 34.93 54.85 36.78 592 0.07 Rate (total disbursed/loan committed) 0.44 0.49 0.44 1135 0.17 Rate at midterm 0.35 0.45 0.36 592 0.00 Rate at quarter 4 0.06 0.07 0.06 1116 0.36 Rate at quarter 8 0.15 0.21 0.16 953 0.00 Rate at quarter 12 0.28 0.36 0.29 785 0.00 Rate at quarter 16 0.43 0.50 0.44 589 0.07 Rate at quarter 20 0.57 0.60 0.57 391 0.54 Rate at closing (closed projects) 0.77 0.86 0.78 297 0.16 Disbursement gap (planned - actual disbursement) Average per quarter 0.01 0.005 0.009 17685 0.00 Average 0.15 0.07 0.14 1135 0.05 Average at midterm 0.27 0.17 0.26 592 0.01 Average at closing(closed projects) 0.00 -0.18 -0.01 297 0.10 Actual disbursement deviation from sector/quarter average Rate 0.99 1.08 0.99 17629 0.25 Rate at midterm 0.89 1.09 0.91 7867 0.10 Time to first disbursement (quarters) 4.15 3.91 4.13 1126 0.23 29 Table 3. SUMMARY TABLE – FIXED EFFECTS ESTIMATES Rate of Disbursement - Rate of Rate of Gap - Up to Mid- Adjusted by Rate of Disbursement - Disbursement - Gap Term S ector Average Disbursement Adjusted by Up to Mid-Term – Up to Mid- S ector Average Term T (quarters with IE==1) 0.060*** -0.078*** 0.40*** 0.038 -0.067** 0.45*** (3.00) (-3.08) (3.39) (1.54) (-2.47) (3.15) TTL switch dummies Y Y Y Y Y Y Polynomial of sector averages Y Y Y Y Y Y Observations 7918 7921 7889 17784 17784 17728 Outcome mean 0.147 0.144 1.00 0.244 0.148 0.917 Effect in % 40.8% 54.2% 40.0% 15.6% 45.3% 49.1% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 30 Table 4. EFFECT OF IE ON RATE OF DISBURSEMENT UP TO MIDTERM DID-OLS DID-RE DID-FE T (quarters with IE==1) 0.082*** 0.056*** 0.082*** 0.056*** 0.063*** 0.056*** 0.064*** 0.056*** 0.060*** 0.036* 0.060*** 0.035 (10.0) (4.59) (9.99) (4.56) (3.43) (3.16) (3.47) (3.15) (2.99) (1.69) (3.00) (1.64) D (project with IE==1) -0.030*** -0.052*** -0.030*** -0.052*** -0.015 -0.052** -0.015 -0.052** (-5.31) (-4.17) (-5.32) (-4.16) (-0.96) (-2.52) (-0.96) (-2.51) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 7918 7918 7918 7918 7918 7918 7918 7918 7918 7918 7918 7918 Outcome mean 0.147 0.147 0.147 0.147 0.147 0.147 0.147 0.147 0.147 0.147 0.147 0.147 Effect in % 55.8% 38.1% 55.8% 38.1% 42.9% 38.1% 43.5% 38.1% 40.8% 24.5% 40.8% 23.8% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 31 Table 5. EFFECT OF IE ON GAP UP TO MIDTERM DID-OLS DID-RE DID-FE T (quarters with IE==1) -0.056*** -0.045*** -0.056*** -0.045*** -0.073*** -0.045 -0.073*** -0.045 -0.078*** -0.051* -0.078*** -0.051* (-4.86) (-2.85) (-4.83) (-2.86) (-3.15) (-1.62) (-3.16) (-1.62) (-3.07) (-1.79) (-3.08) (-1.78) D (project with IE==1) -0.0069 0.035** -0.0069 0.035** 0.0081 0.035 0.0081 0.035 (-0.83) (2.09) (-0.83) (2.07) (0.43) (1.34) (0.44) (1.33) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 7918 7918 7918 7918 7918 7918 7918 7920 7918 7918 7921 7918 Outcome mean 0.144 0.144 0.144 0.144 0.144 0.144 0.144 0.144 0.144 0.144 0.144 0.144 Effect in % 38.9% 31.3% 38.9% 31.3% 50.7% 31.3% 50.7% 31.3% 54.2% 35.4% 54.2% 35.4% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 32 Table 6. EFFECT OF IE ON RATE OF DISBURSEMENT ADJUSTED BY SECTOR AVERAGE UP TO MIDTERM DID-OLS DID-RE DID-FE T (quarters with IE==1) 0.74*** 0.48*** 0.74*** 0.48*** 0.46*** 0.40*** 0.44*** 0.40*** 0.42*** 0.41*** 0.40*** 0.40*** (5.60) (4.03) (5.54) (4.00) (3.88) (3.03) (3.68) (2.93) (3.62) (3.04) (3.39) (2.91) D (project with IE==1) -0.46*** -0.21 -0.46*** -0.21 -0.30 -0.21 -0.29 -0.21 (-3.84) (-1.53) (-3.81) (-1.53) (-1.44) (-1.02) (-1.38) (-0.99) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 Outcome mean 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Effect in % 74.0% 48.0% 74.0% 48.0% 46.0% 40.0% 44.0% 40.0% 42.0% 41.0% 40.0% 40.0% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 33 Table 7. EFFECT OF IE ON RATE OF DISBURSEMENT DID-OLS DID-RE DID-FE T (quarters with IE==1) 0.082*** 0.037*** 0.083*** 0.037*** 0.039* 0.011 0.040* 0.011 0.037 0.0091 0.038 0.0096 (9.21) (3.71) (9.27) (3.75) (1.69) (0.54) (1.73) (0.56) (1.50) (0.44) (1.54) (0.46) D (project with IE==1) -0.029*** 0.0044 -0.030*** 0.0044 -0.0035 0.011 -0.0044 0.011 (-4.66) (0.44) (-4.70) (0.43) (-0.20) (0.45) (-0.25) (0.45) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 Outcome mean 0.244 0.244 0.244 0.244 0.244 0.244 0.244 0.244 0.244 0.244 0.244 0.244 Effect in % 33.6% 15.2% 34.0% 15.2% 16.0% 4.5% 16.4% 4.5% 15.2% 3.7% 15.6% 3.9% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 34 Table 8. EFFECT OF IE ON GAP DID-OLS DID-RE DID-FE T (quarters with IE==1) -0.073*** -0.061*** -0.074*** -0.061*** -0.065** -0.038 -0.066*** -0.037 -0.066** -0.034 -0.067** -0.034 (-8.17) (-5.72) (-8.28) (-5.73) (-2.56) (-1.48) (-2.60) (-1.45) (-2.43) (-1.29) (-2.47) (-1.27) D (project with IE==1) -0.0098 0.0056 -0.0094 0.0055 -0.0090 -0.0080 -0.0079 -0.0083 (-1.43) (0.46) (-1.37) (0.45) (-0.51) (-0.33) (-0.44) (-0.34) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 Outcome mean 0.148 0.148 0.148 0.148 0.148 0.148 0.148 0.148 0.148 0.148 0.148 0.148 Effect in % 49.3% 41.2% 50.0% 41.2% 43.9% 25.7% 44.6% 25.0% 44.6% 23.0% 45.3% 23.0% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 35 Table 9. EFFECT OF IE ON RATE OF DISBURSEMENT ADJUSTED BY SECTOR AVERAGE DID-OLS DID-RE DID-FE T (quarters with IE==1) 0.74*** 0.48*** 0.74*** 0.48*** 0.46*** 0.40*** 0.44*** 0.40*** 0.42*** 0.41*** 0.40*** 0.40*** (5.60) (4.03) (5.54) (4.00) (3.88) (3.03) (3.68) (2.93) (3.62) (3.04) (3.39) (2.91) D (project with IE==1) -0.46*** -0.21 -0.46*** -0.21 -0.30 -0.21 -0.29 -0.21 (-3.84) (-1.53) (-3.81) (-1.53) (-1.44) (-1.02) (-1.38) (-0.99) TTL switch dummies Y N Y N Y N Y N Y N Y N TTL UPI dummies N Y N Y N Y N Y N Y N Y Polynomial of sector averages N N Y Y N N Y Y N N Y Y Observations 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 17784 Outcome mean 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Effect in % 74.0% 48.0% 74.0% 48.0% 46.0% 40.0% 44.0% 40.0% 42.0% 41.0% 40.0% 40.0% Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 36 FIGURE A.1A. PROJECTS WITH IE BY SECTOR AND QUARTER (COUNT) FIGURE A.1B. PROJECTS WITH IE BY SECTOR AND QUARTER (QUARTER RATIO) 37 FIGURE A.2A. PROJECTS WITH IE BY SECTOR AND QUARTER (UP TO MIDTERM) FIGURE A.2B. PROJECTS WITH IE BY SECTOR AND QUARTER (UP TO MIDTERM) 38