WPS8054 Policy Research Working Paper 8054 Do Countries Learn from Experience in Infrastructure PPP? PPP Practice and Contract Cancellation Darwin Marcelo Schuyler House Cledan Mandri-Perrott Jordan Schwartz Public-Private Partnerships Cross-Cutting Solutions Area May 2017 Policy Research Working Paper 8054 Abstract Learning from experience to improve future infrastructure whether general public-private partnership experience has public-private partnerships is a focal issue for policy makers, an impact on reducing the chances of contract cancellation financiers, implementers, and private sector stakeholders. for future projects. The results confirm what the market An extensive body of case studies and “lessons learned” aims intuitively knows, that is, that public-private partnership to improve the likelihood of success and attempts to avoid experience reduces the likelihood of contract cancella- future contract failures across sectors and geographies. This tion. But the results also provide a perhaps less intuitive paper examines whether countries do, indeed, learn from finding: the benefits of learning are typically concentrated experience to improve the probability of success of pub- in the first few public-private partnership deals. More- lic-private partnerships at the national level. The purview over, the results show that the probability of cancellation of the paper is not to diagnose learning across all aspects of varies across sectors and suggests the relative complexity of public-private partnerships globally, but rather to focus on water public-private partnerships compared with energy whether experience has an effect on the most extreme cases of and transport projects. An estimated $1.5 billion per year public-private partnership contract failure, premature con- could have been saved with interventions and support to tract cancellation. The analysis utilizes mixed-effects probit reduce cancellations in less experienced countries (those regression combined with spline models to test empirically with fewer than 23 prior public-private partnerships). This paper is a product of the Public-Private Partnerships Cross-Cutting Solutions Area. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at dmarcelo@worldbank.org, shouse@worldbank.org, cmandriperrott@worldbank.org, and jschwartz3@ worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Do Countries Learn from Experience in Infrastructure PPP? PPP Practice and Contract Cancellation Darwin Marcelo, Schuyler House, Cledan Mandri-Perrott and Jordan Schwartz Keywords: Public-private partnership, PPP, contract cancellation, mixed-effect probit model, linear spline, cubic spline. JEL Classification Codes: C21, C25, O21, H54, R42 Acknowledgements The authors thank Laurence Carter, Luis Andres, Fernanda Ruiz-Nuñez, Francois Bergere, and Stephane Straub for their valuable comments to the paper as well as Chin-Han Chiang for analytical support. Introduction 'Experience is the teacher of all things,' offered Julius Caesar in his commentary on the Roman Civil War of 49-48 B.C. – an adage that has held fast for over 2,000 years as common wisdom about the importance of learning through experience, particularly in situations characterized by complexity. In the case of public infrastructure projects, which are intrinsically complex arrangements with large investment requirements and important implications for economic development and the health and quality of life of the citizen public, the stakes are high to learn from experience to deliver infrastructure services more efficiently and effectively than in the past. Public-private partnerships (PPPs) for infrastructure require that governments learn how to harness the strengths of the private sector while preserving public interest and affordability of infrastructure services, all within long-term contractual relationships subject to inevitable uncertainties over time. The PPP – while a powerful and effective tool for infrastructure delivery – requires sound design and management, a good appreciation of public direct and contingent liabilities, a certain degree of customization to the local context, and the management of relationships between the public and private sectors over long periods. These requirements and their fine calibrations are neither apparent nor fully standardized. Rather, they must be understood, enhanced and designed by the PPP participants – i.e., governments and sponsors – to improve PPP arrangements, with the expectation of increasing the likelihood of future contract success. This paper asks how country experience with public-private partnerships impacts the probability of contract cancellation – an extreme form of PPP failure – for future infrastructure PPP contracts. The results suggest that PPP experience does, indeed, reduce the likelihood of contract cancellation. These results are an important starting point for ongoing research on the multiple channels by which experience and lesson-drawing may be leveraged to improve PPP performance, contract management, and government support over the future. Learning, Experience, and PPP A fairly well-developed management literature – particularly relating to joint venture models, concession models, strategic alliances, and other types of business-to-business partnerships – and a nascent 'policy learning' literature attend to the importance of experience-based learning to the success of complex organizational arrangements. Additionally, a large and growing body of work by PPP practitioners, including those within government, the private sector, and multilateral development organizations, has focused on cataloguing 'lessons learned' for PPP implementation across different sectors, political and economic contexts, and levels of government. The perceived importance of organizational, relational, and technical learning to improved performance is apparent in the proliferation of reports, papers, and other publications focused on case studies and lesson-drawing for infrastructure PPP. Learning is expected to improve the generation and utilization of useful knowledge to help governments avoid future policy failures and increase the potential for greater success with respect to future policy goals and outcomes (Howlett, 2009). Failures may be attributable to inability to anticipate the consequences of a program, particularly in the face of unforeseen risks (Howlett, 2009; Bovens and t'Hart 1996), poor execution during implementation (Mandri-Perrott and Bisbey, 2016; Linder and Peters, 1987), failure to effectively evaluate policies and programs or learn from evaluation to improve future design (May 1992), or the general intractability of a 'wicked problem,' for which there is no clear and apparent cause nor solution (Churchman, 1967). 2 A PPP might be subject to any one of these roots of failure, and the PPP literature quite actively examines why some PPPs fail and others succeed, with studies varyingly focusing on such factors as contract management, regulation, PPP frameworks, governance, principal-agent problems, and government capacity, among other factors. The common thread in these studies is that PPPs are complex arrangements that require the alignment and constant adjustment of many working parts to succeed over the long term –requirements that demand multiple kinds of learning to discover, fine-tune, and maintain workable arrangements. 'Thick learning' for PPP Because PPPs are complex organizational forms that must necessarily balance different and sometimes-conflicting interests and incentives of the public and private sectors, and because successful implementation requires dealing with distinct local legal, financial, regulatory, economic, and physical contexts, it would naturally be expected that multiple kinds of learning would be needed when implementing a PPP in a new market. This multifaceted learning is what policy scholars refer to as 'thick' learning (Howlett, 2012), in that it necessarily entails learning across program, process, and political dimensions (McConnell, 2010). Moreover, because some PPP situations are unique, shared knowledge that emerges through practice and engagement is likely to be important to improving PPP arrangements. While a syntactic perception of knowledge assumes that information can be collected and transferred, and a semantic view emphasizes interpretation, the pragmatic approach sees knowledge as 'localized, embedded, and invested in practice' (Weber and Khademian, 2008). This practical knowledge – or 'metis' as termed by Scott (1998) – evolves through practice and is tied to the experiences and relationships of the participants, who jointly build knowledge to create better- fitting policies, adopt a more unified mental mode, and align values and interests (Weber and Khademian, 2008). Metis is particularly important when information needs are unclear, problems are multifaceted, and differentiated solutions are required. New infrastructure PPPs may incorporate technical components informed by sector-wide best practice or case studies of similar infrastructure projects. But not all PPP-related lessons and best practices can be automatically transferred to new contexts. Some knowledge must be hard won over time as partners collectively generate knowledge specific to the conditions at hand and overcome initial obstacles in the operating context. For example, as countries undertake early PPPs, they may discover regulatory or legal challenges that require resolutions or improved governmental capacities (such as contract management) that require development, which in turn improve the design, implementation, and oversight of projects to follow. Moreover, while participants bring individual knowledge to a new PPP, this knowledge can be difficult to transfer and challenging to integrate into a functioning compilation without experience. For this reason, demonstration projects –those first PPPs implemented by a country– are recognized to be pivotal learning experiences for governments initiating a PPP program and supportive PPP framework (Delmon, 2009). Lesson-drawing from PPP experience Governments, policy think tanks, multilateral development organizations, consultants, ratings agencies, and other parties with interests in infrastructure PPP have produced an extensive body of literature and reporting on lessons learned from PPP experience. The World Bank's Public- Private Partnership in Infrastructure Resource Center (PPPIRC), for example, provides an extensive listing of links to PPP case studies and lessons learned produced by multilateral 3 development organizations and governments. 1 Lesson-drawing and dissemination is geared towards improving the likelihood of future project success and avoiding some of the most deleterious pitfalls that have caused prior PPP distress and cancellation. Extensive reports have been produced that discuss lessons related to contract design, legal frameworks, modes and approaches to risk allocation, pricing, bidding and award methods, financial structures, public support mechanisms, project preparation, political risks, and other factors relevant to PPP implementation. Many developed countries with extensive and long-standing PPP experiences have also seen the value of cataloging lessons learned in order to improve the legislative, regulatory and policy environment and associated governmental mechanisms for PPPs. For example, the United Kingdom, whose Private Finance Initiative of the early 1990s has undoubtedly influenced governments, sponsors and lenders, continues to evolve their program to improve how government facilitates and manages PPP contracts. The UK Treasury's 2011 'Private Finance Initiative' report cited a number of weaknesses in the prevailing Private Finance Initiative (PFI) model, including inflexible contracts, limitations in transparency with respect to investor returns, higher than expected risk premiums incurred by government, and questions over public value for money, all leading to the recommendation of an evolved 'PF2' model of PPP. Given the efforts to evaluate, compile, and disseminate lessons learned through active and concluded PPPs, it is a worthwhile pursuit to ask whether governments do, in fact, learn from PPP experiences to improve future PPPs. While the purview of this paper is not to diagnose learning across all aspects of PPP globally, we more simply ask whether experience has a limiting effect on the most extreme cases of PPP contract failure –premature contract cancellation. To answer this question, we quantitatively model the effect of experience with PPP on future contract performance using a large data set. Data and PPP Country Experience The descriptive statistics and econometric models reported in this study are based on data from the World Bank's Private Participation in Infrastructure (PPI) Project Database, which tracks infrastructure projects in developing regions that entail some form of private participation since 1990. 2 As of August 2016, the database includes 7,192 projects across 139 countries with recorded variables for each project, including project status (active, cancelled, or concluded), project sector and subsector, committed investments, contract duration, contract form, multilateral support, and project sponsors, among other variables. Of the 7,192 projects in the database of projects with some form of private participation in infrastructure (PPI), 5,478 are more narrowly defined as 'public-private partnerships'. This set excludes projects for which the degree of private participation tends toward the extreme-private side of the public-private spectrum. They are either full divestitures (i.e., total privatization of an asset) or 'merchant' projects, where a private sponsor builds a facility for which no government revenue guarantees are provided (thus bearing all construction, operating, and market risks). A further limitation is imposed on the data set: we include only active or cancelled projects that reached financial closure before 2011. The reason for truncating the data set to include only the 1 See http://ppp.worldbank.org/public-private-partnership/lessons-learned-0. 2 The Private Participation in Infrastructure (PPI) Project Database is the leading source of PPI trends in the developing world, covering projects in the energy, telecommunications, transport, and water and sewerage sectors. Projects include management or lease contracts, concessions, Greenfield projects, and divestitures. 4 3,400 PPP projects with at least a six-year history is simple: a project's status (i.e., conclusion after full contract term, early cancellation and ongoing operation) is only reasonably observable and measurable when the project has been (or could have been) in effect for a sufficient amount of time. Contract cancellation rates would undoubtedly be underestimated if one-year operational projects were included, for example, as these projects are likely too early in their development to reveal potential problems that might otherwise lead to cancellation. Upon examining the project data in the PPI Database, cancelled projects exhibit an average duration (between financial closure year and cancellation) of 5.89 years (Marcelo and House, 2016). This average duration is justification for truncating the data set to remove projects whose closure dates are less than six years before the analysis.3 Of the 3,400 PPPs with at least six-year histories, 94% are active – i.e., operational or under construction– while 6% (191 PPPs) were canceled. Figure 1 describes the subsets of data used for modeling, and Table 1 provides summary statistics for the 2,833 projects used in modeling. Figure 1. PPP data selection PPI projects (7,192 projects) PPPs 5,478 projects (76%) PPPs reaching financial closure before 2011 (Excluding 58 distressed and 203 concluded PPPs) 3,400 PPPs (62%): Active PPPs (94%) Canceled PPPs (6%) In order to capture the effect of country-level experience on PPP cancellations, a variable to proxy the degree of familiarity and experience in facilitating public-private partnerships was estimated. The PPP country experience associated with each project is defined as the number of PPPs that reached financial closure within the past decade in the same country. So then, the level of PPP experience associated with any project initiated in country in year , is equivalent to the sum of all PPPs that reached financial closure in that country in the previous 10 years. Following this definition, Figure 2 gives a visual overview of the maximum PPP experience reached by each country over the past decade from 1990 to 2016. African and Central Asian countries have the lowest levels of PPP experience in the developing world, though the region includes three countries (i.e. South Africa, Nigeria and Algeria) with more than 20 PPP deals. In contrast, South American countries have relatively high levels of PPP experience, led by Brazil (387 PPPs). China and India are developing countries with the highest PPP experience, with more than 600 PPP deals each. 3 In the Sensitivity Analysis section, this data set constraint is tested. See also Annex 4. 5 Figure 2. Maximum country-level PPP experience (# projects) within 10 years prior to any project (excluding merchant projects), 1990-20164 Source: PPI Database, August, 2016 4 This map reflects the maximum level of prior PPP experience (over past ten years) within a country for any project within the data set described above (excluding merchant projects). Countries with one PPP (e.g. Paraguay) are classified as ‘zero prior PPP experience’ since the single project is the first in the country (within at least 10 years) during the specified time period. 6 Methodology The question this paper seeks to answer is straightforward: does a country's experience with PPP reduce the probability of project cancellation? In order to answer this question, this study utilizes mixed-effect probabilistic models in combination with linear and cubic splines to examine the role of experience on project cancellation rates. Effects of PPP experience on probability of contract cancellation Within a country, PPP projects are generally subject to the same macroeconomic and legal environments. They typically align to a national development plan and follow common sector and investment policies. At the same time, other important actors and organizations such as multilateral development banks (MDBs), operators and sponsors, private financiers, etc. have an equalizing effect on the PPP environment at the country level (Marcelo and House, 2016). Common exposure to national level factors could mean that PPPs do not behave independently. Rather, they may be significantly correlated at the country level. In this context, the study of their outcomes (e.g. PPP cancellation rates) must follow a strategy that accounts for such a clustered structure – i.e., a multilevel structure – in order to avoid potential biases in the analyses.5 The advantage of using multilevel or mixed-effect models over traditional econometric cross- sectional or pooled regression models is that they allow the correction of biases in parameters and standard errors that result from clustering of projects at the country level. As mentioned above, outcomes of projects in the same country are likely to be correlated, since PPPs are subject to the same socioeconomic, political, regulatory and legal environments. Neglecting this correlation could lead to incorrect statistical inferences. This bias can be corrected by utilizing a mixed-effect regression approach (Guo and Zhao, 2000). This study uses mixed-effect probit models to analyze cancellation rates. We describe the logic and construction of this model in three steps. First, in a mixed-effect or 'multilevel' model, the intercept and estimated coefficients may randomly vary between different clusters (e.g., clusters defined by countries), or even groups of clusters that define a hierarchy (e.g. region–country– project).6 For example, if PPP projects are clustered at the country level, a significant part of the variation in the probability of cancellation would be due to the fact that the PPPs belong to a particular country. This would mean that PPPs do not behave independently at the national level. This clustering effect is hypothesized, but not known in practice. To test for clustering, this study applied standard intra-class correlation tests (see Table 3, 'Empty model', Intra-class correlation coefficient). Second, mixed effect probit models and traditional probit models follow the same basic logic. A dependent variable (the probability of a positive outcome) is a latent (non-observable) variable that can be proxied by an observed binomial phenomenon. Despite the fact that the probability of contract cancellation is not observable at the project level, the current status of each project is observable. In this case, a dummy variable denoted as Status equals 1 if the PPP project is canceled and 0 otherwise. 5 As described by Guo and Zhao, "multilevel modeling corrects for biases in parameter estimates resulting from clustering. In contrast to popular belief, ignoring multilevel structure can result in biases in parameter estimates as well as biases in their standard errors. The more highly correlated the observations are within clusters, the more likely that ignoring clustering would result in biases in parameter estimates" (2000). 6 Mixed-effects or multilevel models are also referred to as hierarchical modes. 7 Third, while a number of factors may explain the behavior of the probability of contract cancellation, this study focuses mainly on the role of country PPP experience. The data presented in the following sections suggest that cancellation rates do indeed decrease as countries gain PPP experience (as PPP country experience is defined in the Data section). To better understand this behavior, the econometric models presented in the results section use several transformations of the variable country PPP experience to allow an accurate understanding of the shape of the relationship between cancellation and experience. Cancellation rate estimation based on splines The observed data suggest that the relationship between PPP cancellation rates and country PPP experience is not linear, but rather asymptotic (see Figure 3). On average, the observed cancellation rate for projects initiated in countries without any prior PPP experience is 22%. This cancellation rate drops to nearly 8% when countries have closed at least 50 PPP deals. Interestingly, most of the reduction in cancellation rate is reached with an approximate range of country experience of only five to ten PPP deals. In a case like this, where the relationship to be analyzed is so markedly curved, the use of linear splines and restricted cubic splines may be more appropriate than a linear equation (Gould, 1993). Figure 3. Rate of cancellation with respect to PPP country experience 30.0% All Sectors Energy Transport Water 27.5% 25.0% PPP Contract Cancellation Rate 22.5% 20.0% 17.5% 15.0% 12.5% 10.0% 7.5% 5.0% 2.5% 0.0% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 Cumulative PPP Country Experience in the Past Decade A linear spline (denoted as LSpline in the Results section) allows the estimation of the relationship between and as a piecewise linear function composed of linear segments. Each linear segment captures the effect on when values of vary within a certain range. That is, the linear spline transforms an explanatory variable into segments and estimates the slope of the linear function between and for each segment. In turn, the linear segments join at pre-defined 'knots', or inflection points, in the slope. In this case, five knots were arbitrarily placed at 12, 24, 48 and 96 PPP projects of country experience (see annexes 4 and 5). 7 7 Annex 4 presents several linear and cubic spline estimations. 8 Transformation of the variable PPP country experience through auxiliary variables –linear segments– was specified following Panis (1994), as described below: = ( , ) = ( , ), ) − = 2, … ,4 = , − where is PPP country experience, and the seven linear segments join at six inflection points, or knots , 12, 24, 48 and 96 PPP projects of country experience. The estimated coefficient associated with each linear segment reflects the effect of PPP country experience, within that range of experience, on the probability of contract cancellation. Moreover, a cubic spline (denoted as RCSpline in the Results section) may be a better choice than a linear spline when working with pronouncedly curved functions. As shown in Figure 3, the observed relationship between PPP country experience and cancellation rate displays a highly curved shape. When using a restricted cubic spline, it is possible to obtain a piecewise smooth cubic polynomial function that connects at pre-defined knots. In this case, the locations of the knots (six in total) were determined by the percentiles strategy proposed by Harrell (2001).8 The resulting auxiliary variables follow the specification bellow: = ( ) ( ) ( ) ( ) ( ) ( ) = ( ) = 1, … , − 2 where − 2 corresponds to the number of auxiliary variables to be created. Unlike the linear spline, coefficients associated with auxiliary variables under a cubic spline specification do not directly reflect the effect of PPP country experience within a specific range on the probability of cancellation. Instead, the marginal effect of PPP country experience on the cancellation rate entirely depends on the value of the variable PPP country experience chosen to evaluate the estimated probabilistic function. 8 Harrell (2001) recommends placing knots at equally spaced percentiles of the original variable's marginal distribution. 9 Descriptive Statistics Tables 1 and 2 summarize the information used in the econometric models presented in the Results section. As mentioned above, about 6% of PPPs that reached financial closure before 2011 were canceled. But remarkable differences were observed when disaggregating by sector, type of PPP, and region. First, the lowest PPP contract cancellation rate is observed in the energy sector (3.2%) – a rate that is less than half the cancellation rates in transport and water and a fifth of that observed in the ICT sector. Second, the cancellation rate for new infrastructure projects or 'greenfield' projects (4.3%) is about 60% of the brownfield concession project cancellation rate (7.1%) and 31% of the management and lease contract cancellation rate (13.7%). Finally, from a regional perspective, the highest cancellation rate was registered in Africa (9.6%), with a rate almost 70% higher than those observed in East Asia and the Pacific (EAP), East Europe and Central Asia (ECA) and Middle East and North Africa (MENA). Compared with South Asia (SAR), Africa has a cancellation rate seven times higher (Table 1). Table 1: Cancellation rates by sector, type of PPP and Region Variable Distribution (%) Cancellation Rate (%) Sector ICT 0.031 0.154 Energy 0.442 0.032 Transport 0.333 0.069 Water and sewerage 0.195 0.074 Type of PPP Brownfield 0.338 0.071 Greenfield 0.619 0.043 Management & Lease 0.043 0.137 Region AFR 0.055 0.096 EAP 0.360 0.056 ECA 0.057 0.057 LAC 0.325 0.072 MENA 0.026 0.056 SAR 0.177 0.013 Obs. (PPPs reaching financial closure before 2011) 3,400 Clusters (Countries) 110 In terms of the size of the PPP projects under study, the average investment committed to a PPP project at the time of financial closure was US$258 million, though half of these projects did not exceed US$78 million in size. Only a quarter of the PPPs that reached financial closure before 2011 surpassed the average investment size (Table 2). Table 2: Investments, GDP and Population Variable Obs. min Average max Total investment (US$m) 3,106 0.04 257.57 6,693.07 GDP (constant 2005 US$) 3,385 192.17 2,799.52 8,942.85 Population (millions) 3,399 0.10 521.39 1,337.71 10 Results Table 3 presents econometric estimations, including the various functional specifications described in the Methodology section. As mentioned before, the objective of the econometric analysis is to provide a better understanding of the effects of PPP country experience on the PPP contract cancellation rates, with particular emphasis on the shape of the relationship between cancellations and PPP country experience. Moreover, it is of interest to examine variations in the influence of experience in different sectors. First, PPP projects do not behave independently. Instead, their outcomes are clustered at the national level. Nearly 25% of the variation in the probability of contract cancellation is due to country-level clustering (see Table 3, Empty model, Intra-class correlation). This means that a significant part of the variation in the probability of cancellation is due to the fact that PPPs belong to a certain country. In this case, multilevel or mixed-effect models allow the correction of potential biases in parameters and standard errors resulting from clustering at the national level. Second, PPP country experience has a negative effect on the probability of PPP contract cancellation. In other words, as countries gain experience in PPPs, the likelihood of subsequent PPP deals ending in cancellation is significantly reduced. Each additional PPP project added to a country's PPP experience is expected to reduce the probability of cancellation by 0.029%, on average (see Table 3, Basic model).9 This simple linear functional specification, however, does not reveal how the rate of reduction varies at different levels of PPP country experience. The data suggest that the relationship between cancellation rates and PPP country experience is markedly curved (Figure 3). Third, cancellation rates rapidly decrease as PPP experience increases, especially over the first few projects initiated in a country (Table 3 and Figure 4). A basic quadratic specification of the relationship between cancellation and PPP experience suggests that the gains associated with PPP country experience are attained only over the first 22 PPPs (Table 3, Square model). 10 However, this specification has a limitation, it does assume that, after a certain point, experience could also be counterproductive, which contradicts intuition and the observed data (Figure 3). Conversely, linear and restricted cubic splines do not impose such an assumption. All the models reveal that the 'experience benefit' from each additional PPP is greatest for the first few deals. After five projects in a country, the reduction in the probability of cancellation from each additional PPP project is considerably diminished. A linear spline specification predicts that the probability of a PPP project to be canceled is 15% when the country to which it belongs has no PPP experience (Table 3, LSpline1 and LSpline2 models). 11 After five PPP projects of country experience, the probability of contract cancellation drops 35% (to approximately 10%). Over the course of the next five projects of the country's PPP experience, the probability of cancellation is expected to fall another 38% (to nearly 6%), with almost no change in the probability of cancellation for PPPs when country experience ranges between 50 and 150 projects. 9 -0.029% is the marginal effect ( ⁄ ) associated with the estimated coefficient -0.003 (Table 3). 10 The quadratic function reaches a minimum when ( ⁄ ) = 0 and the PPP country experience is equal to − ⁄ 2∗ . 11 In models LSpline and Full_LSplinel (Table 3), the probability of PPP contract cancellation is about 15% when the probability function is evaluated at = 0 ∀ = 1, … ,6 and all the other variables at their mean value (see Annex 3). 11 Table 3. Mixed-effect probit regressions on canceled PPPs. Dep. Variable: PPP Status (1=Cancelled PPP contract). Group variable: Country Empty Basic Square LSpline1 LSpline2 CSpline Exp : Country PPP Experience -0.003*** -0.045 ** -0.054*** Exp2 : Exp Square 0.001* LS1 : 6 <= Exp < 12 -0.056*** -0.058*** LS2 : 12 <= Exp < 24 0.026 0.032 LS3 : 24 <= Exp < 48 -0.018** -0.023** LS4 : 48 <= Exp < 96 0.006 0.006 LS5 : 96 <= Exp -0.004*** -0.004*** Cubic Spline 2 30.690** Cubic Spline 3 -44.128* Cubic Spline 4 13.679 Cubic Spline 5 -0.321 Cubic Spline 6 0.207 Total investment 0.419* 0.391*** 0.417*** Total Investment Square -0.025 -0.044 -0.048 Sector12 Energy -1.080*** -0.959*** -0.934*** Transport -0.777** -0.689*** -0.617*** Water and sewerage -0.486 -0.297 -0.254 Type of PPP Brownfield project -0.038 -0.121 -0.031 Greenfield project -0.358 -0.159 -0.144 Country Level GDP per capita® -0.025 -0.084 0.018 Population (millions) 0.001** 0.001 0.000* Region AFR 1.099*** 1.016* 0.384 EAP 1.206*** 1.235** 0.875*** ECA 1.196* 1.183* 0.237 LAC 1.514*** 1.624*** 0.751*** MENA 0.892 0.815 0.112 Constant -1.529*** -1.444*** -1.700*** -1.185*** -1.562** -1.050** Country level variance var(constant) 0.327*** 0.295*** 0.446** 0.261*** 0.283** Wald chi2() 39.45 78.40 55.41 110.77 144.63 Prob > chi2 0.000 0.024 0.000 0.000 0.000 LR test vs. probit : chi2(2) 84.26 63.34 40.08 59.89 21.60 Prob > chi2(2) 0.000 0.000 0.000 0.000 0.000 Multilevel Structure Intra-class Correlation 0.246** # Obs (PPP projects) 3276 3276 1208 3276 2980 2980 # Groups (countries) 109 109 98 109 98 98 Pseudo-R2 0.058 0.091 0.628 0.102 0.256 0.244 * p<.1; ** p<.05; *** p<.01 Robust Std. Err. adjusted for clustering on country ® Constant 2005 US$000 12 ICT is the base category in the regression. 12 A restricted cubic spline functional specification generates very similar results: the probability of cancellation is about 16% when countries have no PPP experience, but the cancellation rate rapidly reduces as countries gain PPP experience (Table 3, CSpline model). The probability of contract cancellation drops 35% and another 33% after five and ten PPP projects, respectively, whereas the reduction in the probability of cancellation is marginal after about 20 projects of PPP experience. After this point, the probability of cancellation is close to 4% (Figure 4). Finally, the probability of PPP contract cancellation is significantly lower in the energy and transport sectors. In a country with no PPP experience, the probability of a PPP contract to be cancelled in the water sector is predicted at 27%, while in the transport and energy sectors, project cancellation rates are expected at around 17% and 11%, respectively. These differences in sector cancellation rates for early projects in new markets suggest the relative complexity of water PPPs as compared to energy and transport projects. More interestingly, the results show that after 20 contracts of PPP country experience, the probability of cancellation in the water sector is reduced by 63%, whereas the reductions in cancellation rates are higher for transport (70%) and energy (74%), with more rapid reductions in the probability of cancellation after only a few deals of PPP country experience (i.e., at five and ten experienced PPPs) (Figure 4). Figure 4. Probability of cancellation with respect to PPP country experience 30.0% All Sectors Energy Transport Water 27.5% Probability of PPP Contract Cancellation 25.0% 22.5% 20.0% 17.5% 15.0% 12.5% 10.0% 7.5% 5.0% 2.5% 0.0% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 Cumulative PPP Country Experience in the Past Decade Sensitivity analysis In order to test the robustness of the results, the models above were additionally estimated under several configurations of the data set. First, PPPs with shorter and longer periods of maturation were considered in a new set of regressions. Second, to eliminate the risk of biased results due to inclusion of extremely high-experience projects, models excluding PPPs with accumulated country experience higher than 200 and 100 PPPs were also estimated. In the first sensitivity analysis, estimations included more recent PPPs that reached financial closure in or before 2013 as well as PPPs with longer maturation periods closing in 2007 or before. In the second sensitivity analysis, the PPP country experience restriction applied to a subset of 13 projects in China, India, and Brazil, where the accumulated experience for some projects was higher than 200 PPPs. The restriction also applied to projects in Argentina, Turkey and Mexico for country experience levels higher than 100 PPPs (see Annex 4). The overall results for this new set of regressions are similar to those obtained in the reported models. The probability of PPP contract cancellation rapidly decreases after a relatively low amount of PPP country experience. The quadratic models predict that the probability of cancellation reaches a minimum at a level between 21 and 23 PPPs of accumulated country experience. Moreover, according to the linear and cubic spline models, there would be almost no reductions in the probability of cancellation beyond the 50th deal, with an average probability of cancellation from this point between 3% and 5% (see Annex 4 and 5 and Figure 5). Figure 5. Sensitivity Analysis: Probability of cancellation with respect to PPP country experience 20.0% Exp≤200 Exp≤100 ≤2013 ≤2010 ≤2007 Probability of PPP Contract Cancellation 17.5% 15.0% 12.5% 10.0% 7.5% 5.0% 2.5% 0.0% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 Cummulative PPP Country Experience in the Past Decade 14 Conclusions Countries learn very quickly from only a little PPP experience. The analysis shows that after a country closes a relatively low amount of PPP contracts, the probability of contract cancellation rapidly declines –a finding that is more pronounced in the energy and transport sectors than in water. The econometric results also reveal how quickly early experience and country-specific PPP knowledge translates into concrete benefits to future PPP performance. The practical experience of bringing PPP contracts to financial closure is most impactful over the first PPP deals (e.g., from one to ten PPPs). The experience dividend, in terms of reduced probability of cancellation, plateaus after about 20 contracts. The sector results suggest that the water sector may require more careful, lengthy preparation, but that early PPPs are nevertheless notably beneficial to subsequent success in the sector. Based on research and experience, the observed sectoral differentiation in cancellation rates may be attributable to the very local and political nature of the water sector, where reforms are subject to the unique urban and geographical conditions as well as local customs, beliefs, and politics (Araral, et al., 2011). Further, water sector projects have tended to transfer demand risk to sponsors whereas transport and energy projects will isolate revenues or payment streams to a single paying entity, typically a government body, ministry or agency (Mandri-Perrott & Stiggers, 2013). Early PPP projects offer important lessons to reduce the likelihood of future contract failures. Implementing interim oversight and evaluation programs can help capture the insights from early PPPs in order to improve contracts, regulation, bureaucratic capacity, and other facets of PPP implementation for future projects. Moreover, while some technical knowledge can be transferred from PPP-experienced countries to other countries without experience (e.g., inclusion of key contractual clauses, regulatory tools, costing and pricing methodologies), some of the knowledge required to sustain healthy PPPs is undoubtedly hard-won and invested in the construction, adjustment, and implementation of the earliest PPP deals within the country. These observations have implications for the advisors that support the development of PPP programs to increase provision of infrastructure. For one, multilaterals and other PPP proponents would do well to focus efforts on supporting countries with limited PPP experience rather than supporting easier 'wins' in countries with more extensive PPP experience. Knowing that public and private resources are at a higher risk when embarking on PPPs in countries with very low levels of experience, MDBs should strategically target PPP support to countries with little or no PPP experience. The results suggest that from a regional perspective, this might include most of the African and Central Asian countries and some of the Central American and Caribbean nations. That said, MDB technical assistance should be provided to those governments that are also willing to create and preserve the conditions that enable a PPP market to develop, including institutional and legal reforms that underpin functional contracts, fair arbitration, and healthy financial markets. With respect to financial support and managing investment risk in new PPP countries, MDBs must also take into account sectoral considerations and may choose to focus initial lending efforts in the energy and transport sectors, where cancellation rates are lower. In doing so, consideration should be given to contractual structures where revenue streams are 'protected' from demand risk. This is done, for example, through the use of direct payments from government agencies in the case of energy IPPs, where payments are made by a single off-taker, or in the transport sector when demand risk is limited to remuneration through a public entity by using some form of availability payment. Moreover, financial support should be offered based on careful examination of the local context prior to proposing PPP as a viable option, and with an eye to develop the 15 earliest PPPs as learning-oriented projects. This means carefully weighing the option of proposing a PPP vis-à-vis other modes of infrastructure provision, including public provision, as well as developing PPP programs with careful consideration of local legal, political, financial, and governmental conditions. Furthermore, it means building in adjustment mechanisms in early PPPs and encouraging lesson-drawing to inform future deals. It is worth noting that a reasonable target cancellation rate for infrastructure PPPs is not necessarily zero percent. Indeed, contract cancellation is sometimes found to be necessary, and cancellation rates for countries with high levels of PPP experience across all sectors are approximately 4%. Nevertheless, project cancellations are associated with high costs and the threat of disruption of critical infrastructure services. Based on the econometric results, an estimated $1.5 billion per year could have been saved with interventions and support to reduce cancellations in less experienced countries (i.e., those with less than 23 prior PPPs) to a 5% cancellation level. The results of this study also have implications for future research. Future research could focus on evaluating the impact of PPP country and subnational government experience on the quality of infrastructure and service provision. At the moment, there exist only some general insights on the effects of PPP country experience that are based on quantitative analysis, and these are limited to the impact on contract sustenance, whereas the impacts of learning for particular facets of PPP implementation are case-based and often anecdotal. Future research could also examine the channels of learning to support future PPPs, including what impact centralized PPP units have on accumulating and disseminating lessons learned to improve the future success of PPPs and how widely-disseminated PPP 'best practice' (e.g., standardized contract clauses) improves the success of future PPPs. Also, it would be helpful to examine what lessons are transferable across sectors and geography – in other words, what PPP factors are inevitably local and unique, demanding customization, and what can be more readily transferred from international experience to directly inform new markets. Lastly, related analyses of the drivers of PPP cancellations could be enriched by analyzing the effects of different contract types, e.g. management contract, lease / affermage, Build Operate & Transfer (BOT) contract and its variants, concessions (including Design-Build-Operate-Finance contracts) and divestitures. 16 References Araral, E., Jarvis, D. S., Ramesh, M., & Xun, W. (2011). Regulating infrastructure: A review of the issues, problems, and challenges, pp. 17-19. Infrastructure regulation: What works, why and how do we know?: Lessons from Asia and beyond. Bovens, M., and P. ’t Hart. (1995). Frame multiplicity and policy fiascos: Limits to explanation. Knowledge and Policy, 8 (4): 61–83. Churchman, C.W. (1967). Wicked problems. Management Science, 14 (4). B141–B142. Delmon, J. (2009). Private sector investment in infrastructure: Project finance, PPP projects and risks. Kluwer Law International. Gould, W. (1993). Linear splines and piecewise linear functions. Stata Technical Bulletin, 15: 13– 17. Reprinted in Stata Technical Bulletin Reprints, vol. 3, pp. 98–104. College Station, TX: Stata Press. Guo, G., & Zhao, H. (2000). Multilevel modeling for binary data. Annual review of sociology, 441- 462. Harrell, F. (2001). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. New York: Springer. Howlett, M. (2009). Policy analytical capacity and evidence-based policy-making: Lessons from Canada. Canadian Public Administration, 52(2), 153-175. Linder, S., & Peters, B. (1987). A design perspective on policy implementation: The fallacies of misplaced prescription. Review of Policy Research, 6(3), 459-475. Mandri-Perrott, X.C., & Bisbey, J. (2016) How to develop sustainable irrigation projects with private sector participation: public-private partnerships toolkits. The World Bank. Mandri-Perrott, X.C., & Stiggers, D.J. (2013). Public Private Partnerships in the Water Sector. London: IWA Publishing. Marcelo D., & House, S. (2016). Effects of multilateral support on infrastructure PPP contract cancellation. World Bank Policy Research Working Paper Series (No. 7751). The World Bank. McConnell, A. (2010). Policy success, policy failure and grey areas in-between. Journal of Public Policy, 30(03), 345-362. Panis, C. (1994). The piecewise linear spline transformation. Stata Technical Bulletin, 18: 27–29. Reprinted in Stata Technical Bulletin Reprints, vol. 3, pp. 146–149. College Station, TX: Stata Press. Scott, J. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press. UK House of Commons Treasury Committee. (2011). Private Finance Initiative: Seventeenth Report of Session 2010–12. Printed 18 July 2011. http://www.publications.parliament.uk/pa/cm201012/cmselect/cmtreasy/1146/1146.pdf. Weber, E. & Khademian, A. (2008). Wicked problems, knowledge challenges, and collaborative capacity builders in network settings. Public Administration Review, 68, 334-349. 17 Annex 1. PPP Experience: PPPs in the Past Decade PPP Experience Countries Afghanistan, Belarus, Belize, Benin, Bhutan, Bosnia and Herzegovina, Botswana, Burkina Faso, Cape Verde, Central African Republic, Chad, Dem. Rep. Congo, Fiji, The Gambia, 0 Guinea, Guinea-Bissau, Guyana, Lesotho, Maldives, Moldova, Mongolia, Montenegro, Niger, Paraguay, Somalia, São Tomé and Principe, Tonga, Vanuatu Comoros, Kosovo, Kyrgyz Republic, Lithuania, Macedonia, Malawi, Mali, Papua New 1 Guinea, Timor-Leste, Uzbekistan, Zimbabwe Azerbaijan, Rep. Congo, Haiti, Iran, Islamic Rep., Liberia, Myanmar, Namibia, Sudan, 2 Syrian Arab Republic, Tajikistan Cuba, Djibouti, Ethiopia, Madagascar, Serbia, Sierra Leone, Togo, Tunisia, West Bank and 3 Gaza, Republic of Yemen, Zambia 4 Cameroon, El Salvador 5 Iraq, Jamaica, Lebanon, Nicaragua 6 Angola, Côte d'Ivoire, Gabon, Kazakhstan, Mauritius, Mozambique, Rwanda 7 Armenia, Georgia, Ghana, Morocco 8 Ukraine 9 Bolivia, Panama 10 Kenya, Senegal 11 Albania, Tanzania, Venezuela, RB 12 Dominican Republic 14 Cambodia, Guatemala, Honduras 15 Jordan 17 Egypt, Arab Rep., Uganda 18 Ecuador, Uruguay 20 Costa Rica 21 Lao PDR 22 Nepal 24 Algeria 29 Romania 32 Russian Federation 34 Nigeria 40 Bulgaria 41 Bangladesh 45 Indonesia, Pakistan 48 South Africa 49 Philippines 55 Sri Lanka 64 Malaysia, Thailand, Vietnam 67 Peru 73 Chile 85 Colombia 109 Mexico 121 Argentina 132 Turkey 387 Brazil 639 India 680 China 18 Annex 2. Maximum PPP country experience over past decade (0 chi2 0.000 0.000 0.000 0.000 0.000 0.000 LR test vs. probit : chi2(2) 41.97 35.95 31.41 12.60 Prob > chi2(2) 0.000 0.000 0.000 0.000 Multilevel Structure # Obs (PPP projects) 1522 3897 3897 929 2390 2390 # Groups (countries) 103 103 87 87 Pseudo-R2 0.672 0.218 0.196 0.616 0.414 0.239 * p<.1; ** p<.05; *** p<.01 Robust Std. Err. adjusted for clustering on country ® Constant 2005 US$000 13 ICT is the base category in the regression. 21 Annex 5. Sensitivity Analysis (part 2) Mixed-effect probit regressions on canceled PPPs. PPP Country Experience ≤ 200 PPP Country Experience ≤ 100 Square LSpline2 CSpline Square LSpline2 CSpline Exp : Country PPP Experience -0.055 ** -0.077*** -0.055** -0.159*** Exp2 : Exp Square 0.001 * 0.001* LS1 : 1 <= Exp < 6 -0.11*** -0.111*** LS2 : 6 <= Exp < 12 -0.003 -0.002 LS3 : 12 <= Exp < 24 0.013 0.012 LS4 : 24 <= Exp < 48 -0.013 -0.012 LS5 : 48 <= Exp < 96 -0.004 -0.005 LS6 : 96 <= Exp 0.006* -0.008 Cubic Spline 2 2.778** 9.168 Cubic Spline 3 -4.089** -13.987 Cubic Spline 4 1.433* 3.587 Cubic Spline 5 2.369 Cubic Spline 6 -1.476 Total investment 0.500** 0.527*** 0.498*** 0.500** 0.525*** 0.474*** Total Investment Square -0.036 -0.068** -0.062** -0.036 -0.062* -0.050 Sector14 Energy -1.237*** -1.195*** -1.122*** -1.237*** -1.234*** -1.128*** Transport -0.862** -0.810*** -0.700*** -0.862** -0.795*** -0.662*** Water and sewerage -0.541 -0.295 -0.258 -0.541 -0.372 -0.326 Type of PPP Brownfield project -0.040 -0.105 -0.045 -0.040 -0.109 -0.060 Greenfield project -0.423 -0.257 -0.247 -0.423 -0.319 -0.283 Country Level GDP per capita® -0.069 -0.133** -0.003 -0.069 -0.128** 0.000 Population (millions) 0.001** 0.000 0.000* 0.001** 0.001 0.000* Region AFR 1.000** 0.949 0.425 1.000** 0.875 0.253 EAP 1.182*** 1.275** 1.005*** 1.182*** 1.169** 0.791*** ECA 1.126* 1.126* 0.293 1.126* 1.055 0.088 LAC 1.553*** 1.675*** 0.880*** 1.553*** 1.572** 0.641** MENA 0.864 0.791 0.190 0.864 0.721 -0.018 Constant -1.505** -1.276* -0.944** -1.505** -1.163* -0.598 Wald chi2() 81.84 117.89 149.41 81.84 108.28 137.04 Prob > chi2 0.000 0.000 0.000 0.000 0.000 0.000 LR test vs. probit : chi2(2) 43.06 34.01 43.06 30.22 Prob > chi2(2) 0.000 0.000 0.000 0.000 Multilevel Structure # Obs (PPP projects) 1576 2697 2697 1576 2358 2358 # Groups (countries) 103 103 103 103 Pseudo-R2 0.672 0.218 0.196 0.616 0.249 0.239 * p<.1; ** p<.05; *** p<.01 Robust Std. Err. adjusted for clustering on country ® Constant 2005 US$000 14 ICT is the base category in the regressions. 22