WPS6906 Policy Research Working Paper 6906 Agreeing on Robust Decisions New Processes for Decision Making Under Deep Uncertainty Nidhi Kalra Stéphane Hallegatte Robert Lempert Casey Brown Adrian Fozzard Stuart Gill Ankur Shah The World Bank Climate Change Group Office of the Chief Economist June 2014 Policy Research Working Paper 6906 Abstract Investment decision making is already difficult for any to contexts of large uncertainty or disagreement on diverse group of actors with different priorities and views. values and objectives. This inverted process promotes But the presence of deep uncertainties linked to climate consensus around better decisions and can help in change and other future conditions further challenges managing uncertainty. Analyses performed in this way decision making by questioning the robustness of all let decision makers make the decision and inform them purportedly optimal solutions. While decision makers on (1) the conditions under which an option or project can continue to use the decision metrics they have is vulnerable; (2) the tradeoffs between robustness used in the past (such as net present value), alternative and cost, or between various objectives; and (3) the methodologies can improve decision processes, especially flexibility of various options to respond to changes in those that lead with analysis and end in agreement on the future. In doing so, they put decision makers back decisions. Such “Agree-on-Decision” methods start by in the driver’s seat. A growing set of case studies shows stress-testing options under a wide range of plausible that these methods can be applied in real-world contexts conditions, without requiring us to agree ex ante on and do not need to be more costly or complicated than which conditions are more or less likely, and against a traditional approaches. Finally, while this paper focuses set of objectives or success metrics, without requiring on climate change, a better treatment of uncertainties and us to agree ex ante on how to aggregate or weight disagreement would in general improve decision making them. As a result, these methods are easier to apply and development outcomes. This paper is a product of the Office of the Chief Economist, Climate Change Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at shallegatte@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Agreeing on Robust Decisions: New Processes for Decision Making Under Deep Uncertainty Nidhi Kalra, Stéphane Hallegatte, Robert Lempert, Casey Brown, Adrian Fozzard, Stuart Gill, Ankur Shah Keyword: decision-making under uncertainty, investment, climate change, adaptation JEL: D81, H54, O22, O18, Q54 Nidhi Kalra1,2, Stéphane Hallegatte2, Robert Lempert1, Casey Brown3, Adrian Fozzard4, Stuart Gill5, Ankur Shah6 1 RAND Corporation 2 The World Bank, Climate Change Group, Office of the Chief Economist 3 University of Massachusetts 4 The World Bank, Poverty reduction and economic management network 5 The Global Facility for Disaster Risk Reduction, The World Bank 6 Centre International de Recherche sur l’Environnement et le Développement The authors would like to thank Roberto Aiello, Arturo Ardila, Ken Chomitz, Leo Dobes, Patrice Dumas, Marianne Fay, Erick Fernandez, Nagaraja Harshadeep, Abas Jha, Todd Johnson, Marcelino Madrigal, Shomik Mehndiratta, John Nash, Niels Holm-Nielsen, Apurva Sanghi, Pascale Scandizzo, Tomas Serebrinsky, and Xiaoping Wang for their time, insights, and contributions. Parts of this article are based on Hallegatte et al. (2012). TABLE OF CONTENTS A. Introduction ....................................................................................................................................... 3 B. Decision and Metrics and decision processes ................................................................................... 7 C. Agree-on-Assumptions Processes ..................................................................................................... 7 Cost-Benefit Analysis and Flood Risk Management ......................................................................... 8 Real Option Analysis ....................................................................................................................... 10 Aggregation and Consensus building in an agree-on-assumption framework ................................. 13 D. Agree-on-Decisions processes ........................................................................................................ 15 Example: Forest Management using an Agree-on-decision Process ............................................... 17 Features of Robust Decisions........................................................................................................... 20 Agree-on-Decisions Applications .................................................................................................... 21 Robust Decision Making for Water Planning in Southern California ............................................. 21 Climate Informed Decision Analysis For Managing the Great Lakes Basin ................................... 23 Info-Gap for Seismic Resilience in Water Systems ......................................................................... 25 E. Conclusions ..................................................................................................................................... 26 Appendix 1. Understanding Climate Uncertainty .................................................................................... 27 References ................................................................................................................................................ 30 TABLE OF FIGURES Figure 1. Three policies in the Paris agglomeration ................................................................................... 17 Figure 2. Steps in an RDM Analysis........................................................................................................... 22 Figure 3. Performance of IEUA’s Urban Water Management Plan in 200 Futures ................................... 23 Figure 4. Decision scaling steps compared to a traditional analytical process. .......................................... 25 Figure A1. Change in annual rainfall in 2080-2100 (with respect to the 1980-2000 period) in Africa according to two climate models. ............................................................................................................... 27 Figure A2. Source of uncertainty over time globally (left) and in the British Isles (right). ........................ 28 TABLE OF TABLES Table 1. Assessment of the performance of the three forest and flood management plans across four scenarios...................................................................................................................................................... 19 TABLE OF BOXES Box 1. The difficulty of predicting a deeply uncertain future....................................................................... 5 Box 2. Uncertainty in climate change projections ........................................................................................ 6 Box 3. Real options for planning in the Thames estuary ............................................................................ 11 Box 4. Setting the Social Cost of Carbon ................................................................................................... 13 2 A. INTRODUCTION 1. Many investment and policy decisions have long-term consequences. Infrastructure like power plants, roads, and dams often last for decades and need to be useful throughout their lifetimes. They will shape and be shaped by the future. These investments can shape development well beyond their lifetimes, sometimes for centuries, because the long-term socioeconomic system reorganizes itself around those changes. The effect of transport infrastructure on urban forms and economic activities can for example be observed over very long timeframes, sometimes even after the infrastructure has become obsolete (Gusdorf, Hallegatte, and Lahellec 2008; Bleakley and Lin 2010). Policies such as urbanization plans, risk management strategies, and building codes and norms can influence development for equally long. Therefore, to make sound plans we must consider the performance of our investments and decisions in the near and long term. 2. Yet deep uncertainty about the future exacerbates the challenge of sound decision making. Developing countries in particular have experienced unprecedented changes in their political economy, land use, demographics, and natural environment. Yet, as Box 1 illustrates, past evidence and current research suggests that our ability to predict the future is limited at best (Silver 2012; Kahneman 2011; Taleb 2007). Compounding the problem, parties to a decision often have competing priorities, beliefs, and preferences. These conditions lead to deep uncertainty. Deep uncertainty occurs when parties to a decision do not know or cannot agree on (1) models that relate the key forces that shape the future, (2) probability distributions of key variables and parameters in these models, and/or (3) the value of alternative outcomes (R. J. Lempert, Popper, and Bankes 2003). In his seminal 1921 paper, Knight offered a similar definition, distinguishing between two kinds of ignorance about our uncertain future – that which we can reliably quantify (called Knightian risk) and that which we cannot (Knightian uncertainty, which corresponds to deep uncertainty). For example, the likelihood of experiencing a car crash is easily calculable from ample historical data and is an example of Knightian risk. In contrast, likelihood estimates of long-term land use patterns or global economic growth would be neither reliable nor verifiable. They are Knightian uncertainties or deep uncertainties. 3. The impact of climate change looms large as a deep uncertainty with global consequences (World Bank 2009; IPCC 2014). There is scientific consensus that the planet is warming due to greenhouse gas emissions. This may bring radical changes in climate, with tremendous implications for the long-term success of near-term decisions in nearly all sectors. Yet, as Box 2 shows, there is deep uncertainty and disagreement among both scientists and policy makers about what the future climate will be in a particular region and the specific effects that climate change will have on different sectors and groups. Continued efforts by climate scientists and others to increase knowledge about the climate and future climate scenarios are valuable. However, uncertainties about climate change and its impacts may increase as scientific inquiry diversifies and deepens (Hallegatte 2009). Therefore, decision makers should accept the irreducible uncertainty about the future climate and formulate adaptation and mitigation policies to manage it. 4. Failing to manage climate and other uncertainties will have serious consequences for adaptation. At the very least it can result in inefficient and ineffective investments. At its worst, it can be maladaptive, hindering effective interventions and leading to social and economic outcomes that are worse than if the interventions had never taken place. Inefficiency may arise from under- adaptation to climate change, i.e., if an intervention fails to consider future climate conditions or does not consider the full range of possible conditions, exposing the population and economy to adverse impacts. Inefficiency may also arise from over-adaptation, i.e. if an intervention turns out to be unnecessary. Construction of a dike provides a simple example of each, since the dike could prove to be either too low (under-adaptation) or too high (over-adaptation). Interventions can also lead to 3 maladaptation, where the intervention subsequently leads to responses that increase vulnerability to climate threats. For instance, building dikes may encourage settlement and investment in the protected area, potentially exposing people to high flood risk if sea level rise is higher than expected and increases in the dike height are not or cannot be undertaken. 5. Failure to manage deep uncertainties and competing beliefs can also hinder much-needed mitigation efforts. Deep uncertainties exist about how much we should mitigate CO2 emissions and at what cost, the effectiveness of different policies, and the availability of future technologies. Indeed, many who are reluctant to address climate change have cited uncertainty as a reason to postpone action. There is an urgent need for methods to manage deep uncertainty and reach sound adaptation and mitigation decisions. 6. In this paper, we seek to help decision makers better manage uncertainty and disagreement, particularly around climate change, by guiding them to the right decision making processes. Traditional decision processes ask us to first reduce uncertainty by agreeing on assumptions about current and future conditions, and then analyze our decision options. When faced with disagreement and uncertainty, these traditional “Agree-on-Assumptions” processes lack transparency, are vulnerable to bias and gridlock, and lead us to brittle decisions – those that perform poorly when the future diverges from our projections. We can instead invert these steps, deferring on agreement until we have analyzed the options under many different assumptions. This inverted “Agree-on-Decisions” process promotes consensus around robust decisions and can help manage uncertainty and disagreement around climate change and other conditions. While some decision-making methodologies assume that the analysts or experts will provide the “best” solution to the decision- maker, agree-on-decisions methodologies usually build on participatory processes and close interactions between experts and decision-makers. As such, they aim to “assist the decision maker to evaluate options, to develop strategies, and to evoke and evolve preferences in light of the analysis of uncertainties” (Ben-Haim 2006). Such an approach is deemed more appropriate in the presence of deep uncertainty, and ensure that decisions are legitimate (Renn 2008; World Bank 2013). 7. By making decisions with “inverted” processes that analyze options first and seek agreement second, we can • Generate buy-in by including diverse beliefs and making analyses more transparent; • Identify strategies that are robust, performing well no matter what the future brings; • Focus decision makers’ attention on the tradeoffs between decision options; • Seek agreement where it matters most: on actions, rather than on assumptions; • Inform the decision while making sure that decision-makers – not experts and analysts – make the decision. 8. The rest of this paper is divided into four sections, B through E. In Section B, we briefly distinguish between decision metrics and decision processes, and we posit that they should be considered separately given that any decision process can be used with any decision metric. Section C then focuses “Agree-on-Assumptions” processes while Section D presents “Agree-on-Decisions” processes. In the latter, we review features of robust decisions around which decision makers can reach consensus and offer three real-world applications. We conclude in Section E. The paper also offers an introduction to climate uncertainty in Appendix 1. 4 Box 1. The difficulty of predicting a deeply uncertain future Analysts and decision makers struggle to make accurate predictions of the future. The figure to the left shows projections made in the 1970’s for future US primary energy use to 2000. The projections were based on a century of data (black data points) that suggested a direct linear relationship (solid orange line) between gross national product (GNP) and energy consumption. The projections extrapolated this relationship and included some uncertainty (dashed orange lines), but nearly all of them were wrong. They significantly overestimated actual US energy use (dashed red line) because they could not anticipate the 1973 oil shock which triggered innovation and behavioral and policy changes that led to large increases in energy efficiency (Craig, Gadgil, and Koomey 2002). Source: Adapted from (Craig, Gadgil, and Koomey 2002) The figure to the right offers an example of the difficulty of predicting even near-term outcomes, and the tendency to be overconfident in those predictions. Each year, the U.S. Federal Reserve conducts the Survey of Professional Forecasters, asking economists to project the next year’s change in GDP. These projections are given as 90% confidence intervals, shown as vertical black lines. In the 18 years from 1993 to 2010, the actual GDP was within this confidence interval in twelve years (green circles) and outside this confidence interval (red x’s) in six years – three times as often as it should have been if the projections offered a true 90% confidence interval (Silver 2012). Source: Adapted from (Silver 2012) 5 Box 2. Uncertainty in climate change projections A cascade of uncertainties plagues climate change, and these uncertainties preclude prediction of the precise nature, timing, frequency, intensity and location of climate change impacts. The chain of increasing uncertainty begins with assumptions about the socio-economic characteristics of the global population, which determine the specification of a range of possible emissions scenarios. Estimates of climatic effects depend not only on the scenarios chosen but also on the configuration of the climate model used and existing knowledge of biophysical responses. Additionally, the farther into the future our projections, the greater the uncertainty. Uncertainty is also compounded by geographical resolution: uncertainty increases as the resolution of effects (i.e., downscaling) increases, from regional to country to local impacts (Gay and Estrada 2010). Even climate experts are unlikely to agree on a prediction of specific impacts of climate change (Arnell, Tompkins, and Adger 2005). Many go even further in rejecting the specification of probabilities for climate change impacts because of the lack of repeated experiments, lack of independent observations, and the fact that all probabilities are conditional on a multitude of socio-economic and other developments. The figure below shows the 25th, 50th (median), and 75th percentile of projections of precipitation in Africa in 2080-2100, compared with 1986-2005 average, using the CMIP5 climate models of the last IPCC report (IPCC 2013) . Some projections show substantially wetter climates while others show substantially drier climates for the same region. Uncertainty in future sea level rise, temperature, precipitation and other climate factors has tremendous implications for policymakers’ near term choices – where to locate key infrastructure such as airports, how to protect coastal areas from flooding, how to ensure water security, and so forth. Maps of precipitation changes in 2016–2035, 2046–2065 and 2081–2100 with respect to 1986–2005 in the RCP4.5 scenario. For each point, the 25th, 50th and 75th percentiles of the distribution of the CMIP5 ensemble are shown; this includes both natural variability and inter-model spread. Hatching denotes areas where the 20-year mean differences of the percentiles are less than the standard deviation of model-estimated present-day natural variability of 20-year mean differences. Source: (IPCC 2013) 6 B. DECISION AND METRICS AND DECISION PROCESSES 9. We begin by making a distinction between decision metrics and decision processes. Often, when we speak of approaches to decision making we are simultaneously referring to two distinct components of approaches: a. The performance metric we use to differentiate between different decision options, and b. The process we use to evaluate each option according to that performance metric and to come to a decision. Consider, for example, a choice between investing in mobile clinics or centralized clinics to provide vaccines to children as cheaply as possible. We might use a metric of cost per vaccination delivered to measure how well each option meets our objective. We can use different processes to assess each option according to this metric. We might first make assumptions about the geographic distribution of people in the region, their costs of traveling to different locations, the safety of different routes, and other conditions relevant to our choice. Based on these assumptions, we could calculate how many vaccinations each approach could deliver and the cost of each option, from which we could calculate our cost per vaccine. We would choose the least-cost approach, and use a sensitivity analysis to identify the main assumptions that drive our conclusion. We could instead use a different process while still using the metric of costs-per-vaccine. For instance, we might start by evaluating each option against a wide range of potential conditions and then choose the one that has the lower cost per vaccine across the widest range of conditions. 10. Treating metrics and processes separately allows us to choose the right tools for a decision. Metrics and processes have distinct strengths and weaknesses. Cost effectiveness metrics like the cost per vaccine can be valuable when it is difficult or impossible to monetize the benefit (such as lives saved) or when our options have the same main result (such as delivering vaccines). This is a characteristic of the metric and not of the process. On the other hand, one might criticize the above steps as too sensitive to the assumptions about geographic distribution and travel costs, particularly when there are large data gaps. This is a characteristic of the process, and not of the metric. In this paper, we focus the discussion on the processes used to evaluate and prioritize investments and policies and we investigate two types of processes: Agree-on-Assumptions processes and Agree-on- Decision processes. C. AGREE-ON-ASSUMPTIONS PROCESSES 11. One approach, which is consistent with most traditional decision processes, is to first reduce uncertainty by agreeing on assumptions about the current and future conditions and second analyze our decision options. The first step in many analyses, regardless of the metric, is to define the assumptions – about the present and the future – under which our investments must perform. This can be a single estimate of the future, as in our vaccine example, or a probability distribution. We then analyze our options, choosing the one that performs best under our assumptions. As one example, urban planners making flood risk investments would first characterize the future urban form, sea level rise, and other factors. They would second evaluate the risk reduction benefits (the metric) of their investment options (e.g. constructing dikes) under these assumptions. We might perform a sensitivity analysis to assess how much influence each assumption has on the outcome. Such approaches have also been termed “predict-then-act” (R. J. Lempert, Popper, et al. 2013) or “science first” (Dessai and Hulme 2007). 12. When faced with disagreement and deep uncertainty, these traditional “Agree-on- Assumptions” processes are vulnerable to bias and gridlock. First, many important assumptions are buried in models, rather than in front of decision makers. This makes it difficult for decision makers to understand and assess potentially critical assumptions on which their investment decisions 7 hinge. Second, many factors are difficult, if not impossible, to predict. Stakeholders also know that the choice of assumptions drives the choice of investment option. They may press for assumptions that will lead to the options they already favor, making consensus difficult (R. J. Lempert, Popper, and Bankes 2003). We risk losing stakeholders’ buy-in early if the foundations of the decision process lack transparency, appear arbitrary, or do not include their beliefs. 13. Agree-on-Assumptions approaches are vulnerable to reaching brittle decisions – ones that are optimal for a particular set of assumptions, but which perform poorly or even disastrously under other assumptions. Sensitivity analyses are often not sufficient for exploring the full range of plausible assumptions and future conditions (Bonzanigo and Kalra 2014), and Agree-on-Assumptions create little opportunity for exploring the performance of our decision options under unexpected conditions. They yield no information about how our “optimal” solution performs if the future surprises us, and they do not guide us to solutions that might work well if the predicted future does not come to pass. Yet, there is a great need for understanding the effect of surprises and unexpected conditions: repeated studies have shown, human beings have a widespread tendency towards over- confidence, believing strongly in our ability to predict the future when we cannot (Kahneman 2011). The following case example of cost-benefit analysis illustrates these shortcomings. C OST -B ENEFIT A NALYSIS AND F LOOD R ISK M ANAGEMENT 14. Cost-benefit analysis is one of the most widely used methodologies to inform investment decision-making. In this methodology, all the costs and benefits of an investment are aggregated into a monetary value in present terms. That is, the methodology (1) calculates the value of each monetary consequence (e.g., a flux of revenues from the project) and non-monetary consequence (e.g., lives that are saved by the investment, the impact of ecosystems and biodiversity, loss of cultural heritage) over time and (2) translates the future costs and benefits into a “present value” using a discount rate. Costs and benefits can be transformed into a single metric such as the benefit-cost ratio (the ratio of benefits to costs which, if larger than one, indicates the investment is desirable) or the net present value of the project (difference between the benefits and the costs which, if larger than 0, indicates the investment is desirable). Investments can be ranked according to these metrics to guide the selection of those that are expected to generate the greatest benefit overall. Cost benefit analysis has a long pedigree and has been honed and refined over many years. It is widely used in the economic analysis for World Bank projects (Tan et al. 2001). For more details, see Boardman et al. (2011). 15. Cost-benefit analysis, as traditionally practiced, is an example of an Agree-on- Assumptions process since it can only be applied if stakeholders agree on how to quantify various impacts (i.e., agree on how to attribute a monetary value to non-monetary consequences such as lives saved) and how to aggregate impacts at different points in time (i.e., agree on the discount rate). If all stakeholders agree on these assumptions, then the methodology provides an unambiguous answer regarding the desirability of an investment and a clear ranking of investment alternatives. 16. Consider the decision of whether to build infrastructure to protect New Orleans against category 5 hurricanes based on a cost-benefit metric. We might ask, “Is the net present value of such an investment positive?” The answer depends on many assumptions about the cost of construction, the probability of a category 5 hurricane losses that would occur in the event of a hurricane, and discount rates. In a traditional approach, we might start by estimating these values. Engineers estimate that it will cost $20 billion to build and operate the protection system. The benefits can be can be estimated as a discounted sum of the benefits (i.e. the avoided losses), over the lifetime of the protection (Hallegatte 2006): 8 = � � � + = where • T is the lifetime of the protection; • pn is the annual probability that a Category 5 hurricane hits New Orleans, which is estimated at about 1/500; • δ is the a discount rate, for which US regulations require assuming two values -- 3 and 7 percent – in two different calculations. • dn is an estimate of the annual direct cost of the New Orleans flooding, estimated at approximately $20 billion in damages and $5 billion in human losses; Under these assumptions, the expected present benefit of a category 5 flood protection system in New Orleans can be calculated at $1.3 billion with a 7 percent discount rate and $6 billion with a 3 percent discount rate. This results in a net present value of -$18.3 billion or -$14 billion, respectively. This rough estimate would rule out the development of a system that protects against category 5 storms. 17. There is no scientific agreement or political consensus on the assumptions upon which this conclusion rests. In general, nearly all parameters in a cost benefit analysis of long-term investments are deeply uncertain. In the above example, this includes: • Discount rate. As illustrated by our comparison of 3 and 7 percent discount rates, the influence of this political choice is large. Since this choice is very controversial and depends on ethical judgment on which it is difficult to reach consensus, the assessment of the system’s benefit will remain very uncertain and controversial. • Probability of occurrence. Suppose climate change and subsidence result in a five-fold increase in the probability of the floods currently caused by category 5 hurricanes over the 21st century, an increase that is within the bounds of current estimates. Then expected benefits from protection against category 5 hurricanes would increase from $1.3 to $2.4 billion or from $6 to $23 billion, using a discount rate of 7 percent and 3 percent, respectively. In the former case, the system would have a negative NPV and remain inadvisable, while in the latter it would have a positive NPV and become justifiable. • Flood damages. Flood damages evaluated by insurance companies are poor proxies of welfare costs, especially for large-scale events (Hallegatte 2014). A conservative estimate of the actual overall cost of the New Orleans floods is at least $60 billion, three times the insurers’ approximation based on direct losses only. Using the new values of event probability and potential damages, the expected benefit of an upgraded protection system would be $4.8 billion with a 7 percent discount rate and $46 billion with a 3 percent discount rate. • Countervailing risks and side effects. The implementation of a large-scale protection system can attract more people in at-risk location and increase exposure to floods if the defense fails. But it can also attract more activities, improve infrastructure, and create jobs and income, thus improving welfare more than an analysis of direct costs only would suggest. The above equation assumes countervailing risks and side effects do not exist, but in reality they can have significant though uncertain implications. These effects are very difficult to estimate and can easily double (or halve) expected benefits, making benefits in the above example range from $0.6 billion (roughly half of 1.3 billion) to $92 billion (twice $46 billion). • Risk aversion. The above equation also assumes that society is risk-neutral. A risk-neutral agent weighs positive and negative risk equally and is not affected by the degree of risk. For example, a risk neutral agent would not see any difference between losing $1 with certainty and having a 10 9 percent chance of losing $10, because the expected loss is the same in both cases. Yet people are often not risk neutral, e.g. preferring to pay disproportionately more to avoid large negative outcomes (Kahneman and Tversky 1979). Including an aversion to risk increases the benefit from protection. 18. In such a situation of deep uncertainty, it is sometimes possible to assign subjective probabilities, i.e. beliefs on the likelihood of different possible future conditions. We could then evaluate the expected benefits as the probability-weighted average of the benefits in the different possible future conditions. For example, we may believe that by 2100 there is a P=1/3 chance that the category-5 hurricane remains a 1 in 500 year event, and a (1-P)=2/3 chance that it increases to 1-in- 100-year event. Then the expected benefit can be written: = � � � ( + ) + ( − ) � ′ � � ( + ) + + = = Where • pn stays constant at 1/500, reflecting a state of the world in which climate change has no influence on hurricanes; • p’n increases from 1/500 to 1/100 by 2100, reflecting a state of the world in which climate change causes a five-fold increase in the likelihood of a category 5 hurricane landfall by the end of the century; • d0 is the present-day damages from a category 5 hurricane landfall; • g is the annual economic growth, estimated at 3 percent. 19. The problem is that – for climate change and many other deep uncertainties – we do not have a strong methodology to assign these subjective probabilities. The probability of a category 5 hurricane under future climate change, for example, cannot be fully based on historical data, because climate change is a new process for which we have no past equivalent. Models of climate change share common flaws and their dispersion cannot be used to assess the real uncertainty. Moreover, when using probability distribution function for different outcomes, the Agree-on-Assumptions approach is extremely sensitive to tails of distribution function. As suggested by Weitzman (2009), for instance, the "optimal" policy regarding climate change mitigation is highly dependent on the low- probability high-impact possible futures, on which knowledge is very limited and uncertainty is particularly large. At the extreme, Weitzman shows that if the probability distribution function has a "heavy tail" (i.e. decreases less than exponentially for increasing damage values), then a cost-benefit analysis suggests that all GHG emissions should be stopped immediately. R EAL O PTION A NALYSIS 20. A decision is often not strictly between “investing” and “not investing” but between “investing now” and “investing later.” To help make this type of decision, some have proposed to mobilize the “real option,” which was initially developed for financial markets (Arrow and Fisher 1974; Henry 1974; Ha-Duong 1998; Pindyck 2001; Gollier C. and Treich N. 2003). 21. Real-options (RO) analysis generalizes cost-benefit analysis by including the possibility of delaying decisions, revising them over time, or making a series of sequential decisions over time as more information becomes available. A real option is the ability, but not the obligation, to undertake a project of uncertain future benefits at a known cost. In finance, an option is the right to purchase an asset at a future point in time for a specific price acquired through the payment of a fee. The option may be realized against the payment of the balance of the agreed price, allowing the 10 option holder to make a profit if market prices for the asset rise above the price of the option. The option holder is not required to exercise this option and so limits losses if the market price falls below the price of the option to the amount of the fee paid to acquire the option. 22. Applied to decision-making, real option values the options created and destroyed by a project, alongside its expected net present value. The analysis itself does not differ from a classical cost- benefit analysis, except that the NPV includes additional consideration, namely the options created and destroyed by the project. The project's Extended NPV is calculated as follows: ENPV= Expected Net Present Value + ( Value of Options created – Value of Options Destroyed ) Thus an investment that has positive net benefits (exploiting existing capabilities) but fails to create new options may be less desirable than an investment with fewer direct benefits but which results in increased options (the ability to explore different opportunities later). In other words, there is a value in implementing a project that does not provide any benefit per se, but makes it possible to implement another project at a later point in time. 23. Applications of real-option approaches are difficult because of the technical complexity of the analysis; however, there are a few applications to real cases. Scandizzo (2011) applied real options to the development of sea defenses in Campeche, Mexico. A comparison of the Extended NPV for building a sea wall and for restoring a mangrove forest revealed that construction of a high sea wall would be uneconomical for several decades. In contrast, the cost of planting and maintaining a bioshield represents an option premium that creates the opportunity to delay the building of an expensive seawall until it is required. Box 3 describes how real options are embedded into a strategic planning exercise for the Thames Estuary. Real options analysis also has value for understanding climate change mitigation options (Ha-Duong 1998). Investing in research and development on renewable energy may have a negative NPV if considered in isolation from other policies. But such investments may create new technologies that could provide large benefits in 20 years when very ambitious climate policies are implemented. So, an investment in renewable energy research and development may be desirable because it will create the option of rapid decarbonization in the future, not because of its own return. 24. Real options approaches allow for a better inclusion of the time dimension in planning, including the option to delay and the introduction of more flexible solutions. Despite these strengths, traditional real-options analysis is an extension of the traditional Benefit-Cost analysis and remains an Agree-on-Assumptions process. Similarly to the cost-benefit analysis, it requires agreement on the value of future options, which may be deeply uncertain, and an agreement on the probability of various outcomes. When the assessment of outcome probabilities is impossible – at least in a consensual way – or when there is strong disagreement on values, real-option approaches are thus as difficult to implement as traditional cost-benefit analysis. In particular, they are vulnerable to the same shortcomings, including bias, gridlock, and brittle decisions. Box 3. Real options for planning in the Thames estuary The UK Treasury’s Green Book (guidance for policy, program, and project appraisal) cites the planning for the Thames Estuary 2100 as an application of Real Options approach to climate change. The Thames estuary floodplain contains 1.25 million people, about £200bn of property, and key transport and infrastructure assets, 16 hospitals and eight power stations. The planning process followed a five step approach: Stage 1: Assess Climate Risks. A series of sea level rise and storm surge scenarios were identified, including a central ‘most likely’ scenario. 11 Stage 2: Design Adaptation Options. A comprehensive range of options were identified, including raising river walls, adapting or building flood barriers or flood storage areas, applying resistance and resilience measures to buildings. These were assembled into portfolios of actions - High Level Options (HLOs) - which worked together to deal with differing degrees of sea level rise. Stage 3: Appraise Options to Address Most Likely View of Risk. All of the HLOs were subjected to cost-benefit analysis under the central sea level rise scenario to select the best generic option to promote under current knowledge of the most likely climate change outcome. Stage 4: Appraise Options Under Other Scenarios. Cost-benefit analysis was repeated for the different climate change scenarios. This demonstrates the potential weaknesses in options as interventions to deal with an uncertain future and highlights critical points in key variables (such as sea level rise) at which a different option may be preferred. Stage 5: Monitoring and Strategic Review. A system of monitoring of key climate change indicators (such as sea level rise) is put in place. Every 5-10 years the strategy will be revisited. If climate change happens more quickly (or slowly) than predicted, decision points may be brought forward (or delayed) as appropriate. At each review, the entire strategy may be reappraised in the light of new information. The figure below shows high-level options and pathways developed by TE2100 (on the y-axis) shown relative to threshold levels increase in extreme water level (on the x-axis). The teal line illustrates a possible ‘route’ where a decision maker would initially follow HLO 2 then switch to HLO4 if sea level increases faster than predicted. Source: (Reeder and Ranger 2011) 12 A GGREGATION AND C ONSENSUS BUILDING IN AN AGREE - ON - ASSUMPTION FRAMEWORK 25. As the above examples show, because Agree-on-Assumptions approaches rely on our being able to come to consensus on the parameters that affect a decision, they break down when there are ethical judgments and differing worldviews. This is particularly the case of cost-benefit analysis, which seeks to aggregate all categories of costs and benefits into a single metric. This encourages analysts to use models and relationships for which we have or can potentially produce consensus. This may mean ignoring or assuming away many large and hidden uncertainties for which there is little consensus or knowledge. Additionally, choosing the costs and benefits to include can be highly subjective and reflect the priorities of analysts, rather than the priorities of stakeholders. For instance, should GHG emissions be a factor in prioritizing new energy generation projects? Those principally concerned with providing low- cost energy may argue that it should be given little weight or ignored, while those concerned with climate change may argue that it should be a key consideration. 26. Decision makers must also ask themselves, “What evidence should we use to evaluate the options?” Cost benefit metrics typically use market values to set the relative importance of different criteria. However, analysts may judge that no market value exists or the market undervalues a good. This is another subjective choice around which there is much disagreement. Do the market prices for developing shoreline adequately reflect the value of ecosystem services that coastal habitats provide? Does the market price of carbon emissions adequately reflect the future impact of climate change? If not, analysts may use shadow prices instead of market prices. But, as Box 4 shows, this too can be contentious, and different preferences for discount rate and other factors may lead to different costs. Box 4. Setting the Social Cost of Carbon The United States has developed estimates of the Social Cost of Carbon (SCC) to analyze regulatory policies. The SCC was constructed using integrated assessment models that value discounted global damages arising from climate change scenarios through to 2050. The selection of the discount rate has a significant impact, given that costs are estimated over fifty years and that costs will increase as more severe impacts are felt over time. The SCC is presented as a range of values with four discount rates, 5 percent, 3 percent, 2.5 percent and 1 percent, with 3 percent as the central estimate. In 2010 the SCC at the 3 percent discount rate is estimated at $21 per ton of CO2 (tCO2) (at 2007 prices) rising 2 to 3 percent a year in real terms. The United Kingdom uses a different approach and value for the SCC. The UK set an official shadow price for Carbon at £25 per ton of CO2 equivalent (tCO2e) in 2007 for the purposes of cost-benefit analysis of government policies, programs, and projects. This was based on the SCC advised by Stern Review, equivalent to £19/tCO2e. This is higher than the values in the US because the UK uses a lower discount rate and equity weighting. The shadow price was set higher than the underlying SCC in order to adequately reflect abatement costs that would be incurred to meet the government’s abatement goal (and thereby incentivize action) and in recognition of the government’s desire to be seen as a leader in climate change action. The SCC would increase by 2 percent a year to capture the rising incremental damage of each unit of carbon as temperatures rise. It would be subject to periodic review to assess progress towards the government’s abatement objectives and target emissions reductions. This policy was substantially revised in 2009, when the UK shifted to a target-consistent approach, based on estimates of the abatement costs that would be incurred to meet specific emissions reduction 13 targets laid out in the Government’s Carbon Budget. SCC would continue to be monitored but would no longer provide the basis for setting the carbon price because of the uncertainty surrounding SCC estimates (Pindyck 2013). Two prices are now in use. For appraising policies in sectors covered by the EU Emissions Trading System (ETS), estimates of future traded carbon price are used, giving a carbon price of £25 in 2020, with a range of £14 - £31. For appraising policies in sectors not covered by the EU ETS, a “non-traded price of carbon‟ will be used, based on estimates of the marginal abatement cost required to meet a specific emission reduction target, with a price of £60 tCO2e in 2020, and a range of £30 - £90. These estimates are periodically revised. Sources: (Wolverton et al. 2012; Price, Thornton, and Nelson 2007; UK Department of Energy & Climate Change 2009) 27. In general, the value of costs and benefits can be deeply uncertain in the absence of readily available market prices or if market prices do not adequately capture value. This includes valuations of environmental costs and benefits and intangibles such as the well-being of persons. The benefits of building a sea wall, for example, are likely to be higher than the damage costs alone because of the suffering and inconvenience from flooding. In these cases cost-benefit analysis requires the construction of ‘shadow’ prices that better reflect utility. There are many different approaches to establishing shadow pricing, each of which may reveal a different shadow price and each of which has different strengths and weaknesses nearly all of which are data intensive. These include surveys, choice modeling (Bennett, Van Bueren, and Whitten 2004), and hedonmic price modeling (Mahan, Polasky, and Adams 2000). The choice of method can be controversial, particularly if the resulting shadow price has a significant impact on the performance of the policies or investments being considered. 28. Choice of discount rate is particularly contentious in the context of climate change. The discount rate determines the viability of climate change mitigation and adaptation projects, given that costs are incurred in the short-term and benefits may only accrue in the distant future. High discount rates will discourage these investments. As with market prices, there is little agreement among economists as to the theoretically ‘correct’ method of setting discount rates, particularly for programs and projects that will generate costs and benefits over several generations. In practice, discount rates are set by the central finance agency as a ‘plug-in value’ for use in policy, program or program appraisal. Most developed countries apply much lower discount rates, mostly in the range of 3 to7 per cent, with many reducing their rates in recent years. Developing countries use a much higher discount rate, reflecting the higher opportunity cost of capital, in the range of 8 to 15 percent. Multilateral development banks follow similar practices with high discount rates (Zhuang et al. 2007). 29. CBA or RO approaches often ignore the distribution of costs and benefits between different social groups, since those who benefit can theoretically compensate those who bear the costs. In practice, however, it will rarely be the case that the beneficiaries of a government project will actually compensate the losers. There are multiple ways of taking this into account, including through distributional weights (Fleurbaey and Hammond 2004; Harberger 1984; Harberger 1978). In many instances, it is extremely difficult to build a consensus on which distributional weights should be applied; and the large impact of these weights on decisions makes it even more difficult to create a consensus. Given the many potential perspectives on the issue of equity, it is unlikely that any particular evaluation tool can provide a satisfactory resolution to the problem: it is typically recommended in CBA that an unweighted analysis be carried out in addition to any weighting exercise. 30. These problems suggest that we should seek to encompass the whole set of possible assumptions to check the robustness of our assessment of options according to whatever metric we have chosen. The example of New Orleans and hurricane protection shows that we can arrive at very 14 different net present values for an investment, even with reasonable parameter values. Here benefits from protection range from $0.6 billion to $140 billion depending on scientific assumptions (e.g., the impact of climate change on hurricanes) and ethical judgments and views (e.g., the aversion to risk and inequality). This situation is common and the Agree-on-Assumptions approach can rarely be used to make a decision in an objective way when uncertainty and disagreement are large. Thus, it is interesting and useful to explore alternative ways to approach such decisions. D. AGREE-ON-DECISIONS PROCESSES 31. We can manage deep uncertainty by seeking a robust decision -- one that performs well across a wide range of futures, preferences, and worldviews, though it may not be optimal in any particular one. Consider two crops: Crop A provides steady yield in drought or excessive rain, while Crop B provides still greater yield under specific conditions consistent with historical precipitation, but fails otherwise. If we could control precipitation or could reliably predict that this year’s precipitation would look like the past, we would do well to plant Crop B and maximize yield. But this decision is likely to be brittle – we can rarely predict precipitation, and we may instead prefer to hedge our bets and plant Crop A if Crop B appears too vulnerable. Robustness becomes important when the consequences of making a wrong decision are high. If crop insurance is available to help protect against potentially poor yields, or if sufficient savings are available, optimizing (and coping with bad years) may be the best strategy. If these tools and resources are not available and the consequences of a few years of low yields are disastrous, then robustness becomes a priority. 32. We can identify robust strategies by inverting the traditional steps, i.e. using Agree-on- Decisions processes. They are sometimes also called “context-first” methods (Ranger et al. 2010). They start by “stress-testing” our options under a wide range of plausible conditions, without requiring us to decide or agree upon which conditions are more or less likely. They evaluate our decision options repeatedly, under many different sets of assumptions. We can evaluate our options under low-likelihood, but high-consequence events. We can treat as uncertain the assumptions buried in models. We can use every stakeholder’s beliefs about the future – we don’t need to agree first on these assumptions. This reveals which of our options are robust – meeting our needs under a wide range of conditions, rather than performing well in only a few. Analytical tools can then help us identify the specific conditions in which each option no longer meets our goals. For example, in stress testing a flood risk management strategy involving dikes, we might find that it results in high risk if two conditions occur: (a) sea level rise proves higher than expected and (b) rapid development results in a larger than forecast population living behind the dikes. This focuses our attention on conditions that matter to the decision and on reaching agreement upon a decision option, without necessarily agreeing on the assumptions that might lead us to prefer that option. 33. This inverted process promotes consensus around decisions and can help manage deep uncertainty around climate change. Analyses performed in this way do not make the decisions for decision makers. Instead, they help decision makers debate important questions: • Are the conditions under which our option performs poorly sufficiently likely that we should choose a different option? • What are tradeoffs do we wish to make between robustness and, for example, cost? • Which options leave us with most flexibility to respond to changes in the future? Decision makers may conclude, for example, that the threat of unexpectedly high sea level rise is sufficiently great to warrant modifying the dike plan or augmenting it with other policies. Knowing that dikes may fail to reduce risk in a future with high sea level rise and extensive urban development, decision makers might modify the current plan to increase dike height or, alternatively, augment the 15 original dike design with policies to shift development away from the dikes. These can be difficult debates. But, they are much more useful than debates about the unknowable future. 34. Importantly, these Agree-on-Decisions methodologies can use the very same metrics as Agree- on-Assumptions methods. For instance, the net present value of a project or the benefit-cost ratio (the metrics often used in cost-benefit analysis) will play an important role in economic analyses that use Agree-on-Decisions methods (e.g., Bonzanigo and Kalra 2014; Lempert, Sriver, and Keller 2012). In other words, the metrics for evaluating and comparing decision options is the same, but the decision-making process is different. 35. One of the potential strengths of Agree-on-Assumptions methods is that they can rank decision options and identify the one that optimize decision makers’ metrics. However, this is only possible if all priorities and concerns are aggregated into a single metric along which the decision options can be monotonically ranked. 36. However, it is often useful to use multiple metrics to evaluate decision options (e.g., the net present value can be complemented with a metrics for distributional impacts), which avoids the difficult problem of having to aggregate all costs and benefits into a unique metric. While Agree-on- Assumptions can use multiple metrics, they often aggregate those metrics into a single weighted utility function in order to achieve a ranking of projects. Agree-on-Decisions are better suited to using multiple disaggregated and distinct metrics since the aim is often to find decisions that work well across a range of assumptions and a range of goals or objectives. 37. Using multiple distinct metrics has at least three major benefits: - First it helps stakeholders with different values reach consensus, since it does not require starting the analysis with an ex ante agreement on valuation techniques and relative prices and does not provide a unique ranking of projects. For instance, evaluating a risk management project through a cost-benefit analysis requires an agreement on the value of life, which is controversial and can result in gridlock. This can be avoided if we instead count the number of lives saved as a separate metric. - Second, by tracking diverse impacts along the analysis, multiple metrics help identify the major trade-offs implied by the decision. Consider for instance the choice between two options to mitigate risks: building a dike or restoring a mangrove forest. While a cost-benefit analysis would produce a ranking as the final outcome, there may be an important tradeoff between these choices. For instance, it may be that the dike has a lower cost but leads to environmental losses while the more expensive ecosystem-based approach creates co-benefits from recreation and biodiversity. A constructive discussion among stakeholders with divergent views is more likely if this trade-offs is identified and explicit. - Third, identifying tradeoffs helps design policy mixes in which complementary policies smooth or mitigate adverse effects for some stakeholders or in some sectors. For instance, Figure 1 shows the impact of various climate-urban policies in the Paris agglomeration – a green belt policy, a public transport subsidy, and flood zoning – as measured by five metrics. Each of these policies has benefits along certain metrics and costs along others. The net benefit depends on how one values these different metrics, and agreeing on an aggregation techniques would be difficult. An analysis that tracks multiple metrics simultaneously makes it possible to add specific actions to correct for the negative outcomes. For example, a green belt policy (in which construction of new buildings is restricted) has a negative impact on housing affordability, and may become socially acceptable only if accompanied by complementary actions to reduce housing and building construction prices (e.g., with a reduction in land acquisition taxes). These options may not be evident when using a single, aggregated metric. 16 Figure 1. Three policies in the Paris agglomeration Note: This figure shows a greenbelt policy, a public transport subsidy, and a zoning policy to reduce the risk of flooding. Each is rated across several criteria. Positive outcomes are towards the outside of the radar plot, while negative outcomes are towards the inside. Source: (Viguié and Hallegatte 2012) E XAMPLE : F OREST M ANAGEMENT USING AN A GREE - ON - DECISION P ROCESS 38. Consider a case in which a local government seeks to manage a heavily forested catchment area to regulate downstream floods and providing irrigation for local farmers. 1 The government is considering several investments and policies, including building one or more dams, constructing irrigation canals, establishing pricing schemes for irrigation, and developing a forest management plan to control erosion and runoff from timber harvesting. The government’s resources are limited, and it is seeking the plan with the highest net present value. Government officials ask a group of four experts, A, B, C, and D to determine the conditions for which the forest management program should be designed. Together, they determine that future rainfall and demands for timber are the main uncertainties that would determine the choice of plan. But their expectations of the future greatly differ. Expert A believes that the past will look like the future. Expert B argues that climate change will result in more rainfall and development will require more timber. Expert C believes that climate change will result in less rainfall but agrees with Expert B that timber demands will increase. Expert D agrees with Expert C that there will be less rainfall, but is convinced that there will be demands for afforestation due to REDD efforts. 39. Under an Agree-on-Assumptions approach, we would ask our experts to first reach consensus on a best projection. Unfortunately, any such agreement may be difficult and arbitrary since these conditions are deeply uncertain. We could invest in research to determine which of the four possible futures is the most likely, and then to select the plan that performs best in this future. Many practitioners report that decision makers often demand this: they want to know what the best prediction is in order to select the best option for that future. If our knowledge base made it possible to make accurate forecasts about the future, this approach would be appropriate. It does not work, however, when it is impossible to determine which scenario will occur. Several scenarios are often equally plausible and none is impossible. As described in the New Orleans case study, one option is 1 Example adapted from Hallegatte et al. (2012) and Ranger (2013). 17 to attribute subjective probabilities to the different scenarios and calculate the expected value of each option. However, as noted earlier, we cannot develop reliable probabilities for future climate or other conditions, and so this approach could also lead decision makers astray. 40. As an alternative, we could identify the plan that is robust, working well across all the scenarios. Table 1 shows the performance of each of three candidate plans in each of the four scenarios developed by experts, rated from a very high positive NPV (+4) to a very negative NPV (- 4). Looking at the performance across all scenarios helps understand the vulnerability and robustness of each of the plans. First, this analysis suggests that no plan is the strongest in all cases. While Plan 1 has the highest NPV in Scenario A, it also has the lowest NPV in Scenario B, if heavy erosion, siltation, and flooding occur. This suggests that Plan 1 is particularly vulnerable to heavy rainfall. It would also have excess flood mitigation capacity in scenarios C and D and might not be able to meet timber demands either. Those concerned about worst-case plausible scenarios might eliminate Plan 1 as an option. Plans 2 and 3, on the other hand, are more robust, performing reasonably well in nearly all futures. 41. One way to quantify the performance is to calculate each plan’s regret. Consistent with our use of the term in everyday life, mathematical regret is the difference between the utility of a decision in a particular scenario, and utility of the best decision that could have been made in that scenario. For example, Plan 1 performs very poorly in Scenario B — resulting in a utility of -2. The best option in Scenario B would have been Plan 2, with a utility of 3. This gives Plan 1 a very high potential regret of 5 (3 – (-2)) in Scenario B. The table calculates regret for each plan in each scenario, where 0 is the lowest possible regret. 42. Plans that have the lowest possible regret across all scenarios are often attractive, as they suggest that the plan will work well no matter what the future brings. Plans 2 and 3 have the same maximum regret of 1. The average regret is also a useful metric. If we assumed that all scenarios are equally probable, a defensible assumption if we have no additional information on the scenarios, Plan 3 has a slightly lower average regret of 0.50, in addition to having a low worst-case regret. Experts could choose between Plan 2 and Plan 3 based on considerations of cost, feasibility, and other factors, or based on later judgments of which scenarios may be more or less likely. 43. In addition, the analysis shows the potential of developing flexible strategies. It is perhaps easier to build a smaller dam first, with expansionary capability, than to build a larger dam that is excessive or inappropriate in various scenarios. We might use a real options metric to calculate the present value of this future flexibility. Other approaches that are more flexible, such as the forest management policy, can be adapted as new information on climate and demand emerges. 18 Table 1. Assessment of the performance of the three forest and flood management plans across four scenarios. 19 F EATURES OF R OBUST D ECISIONS 44. The above example highlights several features that can make decisions robust in the face of deep uncertainties These include but are not limited to (Hallegatte 2009): a. No- and low-regret decisions, b. Reversible and flexible decision, c. Safety-margin decisions, and d. Decisions with reduced time horizons. 45. “No-regret” or “low-regret” decisions have high utility no matter what the future brings. Thus, they can be robust even to deep uncertainties. For example, reducing leaks in water distribution systems is almost always a good investment, regardless of how the climate, future demand, and other factors change. Land-use policies that seek to limit urbanization and development in low-lying, flood- prone areas would reduce disaster losses in the present climate. Climate change, which may increase the frequency or intensity of storms, may make such a policy even more desirable. These examples suggest that finding a system’s existing shortcomings may reveal no-regret or low-regret strategies: such strategies are beneficial over the short term (and thus easier to implement from a sociopolitical point of view) and may offer benefits under a wide range of future conditions. This approach needs to be used with caution, however, to avoid solving near-term problems with decisions that are sensitive to future uncertainties. Building a diversion to transfer water across basins may help reduce near-term water scarcity. However, such investments can take decades and cost billions, and climate change may alter precipitation patterns, reducing or negating the value of these investments by the time they are completed. 46. Reversible and flexible decisions are typically more robust than irreversible ones because they enable us to adjust our decisions as new information becomes available. In this way, reversible and flexible decisions can help us reduce our regret. For example, insurance and early warning systems can be adjusted every year in response to the new information on emerging risks. As another example, when deciding whether to allow the urbanization of an area potentially at risk of flooding due to climate change-induced increases in river runoff, the decision-maker must be aware that one option is reversible while the other is not. Restricting urbanization has a short-term cost, but if new information shows that the area is safe, urbanization can be allowed immediately. This option, therefore, is highly reversible, even though it is not without cost since it may prevent profitable investments from being realized in the near term. Allowing urbanization now, on the other hand, is irreversible or very expensive to reverse. If the area becomes high risk in the future, the choice will be to abandon the area or to protect it, both of which may be difficult and expensive. This does not imply that urbanization should be rejected. Rather, in the decision-making process, the value of the reversibility of a strategy (the option value) should be taken into account. 47. Flexibility can be built into engineering solutions. The Stormwater Management and Road Tunnel (SMART) that runs under the financial district in Kuala Lumpur to relieve traffic congestion offers an example (“What Is SMART?” 2013). The tunnel has three levels: two for road traffic and a lower level for carrying flash floods from the Klang River under the city and out to the Kerayong River. During major storms, cars are excluded from the two traffic lanes and gates are opened to allow stormwater to flow through the upper levels of the tunnel. Traffic can enter again within about 48 hours of closure. The additional construction cost of the multi-purpose tunnel and surface congestion costs during closure represent an option premium. But the cost of the multipurpose tunnel is less than a traffic-only tunnel combined with a duplicate tunnel dedicated solely to channelling intermittent floodwaters. 20 48. Many “safety margin” strategies can reduce the risk of bad options at negative, zero, or negligible cost. For instance, to calibrate drainage infrastructure, in 2008 water managers in Copenhagen used runoff estimates that were 70 percent larger than their current level. This margin helps manage increases in drainage needs due to future population growth and due to potential climate change, which may increase heavy drainage demands in Denmark. This 70 percent increase has not been precisely calibrated because such a calibration is made challenging by climate change uncertainty. But this increase is thought to be large enough to cope with almost any possible climate change during this century, considering the information provided by all climate models. This move is justified because, in the design phase, it is inexpensive to implement a drainage system able to cope with increased precipitation. On the other hand, modifying the system after it has been built is difficult and expensive. It is wise, therefore, to seek to be over-pessimistic in the design phase where possible (Hallegatte 2009). 49. Cheap safety margins are especially important for adaptation measures that are not reversible or flexible. Irreversible decisions (e.g., development in coastal areas) without inexpensive safety margins are particularly brittle to deep climate uncertainties – they can neither be undone nor protected. Adding safety margins to irreversible decisions (e.g., adding coastal defenses or improving urban water-management infrastructure) can improve robustness, but only if we carefully consider the range of potential future climate change scenarios. At the same time, we must recognize that resources are extremely constrained in developing countries, and safety margins with modest or even low-costs may be beyond reach. 50. Reducing the lifetime of investments is one way of reducing uncertainty around a decision. The uncertainty regarding future climate conditions increases rapidly with time. This strategy has already been implemented in the forestry sector with plants species that have a shorter rotation time. Since species choice cannot be made reversible and no safety margins are available in this sector, this option is interesting in spite of its cost. In other sectors, it is also often possible to avoid long-term commitment and choose shorter-lived decisions. For example, if houses are to be built in an area that may become at risk of flooding if precipitation increases, it may be useful to build cheaper houses with shorter lifetimes instead of high-quality houses meant to last one hundred years under historical conditions. A GREE - ON -D ECISIONS A PPLICATIONS 51. There are many practical applications of Agree-on-Decision processes. The specific methodologies share a common underpinning: they manage uncertainty by seeking decisions that are robust, performing well across a wide range of potential futures. This section summarizes three applications of these methodologies to investment decisions: Robust Decision Making applied to water resource planning in Southern California; Climate Informed Decision Analysis applied to management of the Great Lakes; and Info-Gap for seismic resilience in water systems. These examples are not exhaustive – there are many other Agree-on-Decision processes and many other applications of these processes. Rather, we seek to illustrate how these methods can be successfully applied to complex real-world problems that face deep uncertainty. R OBUST D ECISION M AKING FOR W ATER P LANNING IN S OUTHERN C ALIFORNIA 52. Robust Decision Making (RDM) is a decision framework developed specifically for decisions with long-term consequences and deep uncertainty (Schwartz 1996; R. J. Lempert and Schlesinger 2000; R. J. Lempert et al. 2006; Groves and Lempert 2007; R. J. Lempert and Collins 2007; Stéphane Hallegatte 2009; R. J. Lempert, Popper, et al. 2013). RDM is iterative and follows a “deliberation 21 with analysis” process recommended by the U.S. National Research Council for managing climate uncertainty (2009). It has been applied to water resource management (Groves, Fischbach, et al. 2013; Groves, Bloom, et al. 2013; R. J. Lempert and Groves 2010) flood risk management (Fischbach 2010; R. J. Lempert, Kalra, et al. 2013), energy investments (Popper et al. 2009; Bonzanigo and Kalra 2014), and terrorism insurance (Dixon et al. 2007). 53. RDM involves four main steps as shown in Figure 2. First, stakeholders and analysts engage in a decision structuring exercise to identify the goals, uncertainties, and decision options to be analyzed. Second, analysts use computer simulation models to evaluate the performance of the decisions under hundreds or thousands of combinations of uncertainties, which produces a database of performance results. Third, visualizations and analysis of this database help decision makers and analysts understand the vulnerabilities of their potential decisions, i.e. conditions under which their choices would fail to meet their goals. They may suggest new decision options to mitigate those vulnerabilities, resulting in an iteration back to step 1. Alternatively, they may fourth consider tradeoffs between the options based on their vulnerabilities and other considerations. They iterate upon this process until decision makers agree upon a robust strategy. Also, RDM normally involves a detailed quantitative analysis, but the underlying framework can inform more “heuristic” – less resource-intensive – evaluations (R. J. Lempert and Kalra 2011). RDM is especially useful when project plans can be framed as a series of decisions over time, to take advantage of learning explicitly and adjust to new information as it comes along. Figure 2. Steps in an RDM Analysis Source: (R. J. Lempert, Popper, et al. 2013) 54. Robust Decision Making (RDM) was used to help the Inland Empire Utilities Agency (IEUA) of Southern California develop its long-term water management plan (R. J. Lempert, Popper, et al. 2013; R. J. Lempert and Groves 2010). The utility sought a plan that would provide reliable, low-cost water to its customers for the long-term future. It sought to incorporate the impacts of climate change into its planning but was hampered by the deep uncertainty in future climate forecasts. The results of 21 atmosphere-ocean general circulation models scaled down to the Southern California region indicated that the climate could range from 0.1-2.1°C warmer and there could be a - 19 to +8% change in wintertime precipitation. 55. The robustness of the utility’s Urban Water Management Plan (UWMP) to climate and other uncertainties was explored in the range of possible future climates and other socio-economic conditions. The utility sought a plan that would have a low total cost, which included the cost to implement the plan and the potentially additional cost of meeting future water shortages with imports. The RDM analysis evaluated the cost of the UWMP in 200 futures with different assumptions regarding the extent of climate change, future socioeconomic conditions, the agency’s ability to 22 implement its management plan, and costs. This is shown in Figure 3, where each data point represents the supply and shortage costs in a single scenario. Under this analysis, IEUA would face unacceptably high cost in 120 of the 200 scenarios as shown in the shaded high-cost region. 56. The analysis showed the specific conditions under which the UWMP would not meet IEUA’s goals and how augmenting the plan could reduce those vulnerabilities. The UWMP was particularly vulnerable to future conditions that were drier, with reduced access to imported water, and when natural percolation of the ground water basin decreased. IEUA suggested eight additional management options that could be added to the UWMP to potentially reduce these vulnerabilities and make the UWMP more robust. Options included increasing water use efficiency, recycling storm water for ground water replenishment, storing water from surplus years, and developing the region’s water recycling program. In all cases, augmenting the UWMP with additional management strategies led to lower costs and reduced vulnerability. Analysts also evaluated the performance of flexible plans – ones designed to change over time as new information became available. 57. Water managers were able to build consensus around a robust water management plan. Analysts presented water managers with visualizations of the tradeoffs between each management plan’s robustness and its feasibility or difficulty of implementation. These results helped IEUA choose to make its original 2005 UWMP flexible and include near-term enhancements to its recycling and water surplus storage program. This reduced the number of high-cost cases from 120 to 25. RDM also helped the agency build consensus for this plan among its constituents and ratepayers, even those who were dubious about climate. Figure 3. Performance of IEUA’s Urban Water Management Plan in 200 Futures Source: (R. J. Lempert, Popper, et al. 2013) C LIMATE I NFORMED D ECISION A NALYSIS F OR M ANAGING THE G REAT L AKES B ASIN 58. Like RDM, Climate Informed Decision Analysis (CIDA; also known as “decision scaling”) first determines how climate change could affect a project and then, second, assesses the likelihood of those effects using multiple climate information sources (Moody and Brown 2013; Moody and Brown 2012). This is shown in Figure 4. CIDA was first used to improve management of the Great Lakes Basin in the United States. In 2007, the International Joint Commission (IJC) established an independent study board composed of United States and Canadian members to review the operation of structures controlling Lake Superior outflows and to evaluate improvements to the operating rules and criteria governing the system. The study is known as the International Upper Great Lakes Study (IUGLS). As a result of the considerable uncertainty associated with future climate and lake levels, as well as other sources of uncertainty such as ecosystem responses and the state of the navigation industry, a process of selecting the optimal plan based on a most probable future 23 scenario was rejected in favor of a robust decision making process designed to incorporate multiple and at times conflicting sources of climate information. The analysis of the Great Lakes Basin plan comprised three phases. 59. First, stakeholders defined performance benchmarks and identified vulnerabilities that would lead to unacceptable performance. To prioritize concerns for the regulation of Lake Superior, stakeholder experts were convened from state, provincial, federal, and local government agencies, as well as special interest groups (boating, hydroelectricity, navigation), and environmental groups (e.g. The Nature Conservancy). The experts were tasked with identifying the metrics by which they would like the candidate regulation plans to be evaluated. Based on these goals, the stakeholder groups then defined what combination of lake level and duration led to acceptable outcomes, and what levels led to impacts that were either less favorable or unacceptable. In this way, thresholds designating of acceptable performance for candidate regulations plans were established. This allowed the evaluation of regulation plans using a common numeraire for each of the different interests, namely, the occurrence of unacceptable lake levels as defined by each interest. 60. Second, stakeholders quantified the climate sensitivity for each proposed plan. Analysts created a climate response function, which estimated the consequences (lake levels and associated performance metrics) of a given decision (regulation plan) for a set of mean climate conditions that were varied to evaluate a broadly defined plausible range of climate change. The function thus related climate effects to the performance metrics influencing the decision in a way that was independent of any assumptions about the relative likelihood of future climate conditions. Using a parametrically varied set of stochastic time series (each representing a given mean climate), analysts identified those climate conditions that presented risks (unacceptable lake levels) to each regulation plan. Note that climate model projections had not been used in the analysis to this point, yet the vulnerability of proposed plans to potential climate futures was revealed. 61. Third, the plausibility (relative probability) of those conditions was estimated through tailored climate information. Given the uncertainty associated with the probability estimates even after maximizing credibility, the term ‘‘plausibility’’ is used in place of probability. The decision maker, in this case the Study Board, was presented with the risk of each candidate regulation plan, where risk was defined as the plausibility of problematic climate conditions (the climate changes that would cause unacceptable performance). The plausibility estimates were represented as imprecise probabilities, where a range of probabilities was shown for each climate state based on the difference sources of climate information (e.g., climate projections, stochastic analysis, paleo-climate data, and expert opinion). Thus, plans could be evaluated in terms of the range of climate change over which they maintained acceptable lake levels, i.e., their robustness to climate change. The result was a climate-informed robustness index for each candidate plan. The plausibility estimates could be adjusted based on different comfort levels of the Board members with different sources of climate information. For example, some preferred considering only the probabilities generated from climate model projections, while others preferred only probabilities from historical observations. 62. In sum, recommendation for a final regulation plan focused on two plans that performed best over a very wide range of future climates. The analysis also revealed that even these “best” plans were not robust to climate changes that were quite plausible, leading the Study Board to also recommend that the regulation plan be coupled with an adaptive management program to address future uncertainty. The Board expressed satisfaction in the use of climate information to make this decision. 24 Figure 4. Decision scaling steps compared to a traditional analytical process. Source: (Brown et al. 2011) I NFO -G AP FOR S EISMIC R ESILIENCE IN W ATER S YSTEMS 63. Like RDM and CIDA, Info-Gap helps decision makers identify robust options, but it takes a somewhat different approach. RDM and CIDA use models to assess the performance of options in a wide range of potential future conditions and then identify conditions that result in poor performance, i.e., conditions to which the system is vulnerable. In contrast, Info-Gap uses models to compute how options perform as a function of uncertainty. An Info-Gap analysis defines robustness as “the maximum uncertainty in our estimates that can be tolerated while still guaranteeing a particular desired result” (Irias and Cicala 2013). An Info-Gap analysis produces a graph showing the performance we can robustly achieve on one axis, as a function of uncertainty on the other axis. Like RDM and CIDA, Info-Gap does not provide decision makers with the solution; rather it seeks to inform the decision-makers on the trade-offs, risks, and vulnerabilities. 64. Info-Gap typically proceeds through four analytical steps, in collaboration with stakeholders and analysts (Irias and Cicala 2013): a. Identify decision options or alternatives b. Compute performance of each option as a function of increasing uncertainty c. Graph the results by plotting performance versus uncertainty d. Examine the results in light of the degree of uncertainty that stakeholders and analysts believe to exist, and the varying performance of each alternative as uncertainty increases. 65. The East Bay Municipal Utility District, a public water agency in Northern California, used Info-Gap to help it decide how to provide reliable water to Alameda, an island community of 75,000. Alameda relies entirely on four water pipelines from the mainland, but these are vulnerable to damage from corrosion, physical impact, and earthquakes. 66. EBMUD is developing a master plan for the island that will ensure reliable water supply by “not only managing the possibility of critical combinations of outages, but also managing the financial and social consequence of any outages that occur, which likely means keeping the time and cost of 25 recovering from an outage at reasonable levels” (Irias and Cicala 2013). In an initial study, EBMUD used Info-Gap to weigh two options: construct new water pipeline or tunnel crossings to the island or improve the seismic resilience of an existing pipeline. Their goal was to ensure that at any time, one crossing would survive a seismic event, and a second crossing could be returned to service quickly. The key uncertainties were the probabilities of failures and the length of time needed to conduct repairs, which is a sum of the time of individual repair steps: discovery, setup, investigation, and repair. The Info-Gap analysis is currently under way and is helping EBMUD answer two important questions (Irias and Cicala 2013): a. How wrong can we be about the probabilities of various types of failures, and still have an acceptably large overall probability that a crossing survives? b. For any given design and construction alternative, how wrong can we be about the duration for return to service and still be within the maximum allowable down time? E. CONCLUSIONS 67. Investment decision making is already difficult for any diverse group of actors with different priorities and views. But the presence of deep uncertainties linked to climate change further challenges decision-making by questioning the robustness of all purportedly optimal solutions. We suggest that decision makers can continue to use the decision metrics they have used in the past. Cost benefit, cost effectiveness, and multi-criteria metrics are still valid for measuring decision options. They differ in their strengths and drawbacks, but these characteristics are distinct from the challenges of applying these metrics in a deeply uncertain world. 68. We suggest that decision makers consider alternative decision processes to traditional ones, especially those that lead with analysis and end in agreement on decisions. Agree-on-Decision methods start by “stress-testing” options under a wide range of plausible conditions, without requiring us to agree ex ante on which conditions are more or less likely, and against a set of objectives or success metrics, without requiring us to agree ex ante on how to aggregate or weight them. As a result, these methods are easier to apply in contexts of large uncertainty or disagreement on values and objectives. 69. This inverted process promotes consensus around better decisions and can help manage uncertainty. Analyses performed in this way do not make the decisions for decision makers or provide a single “best” solution, deus ex machina. Instead, they help decision-makers debate important questions: “Are the conditions under which an option is vulnerable sufficiently likely that we should choose a different option? What tradeoffs do we wish to make between robustness and cost, or between various objectives and policy goals? Which options leave us with most flexibility to respond to changes in the future?” These can be difficult debates. But, they are much more useful than debates about the unknowable future. 70. A growing set of case studies shows that these methods can be applied in real-world contexts and do not need to be more costly or complicated than traditional approaches. While this paper focuses on climate change, a better treatment of uncertainties and disagreement would in general improve decision-making and development outcomes. 26 APPENDIX 1. UNDERSTANDING CLIMATE UNCERTAINTY 1. Much useful effort has gone into developing climate models. These models confirm that the planet is warming due to greenhouse gas emissions and reveal potential changes in precipitation, temperature, and other climate characteristics around the globe. However, climate models cannot provide the equivalent of reliable historical climate data for future climates. Instead, climate projections can vary widely and it is important to understand why. This appendix summarizes key issues in understanding climate uncertainty. 2. Consider one example. Given the vulnerability of water systems to climate change, a Ghanian urban water manager would be wise to ask climate modelers to predict precipitation rates for the next 100 years, instead of relying on historical data. But using a climate model might be dangerously misleading: projections of future precipitation changes in the region are very uncertain. Figure A1 shows the change in annual rainfall in 2080-2100 (with respect to the 1980-2000 period) in Africa according to two climate models (IPCC 2007). For Ghana, one model (CCSM3) predicts a 20% increase in precipitation, while another (GFDL) predicts a 30% decrease! It would be unwise for our water manager to tailor water management projects to either one of these or any other particular projection. Figure A1. Change in annual rainfall in 2080-2100 (with respect to the 1980-2000 period) in Africa according to two climate models. Source: (IPCC 2007) 3. Such great uncertainties about future climate change stem from three major sources: • Future emissions of greenhouse gases, which will shape future climate change. Future emissions, in turn, are driven by demographic and socioeconomic trends, technology, values and preferences, policies, which are also deeply uncertain. Scientists have developed emissions scenarios to capture a wide range of potential emissions trends that consider these diverse drivers. • Scientific uncertainty and modeling limitations. These limitations are a result of our imperfect knowledge of the climate system and of the systems that climate, in turn, affects, such as lakes, glaciers and ecosystems. For instance, the effect of a given quantity of greenhouse gas on global mean temperature is uncertain. 2 The interactions between the oceans and the atmosphere, rates of melting of ice sheets, and the effect of clouds may have a significant impact on climate change and are still being studied and modeled. There is also uncertainty in the regional effects of global warming. 2 In particular, “climate sensitivity” refers to the increase in global mean temperature from a doubling of the CO2 concentration in the atmosphere. This sensitivity is uncertain. 27 • Irreducible natural variability. Global climate variables have their own dynamics linked to the chaotic behavior of the climate system. Climate models provide climate statistics, e.g. averages, variances, likelihoods to exceed thresholds, etc., but they cannot provide weather forecasts. In other words, they can estimate the average number of rainy days in the summers of 2060’s, but do not say anything about the precipitation in any given day or even any specific summer. 4. These three uncertainties are sometimes referred to as policy, epistemic, and aleatory uncertainty, respectively. Their respective contribution to total uncertainty depend on the timescale and the spatial scale. At a global scale (A2, left), and over the short term, natural variability and model response play the largest roles, and the emission a very small role; over the long term, the emissions dominate other sources of uncertainty. Figure A2. Source of uncertainty over time globally (left) and in the British Isles (right). Note: The green is emission uncertainty, the orange is natural variability, and the blue is (climate) model uncertainty; the variable is temperature change. Source: (Hawkins and Sutton 2009) 5. It is thus critical not to over-interpret the difference between two climate scenarios run with different emissions or different models. The difference might be caused by aleatory uncertainty, with no significance. To rigorously interpret the difference between two scenarios, it is necessary to use ensembles, i.e. a sufficiently large set of simulations run with the same model and the same emission scenario. The spread of these simulations will represent the effect of natural variability as simulated by the model, and only differences that are robust to this effect can be interpreted as the effect of different emissions scenarios or of different models. 6. Also, it is critical to recognize that the spread across models do not represent the full uncertainty. All climate models use the same knowledge base and are based on the same basic methodologies (e.g., the discretization of ocean and atmosphere dynamics in a grid, parameterization of many physical mechanisms such as convection). So it is very likely that all models share common biases, making the epistemic uncertainty larger than the differences across models. For instance, all climate models imperfectly represent convection and monsoons in India or West Africa, making the model projection in these regions particularly uncertain. Thus, testing project robustness, looking outside the range of model results is advisable. 7. At a regional (or continental) scale, the factors’ respective shares are different. Natural variability is much more important regionally than globally, emission uncertainty plays a more 28 moderate role, and climate model uncertainty remains large (A2, right). This suggests that it is much more difficult to predict future climates when looking at one country or one region than globally, regardless of future progress in our understanding of climate change. Natural variability means that the climate signal is more difficult to extract (and – as already mentioned – forecasts of future climate remain out of reach). 8. There is also large uncertainty from differences between global climate models. The IPCC (2013) provides results from 42 global climate models. The models agree on the very big picture (more warming in high latitude than in low latitude; more precipitations in high latitudes; less precipitation around the tropics; more precipitation around the equator). However, the differences can be huge in some regions (e.g., half of the models predict an increase in precipitation over India; half of the models predict the opposite; and – as a consequence – the “average model” predicts no change, showing the risk of averaging projections). 9. When looking locally, we usually do not use global climate models. Instead, we use downscaling techniques which can be done with statistical tools or with regional climate models (RCM). Statistical methods use statistical relationships, calibrated on historical data, to relate large-scale drivers – which climate models can reproduce – to local phenomena – which climate models cannot reproduce (Elsner and Jagger 2006; Mestre and Hallegatte 2009; Nuissier et al. 2012). Even though our knowledge of the laws of physics helps select potential predictors, this method is not directly based on physical laws. Such statistical methods are computationally efficient and reproduce the current climate well. Statistical models, however, have two main drawbacks: first, they need long series of reliable data; second, even with a sufficiently large data range, it is difficult to know whether a statistical relationship will remain valid in a future climate. 10. To avoid the problem of validity of historical relationships, one may use physical models, which are based on physical laws. Physical models, which model and simulate biophysical and atmospheric processes, are of particular interest when investigating extreme patterns and variability changes. Of course, physical models often require calibration and bias correction, so the distinction between physical models and statistical models is sometimes fuzzy. Examples are Regional Climate Models (RCMs); see (Knutson et al. 2010). RCMs are difficult to develop and may only have resolution of about 25 km, but they are less data-dependent. The confidence in RCMs is generally higher because they do not have to assume that the relationship between large-scale and small-scale climate variables remains constant in a future – possibly very different – climate. Such an assumption is questionable. Thus, statistical analyses are more reliable over the short to medium term, while RCMs are necessary to understand large warming over the long term. Nonetheless, in the long term RCMs remain driven by the input from GCMs, and so they do not resolve uncertainty related to climate variability, for example, that is produced by the GCMs. 11. In almost all cases, downscaling improves our ability to reproduce the current climate, but it does not reduce the uncertainty on future changes. Downscaling makes it possible to represent additional mechanisms and is likely to improve both the ability of represent the current and future climate (for instance, downscaling technique is required to look at precipitation in coastal or mountainous areas, or to look at small-scale phenomenon like heavy precipitations). But downscaling cannot help manage uncertainty if global climate models disagree. As in West Africa and the monsoon, downscaling can increase the resolution of scenario with an increase in precipitation scenario and a scenario with a decrease in precipitation, but the difference between both will not be resolved. 29 REFERENCES Arnell, Nigel W., Emma L. Tompkins, and W. Neil Adger. 2005. “Eliciting Information from Experts on the Likelihood of Rapid Climate Change.” Risk Analysis 25 (6): 1419–31. doi:10.1111/j.1539- 6924.2005.00689.x. Arrow, Kenneth J., and Anthony C. Fisher. 1974. “Environmental Preservation, Uncertainty, and Irreversibility.” The Quarterly Journal of Economics 88 (2): 312–19. doi:10.2307/1883074. Ben-Haim, Yakov. 2006. Info-Gap Decision Theory Decisions under Severe Uncertainty. Oxford: Academic. http://www.engineeringvillage.com/controller/servlet/OpenURL?genre=book&isbn=9780123735 522. Bennett, Jeff, Martin Van Bueren, and Stuart Whitten. 2004. “Estimating Society’s Willingness to Pay to Maintain Viable Rural Communities.” Australian Journal of Agricultural and Resource Economics 48 (3): 487–512. doi:10.1111/j.1467-8489.2004.00254.x. Bleakley, H., and J. Lin. 2010. “Portage: Path Dependence and Increasing Returns in US History.” Working Paper No. 16314, National Bureau of Economic Research. Boardman, Anthony E., David Greenberg, Aidan Vining, and David Weimer. 2011. Cost-Benefit Analysis: Concepts and Practice. 4th ed. The Pearson Series in Economics. Boston: Prentice Hall. Bonzanigo, Laura, and Nidhi Kalra. 2014. “Making Informed Investment Decisions in an Uncertain World: A Short Demonstration”. Policy Research Working Paper 6765. World Bank. Brown, Casey, William Werick, Wendy Leger, and David Fay. 2011. “A Decision-Analytic Approach to Managing Climate Risks: Application to the Upper Great Lakes1.” JAWRA Journal of the American Water Resources Association 47 (3): 524–34. doi:10.1111/j.1752-1688.2011.00552.x. Craig, Paul P., Ashok Gadgil, and Jonathan G. Koomey. 2002. “What Can History Teach Us? A Retrospective Examination of Long-Term Energy Forecasts for the United States.” Annual Review of Energy and the Environment 27 (1): 83–118. doi:10.1146/annurev.energy.27.122001.083425. Dessai, Suraje, and Mike Hulme. 2007. “Assessing the Robustness of Adaptation Decisions to Climate Change Uncertainties: A Case Study on Water Resources Management in the East of England.” Global Environmental Change 17 (1): 59–72. doi:10.1016/j.gloenvcha.2006.11.005. Dixon, Lloyd, Robert J. Lempert, Tom LaTourrette, and Robert T. Reville. 2007. The Federal Role in Terrorism Insurance: Evaluating Alternatives in an Uncertain World. RAND Corporation. Elsner, James B., and Thomas H. Jagger. 2006. “Prediction Models for Annual U.S. Hurricane Counts.” Journal of Climate 19 (12): 2935–52. doi:10.1175/JCLI3729.1. Fischbach, Jordan R. 2010. “Managing New Orleans Flood Risk in an Uncertain Future Using Non- Structural Risk Mitigation.” Santa Monica, CA: RAND Corporation, 2010. http://www.rand.org/pubs/rgs_dissertations/RGSD262. 30 Fleurbaey, Marc, and Peter J. Hammond. 2004. “Interpersonally Comparable Utility.” In Handbook of Utility Theory, edited by Salvador Barberà, Peter J. Hammond, and Christian Seidl, 1179–1285. Springer US. http://link.springer.com/chapter/10.1007/978-1-4020-7964-1_8. Gay, Carlos, and Francisco Estrada. 2010. “Objective Probabilities about Future Climate Are a Matter of Opinion.” Climatic Change 99 (1-2): 27–46. doi:10.1007/s10584-009-9681-4. Gollier C., and Treich N. 2003. “Decision-Making Under Scientific Uncertainty: The Economics of the Precautionary Principle.” Journal of Risk and Uncertainty 27 (1): 77–103. Groves, David G., Evan Bloom, David R. Johnson, David Yates and Vishal Mehta. "Addressing Climate Change in Local Water Agency Plans: Demonstrating a Simplified Robust Decision Making Approach in the California Sierra Foothills." Santa Monica, CA: RAND Corporation, 2013. http://www.rand.org/pubs/research_reports/RR491. Groves, David G., Jordan R. Fischbach, Evan Bloom, Debra Knopman, and Ryan Keefe. 2013. “Adapting to a Changing Colorado River: Making Future Water Deliveries More Reliable Through Robust Management Strategies." Santa Monica, CA: RAND Corporation, 2013. http://www.rand.org/pubs/research_reports/RR242. Groves, David G., and Robert J. Lempert. 2007. “A New Analytic Method for Finding Policy-Relevant Scenarios.” Global Environmental Change 17 (1): 73–85. doi:10.1016/j.gloenvcha.2006.11.006. Gusdorf, François, Stéphane Hallegatte, and Alain Lahellec. 2008. “Time and Space Matter: How Urban Transitions Create Inequality.” Global Environmental Change 18 (4): 708–19. doi:10.1016/j.gloenvcha.2008.06.005. Ha-Duong, Minh. 1998. “Quasi-Option Value and Climate Policy Choices”. Post-Print halshs-00002457. HAL. http://ideas.repec.org/p/hal/journl/halshs-00002457.html. Hallegatte, St. 2006. “A Cost-Benefit Analysis of the New Orleans Flood Protection System”. Working paper 184. Regulation2point0. http://ideas.repec.org/p/reg/wpaper/184.html Hallegatte, Stephane. 2014. “Economic Resilience: Definition and Measurement”. Policy Research Working Paper 6852, The World Bank. Hallegatte, Stéphane. 2009. “Strategies to Adapt to an Uncertain Climate Change.” Global Environmental Change 19 (2): 240–47. doi:10.1016/j.gloenvcha.2008.12.003. Hallegatte, Stéphane, Ankur Shah, Robert J. Lempert, Casey Brown, and Stuart Gill. 2012. Investment Decision Making under Deep Uncertainty — Application to Climate Change. Policy Working Research Paper 6193. World Bank. Harberger, Arnold C. 1978. “On the Use of Distributional Weights in Social Cost-Benefit Analysis.” The Journal of Political Economy, S87–S120. ———. 1984. “Basic Needs versus Distributional Weights in Social Cost-Benefit Analysis.” Economic Development and Cultural Change 32 (3): 455–74. doi:10.2307/1153331. 31 Hawkins, Ed, and Rowan Sutton. 2009. “The Potential to Narrow Uncertainty in Regional Climate Predictions.” Bulletin of the American Meteorological Society 90 (8): 1095–1107. doi:10.1175/2009BAMS2607.1. Henry, Claude. 1974. “Investment Decisions Under Uncertainty: The ‘Irreversibility Effect.’” American Economic Review 64 (6): 1006–12. IPCC. 2007. Climate Change 2007: The Physical Science Basis: Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge ; New York: Cambridge University Press. ———. 2013. Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge: Cambridge University Press. ———. 2014. “Working Group 2 Summary for Policymakers - Climate Change 2014: Impacts, Adaptation, and Vulnerability.” http://ipcc- wg2.gov/AR5/images/uploads/IPCC_WG2AR5_SPM_Approved.pdf Irias, Xavier J., and Denise V. Cicala. 2013. “Improving Seismic Reliability of Water Infrastructure Using Info-Gap Robustness.” In 8th WaterRF/JWWA/CTWWA Water System Seismic Conference. Oakland, CA. Kahneman, Daniel. 2011. Thinking, Fast and Slow. MacMillan. http://www.amazon.com/dp/B00555X8OA/ref=pe_245070_24466410_M1T1DP. Kahneman, Daniel, and Amos Tversky. 1979. “Prospect Theory: An Analysis of Decision under Risk.” Econometrica: Journal of the Econometric Society, 263–91. Knutson, Thomas R., John L. McBride, Johnny Chan, Kerry Emanuel, Greg Holland, Chris Landsea, Isaac Held, James P. Kossin, A. K. Srivastava, and Masato Sugi. 2010. “Tropical Cyclones and Climate Change.” Nature Geoscience 3 (3): 157–63. doi:10.1038/ngeo779. Lempert, Robert J, and Myles T Collins. 2007. “Managing the Risk of Uncertain Threshold Responses: Comparison of Robust, Optimum, and Precautionary Approaches.” Risk Analysis: An Official Publication of the Society for Risk Analysis 27 (4): 1009–26. doi:10.1111/j.1539- 6924.2007.00940.x. Lempert, Robert J., and David G. Groves. 2010. “Identifying and Evaluating Robust Adaptive Policy Responses to Climate Change for Water Management Agencies in the American West.” Technological Forecasting and Social Change 77 (6): 960–74. doi:10.1016/j.techfore.2010.04.007. Lempert, Robert J., David G. Groves, Steven W. Popper, and Steve C. Bankes. 2006. “A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios.” Management Science 52 (4): 514–28. doi:10.1287/mnsc.1050.0472. Lempert, Robert J., and Nidhi Kalra. 2011. “Managing Climate Risks in Developing Countries with Robust Decision Making”. World Resources Institute. http://www.worldresourcesreport.org/files/wrr/papers/wrr_lempert_and_kalra_uncertainty.pdf. 32 Lempert, Robert J., Nidhi Kalra, Suzanne Peyraud, Zhimin Mao, Sinh Bach Tan, Dean Cira, and Alexander Lotsch. 2013. “Ensuring Robust Flood Risk Management in Ho Chi Minh City: A Robust Decision Making Demonstration”. Policy Working Paper 6465, World Bank. http://elibrary.worldbank.org/doi/book/10.1596/1813-9450-6465 Lempert, Robert J., Steven W. Popper, and Steven C. Bankes. 2003. Shaping the next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis. Santa Monica, CA: RAND. Lempert, Robert J., Steven W. Popper, David G. Groves, Nidhi Kalra, Jordan R. Fischbach, Steven Bankes C., and Benjamin P. Bryant. 2013. “Making Good Decisions Without Predictions | RAND”. RB-9701. RAND Corporation. http://www.rand.org/pubs/research_briefs/RB9701/index1.html. Lempert, Robert J., and Michael E. Schlesinger. 2000. “Robust Strategies for Abating Climate Change.” Climatic Change 45 (3-4): 387–401. doi:10.1023/A:1005698407365. Lempert, Robert, Ryan L. Sriver, and Klaus Keller. 2012. Characterizing Uncertain Sea Level Rise Projections to Support Investment Decisions. California Energy Commission Sacramento, CA, USA. http://www3.geosc.psu.edu/~kzk10/Lempert_et_al_SLR_12.pdf. Mahan, Brent L., Stephen Polasky, and Richard M. Adams. 2000. “Valuing Urban Wetlands: A Property Price Approach.” Land Economics 76 (1): 100. doi:10.2307/3147260. Mestre, Olivier, and Stéphane Hallegatte. 2009. “Predictors of Tropical Cyclone Numbers and Extreme Hurricane Intensities over the North Atlantic Using Generalized Additive and Linear Models.” Journal of Climate 22 (3): 633–48. doi:10.1175/2008JCLI2318.1. Moody, Paul, and Casey Brown. 2012. “Modeling Stakeholder-Defined Climate Risk on the Upper Great Lakes: Modeling Climate Risk On The Upper Great Lakes.” Water Resources Research 48 (10): n/a–n/a. doi:10.1029/2012WR012497. ———. 2013. “Robustness Indicators for Evaluation under Climate Change: Application to the Upper Great Lakes.” Water Resources Research 49 (6): 3576–88. doi:10.1002/wrcr.20228. National Research Council. 2009. Informing Decisions in a Changing Climate. Washington, D.C.: The National Academies Press. Nuissier, O., B. Joly, B. Vié, and V. Ducrocq. 2012. “Uncertainty of Lateral Boundary Conditions in a Convection-Permitting Ensemble: A Strategy of Selection for Mediterranean Heavy Precipitation Events.” Nat. Hazards Earth Syst. Sci. 12 (10): 2993–3011. doi:10.5194/nhess-12-2993-2012. Pindyck, Robert S. 2001. “Optimal Timing Problems in Environmental Economics”. Working Paper. MIT Center for Energy and Environmental Policy Research. http://dspace.mit.edu/handle/1721.1/44973. ———. 2013. “Climate Change Policy: What Do the Models Tell Us?” Journal of Economic Literature 51 (3): 860–72. Popper, Steven W., Claude Berrebi, James Griffin, Thomas Light, Endy M. Daehner and Keith Crane. 2009. Natural Gas and Israel's Energy Future: Near-Term Decisions from a Strategic Perspective. 33 Santa Monica, CA: RAND Corporation. http://www.rand.org/pubs/monographs/MG927. Also available in print form. l. Price, Richard, Simeon Thornton, and Stephen Nelson. 2007. “The Social Cost of Carbon and the Shadow Price of Carbon: What They Are, and How to Use Them in Economic Appraisal in the UK.” Department for Environment Food and Rural Affairs, UK. Ranger, Nicola. 2013. “Topic Guide. Adaptation: Decision Making under Uncertainty”. Evidence on Demand. http://www.evidenceondemand.info/topic-guideadaptation-decision-making-under- uncertainty. Ranger, Nicola, Anthony Milner, Simon Dietz, Sam Frankhauser, Ana Lopez, and Ruta, Giovanni. 2010. “Adaptation in the UK: A Decision-Making Process”. Centre for Climate Change and Economics Policy. http://www.cccep.ac.uk/Publications/Policy/docs/PB-adaptationUK-ranger.pdf. Reeder, T., and Nicola Ranger. 2011. “How Do You Adapt in an Uncertain World? Lessons from the Thames Estuary 2100 Project.” World Resources Report. World Resources Institute. Renn, Ortwin. 2008. “White Paper on Risk Governance: Toward an Integrative Framework.” In Global Risk Governance, 3–73. Springer. Scandizzo, Pasquale L. 2011. “Climate Change Adaptation and Real Option Evaluation”. SSRN Scholarly Paper ID 2046955. Rochester, NY: Social Science Research Network. http://papers.ssrn.com/abstract=2046955. Schwartz, Peter. 1996. The Art of the Long View: Paths to Strategic Insight for Yourself and Your Company. St Leonards, N.S.W.: Australian Business Network. Silver, Nate. 2012. The Signal and the Noise: Why so Many Predictions Fail--but Some Don’t. New York: Penguin Press. Taleb, Nassim Nicholas. 2007. The Black Swan the Impact of the Highly Improbable. New York: Random House. Tan, Jee-Peng, Jock R. Anderson, Pedro Belli, Howard N. Barnum, and John A. Dixon. 2001. Economic Analysis of Investment Operations: Analytical Tools and Practical Applications. WBI Development Studies. Washington, D.C: World Bank. UK Department of Energy & Climate Change. 2009. “Carbon Valuation in UK Policy Appraisal: A Revised Approach.” https://www.gov.uk/government/publications/carbon-valuation-in-uk-policy- appraisal-a-revised-approach. Viguié, Vincent, and Stéphane Hallegatte. 2012. “Trade-Offs and Synergies in Urban Climate Policies.” Nature Climate Change 2 (5): 334–37. doi:10.1038/nclimate1434. Weitzman, Martin. 2009. “On Modeling and Interpreting the Economics of Catastrophic Climate Change.” Review of Economics and Statistics 91 (1): 1–19. “What Is SMART?” 2013. Accessed December 6. http://smarttunnel.com.my/smart/what-is-smart/. 34 Wolverton, Ann, Elizabeth Kopits, Chris Moore, Alex Marten, Steve Newbold, and Charles Griffiths. 2012. “The Social Cost of Carbon: Valuing Carbon Reductions in Policy Analysis.” World Bank. 2009. “Climate Change and Development.” World Development Report. The World Bank. ———. 2013. “Risk and Opportunity - Managing Risk for Development.” World Development Report. The World Bank. Zhuang, Juzhong, Zhihong Liang, Tun Lin, and Franklin De Guzman. 2007. “Theory and Practice in the Choice of Social Discount Rate for Cost-Benefit Analysis: A Survey.” http://trid.trb.org/view.aspx?id=1154582. 35