WPS6722 Policy Research Working Paper 6722 The Method of Randomization and the Role of Reasoned Intuition Kaushik Basu The World Bank Development Economics Vice Presidency Office of the Chief Economist December 2013 Policy Research Working Paper 6722 Abstract The method of randomization has been a major driver in it is not able to demonstrate generalized causality. Nor the recent rise to prominence of empirical development does it, in itself, lead to policy conclusions, as is often economics. It has helped uncover patterns and facts that claimed by its advocates. To get to policy conclusions had earlier escaped attention. But it has also given rise to requires combining the findings of randomized debate and controversy. This paper evaluates the method experiments with human intuition, which, being founded of randomization and concludes that, while the method in evolution, has innate strengths. Moreover, even non- of randomization is the gold standard for description, and randomized empirical methods combined with reasoned does uncover what is here called ‘circumstantial causality,’ intuition can help in crafting development policy. This paper is a product of the Office of the Chief Economist, Development Economics Vice Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at kbasu@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The Method of Randomization and the Role of Reasoned Intuition [Albert Hirschman Lecture, LACEA-LAMES, Mexico City, 31 October, 2013] Kaushik Basu Senior Vice President and Chief Economist, World Bank and Professor of Economics and C. Marks Professor, Cornell University Contact: 1818 H Street, NW Washington, DC 20433 Email: kbasu@worldbank.org Key words: randomization, instrumental variables, development policy, causality, intuition JEL Classification Number: B41, O20, Z18 Acknowledgements: The method of randomization has been a topic of interest to me for quite a while. During this long period and, more recently, in the course of preparing for the Albert Hirschman lecture and in writing it up as a paper, I have accumulated many debts and would like to thank, in particular, Abhijit Banerjee, Talia Bar, Alaka Basu, Karna Basu, Tito Cordella, Karla Hoff, Vamsee Krishna Kanchi, Luis-Felipe Lopez-Calva, Mattias Lundberg, Celestin Monga, Augusto de la Torre, Daniel Lederman, Nora Lustig, Dilip Mookherjee, Ted O’Donoghue, John Roemer, Vito Tanzi, and Merrell Tuck-Primdahl. Preamble I am greatly honored to be invited to give the Albert Hirschman lecture at the LACEA-LAMES conference in Mexico City on October 31, 2013. What those who invited me to deliver this lecture would not have known is that I owe a special debt to Albert Hirschman. I met Albert for the first time in early 1985 when he came to India on a visit organized by the Ford Foundation. I was familiar with his writings and after his lecture in Delhi got into a long discussion and some argument too. The next day, at a social gathering, he drew me aside and asked me if I would consider spending a year at the Institute for Advanced Study in Princeton. I answered with indecent haste. The one year, 1985-86, that I spent in Princeton opened up for me a lasting interest in political economy and related social sciences. Following Hirschman’s lead, and stealing a favorite expression of his, I learned to trespass on neighboring disciplines and have never given up on that since. My wife and I got to know him and his wife, Sarah, well and greatly appreciated their kindness and warmth during that wonderful year. I am grateful to be able to pay a tribute to Albert Hirschman with this lecture and have deliberately chosen to speak on an area of wide methodological significance to economics, especially in the context of development policymaking. The topic is the method of randomized trials and its role in discovering causal links and designing policy. I chose this topic somewhat self-consciously because there are certain things in life that are easier to do than to explain how to do. Cycling and research fall in this category. This paper is about the “how to do” of research. 1. Introduction Human knowledge owes a lot both to the ability of human beings to think and to their inability to think deep enough. This is not as paradoxical as it may sound. It implies people typically know less than they think they do. A careful examination of how we acquire knowledge, even of the most scientific nature, compels us to recognize that a lot of what we take to be knowledge is illusory. 2 The present paper is focused on one of the most important and rapidly-spreading scientific methods of knowledge acquisition in economics, that of randomized trials. The method of randomization has long been a part of the instruments of investigation in epidemiology, and it is often treated as the gold standard for making causal inferences. Randomization and probabilistic inference as a method for acquiring scientific knowledge goes back to, at least, the nineteenth century. An interesting early example of probabilistic inference occurred in studying telepathy and parapsychology. In an experiment in 1884, Charles Richet wanted to see if one person’s drawing and looking at a playing card and the information thus acquired by the person gets transmitted to others. His study showed that of the 2,927 guesses about the card, 789 were correct whereas the expected number of correct guesses if they were completely random would have been 732. This allowed him to conclude that there is a small and error-ridden but, nevertheless, positive transmission of knowledge that occurs between people’s minds. These early parapsychology studies and debates drew in some prominent thinkers including philosophers and economists, like Henry Sidgwick and Francis Ysidro Edgeworth. For a fascinating account of this early history of the subject, including some proper randomization studies, see Hacking (1988) 1. The arrival of randomization as a systematic method of investigation in economics and, in particular, development economics, is, however, relatively new. But it has arrived here with such gusto that this has become a major story of our time, giving rise to widespread use around the world by development economists, and, in equal measure, debate and discussion about its value and validity. 2 1 As must be obvious from this paragraph, what constitutes a proper randomization experiment is not beyond dispute. Even in modern development economics many variants, around the broad idea of randomization, have been tried and critically evaluated (see Bruhn and McKenzie, 2009). 2 See, for instance, Banerjee, 2005; Banerjee and Duflo, 2009, 2012; Bardhan, 2013; Barrett and Carter, 2010; Basu, 2005, 2011; Cartwright, 2010; Cartwright and Hardie, 2012; Deaton, 2009, 2010; Duflo, Glennerster and Kremer, 2008; Elbers and Gunning, 2013; Heckman and Smith, 1995; Mookherjee, 2005; Ravallion 2009; and Rodrik, 2008. 3 It will be argued here that the rise of randomization and randomized control trials (RCTs) is of immense value to economics and to the crafting of economic policy. It gives us insights into history that we did not have earlier. At the same time, many of the claims widely made on behalf of RCTs are exaggerated and invalid. It will be shown that RCTs do not give proof of any universal causality. Provided that we treat as axiomatic that the world is causal, it enables us to establish what will be here called ‘circumstantial causality.’ Some of the help that, in indirect ways, it provides to shaping policy is also provided by non-randomized methods and studies of pure correlations. What RCTs do well, and on this it is indeed the gold standard, is to describe large populations over multiple periods. Its true value lies in good description; and the worth of good description is not something to be dismissed (Sen, 1980). From this to go to policy requires the use of intuition. If we deny intuition any ground, RCTs are of no value. But the paper argues that the use of intuition is justified and, hence, RCTs do act as ‘aid’ to policymaking. Why do we make the mistake of thinking that studies based on exogenous randomizations, RCTs and even the method of carefully-selected instrumental variables give us insights into universal causality? The next three sections try to answer this question. 2. Randomization and Causality To understand why we often tend to think that RCTs demonstrate universal causal links, consider two celebrated studies that used randomization and led researchers to some striking findings--Chattopadhyay and Duflo (2004), and Miguel and Kremer (2004). Chattopadhyay and Duflo raise a pertinent question: Does having an elected woman leader to head a local government make a difference to the quality of life in the village or locality? There is, it can be shown, a positive correlation between having women leaders and the quality of provision of public goods, such as water. But, a priori, it would be reasonable to say that causality can run either way. On the one hand, it is possible that women have a 4 greater sense of social responsibility and so once elected they make sure that villagers get good water. On the other hand, it is not difficult to think of the causality running the other way around. In a society like India’s, it is traditional for women to be in charge of getting water for the household. It is arguable that if fetching water takes up a lot of time in a village, women are unlikely to have the time to run for office, and so would be unlikely to be the elected head of the local government, the panchayat. It is therefore possible to argue, a priori, that the correlation between the election of a woman leader and the better provision of public goods, such as water, reflects causality running either way. Chattopadhyay and Duflo (2004) sorted this out through a deft use of a natural randomization exercise.3 The Indian government had decreed through a constitutional amendment that in one-third of the panchayats all over India, selected by lottery, the leader must be a woman. Utilizing this random choice of women leaders, they showed that one of the a priori hypotheses, namely, that the choice of elected women leaders was caused by the better provision of water, was false. And, from this, they concluded that it was the choice of a woman leader that caused the better provision of public goods and, in particular, water. What their study did, virtually beyond question, was to rule out the hypothesis that causality ran from villages with superior provision of water to women being elected leaders. However, it never proved the causality ran in the reverse direction. That impression was created in our minds by the unwitting use of a set of prior possibilities and picking the residual after all other possibilities were ruled out by trials and experiments. To jump from this to the policy conclusion that we should elect a woman to the head the panchayat to improve village water supply would be hasty. 3 This rightly-celebrated paper has spawned other papers shedding interesting light on the status of women in some parts of India (for instance, Beamen, Chattopadhyay, Pande and Topalova, 2012; Iyer, Mishra, Topalova, 2012). 5 Similarly, consider the seminal work of Miguel and Kremer (2004). In developing countries there tends to be a correlation between the use of deworming medicine by school children and superior participation in schools by children. Why may this be so? Regular participation in school may teach the children the importance of good hygiene and, in particular, keeping oneself free of stomach worms and so they take deworming medicines. It can be that some households are smarter and smartness means being aware of the importance deworming and going to school regularly. It can also be that deworming makes children fit and fitness enables them to attend school regularly. Miguel and Kremer used an excellently designed RCT. They administered deworming medicine to children in a bunch of randomly selected schools in Kenya 4 and found that participation rates of students improved. This clearly ruled out the first two causal hypotheses. 5 It should be clear that in neither of the two studies just discussed is causality established. Based on these findings, we cannot assert that electing a woman leader in a Tamil Nadu village will improve water provision there and administering deworming medicine to all school kids in a school in Kenya in 2014 will improve school attendance. All these two RCTs do—and RCTs are ideal for this—is to rule out certain possible causal links. They give the impression of having pinned down causality only because we carry certain priors about causal links and when all but one of our priors are ruled out, our mind jumps to the conclusion that causality has been established; 6 and we feel that we are now ready to use these findings to design actual policy interventions. 7 4 In this brief statement I am, of course, leaving out important details of the experiment. 5 These are just two papers I am using for my argument. But RCTs have given us a rich haul of findings on past regularities in a range of domains in development economics: see, for instance, Ashraf, Karlan and Yin (2006), Bernard, Dercon, Orkin and Taffesse (2013), Habyarimana and Jack (2012), Jensen (2012), Hoff and Pandey (2006), Ifcher and Zarghamee (2011). 6 The reason magic surprises us has roots in a similar phenomenon of the human mind. The skillful magician instills in us certain prior beliefs about how a rabbit may have got into a hat. Then he or she demonstrates that none of these are possible and when the rabbit is pulled out of the hat, we are left wondering if this was a miracle. 7 The journey from (allegedly) finding causes to designing policy is fraught with dangers and is the subject matter of considerable writing (see, for instance, Cartwright, 2007; Basu, 2000; Das, Devarajan and Hammer, 2011). I shall return to this later. 6 To be aware of our own minds’ propensities, it is at times useful to take cognizance of the very different beliefs found in other cultures, which those cultures consider perfectly reasonable. The Gurung tribesmen of Nepal spend a lot of time climbing dangerously high trees to gather honey. When the French photographer, Eric Valli, who worked extensively among the Gurung, asked them if they do not ever fall down from their high perch, they replied: “Yes, you fall when your life is over.” (National Geographic, vol. 193, no. 6, June 1998, p.92) The above argument warns us that these “conclusions,” about causal links and policy making can be wrong. By the use of examples and a simple argument, I now show they are wrong. Suppose researchers come to a town and do a randomized control trial on the town population to check if the injection of a green chemical improves memory and has adverse side effects. And suppose it is found that it has no side effects and improves memory greatly in 95% of cases. If the study is properly done and the random draw is truly random, it is likely to be treated as an important finding and will, in all likelihood, get published in a major scientific journal. Now consider a particular woman called Eve who lives in this town and is keen to enhance her memory. Can she, on the basis of this scientific study, deduce that there is 0.95 probability that her memory will improve greatly if she takes this injection? The answer is no, because she is not a random draw of an individual from this town. All we do know from the law of large numbers is that for every randomly drawn person from this population the probability that the injection will enhance memory is 0.95. But this would not be true for a specially chosen person in the same way that this would not be true of someone chosen from another town or another time. To see this more clearly, permit me to alter the scenario in a statistically- neutral way. Suppose that what I called the town in the above example is actually 7 the Garden of Eden, which is inhabited by snakes and other similar creatures, and Eve and Adam are the only human beings in this place. Suppose now the same experiment was done in the Garden of Eden. That is, randomizers came, drew a large random sample of creatures, and administered the green injection and got the same result as described above. It works in 95% of cases. Clearly, Eve will have little confidence, on the basis of this, to expect that this treatment will work on her. I am assuming that the random draw of creatures on whom the injection was tested did not include Eve and Adam. Eve will in all likelihood flee from anyone trying to administer this injection to her, because she would have plainly seen that what the RCT demonstrates is that it works in the case of snakes and other such creatures; and the fact that she is part of the population from which the random sample was drawn is in no way pertinent. Indeed, and the importance of this will become evident later, suppose in a neighboring garden, where all living creatures happen to be humans, there was a biased-sample (meaning non-random 8) trial of this injection, and it was found that the injection does not enhance memory and, in fact, gives a throbbing headache in a large proportion of cases, it is likely that Eve would be tempted to go along with this biased-sample study done on another population rather than the RCT conducted on her own population in drawing conclusions about what the injection might do to her. There is as little hard reason for Eve to reach this conclusion as it would be for her to conclude that the RCT result in her own Garden of Eden would work on her. I am merely pointing to a propensity of the human mind whereby certain biased trials may appear more relevant to us than certain perfectly controlled ones. We shall return to this later. For now what is important to note is that even if there is a result proved by a proper randomized controlled trial conducted on a population to which you belong, you have no reason, on the basis purely of this study, to believe that its finding will apply to you. 8 For instance, the trial was done early in the morning and so, inadvertently, only early risers were chosen for the test. 8 The same reasoning carries over when one moves from a finding based on one population to another population. The fact that the choice of a woman leader of the village council in Bengal and Rajasthan led to the better provision of public goods does not mean that you have reason to expect this will be true in Bihar, Bahia or Bahrain, or, for that matter, in Bengal and Rajasthan tomorrow, because tomorrow’s population is a different one. At least, if we were as fastidious about deducing conclusions as the practitioners of randomized control trials want us to be, we could not draw those conclusions about other places and other times. That would be like drawing lessons from biased samples. This, in an interesting converse way, is related to an important problem noted by Deaton (2010). Even if we learn the facts from properly controlled randomized trials, if the intervention is then to be made not on a random draw from the population but to a specially- selected group, we should be prepared for failure. 9 3. A Limited Causality Claim The claim that can be made, but after some prior assumptions are made, is that randomized trials can establish certain ‘circumstantial’ causalities. The prior assumption that we need is to accept the determinist claim that the universe proceeds by causality and so the future that lies ahead of us is as determined as our history. To state this more precisely, assume time progresses in pixels, instead of a continuum, so that for each point of time, there is a well-defined immediate previous point of time. Let X be the set of all possible states of the world at all possible points of time. Further, X is a Cartesian product of n sets, X1, X2, … , Xn, where each of these sets consists of all possible states of some particular feature of the world. Thus X1 could consist of positions of the sun; X2 could be states of the weather, etc. Hence, at any point of time, an x ε X is a full description of the 9 A stark example of this is provided by Worrall (2007). The drug, benoxaprofen, for arthritis and muscular skeletal pain, passed the test of randomized trial very well. But when the drug was tried on actual patients it was found to cause a disproportionate number of deaths. What later became clear is that this is a drug that is not used on a random sample of people, but pre-dominantly on the older persons and they are the ones most likely to get an adverse reaction. This powerful argument is also discussed and elaborated upon by Deaton (2010). 9 state of the world10. Of course, n will be a very large number; it is conceivable that n is infinity. In its minimal form, the determinist claim says that if at two points of time, t and k, the world is in state x ε X, and at time t+1 it is in state y ε X, then at k+1 it must also be in state y. In other words, being at x causes the world to be at y in the next instant. This simple axiom is untestable but has many implications. It means, for instance, that if at two consecutive points of time the world is in the same state, then onwards it will forever remain in that state. It follows that, since the world thus far has been changing, it must have always been changing. This in itself does not mean that the world never comes back to the same state. Put simply, it is not that history never repeats itself but either history never repeats itself or it repeats itself endlessly. The determinist axiom also has large implications for our attitude to crime, responsibility and punishment, and much has been written by philosophers on this. But that is not my concern here. To see what randomized control trials show, let me define the Cartesian product of X2, X3, … , Xn by Z. What RCTs show is that there exists some z ε Z, such that if we have the world in state (x, z) ε X instead of (y, z) ε X, the world in the next period will be in some a ε X instead of b ε X. This is like saying, other things being the same (that is, z) if you vaccinate people, in the next period, there will be no influenza. But if you did not vaccinate them, there would be influenza. If, we accept the determinist axiom, as many do11, then this demonstration means that whenever we switch from (y, z) to (x, z), the world will in the next period switch from b to a. It is the ‘whenever’ that makes this a causal claim. And, this is what I am referring to as ‘circumstantial causality’. Given a certain set of circumstances, changing y to x, has a predictable consequence. The discovery of circumstantial causal connections, as has happened with the rise of RCT studies, is extremely valuable and, at the same time, of limited consequence, more so than the proponents believe. As such, the RCTs have given 10 It should be clarified that the time when a state exists is not a feature of the state and so there cannot be a reference to the temporal position of the state. 11 I personally do believe in the determinist axiom, while remaining prepared to be corrected (Basu, 2000). 10 us numerous valuable descriptions of what happened in the past and numerous instances of causes in the past (provided of course that one is willing to accept the determinist axiom). On the other hand, what they show is very limited. This is because when they show that it was the switch from y to x that caused the switch from b to a, what they are saying is that this was true under certain historical conditions (z), but they cannot tell you what those historical conditions are. RCT discoveries never graduate from something “was a cause” of something else to “is a cause.” RCTs give us no insight into universal causality because it cannot tell us what is it that was being held constant (z in the above example), when we switched some intervention b to a. For Bengal in a certain period, electing a woman leader of the local government caused water provisioning to be better. This is no guide to the future, since we do not fully know what Bengal in a certain period is like. Henceforth, a reference to causality without a qualifying epithet should be taken to be a reference to universal causality, since for policy purposes that is what is of essence. One clarification is called for here. When we hold the z constant and change y to x, in the above example, why are we saying that it is not known what is being held constant and that we simply know that something is being held constant? In the Chattopadhyay and Duflo (2004) example, for instance, that this is true for Bengal of a certain period. Why is this not a useful description? The reason is that this is a ‘dated description’—one in which the date or period of the experiment is a part of the description of what is being held constant. Since a date never recurs, we can never use this description in the future. So we will never be able to say that the same conditions now hold and so the same result is to be expected (recall the determinist axiom). What is minimally be needed is a description of what is being held constant when a particular experiment is being run without reference to a date. RCTs do not give this to us and this is what makes the findings of RCTs non-portable. Cartwright (2010) is right in her broad reservations about RCTs. However, contrary to Cartwright’s claims, I am arguing that RCTs do provide insight into circumstantial causality and is the gold standard for describing large populations 11 over time. On the other hand, they fail in demonstrating any form of universal causality. They show us that, by the use of the law of large numbers, we can describe the average characteristics of a large population and changes over time, by appropriately studying a small sample drawn from the population. RCTs do this extremely well. In other words, what I have tried to demonstrate is a critique similar to that of Cartwright (2010, 2011) but with a significant difference. On description and on historical causal connections (without being able to show what needs to be held constant for the causal link to work), RCTs are quite exemplary; but on universal causality, which is the main claim of the practitioners of the method of randomization and which is the crux of going from research to policy, RCTs get no marks. On universal causal connections that tell us how to design policy interventions the method of randomized controlled trials is silent. 12 4. Learning from Success and Failure The advantage of randomized experiments in describing populations creates an illusion of knowledge, which is not always easy to unravel. This happens because of the propensity of scientific journals to value so-called causal findings and not value findings where no (so-called) causality is found. In brief, it is arguable that we know less than we think we do. 12 Causality in science and economics has long been the source of philosophical debate and skirmish: see, for instance, Sen (1959), Hicks (1979), Hacking (1975), Skyrms (1980), Hoover (2001). In its purest form, the existence of causality simply means that if there were two worlds which were identical just before time t, they cannot be different at time t. If one buys into this as an axiom, as many prominent philosophers through the ages have done, one adopts a 'deterministic' view of life. Among economists, Amartya Sen took a similar line in one of his earliest writings—Sen (1959). I should hasten to add that though he has not directly written on this subsequently, there is some smoking-gun evidence from his other writings that his views on this have changed. It may appear at first sight that this pristine form of causality does not have any implication for what we do and how we behave and so would be dismissed at least by logical positivists as a debate of no consequence. This is however not right since whether or not one is a determinist has important implications concerning the moral responsibility of individuals for their actions, with implications for systems of justice and the punishment that is meted out to those whose actions are found unacceptable to society. 12 To see this, suppose--as is indeed the case in reality--that thousands of researchers in thousands of places are conducting experiments to reveal some causal link. Let us in particular suppose that there are numerous researchers in numerous villages doing randomized experiments to see if M causes P. Words being more transparent than symbols, let us assume they want to see if medicine (M) improves the school participation (P) of school-going children. In each village, ten randomly-selected children are administered M and the school participation rates of those children and also children who were not given M are monitored. Suppose children without M go to school half the time and are out of school the other half. The question is: Is there a systematic difference of behavior among children given M? I shall now deliberately construct an underlying model whereby there will be no causal link between M and P. Suppose Nature does the following. For each child, whether or not the child has had M, Nature tosses a coin. If it comes out tails the child does not go to school and if it comes out heads, the child goes to school regularly. Consider a village and an RCT researcher in the village. What is the probability, p, that she will find that all ten children given M will go to school regularly? The answer is clearly p = (1/2)10 since we have to get heads for each of the ten tosses for the ten children. Now consider n researchers in n villages. What is the probability that all n villages will have mixed outcomes, by which I mean, some children will participate in school and some will not (i.e., some of the ten tosses in each village will result in heads and some in tails)? Clearly, the answer is: (1 – p)n. Hence, if ϕ(n) is used to denote the probability that among the n villages where the experiment is done, there is at least one village where all ten tosses come out heads, we have: 13 ϕ(n) = 1 – (1 – p)n. Check now that if n = 1, that is, there is only one village where this experiment is done, the probability that all ten children administered M will participate in school regularly is ϕ(1) = 0.001. In other words, the likelihood is negligible. It is easy to check the following are true: ϕ(100) = 0.0931 ϕ(1,000) = 0.6236 ϕ(10,000) = 0.9999 And therein lies the catch. If the experiment is done in 100 villages, the probability that there exists at least one village in which all tosses result in heads is still very small, less than 0.1. But if there are 1,000 experimenters in 1,000 villages doing this, the probability that there will exist one village where it will be found that all ten children administered M will participate regularly in school is 0.6236. That is, it is more likely that such a village will exist than not. And, if the experiment is done in 10,000 villages, the probability of there being one village where M always leads to P is a virtual certainty (0.9999). This is, of course, a specific example. But that this problem will invariably arise follows from the fact that limn→∞ ϕ(n) = 1 – (1 – p)n = 1. Given that those who find such a compelling link between M and P will be able to publish their paper and others will not, we will get the impression that a true causal link has been found, though in this case (since we know the underlying process) we know that that is not the case. With 10,000 experiments it is close to certainty that someone will find a firm link between M and P. Hence, the finding of such a link shows nothing but the laws of probability being intact. Yet, thanks to the propensity of journals to publish the presence rather than the absence of 14 “causal” links, we get an illusion of knowledge and discovery where there are none. One practical implication of this observation is that it spells out the urgent need for a Journal of Failed Experiments or at least a publicly-available depository of such experiments so that we know how many people found a relation between M and P and how many did not. This will not solve all problems, since arguably, in each of the villages where researchers were looking for the relation between M and P, it is possible that M and P stood for different things from what others were looking for in other villages. If each underlying process entails the toss of a coin as in the above description, then we will not have evidence on the same test in many places, but different tests in different places. Nevertheless, it is important to keep track of failed experiments, in the sense of experiments that did not reveal any link between two variables. And such a journal, there can be little doubt, will have a sobering effect on economics, making evident where the presence of a result is likely to be a pure statistical artifact. 5. Knowledge, Evolution and the Induction Principle Despite the skepticism of the previous sections, the fact remains that human beings know a lot. We “know” when we release a ball in mid-air, it will move down; we “know” what time the sun will rise tomorrow; we “know” that it is a virus that is the source of the common cold. Of course, all this can be the luck of the draw, implying that all this supposed knowledge worked thus far, but will not tomorrow. This cannot be ruled out formally, but intuition suggests there is more to these bits of knowledge than pure luck. At any rate, that is the presumption under which I will work. Human beings know a lot about nature or, more minimally, our minds are somehow synchronized with nature. We tend to observe patterns in nature, where indeed there are patterns. And most of what our mind knows it does simply by picking up knowledge along the way without pausing to think if the knowledge has scientific basis and if it will stand the test of randomization. 15 A child soon learns that a person rolling over with laughter is happy, the woman weeping quietly in a corner is sad, the man with ruffled hair running toward another man with an open knife is likely to do harm. If we acted as custodians of knowledge by randomization, and stopped the child at each stage and asked the child to reject these pieces of knowledge until they had verified them by conducting RCTs, that is by making sure that randomly-selected persons rolling over with laughter were happier than others, and so on, then it is likely that the child’s cognitive abilities would be deeply damaged. If we truly paused to think, we would recognize that the proportion of knowledge that we informally absorb is substantially greater than what we know based on scientific enquiries. To dismiss the former out of hand would greatly deplete our knowledge. As the example of the Garden of Eden, above, shows, in acquiring knowledge there is scope and indeed need for a catholicity of methods, since our intuition can help fill the gaps. A recent study (Paul and Dredze, 2011) shows that there is an amazingly strong correlation between the incidence of flu in the US and the extent of chatter related to the flu on twitter. Seeing this, most of us will be prone to assuming that it is not the chatter that causes the flu. And this would seem to me to be the right conclusion, and there would be no reason to insist that this be established by conducting proper randomized trials. 13 But how do children pick up knowledge from such ascientific processes? While there is no clear answer, one incomplete answer relates to evolution. Only those human minds that are in reasonable synchrony with nature and can glean knowledge from experience have survival value. Hence, over long stretches of 13 One can go further and argue that even without correlation, the pure act of measurement facilitates understanding. Gates (2013) gives numerous examples, though without going into why measurement works. It is arguable that measurement makes it easy for our minds to process information and develop and intuitive and heuristic understanding, the role of which must not be minimized (Gigerenzer and Gaismaier, 2011; Gilovich, Griffin and Kahneman, 2002). 16 time, minds with this ability have been selected; and we, at the start of the twenty-first century, are lucky to be endowed with such ability. 14 What will be argued here is that we have no choice but to use this faculty of ours to go from experience—both statistically sound ones based on RCTs and also ones based on biased samples—to the crafting of policy. There is actually no other option. As we saw in the previous sections, RCTs do not in themselves tell us anything about the traits of populations in other places and at other times. Hence, no matter how large the population is from which we draw our random samples, since it is impossible to draw samples from tomorrow’s population and all policies we craft today are for use tomorrow, there is no “scientific” way to go from RCTs to policy. En route from evidence and experience to policy, we have to rely on intuition, common sense and judgment. 15 It is evidence coupled with intuition and judgment that gives us knowledge. To deny any role to intuition is to fall into total nihilism. 16 Some critics have pointed out that the method of experiments and random trials deflects us from the important aim of explaining “why” certain regularities in nature occur instead of merely finding such regularities. This is an important observation made by very prominent commentators in this debate (see, for instance, Deaton, 2010; Heckman and Smith, 1995) and answering to the “why” is indeed an important ingredient of human understanding. However, I would argue that as a critique of RCTs, this is invalid (see also McKenzie, 2012). To see this, let me point out, as will become evident later, that some of our so-called understanding is delusional. It is based on the mind observing regularities and, after some time, treating these regularities as facts of nature. Then when we see 14 I tried to develop this idea in Basu (2000), utilizing the prior work of Lorenz (1977). 15 As I venture into this controversial territory, it is useful to marshal some support from Leonard Savage: “I believe that here, as elsewhere, catastrophe is avoided, primarily because in practical situations commonsense generally saves all but the most pedantic of us from flagrant error.” (Savage, 1954, p. 1). 16 Total nihilism regarding knowledge is not a line to be dismissed out of hand and several prominent philosophers have taken such a line and there are probably even more philosophers who have taken such a line than we know since it is reasonable to expect that many of them would not write anything and maybe not even speak about it. 17 something new happen and say that we have understood it, it is because we have managed to fit this new observation into the partly solved jigsaw puzzle where the other parts are provided by our previous observation of regularity and our propensity to treat those as immutable parts of nature. To the extent that RCTs feed our minds with more and more observed and statistically verified regularities, they actually help with our understanding of why certain things happen, instead of deflecting us from such an understanding. So this arrow, despite having been shot by such shrewd observers, does not slay the claims made on behalf of the method of randomization. But there are other arrows. The first ingredient for going from past regularities to expectations about the future is the “induction principle.” Broadly, this says that if we see something that has happened repeatedly in the past, like the sun having risen every day, we have reason to expect this will happen in the future. We have no escape from using the induction principle in acquiring (falsifiable) knowledge. 17 That qualifier, “falsifiable,” is necessary because there are certain kinds of deductive knowledge that do not require any evidence or data. This is true of, for instance, the Pythagoras theorem on right-angled triangles, or Arrow’s Impossibility theorem on voting. The knowledge of these results does not require facts and so is independent of the principle. But, in this paper, I am keeping such non-falsifiable knowledge aside and focusing attention exclusively on knowledge which is, in principle, falsifiable. The problem with the induction principle, so essential for this kind of knowledge, is that it is inherently fuzzy; and it is possible for intelligent people not to accept it. This stems from the fact that the validity of the induction principle depends on our priors, and priors, by their very nature, cannot be proved. 18 17 Philosophers have long battled with analyzing and trying to understand induction (for instance, Mill, 1843; Bunge, 1959; Skyrms, 1840). For an historical account, see Hacking (1975). 18 Moreover, as John Stuart Mill had noted in the mid-nineteenth century (1843), the induction principle, unlike deduction, does not lead us to apodictic certainty. 18 Let us see what kinds of priors get us to the induction principle. Suppose you have the prior that nature sits with an urn containing a very large number of balls which could be (1) all white, (2) all black and (3) some white and some black. You believe that just before dawn nature picks a ball. If it is white, the sun is made to rise and if it is black, the sun is stopped from rising. Now consider your first dawn. If the sun rises, you know (2) cannot be valid. Then as the sun continues to rise every morning, you gradually begin to rule out (3) and base your expectations on (1). The more ample the evidence is, the harder your belief becomes that the sun will rise tomorrow as it has done in the past. 19 But consider a person with different priors. She believes either (2) or (3), above, is true. Now when the sun rises the first day, (2) is ruled out. Then as the sun rises each morning, her prediction with every passing day will go the other way around. She will become more and more firm in the expectation that the sun will not rise tomorrow. 20 She will reject the induction principle and there is no way to persuade her that she is wrong. It is my prior versus hers. 21 19 Russell (1912), in an essay on induction, has made this very clear with the example of the chicken trained in the art of induction who expects to have its neck patted every day only to have it wrung one fine morning. This risk will always be there but one can amass greater and greater amount of data from the past to provide strength to the inductive process. Thus we may have begun by arguing that the sun will rise tomorrow because it has done so in the past. But if we could base this on the larger body of evidence that we now have about spinning bodies continuing to spin, we will have an even longer data base to base our induction on. In some sense, this is the heart of the scientific enterprise. 20 If her prior is that the urn contains a large and finite number of balls, will not be able to compute the probability that the sun will not rise tomorrow but with will be sure with each passing day that the probability of the sun not rising will become larger. If the ball has z balls and w of those are white, then after the sun has risen for y days, her estimated probability of the sun not rising the next day will be η ≡ (z - w)/(z – y). Clearly, as y increases, so does η. 21 And there are many examples where a blind adherence to the induction principle does not work. Folklore attributes the example that follows to Galileo. Modernizing the example a little, suppose a man, rooted to the induction principle and impatient about wasting time understanding calendars, asks his neighbor each morning, “Is st today 1 November 2013?” and gets the answer “no.” This continues for days, months and years and he finally st predicts: “No morning will be 1 November 2013.” But of course he will, as I pointed out in my Albert Hirschman st lecture delivered on 31 October 2013, “get a big surprise tomorrow.” This is a light-hearted argument but cannot be dismissed out of hand. This along with the argument alerts us to the fact that science, which is so rooted in induction, should be taken with a dose of skepticism. This is my one argument with Dawkins (2006). The skepticism he expresses about religion and faith is right; but he falls into some of the same traps he warns us against when it comes to science. The human proclivity for superstition does not end with religion. 19 The induction principle cannot of its own reach definite conclusions. As Manski (2013, p. 30-31) observes, “The logic of inference does not enable any conclusions about future or hypothetical situations to be drawn based on observed tendencies per se. Assumptions are essential.” This inherent ambiguity of the induction principle and the need for tempering it with assumptions or human judgment opens us to the risk of “anything goes.” Human beings have often been subject to blind faith and strange delusions of knowledge and superstitions and large mistakes in decisions have been made and continue to be made because of this. Hence, it needs to be emphasized that the method of using the best available evidence and intuition to derive policy has to guard against this risk. Just as we have intuition, we also have reason. While, as I have tried to show, there is no getting away from the use of intuition, what we need to do is to harness reason to vet our intuition. What we have to rely on is best described as reasoned intuition. 6. Reasoned Intuition Making good policy entails the use of the best available evidence. But to do this right we need reason and that is often a stumbling block. Moreover, the fact that we should use the best available evidence does not imply that policies must invariably be based on evidence. To see this, consider the common criticism whenever someone proposes a new policy. Assume that a person recommends a new policy intervention, called X, without providing any evidence on whether or not X works. X could be a new way to make conditional cash transfers to the poor. It is common in such a situation to find critics who will oppose this policy X on the ground that there is no evidence whether or not X works. What these critics do not realize is that such a criticism is self-contradictory and thus invalid. To understand this, let “not doing X” be called “doing Y.” Clearly, if there is no evidence whether or not X works, there is no evidence whether or not Y works. If it is wrong to implement X if there is no evidence whether or not X works, it is wrong to implement Y if there is no evidence whether or not Y works. But since Y 20 is the negation of X, it is meaningless to say that we should not implement X or Y. This contradiction establishes that the initial criticism must be invalid. This is simply a cautionary tale about the ubiquity of unreason and the errors it leads us into. Of course, the best available evidence must be used, and we must learn from past patterns and regularities about what to expect in the future. What is often not recognized is that when people try to follow the widely- used dictum, “policies should be evidence based,” the hardest part of the rule is contained in the word “based.” This is the reason why we so often hear glib references to some evidence and then a hand-waving switch to one’s favored policy, ignoring the fact that the policy should be based on the best available evidence. But how do we make this transition from evidence to policy? As I argued earlier, our intuition has something innately valuable in it. This may well be an outcome of evolution. If the human intuition were completely out of line with nature and read patterns wrong, presumably it would have been weeded out by the process of natural selection. However, as the above example shows, our pure intuition and gut feeling may not be quite flawless. They need to be held under the scanner of reason before we use them to translate experience and evidence into rules of behavior and policy. I use the expression “reasoned intuition” to refer to intuition after it has been subjected to thought and scrutiny by the other faculty that we all possess, namely, reason. It is not possible to give “reasoned intuition” a hard definition. Its importance is founded on the realization that our intuitions are not flawless. Major blunders have been committed in the past by humankind on the basis of intuition and a refusal to subject intuition to scrutiny. Fortunately, we also possess the ability to reason. What I am arguing is that by bringing these two faculties—intuition and reason—together, we can greatly improve our ability to make policies. Is my intuition consistent with the evidence we have and the other things I already know? Is it consistent with the other intuitive beliefs I hold? Reasoned intuition is intuition that has been subjected to this kind of interrogation. 21 Indeed, a large part of the present paper tried to show that there is no getting away from the use of reasoned intuition. Empirical studies, no matter how carefully done, cannot in themselves lead us to particular policies. Going from evidence, data, and statistics to policy will invariably entail a leap of imagination. This is facilitated by our intuition, honed over millennia by the processes of evolution. But we must stand ready to subject the intuition to reason. This cannot be stated as a hard rule; but to ignore it on that ground would be a mistake. I can illustrate some kinds of common mistakes that our intuition often leads us to. The remedy is not to discard the use of intuition but to pause and check that it stands the test of reason. One empirical finding which has been at the base of a lot of development policy discussion is the role of growth in mitigating poverty. Dollar, Kleineberg and Kraay (2013) have recently shown, using a remarkably comprehensive data set, spanning multiple countries and several decades, that the bulk of poverty reduction that the world has seen in the past was due to overall GDP growth. Intuition then prompts us to propose that we should therefore rely on growth rather than specially designed market interventions to battle poverty. For the untutored, the growth v. poverty debate is an emotive one and this observation quickly gathers support among those who have a predilection to leave it all to the market. Without going into this larger debate here (see Basu, 2013, for such a discussion), let me simply point to an obvious mistake that leads us to make this policy conclusion from this robust empirical finding. The fact that 75% of the drop in poverty in the past occurred because of growth in no way shows that growth is more effective in eradicating poverty than say a new conditional cash transfer program. The mistake in making that deduction would be the same as someone studying infections in the 1930s asserting that we should rely on non-penicillin medicines because past data shows that 99% of all cures were because of non-penicillin drugs. This ignores the fact that penicillin was discovered in 1928 and so the lack of evidence of the success of penicillin is not a sign of penicillin not working but of penicillin not existing. 22 In other words, if we want to use the induction principle to decide if policy X is likely to achieve a certain result R, it is not enough to show that R was achieved in the past because of Y, and from this conclude X will not work. It is important to check if X was tried in the past and then make the deduction. In case X never achieved R because X was never tried, we do not have evidence on the efficacy of X one way or the other. To assume otherwise is to make the same mistake as the one discussed at the start of this section, whereby the lack of evidence in favor and against a new policy is taken to be a verdict against the new policy. As was pointed out earlier, the induction principle does not have a precise definition; but, upon cogitation, we can nevertheless conclude that supposedly inductive conclusions are wrong. It is this which makes the inductive principle useful. The mistake about growth and poverty just discussed is the same one that we so often hear being used for job creation. People often assert that we have to rely on the private sector for job creation because past data show that 80% of jobs were created by the private sector. If this logic were correct, we would also have to accept the logic of an economist in Soviet Russia in 1980 arguing that we have to rely on the state for job creation because past data show that 90% of all jobs were created by the state. These are all examples of using our intuition wrongly to go from induction to certain policy prescriptions. Since the induction principle is not one that can be formally defined, mistakes are also not always possible to formally define. Yet, if we pause to reason carefully or listen to someone else provide the reasoning, we would, on our own, back off from many a wrong policy. This is what reasoned intuition is all about. What is the role of theory in all this? It is common to pay lip service to this and assert that we need both theory and data to make policy prescriptions. There are many instances where we do not need any theory. Spotting regularities in data, coupled with reasoned intuition, can take us to useful policy prescriptions. 23 There can be no need for theory. Likewise, there are areas where theory, coupled with what we intuitively know, can take us to useful policy decisions. The real role of theory is to make consistency checks on our intuitive beliefs22 and, based on our intuitive beliefs, to take steps toward a deeper understanding using pure deduction. Can all the beliefs we hold be together valid? Theory is a method for checking this. It also helps us take deductive steps in conjunction with what we know, whether through formal empirical studies or simply through the informal acquisition of knowledge. The role of theory and empirics is best understood with an analogy with Sudoku. Suppose we have a slightly different Sudoku game in which the numbers that are typically entered in the squares to start with are not shown explicitly but have to be found out using clues (“look under a pillow for the number that goes into the left-hand square). The search for these preexisting but hidden numbers is akin to empirical investigation. However, once we have found out all the numbers specified in advance, the rest of the Sudoku proceeds by pure reasoning. All the other numbers have to be deduced rather than found. This is akin to what theory does. It enables us to move on using the given facts and by using pure deduction. 7. Concluding Remarks The rise of the method of randomization has been a major stimulus to empirical development economics and as such it is a welcome advance. Akin to what this method did earlier for epidemiology, it has in its short life in economics brought to light several new facts that were earlier languishing unseen. This deserves credit and further use. However, the claims made by its proponents concerning its ability to reveal causal links that directly lead to policy conclusions are wrong. That is what this paper tried to argue. RCTs have no advantage on these scores. What RCTs do usefully and well is to describe. They give us ways to describe static features and also temporal features (M happened in period 1 for 22 As Rubinstein (2012, p. 129) observes, “My view of game theory is consistent with my approach to economic models in general … . Game theory does not try to describe reality or be normative. Game theory investigates the logic of strategic thinking.” 24 some members of the population and P happened in period 2 for 90% of those to whom M happened and 10% of those to whom M did not happen) with no clues to universal causality. They can reveal circumstantial causality, namely, that, in certain historical circumstances, doing x instead of y, led to a instead of b, in the next period, but since we do not have a full description of the historical circumstances, there is, strictly, no lesson in this which is portable through time. Universal causality is a mental construct for observers, something that helps them apply intuition and judgment, in conjunction with what they know or think they know, to decide how to behave and which policies to choose. What human beings know comes from many sources, and to deem only one method valid and all others invalid is to slow the process of knowledge acquisition. The catholicity of methods currently used—from anthropological notes, analysis of large data sets, everyday experience and randomized trials—all have a role to play in this enterprise. When it comes to seeking good descriptions of past facts, RCTs constitute the best method to aspire to. When it comes to the search for universal causality and best policy, there is a role for multiple sources, including reasoned intuition, honed over millennia of human evolution. And all this should be tempered with a shot of skepticism—an awareness that for all our best efforts, we may be wrong. 25 References Ashraf, Nava, Karlan, Dean and Yin, Wesley (2006), ‘Tying Odysseus to the Mast: Evidence from a Commitment Savings Product in the Philippines,’ Quarterly Journal of Economics, vol. 121. Banerjee, Abhijit (2005), ‘‘New Development Economics’ and the Challenge to Theory,’ Economic and Political Weekly, October 1. Banerjee, Abhijit and Duflo, Esther (2009), ‘The Experimental Approach to Development Economics,’ Annual Review of Economics, vol. 1. Banerjee, Abhijit and Duflo, Esther (2012), Poor Economics: A Radical Rethinking of the Way to Fight Poverty, Public Affairs. Bardhan, Pranab (2013), ‘Two Ideas about Fighting Global Poverty,’ Boston Review, May/June. Barrett, Christopher and Carter, Michael (2010), ‘The Power and Pitfalls of Experiments in Development Economics: Some Non-Random Reflections,’ Applied Economic Perspectives and Policy, vol. 32. Basu, Kaushik (2000), Prelude to Political Economy: A Study of the Social and Political Foundations of Economics, Oxford University Press, Oxford. Basu, Kaushik (2005), ‘New Empirical Development Economics: Remarks on Its Philosophical Foundations,’ Economic and Political Weekly, vol. 40, October 1. Basu, Kaushik (2011), Beyond the Invisible Hand: Groundwork for a New Economics, Princeton University Press, Princeton, and Penguin, New Delhi. Basu, Kaushik (2013), ‘Shared Prosperity and the Mitigation of Poverty: In Practice and in Precept,’ World Bank Working Paper. http://documents.worldbank.org/curated/en/2013/11/18506691/shared- prosperity-mitigation-poverty-practice-precept Beamen, Lori, Chattopadhyay, Raghav, Duflo, Esther, Pande, Rohini and Topalova, Petia (2012), ‘Powerful Women: Does Exposure Reduce Bias?’ Quarterly Journal of Economics, vol. 124. Bernard, Tanguy, Dercon, Stefan, Orkin, Katie, and Taffesse, Alemayehu (2013), 26 ‘Learning with Others: A Field Experiment on the Formations of Aspirations in Ethiopia,’ mimeo: Oxford University. Bruhn, Miriam and McKenzie, David (2009), ‘In Pursuit of Balance: Randomization in Practice in Development Field Experiments,’ American Economic Journal: Applied Economics, vol. 9. Bunge, Mario (1959), Causality and Modern Science, 4th Edition, 2009, Transaction Publishers, New Brunswick. Cartwright, Nancy (2007), Hunting Causes and Using Them: Approaches to Philosophy and Economics, Cambridge University Press, Cambridge. Cartwright, Nancy (2010), ‘What Are Randomized Trials Good For?’ Philosophical Studies, vol. 147. Cartwright, Nancy (2011), ‘The Art of Medicine: A Philosopher’s View of the Long Road from RCTs to Effectiveness,’ The Lancet, vol. 377, April 23. Cartwright, Nancy and Hardie, Jeremy (2012), Evidence Based Policy: A Practical Guide to Doing It Better, Oxford University Press, Oxford. Chattopadhyay, Raghabendra and Duflo, Esther (2004), “Women as Policy Makers: Evidence from a Randomized Policy Experiment in India,” Econometrica, Vol. 72(5). Das, Jishnu, Devarajan, Shanta, and Hammer, Jeffrey (2011), ‘The Welfare Economics of Behavioral Economics and Randomized Trials,’ World Bank Blog. http://blogs.worldbank.org/files/africacan/DDH_The%20Welfare%20Econo mics%20of%20Behavioral%20Economics%20and%20Randomized%20Contr ol%20Trials(1).pdf Dawkins, Richard (2006) The God Delusion, Bantam Books, London. Deaton, Angus (2009) “Instruments of development: Randomization in the tropics and the search for the elusive keys to development,” Keynes Lecture, British Academy, Proceedings of the British Academy 2008 Lectures, 162, Oxford University Press, Oxford. Deaton, Angus (2010), ‘Instruments, Randomization, and Learning about Development,’ Journal of Economic Literature, vol. 48. 27 Dollar, David, Tatjana Kleineberg and Kraay, Aart (2013) “Growth Still Is Good for the Poor,” World Bank Policy Research Working Paper 6568. World Bank: Washington D.C. Duflo, Esther, Glennerster, Rachael and Kremer, Michael (2008), ‘Using Randomization in Development Economics Research: A Toolkit,’ in T. Paul Schultz and J. Strauss (eds.), Handbook of Development Economics, vol. 4, Elsevier. Elbers, Chris and Gunning, Jan (2013), ‘Evaluation of Development Programs: Randomized Controlled Trials or Regressions?’ World Bank Economic Review, vol. 27. Gates, Bill (2013), ‘Measuring Progress,’ in Annual Letter, Gates Foundation. http://annualletter.gatesfoundation.org/#nav=section1&slide=0 Gilovich, Tom, Griffin, Dale and Kahneman , Daniel (2002), Heuristics and Biases: The Psychology of Intuitive Judgment, Cambridge University Press, Cambridge. Gigerenzer, Gerd and Gaissmaier, Wolfgang (2011), ‘Heuristic Decision Making,’ Annual Review of Psychology, vol. 62. Habriyamana, James and Jack, William (2012), ‘State vs. Consumer Regulation: An Evaluation of Two Road Safety Interventions in Kenya,’ NBER Working Paper. Hacking, Ian (1975), The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference, Cambridge University Press. Hacking, Ian (1988), ‘Telepathy: Origins of Randomization in Experimental Design,’ Isis, vol. 79, No. 3. Heckman, James and Smith, Jeffrey (1995), ‘Assessing the Case for Social Experiments,’ Journal of Economic Perspectives, vol. 9. Hicks, John (1979), Causality in Economics, Clarendon Press, Oxford. Hoff, Karla, and Pandey, Priyanka (2006), ‘Discrimination, Social Identity, and Durable Inequalities,’ American Economic Review, vol. 96. Hoover, Kevin (2001), Causality in Macroeconomics, Cambridge University Press, Cambridge. Ifcher, John and Zarghamee, Homa, (2011), ‘Happiness and Time Preference: The 28 Effect of Positive Affect in a Random Assignment Experiment,’ American Economic Review, vol. 101. Iyer, Lakshmi, Mani, Anandi, Mishra, Prachi and Topalova, Petia (2012), ‘The Power of Political Voice: Political Representation and Crimes against Women in India,’ American Economic Journal: Applied Economics, vol. 3. Jensen, Robert (2012), ‘Do Labor Market Opportunities Affect Young Women’s Work and Family Decisions?’ Quarterly Journal of Economics, vol. 127. Lorenz, K. (1977), Behind the Mirror: A Search for the Natural History of Human Knowledge, (translated from German by R. Taylor), Harcourt Brace and Jovanovich, New York. Manski, Charles (2013), Public Policy in an Uncertain World, Harvard University Press, Cambridge, MA. McKenzie, David (2012), ‘Does Angus Deaton Worry Too Much about Wolves Eating His t-Values?’ World Bank Blog. http://blogs.worldbank.org/impactevaluations/does-angus-deaton-worry- too-much-about-wolves-eating-his-t-values Miguel, E. and Kremer, M. (2004), ‘Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,’ Econometrica, vol.72. Mill, John Stuart (1843), A System of Logic, Ratiocinative and Inductive. Cambridge Books Online edition 2011. Mookherjee, Dilip (2005), ‘Is There Too Little Theory in Development Economics Today? Economic and Political Weekly, vol. 40, October 1. Paul, Michael and Dredze, Mark (2011), ‘You Are What You Tweet: Analyzing Twitter for Public Health,’ Association for the Advancement of Artificial Intelligence. Ravallion, Martin (2009), ‘Should the Randomistas Rule?’ Economists’ Voice. http://siteresources.worldbank.org/INTPOVRES/Resources/477227- 1142020443961/2311843- 1229023430572/Should_the_randomistas_rule.pdf Richet, Charles (1884), ‘La suggestion mentale et le calcul des probabilities,’ Revue philosophique de la France et de l’etranger, vol. 18. Rodrik, Dani (2008), ‘The New Development Economics: We Shall Experiment but How Shall We Learn?’ in J. Cohen and W. Easterly (eds.), What Works in 29 Development? Thinking Big and Thinking Small, Brookings Institution Press, Washington, DC, 2009. Rubinstein, Ariel (2012), Economic Fables, Open Book Publishers. Russell, Bertrand (1912), The Problems of Philosophy, Re-published by Project Gutenberg, 2009. Savage, Leonard (1954), The Foundations of Statistics, John Wiley & Sons. (Page references are to the revised Dover publication, 1972.) Sen, Amartya (1959), ‘Determinism and Historical Predictions,’ Enquiry, vol. 2. Sen, Amartya (1980), ‘Description as Choice,’ Oxford Economic Papers Skyrms, Brian (1980), Causal Necessity, Yale University Press, New Haven. Worrall, John (2007), ‘Evidence in Medicine and Evidence-Based Medicine,’ Philosophy Compass, vol. 2/6. 30