CHAPTER 3-3 USING SCIENTIFIC DISCIPLINE TO OBTAIN INFORMATION Last month, an Atlanta newspaper columnist named Lewis Grizzard came here for a visit. He wishes he hadn't. Reason: our wondrous, knowledgeable, kind, efficient taxi drivers. Grizzard took four taxi rides during this stay. The first driver took him to Union Station when he had asked to go to National Airport. The second charged Grizzard $10 to drive around in search of a restaurant that they never found. The third ran up $11 looking for the Key Bridge Marriott -- having started all of 200 yards away, in Georgetown. The fourth couldn't locate a rather well-known local landmark, the U. S. Capitol. When Grizzard got home, he wrote a column that was more like a fragmentation grenade. Conclusion: Washington is the worst taxi city in the country. ...So I thought I would go to Atlanta for a day, take as many cabs as I could between my flight in and my flight out, and see whether Grizzards's home town cabs were worse than ours. It wasn't even close. [The author goes on to describe nothing but good experiences in Atlanta.] Ladies and gentlemen, the loser, and still champion: Washington, D. C. (Bob Levey, Washington Post, Feb 16, 1987, p. C17) Anyone who has ever taken a cab from the Atlanta airport will be exceedingly reluctant to do so again. For years, it has been unusual to find a cab driver at Hartsfield [Atlanta] who could speak English, make change, or find his way around town. And the cabs themselves were, if anything, worse than the drivers. (Geoffrey Norman, "The Hustle and Hypocrisy of Andrew Young's Atlanta", The American Spectator, June, 1988, p. 24) Casual observation satisfactorily provides most of the information we need for our work and our personal lives. Is it raining? Stick your head outside the door. Can J. J. fix bicycles? Give her your bicycle and try her out. Will the flower plants survive the winter? Put them out on the porch and check when Spring comes. No special knowledge is necessary to get competent answers to questions like these. Indeed, as Chapter 3-1 noted, investigation of every question should begin with such first-hand observation -- actually looking inside the horse's mouth to check out the dental situation. But the investigation should not end with first-hand investigation in many cases. Many questions cannot be answered with casual observation. Is taxi service worse in Washington, D. C. than in Atlanta? A journalist takes a few rides in each city, and writes a column announcing that the service is worse in Washington (or in Atlanta). But he cannot base a valid conclusion on that evidence, and the column is simply amusing foolishness. To get a valid answer to this question one must take a representative sample of rides by a representative sample of customers, and the sample must be large enough to allow for the considerable variability from ride to ride; the reporter's haphazard sample of a few rides is quite inadequate. Scientific discipline, and the sampling techniques that are part of it, must be brought to bear in this case. size; see The representative sample need not be constituted perfectly, nor need it be huge. A show of hands in a class of 200 students asked whether the previous cab ride was bad would provide a much more valid estimate of the proportion of bad rides in one city, and the same one-minute survey in two cities would be sufficient for a much more satisfactory comparison, than the journalists' method. Or consider whether brushing your teeth horizontally is more effective in reducing gum disease than brushing vertically. The important effects do not occur until months or perhaps years afterwards, and the results may also be very variable from person to person. Only a carefully controlled experiment on two samples of subjects chosen randomly from the same population can provide an adequate answer. Theoretical reasoning, and even short-run observation of one or even both groups, are almost surely inadequate to provide a valid answer. When immediate observation is insufficient, and when experts and libraries do not yield the answers you need, you must turn to scientifically-disciplined research. And when I say "must" I mean that failure to use scientifically-sound methods means fooling yourself into potential difficulty, or fooling others with results that will be fraudulent at best and disastrous at worst. Please note, however, that "scientific" does not mean experimental. Astronomy and population censuses are two important examples of scientific enterprises that do not use experimentation. Experiments have great advantages when they are feasible, but when they are not feasible other methods are usually available. The failure to use scientifically-sound methods is an ever- greater affliction upon the public as newspapers and television become more important influences upon us and as our attention is increasingly upon events outside of our own immediate surroundings where we can check the situation for ourselves. The public is systematically misled about such issues as the extent of welfare abuse by immigrants, the dangers of nuclear plants and nuclear waste, and trends in the availability of natural resources and the cleanliness of our environment, among many others, because journalists apply to these issues the same techniques that their profession has used so successfully in covering warehouse fires and corrupt politicians. But these non- scientific techniques systematically provide unsound answers to the more global questions. Nowadays newspapers recognize that scientific polling methods are required for useful forecasts about the outcomes of elections; gathering opinion the old way in bars and barber shops is not enough. But in too many other cases, journalists and others barge ahead without the necessary scientific techniques. The aim of this chapter is teach you what you need to know as a consumer of scientific research in any of several roles -- as a manager who must make judgments based on the information that research yields, as a politician who must make policy decisions that require scientific background, and as a citizen who reads newspapers and watches television news and then votes. The aim is not to teach you to perform research; for that knowledge I commend you to my long book on the subject (3rd edition, Simon and Burstein, 1985). The chapter does, however, list the basic principles of scientific research that are applicable when you must create reliable knowledge on your own. As a checklist to keep in mind, and as a sequential order in which you should consider them, these principles can be useful. Violations of these same principles are much the same as the errors we make in drawing everyday conclusions, as will be discussed in Chapters 4-5 and 4-6. And many of the same principles are the converse of the logical fallacies that have been known to philosophers since the Greeks. This is a nice example of how the same principles of thinking appear in several different contexts. The most important knowledge about the scientific method is knowing when the special discipline and techniques of science are necessary. The reporter should know that scientific sampling is necessary to get a valid answer about comparative taxi service, though casual observation rather than special techniques is sufficient for covering a fire. Therefore, we proceed to discuss the characteristics of situations that indicate the need for scientific discipline, or instead indicate that casual observation should suffice. In general, scientific discipline is necessary when the chunk of the world which you wish to understand presents to you a complex, varied, off-again-on-again picture, rather than a simple, tight, immediate cause-and-effect pattern. Estimating the mortality rate of a nation requires scientific census-taking techniques whereas finding out who died in a fire does not. Learning the effects of last week's heatwave on ice cream sales needs no special methods, whereas determining whether there has been a rise in the earth's temperature, and whether the summer of 1988 was unusual due to the supposed greenhouse effect requires statistical techniques not known even to many climatologists. You can see whether there are cockroaches in your kitchen without any special equipment, but determining how many bacteria there are in the water requires a microscope. We will see that the presence of the same factors that require scientific techniques lead to the pitfalls in our everyday thinking that will be discussed in Chapters 4-5 and 4-6. Variability in the Phenomenon When every case you look at yields the same result -- when every time you hit the window with a baseball it shatters -- immediate observation tells you what you need to know. But when there is considerable variation from case to case -- when some taxi rides are nasty whereas most are fine, or when some people will vote Republican whereas others will vote Democratic -- scientific discipline is needed to assess just how much there is of one or the other. Laypersons tend to underestimate the difficulty of reaching a sound conclusion in the face of variability. Can stock-market experts forecast with any accuracy which way the market will go next week? Because many kinds of variability are involved, that very subtle statistical techniques are needed to establish that supposed experts cannot forecast better than chance, and laypersons therefore tend to believe that expert forecasting is possible. People tend to underestimate the amount of variability, and the likelihood of apparently-improbable events such as a person who has no ability to forecast the market being right ten times in a row -- though they would not be so surprised if a coin comes up heads ten times in a row. A very large proportion of television sportscasters' comments during basketball and football games is sheer nonsense deriving from not understanding the extent and nature of variability. And it is almost impossible to convince people of the operation of this variability. To convince you that there is no such thing as a slump in baseball batting, and no such thing as a hot hand in basketball, is most unlikely, indeed, based on my experiences in trying to convince classes and newspaper audiences. Yet it is so. Casual observation breaks down entirely in such matters, and the techniques of science are necessary when there is so much variability from at-bat to at- bat. Many apparent phenomena really are no more than coincidence -- heightened mortality from cancer in particular localities, for example, or from Agent Orange. Yet the coincidences seem so striking to the non-statistician, and the results are so frightening, that many people do not recognize the need for scientific techniques and will not accept the results even when scientific techniques are used correctly. The effect of a given amount of variability depends upon the circumstances. Variability in height within the two populations is sufficiently great that scientific methods are necessary to compare the average height of Americans and Yugoslavs. But the difference in the averages among Japanese and Americans is sufficiently great so that casual observation is sufficiently powerful to say with accuracy that Japanese (in this generation, at least) tend to be smaller. Large Volumes of Events and Data When there are just fifty people in a small town, you may be able to judge whether the current cohort of women are having more children than did the cohort ten years earlier. But it is not usually possible to make a valid judgment about fertility change for a large population such as the U. S. without careful measurement techniques. It is very difficult to draw a sample of a large and varied population that will be sufficiently representative. You must sample from all the sub-groups in proportion to their representation in the population. Lack of recognition of the difficulty of drawing a representative sample from the large "universe" of taxi rides -- perhaps along with lack of recognition of the problem of variability -- probably is responsible for the journalistic foolishness of drawing conclusions about the quality of rides in different cities on the basis of casual observation. Amazing blunders are made by people untrained in a field -- even though highly trained in other fields -- by not recognizing and overcoming the difficulties of representative sampling of a large set of data. For example, biologists wished to determine how much of the Amazon forest had been cut down. One of the estimates of deforestation most relied upon by biologists who forecast the denuding of tropical forests relied upon observations only from the sides of roads through the Amazon and elsewhere. No matter how clever the theoretical assumptions one makes to supplement such observations, one is most unlikely to soundly judge the state of the forests that are not near roads with such evidence. When a Check on Personal Bias is Necessary Whenever personal interests are present in a situation -- and this is most of the time -- the objectivity feature of scientific research offers protection. Fundamental to every piece of research deserving the label "scientific" is that the procedures are stated publicly in such fashion that another investigator may repeat the essentials of the research. This enables others to check upon the claims that are made by the investigator. And the objective statement of the procedures itself -- if given honestly -- enables others to judge whether the research was done in a fair and unbiased manner. This matter comes up frequently in commercial research which is intended to prove the value of a particular product. For example, a firm that develops a new drug for high blood pressure has a stake in research results that prove the drug's efficacy. If such research is to be meaningful it must be conducted and written up in a sufficiently objective manner that other investigators can repeat the work and verify the results. Multiple Causation, Hidden Causation, and Reverse Causation Two gentlepeople got drunk on successive nights with brandy and soda, whiskey and soda, and bourbon and soda. Should they have agreed that the soda must have caused their inebriations. More seriously now: most countries with high birth rates are poor. Does this mean that high birthrates cause countries to be poor? Perhaps there are other lines of causation, perhaps involving one or more other causes, such as the social system of the country. If the social system and (contrary to fact) high birthrates cause slow economic development, that would be a case of multiple causation. It is nearly impossible to separate the effects of such two possibly-causal variables from each other without scientifically-disciplined experiment or statistical analysis. Another possibility is that the social system (which had not been one of the factors you originally considered) causes both high birthrates and low income (and low growth of income); this would be a case of hidden causation. Still another possibility is that low income causes high birth rates, a case of reverse causation. In these cases, too, scientific techniques would be necessary to disentangle the causal relations. THE STEPS IN A SCIENTIFIC INVESTIGATION Here is a checklist of steps for a scientific investigation. The sequence is not rigid. In practice you often skip some of the steps --though sometimes you fall into error because you do so. And in practice you go back and forth among the steps, rather than proceeding smoothly through the sequence. 1. Make Sure The Question is Worth Studying. The best- chosen methods and the fanciest, most rigorous, and sophisticated techniques are worthless unless the problem is important. I once heard a commercial researcher put it this way: Study issues that scream, not issues that whisper. In the back of your mind should be a cost-benefit framework, such as discussed in Part I, applied to the evaluation of the research the benefits not being limited to private benefits, of course. 2. Ask: "What Do I Want to Find Out?" This is another of the intellectual devices that pops up in many different contexts in this book. When you are unclear of the direction to take, ask yourself the appropriate variation of the question: What am I trying to do? Answering this question with reasonable precision is hard work. It is easier to wave your hands around and say something vague about how "I'm interested in getting some answers about juvenile delinquency." What answers? To what questions? Do you want to know how many delinquents there are? Or whether parents' incomes affect the rate of juvenile delinquency? Or whether delinquents enjoy their delinquency? Or what? 3. Establish the Purpose of the Project. Why do you want to know the answers to the research question you are asking? How will the information be used, and by whom? Is the information about taxi rides intended to affect legislation regulating taxi service, or is it just for the amusement of newspaper readers? The purpose is likely to determine the methods, including how big a sample you take. 4. Determine the Value of the Research. If you have several projects you are considering, you would like to do the most valuable one. Again, a cost-benefit analysis is appropriate. This is another example of how cost-benefit thinking has application far beyond business. Indeed, we do cost-benefit thinking all the time, but because we do not do it explicitly we often do it poorly. 5. Saturate Yourself in the Problem. It is amazing what foolish ideas you can hold until you experience a situation at first hand. The sociologist Caudill committed himself to a mental institution to increase his intuitive understanding of the situation, though I do not recommend such heroic measures in all cases. This is an example of a principle of thinking that appears in several contexts in the book, the necessity for concrete experience. Years ago the advertising agency Cunningham and Walsh sent every writer and account executive to work in a retail store one week in every year to hear what consumers had to say at first hand, and to observe buyer reactions to the products they worked on. Unfortunately we often are too hurried or too lazy to enforce this sort of discipline upon ourselves, however. 6. Choose empirical variables. This is another stage in the sequence of getting more specific. At this point you decide what you will actually measure. Will you count the number of complaints to the taxi commission about bad rides, or the verbal comments of riders, or a sample of rides that you take yourself? 7. Calculate the benefits of accuracy and the costs of error. This is another stage in the sequence of relating your actions to their values. How much would it be worth to the public if four times as large a survey were taken and the error in the results of the taxi survey were cut in half? It is difficult to render this calculation in the money terms that are necessary to relate to the costs of increasing the sample survey, but the exercise of doing so is very valuable, just as in the case of cost-benefit analyses for non-profit firms described in Chapter 1-6. 8. Determine the most important research obstacles. The patterns of nature and social life are often obscure. The factors discussed earlier that make scientific discipline necessary constitute obstacles to getting knowledge in casual fashion, and require special devices to surmount them. For example, a randomly-drawn sample is a device to overcome the obstacle of an unrepresentative collection of taxi rides. There are also many other obstacles, such as people's unwillingness to tell the truth about delicate matters such as their incomes, and the ethical barrier against human experimentation in some drug investigations. My book on research methods catalogs these methods and discusses devices to overcome the various obstacles. 9. Choose methods. After the important obstacles have been identified, you are ready to select the methods that will surmount the obstacles most effectively. It is important to consider the widest possible range of methods even if they are not commonly used in your trade. For example, though experimentation is seldom used in mainstream economics, economists should keep this method in mind as a possibility for unusual circumstances. Often it is best to use two or more quite different methods so that one imperfect method (all methods always are imperfect) can serve as a check upon another. For example, a journalist might use records of complaints to the taxi bureau along with a survey of riders. 10. Check the ethics of your proposed research. It is easy to forget the ethical problems when doing studies such as surveys of people's private behavior. Remembering to think through the ethics can avoid a lot of grief. 11. Prepare a detailed design of the method. Now it is time to get specific and operational. It is important to identify problems before they occur. If an agronomist forgets to try variations in fertilizer as well as different seeds, there will be no way to save the experiment later. 12. Collect the data. Carefully. 13. Analyse the data. Look at the data every which way. Use as many tables as possible. You will find that designing tables and graphs is very demanding work, which is a proof of its importance in clarifying your thinking. 14. Write up the research work. Do this as early as possible, even before the data have been collected. Doing so will help you do a good job collecting the data and analysing it. Warning Here the same warning is needed as about formal cost-benefit analysis: Please do not conclude from the neat, clean look of this set of steps that most scientific thinking goes forth on this model. Skilled scientists jump back and forth from one step to the other, and you will constrain yourself in a mental straitjacket if you insist on proceeding strictly by the numbers. Indeed, the philosophy of science threw sand into the gears of science for a while when it insisted on the "hypothetico- deductive" process -- that is, first making a formal speculation, and then limiting the empirical and statistical work to the test of the original hypothesis. This is a perfect recipe for blinding yourself to new discoveries. On the other hand, working systematically down the series of steps can be extremely useful in helping you find your way out of confusion, or saving you from forgetting some important activities. SUMMARY The feature which distinguishes scientific from non- scientific knowledge-getting is not a special body of techniques, but rather the use of procedures that are disciplined instead of casual. The most important knowledge about the scientific method is knowledge of when the special discipline and techniques of science are necessary. The characteristics of a situation that call for such disciplined procedures are when the situation is complex, varied, and off-again-on-again, rather than simple, tight, and with immediate cause-and-effect. Other characteristics that call for scientific methods are the existence of possible personal bias, and multiple, hidden or reverse causation, and the presence of large volumes of events and data. method -- repeated effort building on prior work which has been carefully studied look at encycl for def small prob events, differences in make chart of methods, errors, rhetoric, cleardon't use; too specialized on journalism; see file journalism on article9 if interested.****lettr to Hoagland. This is non-trivial problem because refers to problems seen as most pressing (see ult res. survey files. Page # thinking scien33# 3-3-4d