I'm heading over to C-SPAN world headquarters later today to do a Book TV interview with Steven Levitt and Stephen Dubner, the co-authors of "Super Freakonomics." As such, I've been reading up on the controversies not only around this book, but around the series more generally. The most interesting criticism I've read comes from Rutgers sociologist Ted Goertzel, who used the original "Freakonomics" to launch a general attack on econometrics.
The acid test in statistical modeling is prediction. Prediction does not have to be perfect. If a model can predict significantly better than random guessing, it is useful. For example, if a model could predict stock prices even slightly better than random guessing, it would make its owners very wealthy. So a great deal of effort has gone into testing and evaluating models of stock prices. Unfortunately, researchers who use econometric techniques to evaluate social policies very seldom subject their models to predictive tests. Their excuse is that it takes too long for the outcomes to be known. You don’t get new data on poverty, abortion or homicide every few minutes as you do with stock prices. But researchers can do predictive testing in other ways. They can develop a model using data from one jurisdiction or time period, then use it to predict data from other times or places. But most researchers simply do not do this, or if they do the models fail and the results are never published.
The journals that publish econometric studies of public policy issues often do not require predictive testing, which shows that the editors and reviewers have low expectations for their fields. So researchers take data for a fixed period of time and keep fine tuning and adjusting their model it until they can "explain" trends that have already happened. There are always a number of ways to do this, and with modern computers it is not terribly hard to keep trying until you find something that fits. At that point, the researcher stops, writes up the findings, and sends the paper off for publication. Later, another researcher may adjust the model to obtain a different result. This fills the pages of scholarly journals, and everybody pretends not to notice that little or no progress is being made. But we are no closer to having a valid econometric model of murder rates today than we were when Isaac Ehrlich published the first model in 1975.
The scientific community does not have good procedures for acknowledging the failure of a widely used research method. Methods that are entrenched in graduate programs at leading universities and published in prestigious journals tend to be perpetuated. Many laymen assume that if a study has been published in a peer reviewed journal, it is valid. The cases we have examined show that this is not always the case. Peer review assures that established practices have been followed, but it is of little help when those practices themselves are faulty... When presented with an econometric model, consumers should insist on evidence that it can predict trends in data other than the data used to create it. Models that fail this test are junk science, no matter how complex the analysis.
I wouldn't go quite that far. Building models to fit data is a suggestive pursuit that's often presented as a definitive analysis. There's nothing wrong with being suggestive, or doing your best to explain trends. But Goertzel is right that fitting a model to the data is like tailoring a suit to the customer: It doesn't mean that model will fit all similar data any more than it means that that suit will fit all future customers.
Posted by: Drew_Miller_Hates_IDs_That_Dont_Allow_Spaces | October 26, 2009 1:28 PM | Report abuse
Posted by: osiuerer | October 26, 2009 1:38 PM | Report abuse
Posted by: reader44 | October 26, 2009 1:39 PM | Report abuse
Posted by: alex50 | October 26, 2009 1:53 PM | Report abuse
Posted by: pseudonymousinnc | October 26, 2009 1:56 PM | Report abuse
Posted by: jamusco | October 26, 2009 2:06 PM | Report abuse
Posted by: JWHamner | October 26, 2009 2:37 PM | Report abuse
Posted by: rjw88 | October 26, 2009 8:07 PM | Report abuse
Posted by: nylund | November 1, 2009 5:56 PM | Report abuse
The comments to this entry are closed.