Network News

X My Profile
View More Activity

Freakostatistics

I'm heading over to C-SPAN world headquarters later today to do a Book TV interview with Steven Levitt and Stephen Dubner, the co-authors of "Super Freakonomics." As such, I've been reading up on the controversies not only around this book, but around the series more generally. The most interesting criticism I've read comes from Rutgers sociologist Ted Goertzel, who used the original "Freakonomics" to launch a general attack on econometrics.

The acid test in statistical modeling is prediction. Prediction does not have to be perfect. If a model can predict significantly better than random guessing, it is useful. For example, if a model could predict stock prices even slightly better than random guessing, it would make its owners very wealthy. So a great deal of effort has gone into testing and evaluating models of stock prices. Unfortunately, researchers who use econometric techniques to evaluate social policies very seldom subject their models to predictive tests. Their excuse is that it takes too long for the outcomes to be known. You don’t get new data on poverty, abortion or homicide every few minutes as you do with stock prices. But researchers can do predictive testing in other ways. They can develop a model using data from one jurisdiction or time period, then use it to predict data from other times or places. But most researchers simply do not do this, or if they do the models fail and the results are never published.

The journals that publish econometric studies of public policy issues often do not require predictive testing, which shows that the editors and reviewers have low expectations for their fields. So researchers take data for a fixed period of time and keep fine tuning and adjusting their model it until they can "explain" trends that have already happened. There are always a number of ways to do this, and with modern computers it is not terribly hard to keep trying until you find something that fits. At that point, the researcher stops, writes up the findings, and sends the paper off for publication. Later, another researcher may adjust the model to obtain a different result. This fills the pages of scholarly journals, and everybody pretends not to notice that little or no progress is being made. But we are no closer to having a valid econometric model of murder rates today than we were when Isaac Ehrlich published the first model in 1975.

The scientific community does not have good procedures for acknowledging the failure of a widely used research method. Methods that are entrenched in graduate programs at leading universities and published in prestigious journals tend to be perpetuated. Many laymen assume that if a study has been published in a peer reviewed journal, it is valid. The cases we have examined show that this is not always the case. Peer review assures that established practices have been followed, but it is of little help when those practices themselves are faulty... When presented with an econometric model, consumers should insist on evidence that it can predict trends in data other than the data used to create it. Models that fail this test are junk science, no matter how complex the analysis.

I wouldn't go quite that far. Building models to fit data is a suggestive pursuit that's often presented as a definitive analysis. There's nothing wrong with being suggestive, or doing your best to explain trends. But Goertzel is right that fitting a model to the data is like tailoring a suit to the customer: It doesn't mean that model will fit all similar data any more than it means that that suit will fit all future customers.

By Ezra Klein  |  October 26, 2009; 1:03 PM ET
 
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: The end of the old boys golf club
Next: Reid to announce a public option?

Comments

Link us to the interview if you can - I would be very interested in checking it out.

I think you are being too nice about fitting models to data. It isn't that hard to separate out learning sets and test sets and then follow good practices on modeling; machine learning people do it all the time.

Posted by: Drew_Miller_Hates_IDs_That_Dont_Allow_Spaces | October 26, 2009 1:28 PM | Report abuse

Hi Ezra,

The "suggestive" tactic you find reasonable is mostly used by pop economists to make otherwise ridiculous assertions seem reasonable. By setting up euphemistic proxies for barbaric claims, economists are legitimating an old school W.A.S.P.-y worldview that estimates people to be selfish, calculating, hyperrationalists who operate only according to incentives which straight, white, upper class men perceive as valid. Economics is the business equivalent of the "states rights" argument.

It technically can be used by all groups of people, but the group of people who are constantly developing and complicating this line of argumentation are part of an insular echo chamber who (to paraphrase Lois Griffin from the family guy) see the rise of racial diversity, gender equity, and alternative sexualities as "trouble ahead."

Give 'em hell, man.

Warm Regards,
Brown Bourne
Blog: http://brownbourne.wordpress.com
Roll: http://brownbourne.wikidot.com

Posted by: osiuerer | October 26, 2009 1:38 PM | Report abuse

"It doesn't mean that model will fit all similar data any more than it means that that suit will fit all future customers."

Actually I think the more worrisome problem is something like the exact opposite. It's that many models will fit the data and there's no reason to believe one over the other. Except that no one actually looks for those alternative models because once the first one fits you publish it and move on. This is definitely one of the reasons why statistical modeling shouldn't be taken so seriously. It's a great way to generate hypotheses (they can be very suggestive, as you point out Ezra), but the real work should go in confirming the hypothesis mechanistically. Now in the social sciences this is in general impossible (unlike in the hard sciences where you can tune a knob in a completely controlled fashion and see if your prediction holds, etc.). But just because it's impossible and just because people are trying their hardest to get around that impossibility doesn't mean that we should suspend disbelief and the laws of logic and accept the fruits of this research. Sometimes, as Confucious said, half truths are more dangerous than none at all. Economists, especially, should be a bit more honest (blaring disclaimers would be a start) in talking about their findings.

Posted by: reader44 | October 26, 2009 1:39 PM | Report abuse

Thanks, reader44. A good illustration of the pitfalls of statistical modeling without any reference to underlying mechanisms is this proof that sun spots cause Republicans to be elected to the senate:

http://www.realclimate.org/index.php/archives/2007/05/fun-with-correlations/

But if you're getting published in the right journals, and even get some best sellers to boot, why worry about this silly science stuff?

Posted by: alex50 | October 26, 2009 1:53 PM | Report abuse

You should, of course, be reading (and presenting to Levitt and Dubner) Daniel Davies' multi-year review of the original book, and at very least his conclusion, describing it as "the disciplinary equivalent of the battery chicken":

"We stopped doing economics and started doing awful amateur-hour sociology, basically, because we believed that all the major problems had been solved, that some form of dynamic general equilibrium was all that there was to be said about the economy considered as a system, and that the only interesting things to do were growth theory and finance. It is no coincidence that Freakonomics began in Chicago; for a guy like Levitt who doesn't possess the engineering-maths to be a finance theorist or the empirical skills to do endogenous growth, there was literally nothing to do."

http://d-squareddigest.blogspot.com/2009/10/hell-freezes-over-yes-folks-its-last.html

Davies regards the 'Freakonomics era' as symptomatic of a complacency and decadence in economics, combined with a distaste for macro, which to some extent made it the herald of its own downfall. You can combine his critique with that of Paul Krugman on the 'Dark Age of Macroeconomics', which was the germ of his NYT Magazine piece 'How Did Economists Get It So Wrong?' in September.

Posted by: pseudonymousinnc | October 26, 2009 1:56 PM | Report abuse

Get in a baseball question or two. It's that time of year and the viewers will like it. Baseball is a lot like the stock market in that there is a ton of data generated every day and you can test your models.

Posted by: jamusco | October 26, 2009 2:06 PM | Report abuse

Cross-validation isn't the only way to check goodness of fit, and it's not appropriate choice for many datasets. It seems to me that Goertzel is complaining about the misuse of math and statistics in a field where most "peers" aren't math/stats literate enough to evaluate the work. This is sadly common in my field as well, but not really a reason for us all to only accept papers where there's a training and validation set.

Posted by: JWHamner | October 26, 2009 2:37 PM | Report abuse

I don't think it's quite that bad. It is true that papers can get published even if they predict nothing. However, their conclusions are not accepted as fact. Only if later research with new data confirms the old research do economists get convinced. This means roughly never, but at least we know how little we know.

I would consider fitting known trends to be developing a hypothesis not testing the hypothesis. That doesn't mean that such work shouldn't be published at all.

Now on abortion and murder Goertzel is totally right. There wasn't much data to fit so it was too easy. I personally glanced at a different jurisdiction -- the UK. They legalized abortion slightly earlier, but their violent crime rate increased in the 90s.

Notably, the required new cars to run on unleaded gasoline later than the USA. A competing freaky theory is that switching from leaded to unleaded (starting in 1973 in the USA) caused the decline.

In 2008 I predicted a UK crime peak in around 2008. http://tinyurl.com/yljmm87

Most recently avaible data were from 2007. So I'm off to check

(tense pause)

Sigh. According to a victimization survey there was a statistically insignificant 4% decline in violent crime in the past year (they don't use calender years so it is 2008/9 vs 2007/8 the study was published in July. Police recorded violent crime declined 6%.

I predicted about zero, so OK so far.

Posted by: rjw88 | October 26, 2009 8:07 PM | Report abuse

Just like people criticize economists for not having a good background in the side areas (like social policy, global warming) that they often delve into, much of the criticism of economists from those outside the field also suffers from not properly understanding what exactly economists and econometricians are doing. Like any other field of academics, it takes years and years of specialized study to actually understand what is happening within the field. Everyone looks a fool when they assume to know more than those dedicated to another field. Economists should stop straying outside economics and sociologists should stop pretending they perfectly understand the field of econometrics.

Posted by: nylund | November 1, 2009 5:56 PM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company