Nate Silver: The Polls Aren’t Wrong

In the end, it does boil down to the numbers. Right now, you can say Obama’s the favorite, but Romney is not out of it, right? I think everyone can agree with that. But what’s “not out of it”? Does that mean he has a 3 percent chance? A 5 percent chance? It’s that margin between the 5 percent and the 45 percent chance where I’m trying to make my trade.

Do we poll too much? Or pay too much attention to the results?

Maybe. I think there are definitely diminishing returns. Part of what we’ve found, as well, is that the more polls you have in any one state, the more they converge, which shouldn’t happen statistically. It reflects the fact that once there is a consensus of what’s supposed to happen, then most polls just want to stay within that consensus. Polls just start using chintzy methodology that will just calibrate to the pollsters who were good. What we find is you get to go from one poll to two, two to three, three to four, that helps a lot. After that, you really start to encounter diminishing returns.

You can have catastrophic errors from time to time. And, you know, we could have one of those this year; obviously Unskewedpolls.com thinks it might be in one direction. But if the independence of pollsters is compromised, that makes everything more difficult. I do worry sometimes that this tendency for pollsters to herd together has picked up since about 2004, when RealClearPolitics came online.

Best case: people are cheating off one another when taking the test, and you have to adjust. That’s why I started going back and evaluating pollsters on quality-related factors. Are they following industry standards for disclosure? Are they calling cell phones? Those are two basic ones. So it’s not too complicated, but that gives you some sense for who is doing polls the traditional way and who isn’t. It used to be, well, there wasn’t much difference between what the traditional polls did and what the newer ones did, so of course they got the same results. But now you do see a divergence — and we’ll see a big test of it this year.

As important as it is to predict baseball playoffs and election results, you’ve got a lot of other examples in your book of perhaps more serious predictions where the decisions made by expert analysts really matter: recessions, the housing bubble, terror attacks. And lots of these experts don’t have better results than John McLaughlin or Dick Morris.

Economic prediction is very, very tricky. When you say GDP grew by 1.7 percent, that figure has a margin of error based on the actual revisions of plus or minus three or four points worth of GDP. It’s hard to know where we are today. But in economic forecasting, you don’t seem to see much self-awareness of that. At least when we have a poll, people say here’s the margin of error. That is sometimes misinterpreted.

But economists might say that there will be 115,000 jobs created next month — and on average that figure misses by 70,000. That’s a huge miss, right? Really it’s fairly average for it to be off that much. Because in economics there is so much data, I think people feel like you can be very precise about these things. And some things you can’t; some things just aren’t predictable beyond a certain degree because they have an intrinsic complexity and they’re evolving fast. Whenever you have dynamic interactions between 300 million people and the American economy acting in really complex ways, that introduces a degree of almost chaos theory to the system, in a literal sense.