With Exit Polls: President Dukakis

From the Center for American Progress and The Century Foundation:

In this edition of Public Opinion Watch:

(covering polls and related articles from the weeks of November 8-14, 2004)

• Do the Exit Polls Indicate Voter Fraud?
• Where Did Bush's Gains Come From?
• Solving the Paradox of 2004

Do the Exit Polls Indicate Voter Fraud?

There are two lines of analysis that are typically used to justify the claim that the 2004 election result was somehow stolen by the GOP. The first is various bits and pieces of "evidence"" – the precincts in Cuyahoga County, Ohio, with more votes than registered voters; the counties in Florida (Baker, Holmes) with huge Bush margins but big Democratic registration advantages, and so on – that supposedly indicate vote tampering. I find this evidence profoundly unconvincing and think Farhad Manjoo and others have it basically right: there's not a lot of there there. Vote tampering does not appear to have happened on the scale necessary to affect this election.

The second line of analysis invokes the now-infamous early releases of the National Election Pool (NEP) exit poll data, which showed Kerry with a three-point national lead, solidly ahead in Ohio and also leading in Iowa, Nevada, and New Mexico. The reasoning, laid out most clearly in a paper, "The Unexplained Exit Poll Discrepancy," by Steven Freeman of the University of Pennsylvania, is that exit polls are very accurate surveys and highly unlikely to produce the results referred to above by chance if the real world results truly were +3 Bush, and so on. Therefore, the reasoning goes, our measurement of the real world (the actual vote counts) must be wrong and the original exit poll results right. Conclusion: there's something very funny going on with this election.

But there is a huge problem with this line of reasoning. The exit polls have always drawn samples that are off the real world results and have always had to be corrected (weighted) to eliminate bias, reflect new turnout patterns, and, in the end, just flat-out conform to the election results. This year is no different (though it is possible that the magnitude of these corrections has been greater than normal).

Here is my understanding of how the exit poll samples are weighted, based on what I have been able to ferret out so far. (No doubt, I'm not getting it entirely right, but it's damnably difficult to track down good information about this – exit pollsters have never made much effort to publicly explain and document their methods.)

1. Samples are weighted to correct for oversampling of precincts (for example, exit polls have historically selected minority precincts in some states at higher rates than other precincts) and for nonresponse bias (exit poll interviewers try to keep track of refusers by sex, race, and age).

2. Samples are weighted to correct for changing turnout patterns in the current election, since the sample design is based on past turnout behavior.

3. Samples are, in end, simply weighted to correspond to the actual election results. This is done by first weighting exit poll results in sample precincts to the true precinct results, as they are known, and then weighting the overall sample to the overall election result, once it is known.

At what point are these various weighting procedures performed? That's difficult to say because of the lack of public documentation of exit pollsters' methods. But it appears to be the case that weighting of flavors one and two takes place at least partially during the day (and continuously through the day), while the third flavor naturally has to wait until actual election results start to become available.

So where were we in this extremely complicated weighting process when those first +3 Kerry exit polls hit the CNN website? Who knows? (And exit pollsters have not exactly clarified the issue since.)