Bad Gallup! No Biscuit!

By Ruy Teixeira, The Center for American Progress and The Century Foundation. Posted October 19, 2004.

From the Center for American Progress and The Century Foundation:

In this edition of Public Opinion Watch:
(covering polls and related articles from the week October 11-17, 2004)

• Bad Gallup! No Biscuit!


• Time, Newsweek Polls Have Race Tied


• A Note on Nader


• Final Verdict on the Debates

Bad Gallup! No Biscuit!

Readers of the USA Today were treated on Monday to the following headline splashed in huge type across the front page today: "Poll: Bush leads by 8 points." The headline referred to a 52 percent to 44 percent lead that the new Gallup poll found among likely voters (LVs). The accompanying story pointed out this was quite a turnaround compared to the Gallup poll of one week earlier, which had Kerry ahead by a point among LVs.

A nine-point swing. That's pretty impressive. Of course, if you read the story closely, it does mention that Bush was ahead by just three points (49 percent to 46 percent) in their registered voter (RV) sample. And, as it turns out – though this isn't mentioned in the story – that's a shift of only three points from a week ago, when Kerry and Bush were dead-even in the RV sample.

Much less impressive. Well, which is more believable? I think this a good time to review the basic case against Gallup's LV data.

Sampling likely voters is a technique Gallup developed to measure voter sentiment on the eve of an election and predict the outcome, not to track voter sentiment weeks and months before the actual election. There is simply no evidence, and no good reason to believe, that it works well for the latter purpose. In fact, the evidence and compelling arguments are on the other side: that the registered voters are the more reliable gauge of voter sentiment during the course of the campaign.

Here's why. Gallup decides who likely voters are based on seven questions about their interest in voting, attention to the campaign and knowledge about how to vote (for example, where their polling place is located). The interested/attentive/knowledgeable voters are designated "likely" and the rest are thrown out of the sample. But as a campaign progresses, the level of interest among voters tends to change, particularly among those with partisan inclinations whose interest level will rise when their party seems to be mobilized and doing well and fall when it is not. Because of this, partisans of the mobilized party (lately, Republicans) tend to be screened into the likely voter sample and partisans of the demobilized party (lately, Democrats) tend to get screened out. But tomorrow, of course, the Democrats could surge, in which case their partisans may be the ones over-represented in likely voter samples.

That suggests the uncomfortable possibility that observed changes in the sentiments of "likely voters" represent not actual changes in voter sentiment, but rather changes in the composition of likely voter samples as political enthusiasm waxes and wanes among the different parties' supporters. And that is exactly what political scientists Robert Erikson, Costas Panagopoulos, and Christopher Wlezien find in their analysis of Gallup's 2000 RV/LV data in their forthcoming paper, "Likely (and Unlikely) Voters and the Assessment of Campaign Dynamics" in Public Opinion Quarterly: "shifts in voter classification as likely or unlikely account for more observed change in the preferences of likely voters than do actual changes in voters' candidate preferences."

That means that, instead of giving you a better picture of voter sentiment and how it is changing than conventional registered voter data, likely voter data give you a worse one since true changes in voter sentiment are swamped by changes in who is classified as a likely voter.

So, where both are available: focus on the RV data, ignore the LV data. Indeed, in my view, it's time for Gallup to drop reporting these data altogether because they are highly likely to give an inaccurate picture of the state of the race and, by doing so – especially given the high profile of Gallup's polls and how they tend to drive media coverage – unfairly pump up one side of the race and demoralize the other. That doesn't seem acceptable to me. At a minimum, Gallup and other polling organizations that use similar approaches to defining likely voters should lead with their RV data and provide the LV data as a supplement, not the other way around.

That would make a difference in how the race is covered. Based on the Gallup LV result plus a couple of other recent LV results with fairly solid Bush leads (50 percent to 44 percent among Newsweek LVs and 50 percent to 46 percent among Washington Post LVs), other media outlets had stories Sunday and Monday on how Bush was surging and even breaking the critical 50 percent barrier in voter support (see, for example, this story by Ronald Brownstein in the Los Angeles Times which prominently features the three 50+ LV results).

But here are the same three polls, with RV results (all three-way to match the data released by Gallup):

Gallup: 49 percent to 46 percent
Newsweek: 48 percent to 46 percent
Washington Post: 48 percent to 47 percent

So these three polls would then all have Bush under 50 percent (average: 48 percent) with only a one- to three-point lead. A very different picture and much dicier for Bush.

And probably much more accurate. Consider these other recent data, all of which paint a very different political picture than that implied by the Gallup LV data and the gaudy USA Today headline:

1. The Gallup RV data, while preferable to their LV data, may themselves be flawed. Their RV sample has a two-point edge for the Republicans in party identification. If that sample is reweighted to conform to the 2000 exit poll party identification distribution, Kerry leads Bush by two points, 49 percent to 47 percent.

2. A Democracy Corps survey, conducted October 14-16, the exact same dates as the Gallup survey, has Kerry up by three points, 50 percent to 47 percent. (And see this analysis by political scientist Alan Abramowitz for evidence on the superior accuracy of the DCorps survey, relative to Gallup, in the 2000 election.)

3. The Zogby and Rasmussen tracking polls both closed by four points on Sunday and Monday, right as all the media stories about Bush's surge were being written, eliminating Bush's leads in these polls. Indeed, as Jerome Armstrong points out over at MyDD, if you include leaners in their trial heats (as most reported national results do), Kerry was slightly ahead in both polls on Monday: 47.2 percent to 46.6 percent in Zogby and 49.5 percent to 47.7 percent in Rasmussen.

Doesn't sound like an eight-point Bush lead to me. Or much of a lead at all, for that matter.

Source used for this section:

Gallup poll of 1,013 adults for CNN/USA Today, released October 17, 2004 (conducted October 14-16, 2004)

****

Time, Newsweek Polls Have Race Tied

The new Time poll, conducted October 14-15, has the race tied 46 percent to 46 percent in a two-way RV matchup. That's pretty bad for an incumbent seeking reelection, but the rest of the poll has even worse news for Bush.

Start with the debates. The poll confirms that voters see Kerry as the winner of the final debate (37 percent to 28 percent), though not by the crushing margin of the first debate (59 percent to 23 percent). But when asked to consider all three presidential debates, voters do indeed see Kerry as the victor by a crushing margin, 57 percent to 27 percent.

Moreover, voters give Kerry very high marks on specific aspects of the last debate, despite the fact that they were less likely to see him as the overall winner. This presumably reflects the extent to which (positive) impressions of Kerry are settling in voters' minds.

For example, by 49 percent to 40 percent voters thought Kerry, rather than Bush, had the best understanding of the issues. That's actually better than after the first debate, when voters saw the candidates tied on this attribute.

And then there's this one: on who "took positions on issues that are closer to your own," voters gave Kerry a wide 54 percent to 39 percent margin after the last debate, compared to 48 percent to 42 percent after the first debate.

And how about this one: after the first debate, voters gave Bush a slight one-point edge on who seemed more presidential; after the last debate, voters gave Kerry the edge, 49 percent to 44 percent.

On which candidate can be trusted more on different issues and in different areas, the poll finds little change from their post-first debate poll. Kerry's gains after that debate apparently have stabilized.

Here are some of these gains, as summarized in the SRBI release on the poll:

Handling of the economy: Kerry has opened a six-point lead over Bush, 49 percent to 43 percent. Just before the first debate, the candidates were even, 44 percent for each.

Health care: Kerry has widened his lead to thirteen points, 51 percent to 38 percent. Before the debates, Kerry had an eight-point edge.

Understanding people's needs: Kerry is up by seven points, 49 percent to 42 percent. Before the debates, he was up by just four points.

Commander-in-Chief: Bush is ahead by ten points, 51 percent to 41 percent, but this has narrowed from a sixteen-point advantage before the debates.

Providing leadership in difficult times: Bush leads by eight points, 52 percent 40 percent. Before the debates, he dominated by as much as twenty-one points.

War on terrorism: Bush tops Kerry 51 percent to 40 percent, after leading Kerry by as much as eighteen points before the debates.

The poll also asked about some of the specific issues Kerry and Bush differed on in the last debate.

Assault weapons: By 73 percent to 22 percent, voters favor the ban on assault weapons; by 49 percent to 8 percent, they feel that gun control laws should be more strict, not less strict; and by 41 percent to 40 percent they say that Kerry is closer to their position on gun control than Bush.

Embryonic stem cell research: By 69 percent 22 percent voters favor using discarded embryos to conduct stem cell research; by 49 percent to 34 percent they say that Kerry is closer to their position on this issue than Bush.

Abortion: Voters say by 45 percent to 40 percent that Kerry is closer to their position than Bush on this issue.

Gay rights: Voters say by 44 percent to 41 percent that Kerry is closer to their position than Bush on this issue; by 54 percent to 41 percent they oppose amending the U.S. Constitution to ban same-sex couples from marrying.

Supreme Court appointments: By 43 percent to 38 percent, voters say the issue of Supreme Court appointments makes them more likely to vote for Kerry rather than Bush.

Over at Newsweek, I think we're finally getting to them about the potential problems with LV data. Their latest poll has Bush up by five points (50 percent to 45 percent) in their two-way LV matchup (50 percent to 44 percent in their three-way). But here's the headline and lead of their polling release:

Bush/Cheney in Dead Heat with Kerry/Edwards in Two-Way and Three-Way Matchups among Registered Voters

In a two-way matchup, the presidential race remains in a dead heat in the latest Newsweek Poll. Among registered voters, Bush/Cheney gets 48 percent and Kerry/Edwards 47 percent of the vote....

And here's the headline and lead of the polling story they posted on their website:

Too Close to Call: With the debates behind them, the contenders in the race for the White House remain locked in a dead heat in the latest NEWSWEEK poll

With just 17 days remaining in the race to the White House, President George W. Bush and Sen. John Kerry remain locked in a dead heat, according to the NEWSWEEK poll, taken after Wednesday's final debate in Arizona. In a three-way race with Independent candidate Ralph Nader, 48 percent of all voters say they would reelect Bush while 46 percent prefer Kerry....

Good job, Newsweek (and in stark contrast to the disgraceful conduct of Gallup and USA Today, discussed above)! They don't even get to their LV results until the third paragraph of the above story.

As to the data themselves – leaving aside the LV issue – there are some rather odd things about it. One is that their RV sample – where the race is close to even – shows a strong swing to the Republicans in party identification, compared to Newsweek's last survey, conducted right after the first debate.

While Newsweek rather unhelpfully only provides their (demographically) unweighted party identification distributions for RVs, one can infer from other data they provide that there has been about a six-point swing toward the Republicans in their demographically weighted party identification distributions between the two polls. That means that, if one weighted their current poll to match their previous party identification distribution (which was pretty close to that of the 2000 exits), the horse race results between the two polls would look remarkably similar: a two- to three-point Kerry lead (similar also to the party-weighted Gallup results-see above).

Newsweek provides a number of interesting subgroup horse race numbers for their three-way RV matchup that are worth taking a look at:

1. Independents favor Kerry-Edwards by eleven points, 51 percent to 40 percent.

2. First-time voters favor Kerry-Edwards by twenty-one points, 57 percent to 36 percent.

3. Early voters favor Kerry-Edwards by nine points, 52 percent to 43 percent.

4. Young voters (aged eighteen to twenty-nine) favor Kerry-Edwards by nine points, 50 percent to 40 percent, and seniors (aged sixty-five and over) favor Kerry-Edwards by fifteen, 54 percent to 39 percent.

5. Men favor Kerry-Edwards by 50 percent to 46 percent and women favor Bush-Cheney by 49 percent to 43 percent.

Huh? Boy, I had to look over those gender breakdown data several times to make sure I wasn't seeing things. And I'm still not sure they didn't somehow mislabel their categories. But if they didn't, it's certainly a head-scratcher. Perhaps they not only oversampled Republicans in general but Republican women in particular. Who knows.

It's also interesting to note that, even with a Republican-leaning sample, the poll still gives Bush only a 47 percent approval rating, basically the same as he received in their last poll. And his reelect number, at 47 percent, is also about what he received previously.

As for the rest of the poll, it generally shows Kerry and Bush with advantages where you'd expect them to have them. However, in contrast to the Time poll discussed above, Kerry's margins on given issues and attributes are smaller and Bush's larger than they were in Newsweek's previous poll. But it is difficult to assess how much of this is real movement and how much of can be attributed to the effects of a substantially more Republican sample.

Sources used for this section:

SRBI poll of 1,131 adults for Time magazine, released October 16, 2004 (conducted October 14-15, 2004)

Princeton Survey Research poll of 1,004 registered voters for Newsweek, released October 16, 2004 (conducted October 14-15, 2004)

****

A Note on Nader


A front-page story in last Friday's New York Times (and see also Ryan Lizza's piece in the new TNR) on the threat Nader poses to Kerry no doubt caused some rending of garments in Democratic circles. And there's no doubt he does pose some kind of threat – pretty much by definition even a very small Nader vote could tip a state to the Republicans if that state were close enough.

But the smaller Nader's vote, the less likely such an outcome will occur. And Nader's vote is likely to be quite small indeed. Currently, he is running at 1 percent to 2 percent in the national polls and most relevant state polls as well. Looking at analagous polls from the analagous time period in the 2000 campaign, he was running at 3 percent to 5 percent of the vote.

Take Zogby's tracking poll as an example. In 2000, Nader was polling 5 percent at this time in October. This year in the same poll, he's getting 1 percent support, very close to the support being received by Badnarik, the Libertarian candidate, in some of Zogby's polls.

So: Democrats are right to worry some about Nader. But not too much. They should worry much, much more about turnout and mobilization, because these factors are likely to be more consequential.

Final Verdict on the Debates

Here are the basic results from the immediate post-debate "snap" polls for all three debates (thanks to Kos for organizing these data, who originally posted them over at the Daily Kos). As the data show, Kerry won every single poll after all three debates and consistently did even better among uncommitted voters and independents.

Debate 1

ABC: Kerry won, 45 percent to 36 percent.

CBS, uncommitted voters: Kerry won, 44 percent to 26 percent.

CNN/USA Today Gallup: Kerry won 53 percent to 37 percent, 60 percent to 29 percent among independents.

Democracy Corps: Kerry won, 45 percent to 32 percent.

ARG: Kerry won, 51 percent to 41 percent.

Debate 2

ABC: Kerry won, 44 percent to 41 percent.

CNN/USA Today Gallup: Kerry won, 47 percent to 45 percent.

Democracy Corps: Kerry won, 45 percent to 37 percent.

Debate 3

CBS, undecideds: Kerry won, 39 percent to 25 percent. Before the debate, 29 percent said that Kerry had clear positions on the issues, after, that number doubled to 60 percent.

ABC: Kerry won, 42 percent to 41 percent in a poll that surveyed 8 percent more Republicans than Democrats. Independent voters thought Kerry won, 42 percent to 35 percent.

CNN/USA Today Gallup: Kerry wins 52 percent to 39 percent. Among independents, Kerry won 54 percent to 34 percent.

Democracy Corps: Kerry won, 41 percent to 36 percent.

These were the immediate post-debate results. And Kerry's winning margins typically widened after each debate as impressions of the debates "settled" in voters' minds. For example, after the first debate, Kerry's winning margin widened dramatically to 57 percent to 25 percent in the Gallup Poll and a crushing 61 percent to 19 percent in the Newsweek poll. After the second debate, Kerry's winning margin rose to fifteen points from Gallup's initial debate night margin of just two points. And, as reported above in the discussion of the Time poll, voters see Kerry as the overall winner of the three debates by 57 percent to 27 percent.

Not a bad performance for Mr. Kerry. Not bad at all.

              

Ruy Teixeira is a senior fellow at the Center for American Progress and The Century Foundation.

Liked this story? Get top stories in your inbox each week from AlterNet! Sign up now »