Chris Blattman

Phantom poll changes


From Benjamin Lauderdale and Doug Rivers writing at YouGov:

We believe that most of the bounces seen in surveys this year represent sampling noise that can be reduced or eliminated by adopting by better statistical methodology. We risk a repetition of 2012 where polling swings were largely statistical mirages. The convention and first debate bounces in 2012 were mostly the consequence of transitory variations in response rates. Fewer voters were changing their minds than were changing their inclination to respond to surveys.

Most telephone polls use independent samples, so the respondents in one week’s poll are different from those in another week’s. This makes it impossible to distinguish change in individual vote intentions from changes in sample composition from week to week. It is possible that five percent of the electorate switched from Clinton to Trump over the past week (decreasing Clinton’s lead by 10 points). But it’s also possible that nobody switched and apparent swings are due to differences in sample composition.

YouGov draws its samples from a large panel of respondents. In most of our polls, there is little overlap from one sample to another. However, sometimes the same respondents are recontacted to see whether their opinions have changed. For example, after the first presidential debate in September, we reinterviewed 2,132 people who had told us their vote intentions a month before. 95 percent of the September Clinton supporters said they intended to vote for her. None of them said they intended to vote for Donald Trump, but five percent said they were now undecided, would vote for a third party candidate, or would not vote. Of the Trump supporters, only 91 percent said they were still planning on voting for Trump. Five percent moved to undecided, one percent to Clinton, and the rest to third party candidates or not voting. The net effect was to increase Clinton’s lead by almost four points. That was real change, though significantly less that the ten point change to Clinton’s lead seen in some polls.

Other events, however, have not had any detectable impact on voting intentions. We did not see any shifts after the release of the Access Hollywood video, the second or third presidential debates, or the reopening of the FBI investigation into Clinton’s emails. When the same people were reinterviewed, almost all said they were supporting the same candidate they had told us they were supporting in prior interviews. The small number who did change their voting intentions shifted about evenly toward Clinton and Trump so the net real change was close to zero.

Although we didn’t find much vote switching, we did notice a different type of change: the willingness of Clinton and Trump supporters to participate in our polls varied by a significant amount depending upon what was happening at the time of the poll: when things are going badly for a candidate, their supporters tend to stop participating in polls. For example, after the release of the Access Hollywood video, Trump supporters were four percent less likely than Clinton supporters to participate in our poll. The same phenomenon occurred this weekend for Clinton supporters after the announcement of the FBI investigation: Clinton supporters responded at a three percent lower rate than Trump supporters (who could finally take a survey about a subject they liked).

The bolded italics are theirs.

7 Responses

  1. We are looking forward to connect entrepreneurs that have viable projects to invest in and those that needs loan between the ranges of US$1 Million to US$50 Billion for the maximum period of 3-10yrs.

    Ford Mortgage Services Co. Intl has over $50 billion private and corporate investment portfolios with an Excess Maximum Return Capital Profit (EMRCP) of an average of 4.6% on each private investment and portfolio. Ford Mortgage Services Co.Intl wishes to re-invest this fund by putting it into the management of entrepreneurs, corporate entity(s) and viable professionals with good business ideas that can generate at least 10% ROI per annum over a maximum period of 10years or more.Contact us:

  2. I think the polling industry is licking their wounds right now. Gallup missed badly the last time and pretty much got out of the Presidential Election prediction business. From my view point, the biggest factor in this election is how the working, blue color class were pushed by fear and helplessness to vote and hand over the entire government to the very party that is determined to destroy the very safety net the need and rely on – Social Security, Food Stamps, Medicaid, WIC etc. People are going to wake up to find that the jobs that are gone are not coming back and the very safety net that helped them make ends meet has been taken away by the people the voted for.

  3. Despite the borderline fetish with data in the past 3 political cycles, I think that most just rely on their gut when making these sorts of predictions. No on believed the polling (that turned out to be correct) that showed Trump wining the nomination yet everyone believed the polling (which turned out to be incorrect) which showed Hillary would win the presidency.

  4. If the swings in 2012 and 2016 are due to respondent composition and are not real swings, it seems like this could be an actual trend or a new normal. Any thoughts why?

    I think it’s easy to point to it being harder to reach people and you have the phone vs internet debate and how to handle each of those methods heating up. But I’m sure that’s not the entire story.

    Thinking aloud, I can come up with a few other possible influences:
    – Increased partisanship
    – The election seasons are longer and the news cycles are so focused on the election that there’s a lot of voter fatigue. So unless you’re feeling pretty strongly at that point in time, you’re not going to participate in a poll.
    – Data and polling really came onto the scene as a popular focus as far as I can tell in 2008 (but 2008 was the first election I could vote in). Could the focus on polling in the news and by candidates impact attitudes towards polls and taking polls?

    I work in corporate market research where most of these issues don’t apply, so I find the unique challenges of public opinion and political polling particularly fascinating. And I think it’s only going to get harder in the immediate future.

Comments are closed.