Going into last week’s parliamentary election in the UK, commentators were predicting an election so close it would precipitate a constitutional crisis: opinion polls predicted no party would have a majority in the house of commons or an obvious coalition to form one. Yet when the votes were counted, it turned out that the Conservative Party actually gained 25 seats, enough to form a government with a comfortable majority, rather than losing seats as most expected. As Nate Silver points out at 538, this marks the fourth time in the past year that pre-election polls have gotten the actual results wrong. What’s going on with pollsters that they’re missing the mark so badly?
1. Technology has made respondent bias worse
Twenty years ago, pollsters had a fairly reliable way to reach people: call them on a landline telephone. Nearly every household had a phone, and according to the Pew Center for People and the Press, 90% of them would pick up when a pollster called, with 36% completing the survey. While that still leaves a good sized non-response bias (what pollsters call the people who have the option to participate in a survey but choose not to), it’s nothing compared to what things are like today. Since that time, CallerID has become ubiquitous, so that as of 2012, phone response rates had declined to 9%, and were still trending downwards. When 90% of the population chooses not to participate, it’s pretty likely that instead of talking to the general population, you’re talking to the weirdos who like answering questions on the phone.
If CallerID made things bad, cell-phone only households, which are also on the rise, have made things even worse: 60% of 25-29 year olds live in a household without a landline phone. Calling cell phones is more expensive, more complicated legally, and has a lower response rate than traditional land line phones. According to Pew, doing good research is still possible, but it’s hard, time consuming, and expensive. Years of experience have taught me that when told that a survey can be fast, affordable, and accurate, but you only get two of the three, accuracy is usually the first to go.
Some pollsters have tried to use the internet to compensate, but internet polling has two problems. First, it tends to undercount groups with lower levels of internet access, most notably the poor and elderly. Second, internet research relies on convenience samples, which were the cause of the famous “Dewey Defeats Truman” headline. Rather than randomly dialing phone numbers, online pollsters rely on people who have volunteered to join a panel and take surveys in exchange for some sort of compensation. When that happens you’re back to the same respondent-bias you had with phones: the people who want to take surveys in exchange for Amazon gift-cards or airline points seem to be a little different than the rest of us.
2. The models behind the predictions are bad
To compensate for all the peculiarities in their samples, pollsters have developed elaborate models in order to improve the predictive power of their research. For instance, most pollsters will weight their data based on demographic comparisons to census data. But this can have unintended affects, as a former coworker found out when he applied a weighting model to a survey with only one black respondent. In other cases, pollsters will use likely voter models to help distinguish between people who answer surveys and those who will actually vote. But getting these models wrong can have significant consequences: Gallup had to completely rethink their likely voter model after they botched the 2012 US Presidential Election.
The models are also complicated by peculiarities in each country’s electoral system. In the United States, the electoral college system means that national polls are less important than local polls in a handful of swing states. Israeli election laws preclude polling [fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][howlong?] before the race, which is why most of the polls accurately predicted the left/right divide in the election, but weren’t able to account for a last minute effort by the Likud to go after votes at the expense of smaller right-wing parties. Polling each of the United Kingdom has 650 single member constituencies – to get a statistically reliable poll, you’d need to survey 1,000 people in each one. Each of these little nuances mean that polls themselves aren’t capable of predicting elections, they’re just data points that need to be added into a more complicated model.
3. The pollsters have their thumbs on the scales
Both of these challenges affect the entire polling industry. But they don’t explain some other worrying trends that we’re seeing in election polls. Most noticeably, as Nate Silver and Harry Enten point out, pollsters are deliberately herding their results together. Nate Silver explored this and found that its’ very likely that, towards the end of an electoral cycle, pollsters are selectively releasing polls to avoid embarrassment. Pollsters whose results deviated from the average held them back rather than risk embarrassment for getting it wrong. It’s easier to be wrong if everyone else is wrong too, and the benefits of being right don’t usually outweigh the risks. As troubling as this is, Silver himself is part of the reason a pollster may want to hold back: he specifically called out Gallup in 2012, leading to significant embarrassment for the organization.
In addition to avoiding embarrassment, pollsters also have a strong incentive to show a close race, which is why most of the polling tends to converge around a dead heat. Media outlets, who sponsor most polls, benefit because tight races lead to viewership, for the same reason that close games are better for sports networks. And candidates benefit because tight races are better for fundraising: nobody wants to waste money on a loser, and winners don’t seem to need it. Behind the scenes is another interesting incentive: the pollsters themselves aren’t paid for accuracy; in the United States they’re usually paid as a percentage of the media buy. The closer a race looks, the more candidates have to spend on advertising, the more the pollster makes.
Do these perverse incentives mean everyone is lying? Doubtful. But as with most other biases, it’s very likely that we’re seeing decision made at the margin – using certain models, only reporting certain polls, reporting the responses to different questions, or structuring a sample a certain way – that are independently innocuous, but that can lead certain outcomes to be reported above others. While research carries with it an air of certainty, methodological decisions made by the researcher can have a real impact on how the data comes out. Beyond deliberate manipulation, the same factors that lead to herding also make it difficult for any pollster to experiment with new forms of data collection, and that inhibits the type of innovation that would find a solution to this problem.
This inability to innovate is a real risk to pollsters, both in the public and private sectors, who are increasingly seeing their positions of authority eroded by those with new technology and new ways of seeing the world. Today, those new methodologies aren’t much better than a phone poll, and are usually worse. But unlike traditional polling, the new approaches are getting better, not worse.
Disclosures: I worked in Gallup’s consulting organization early in my career, and have since advised a number of former political pollsters on their private sector businesses.[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]