Were the Polls Wrong in 2020? And, if so, Why?

South Carolina Senator Lindsey Graham just beat his Democratic opponent 57%-42%, when the polls gave him only a 3% advantage. Graham gave a belligerent speech on November 4 attacking the press and making fun of the polls:  

“I’ve had two calls already, one from President Trump, he’s gonna win. He’s gonna win.” 

Graham added: 

“So all the pollsters out there, you have no idea what you’re doing. And to all the liberals in California and New York, you wasted a lot of money. This is the worst return on investment in the history of American politics.”

It does sound as if the 2020 polls were wildly wrong. Well, they were wrong, but most of them were not wrong as much, or in the ways, that either Graham or media talking heads seem to think. Still, in 2020, as in 2016, political opinion polls undercounted Republican voters. This is despite the fact the pollsters were on the lookout for that problem and tried to compensate statistically. 

Poll aggregator Nate Silver’s last prediction for the 2020 presidential election posited that Joe Biden would beat Donald Trump in the popular vote by about 8%. Votes are still being counted, but it seems likely that Biden will end up with about a 3% popular vote advantage over Trump, or maybe slightly less. The Blue Wave that optimistic Democrats predicted never happened. (Note that pollsters did not predict a blue wave.) As I write this, it seems probable but not certain that Biden will have more than the 270 electoral college votes that he needs to win the presidency. The Democrats will probably keep control of the House of Representatives, but will not, barring a real surprise, gain the Senate. 

Some people think that polls are a conspiracy against our republic. The truth, as always, is more complicated. National polls were less accurate than usual in two major elections in a row. Worse, they missed in the same direction, underestimating Republican turnout. Social scientists don’t like coincidences any more than TV detectives. So, let’s look more deeply. 

First, opinion polls are just estimates based on a sample. The quality of the poll depends on how good the sample is. No one is going to survey hundreds of millions of voters. Instead, a poll will survey a few thousand voters, at the most, using those results to estimate the outcome. Let’s suppose that a poll samples a thousand voters. Let’s suppose that half support Biden and half support Trump.  But do they fairly represent the population at large? Maybe yes, maybe no.

It is always possible that, by sheer bad luck, the pollsters accidentally surveyed people who are not typical of the population. After all, expecting a couple thousand people to speak for the entire nation is kind of tricky. That’s why pollsters calculate the chance that their sample was wrong. Opinion polls typically run with a 3% to 5% predicted error rate. What that means is that there is no more than a 5% chance that the poll’s findings will be within 3% to 5% of the entire population. So:

(a) A 3% to 5% error predicting the voting results isn't surprising, and

(b) There is one chance in twenty that the error will be even more than 5%.  

Since the polls predicted an 8% win for Biden, and he only won (we think) by 3%, that doesn’t prove that the polls were wrong. It might only show bad luck. Pollsters can reduce the risk of sampling error by surveying more people, but, strangely enough, that doesn’t help as much as you might think. Not unless the sample is enormous, and that causes other statistical problems. 

Second, poll aggregators like Nate Silver look at many polls, not just one. That makes up for the danger that one or two polls might be wrong. Silver gives more weight to quality polls and statewide polls. That usually helps, but there are no guarantees.  Pollsters all read the same statistics books, and they might have all made the same sampling errors.  

Third, people don’t always cooperate with polls. The election polls coincided with healthcare open enrollment, so most of us are sick and tired of harassing phone calls. I accidentally hung up on a legitimate pollster last summer, just because she called at the end of a string of robocalls. I realized my mistake a second later, but it was too late. Pollsters have massive databases that help them to compensate for uncooperative respondents, but that’s never precise.

Fourth, the exact way that a question is asked makes a big difference in how people respond. Suppose that one pollster asks, “who do you plan to vote for in the upcoming presidential election?” Another pollster asks, “do you plan to vote for Trump or Biden in the next election?” Even if they survey the same people, they won’t get exactly the same responses because they did not ask identical questions. When you hear, for example that Rasmussen tends to get more conservative results than the Washington Post poll, part of the explanation is probably slight differences in the way they ask questions. Tiny differences in the question can produce big changes in the poll’s results. 

Fifth, it matters what order questions are asked in. If a pollster asks a series of positive questions, the respondent could be in a more favorable frame of mind when asked a hard question. Again, tiny differences in the way the questions are organized can cause huge changes in poll results. Good pollsters work hard to organize their questions well, but it’s an art as much as a science. 

Sixth, people respond differently to different poll formats. In-person polls, like what Harris and Gallup used to do, are often the most accurate. But that’s an expensive way to gather information, and it would be unsafe during the pandemic. Telephone polls with a live operator are the next best, but, as I said, people often hang up on them. Robocall polls are much cheaper – and therefore can get larger sample sizes – but no one feels bad when they hang up on a computer. Again, pollsters try to compensate for that statistically, but that’s never as good as getting accurate answers. Internet polls are always suspicious. Polls that you see on Twitter or Facebook are totally useless. Polls issued by politicians are worse than useless.

Think about the obvious. Telephone polls only work if people answer their phones. Nowadays, many people screen their calls. Internet polls only work for people who have Internet access.

Seventh, people sometimes flat-out lie to pollsters. There are Internet rumors that some Trump voters are embarrassed to support him and lie to pollsters. Other rumors say some Trump voters lie to pollsters because they want to “own liberals” or “make liberals heads explode.” I don’t know whether that’s true, but pollsters have known for decades that people do not always tell them the truth. (The technical term for this is “demand characteristics,” which means that some people respond to polls socially instead of truthfully.)

Eighth, there is what statisticians call regression toward the mean. In simple terms, as election day gets close, most people shrug their shoulders and vote for the political party they’ve always voted for. No matter who the candidate is. 

Earlier Post: Trump's Polarizing Rally Was All About Getting People to Vote

Ninth, and this could be the most important problem: it’s hard to predict who is going to vote. Most polls ask whether you plan to vote, but those responses are not always right. People might feel sick or discouraged and stay home. Trump voters might get excited by a Trump boat parade and show up in force. That’s important because the #1 factor in elections is voter turnout. Few voters change their minds during the campaign. Disgruntled Republicans who switched and voted for Biden were few and far between – and vice versa. Most people choose candidates by party loyalty. But people don't always submit a ballot. Enthusiasm is hard to measure, but pollsters probably need to work harder to predict who will and will not actually vote. 

Also, voter suppression states, like Texas, where I live, or Lindsey Graham’s South Carolina, make it harder for people to vote. They make it especially hard to vote in Democratic communities. It’s possible that some people who planned to vote got discouraged and never submitted a ballot. 

Earlier Post: Kimberly Guilfoyle Tried to Drum up Enthusiasm in her 2020 RNC Speech

In decades past, presidential opinion polls rarely missed by more than 3%-4%. This year, as in 2016, they seemed to be off by 5%, or even a little more. Some statewide polls missed by more than that. Of the hundreds of polls, some came close to the mark and some missed by a country mile. 

So, overall, polltaking is not an exact science. Sampling error, errors in technique, and problems with the respondents themselves all affect the accuracy of opinion polls. Pollsters spent the last four years trying to avoid the mistakes of 2016. It appears they only partly succeeded. At the same time, the polls were – for the most part – more accurate than what many people might think. Biden and Trump mostly carried the states that the polls predicted. 

So, yes, the polls missed Graham’s landslide victory. But Graham also griped that the polls were wrong about Trump, and yet Trump is losing. In the long run, the only count that matters is the official election. 

Update: Once all the votes were counted, Biden's advantage was about 4.5%.