Polls Are Not Rigged, But They Also Aren't 'Scientific'

Every four years, Americans become obsessed with The Polls. What do the polls say? Have the polls shifted? Which presidential candidate is up, and which is down? Entire careers have been built (and destroyed) by analyzing The Sacred Polls.

Savvy politicos know that not just any poll will do. Online polls, in which anybody can vote, are not legitimate. The reason is because they do not accurately reflect the voting public. Imagine, for instance, a poll on Starbucks' website asking readers if they like to drink coffee every day. In this hypothetical poll, it would not be a surprise if nearly 100% of respondents said "yes," even though only 64% of Americans drink coffee every day. The reason is obvious: People who are using Starbucks' website are probably coffee drinkers.

That's why Donald Trump's claim that he won "every poll" after the second debate earned him a "Pants on Fire" rating by PolitiFact. He was citing non-scientific polls.

So, what is a scientific poll? First, it is a misnomer. There is nothing scientific about a poll. (More on that later.) Second, it is conducted using sound statistical techniques.

Properly done, an election poll tries to determine as accurately as possible how voters will vote in an upcoming election. To do this, pollsters survey a diverse sample of people who they hope are representative of the entire population. But since this group is never a perfect representation of the population (because it may oversample some voters and undersample others), pollsters "weight" the results. If a sample has few minority voters, for example, the pollsters will inflate the answers provided by minority respondents in order to counteract the effect of undersampling. More complex polls, such as "likely voter" polls, ask a series of questions to screen out people who are unlikely to vote. 

These are tried and true methods but they are far more art than science. If calculated using wrong assumptions, weighting can skew the results. And screening voters to determine who is likely to vote is guesswork. It is sophisticated guesswork, but it is still guesswork. An analysis at UK Polling Report explains how various weights and turnout models incorrectly skewed most of the Brexit polls in favor of Remain, when in actuality Leave won by nearly 4 points.

In 2012, many Republicans thought that Mitt Romney was going to win. They believed that the 2012 electorate would not resemble the 2008 electorate, so they claimed pollsters were basing their models off incorrect assumptions. In response, some excitable partisans called Republicans "anti-science" for rejecting polling data. But that's a silly accusation. Questioning the assumptions underlying a poll is perfectly legitimate. However, the Republicans were wrong and the pollsters were right. President Obama won rather easily.

Still, as Brexit proved, skewing can happen due to faulty sampling or weighting. That is why there is no such thing as a "scientific poll." Tweaking turnout models is more akin to refining a cake recipe than doing a science experiment. Therefore, polls should be referred to as "statistically correct," not "scientific."

If it's more art than science, then why are presidential polls (particularly polling averages) correct more often than not? Like a painter or sculptor, American pollsters are rather good at their craft.