In 1950, researchers collected official data on the residents of Denver, CO to determine the percentage of people who voted, gave to charity, registered to vote and owned a library card.
Then they surveyed those residents to see if the actual data matched what people said they did:
Even though this whole project was anonymous, people still exaggerated in large numbers.
In his book Everybody Lies Seth Stephens-Davidowitz talks about why this happens:
People lie about how many drinks they had on the way home. They lie about how often they go to the gym, how much those new shoes cost, whether they read that book. They call in sick when they’re not. They say they’ll be in touch when they won’t. They say it’s not about you when it is. They say they love you when they don’t. They say they’re happy while in the dumps. They say they like women when they really like men.
People lie to friends. They lie to bosses. They lie to kids. They lie to parents. They lie to doctors. They lie to husbands. They lie to wives. They lie to themselves.
And they damn sure lie to surveys.
Sometimes we lie to others to make ourselves look good. Sometimes we lie to ourselves. Sometimes we don’t know the exact truth so we make something up. Other times there is no true incentive to tell the truth.
Stephens-Davidowitz analyzed data from the General Society Survey which is one of the authoritative sources for information on the behavior of Americans. Extrapolating data from that survey, men said they use 1.6 billion condoms every year while women claimed to use 1.1 billion. If 2.7 billion condoms every year sounds like a big number that’s because it is. Unfortunately, the actual number of condoms sold is just 600 million per year.
Which brings us to the election polls.
The analytics on the polls is now its own industry. And while it does appear the number-crunchers were right about Joe Biden winning, there were plenty of people who were upset that the numbers didn’t match their initial estimates of a blowout:
https://twitter.com/DKThomp/status/1324017800822824960?s=20
There are 3 ways to think about these results:
(1) Nate Silver and crew were wrong yet again. Why do we ever listen to these experts? They obviously have no idea what they’re talking about and the polling errors are only getting worse.
(2) No, actually you just don’t understand probabilities. High probabilities do not always indicate the certainty of an outcome occurring. The 2016 election was an anomaly and this year’s results were skewed by polling errors. And sometimes low probability events happen.
(3) Our country is far too complex for polling to be as accurate as people would like. This is not Nate Silver’s fault. Five Thirty-Eight is using the data that’s available to them. There’s simply too much margin for variance in something like this. I mean who the hell answers their phone anymore if the pollsters call?
I don’t envy the position these firms are in. Voters and pundits want answers and they want the polling analytics people to provide them a crystal clear prediction of the future.
This is impossible in part because no one knows what’s going to happen but also because you can’t place specific probabilities about the future. Life does not play out like a casino game where the house knows the exact odds.
There’s far too much variance during actual events because humans are often unpredictable.
You could say a candidate has the odds in their favor to win the election 9 out of every 10 times but we don’t get to test those odds by re-doing the election 10 times. You only get one shot at these things. No do-overs.
People are not going to ignore the polls in the future because there is no alternative. But it’s worth remembering these models always require more nuance than looking at a single number. Sometimes the polls will be wrong. Sometimes people will change their minds before they vote.
And sometimes people will lie to the pollsters because that’s what people do.
Further Reading:
No One Knows What Will Happen
