Political polls give indications of voter attitudes, not predictions of election outcomes, says Anthony Salvanto in his new book, Where Did You Get This Number.
Salvanto, the director of elections and surveys for CBS News, says he wrote his book to explain how polling works after skepticism arose following the 2016 presidential election that polls suggested was a lock for Hillary Clinton. She did win the popular vote, but lost in states critical to a victory in the Electoral College. The polls were right and wrong at the same time.
In an interview on Face the Nation, Salvanto said he is often asked how national poll numbers are generated based on as few as 1,000 ten-minute telephone interviews. He explains representative samples can produce reliable results. Pollsters may not interview you, but they interview people like you.
A representative sample is just part of the best practices followed by professional pollsters. Clear, objective questions must be asked. Individual questions should test a single variable. Conclusions should be tempered by statistical validity. For example, a national poll with a 1,000-respondent sample may provide a valid national picture, but not a statistically valid picture of voters in Colorado.
Even the most scrupulous professional pollsters don’t always get the numbers exactly right. There often is a slight, but significant skew as a result of the specific methodology a pollster uses. For example, failure to include a representative number of random sample calls to cell phone users could under-represent younger people, low-income families and minorities.
Nate Silver of FiveThirtyEight.com argues it is more reliable to look at groups of polls through the lens of a probability model. He claims analyzing a pool of polls and weighting each one by their history of accuracy can burp out a more accurate polling results. Even then, Salvanto would say, it is not a prediction, just a reflection in time.
Then there are the polls that aren’t really polls. Push-polls ask questions, less to get an answer and more to deliver a message, often a negative one, about a political opponent. Cheap robopolls get lower than average response rates, which can skew results. Because they are prohibited by law from calling cell phone users randomly, they have a built-in bias.
The bottom line: Purchasers need to be smart consumers of research. Before looking at results, look at the sample so you know whose views are represented in the results. Understand the methodology being used and the statistical confidence it will yield. Know the benefits and limitations of different types of research, and certainly between qualitative and quantitative research. Collaborate with a pollster on the questions that need to be asked and let him or her advise you how to ask them fairly so you get usable responses, not just what you want to hear.
Salvanto’s book may be the place to start on your journey to understanding polling’s potential and limitations.