unbiased questions

New Book Says Polls Provide Indications, Not Predictions

Anthony Salvanto with CBS News has written a new book that explains what polling can and can’t do. It’s a good place to begin to become a savvy research consumer.

Anthony Salvanto with CBS News has written a new book that explains what polling can and can’t do. It’s a good place to begin to become a savvy research consumer.

Political polls give indications of voter attitudes, not predictions of election outcomes, says Anthony Salvanto in his new book, Where Did You Get This Number.

Salvanto, the director of elections and surveys for CBS News, says he wrote his book to explain how polling works after skepticism arose following the 2016 presidential election that polls suggested was a lock for Hillary Clinton. She did win the popular vote, but lost in states critical to a victory in the Electoral College. The polls were right and wrong at the same time.

In an interview on Face the Nation, Salvanto said he is often asked how national poll numbers are generated based on as few as 1,000 ten-minute telephone interviews. He explains representative samples can produce reliable results. Pollsters may not interview you, but they interview people like you.

A representative sample is just part of the best practices followed by professional pollsters. Clear, objective questions must be asked. Individual questions should test a single variable. Conclusions should be tempered by statistical validity. For example, a national poll with a 1,000-respondent sample may provide a valid national picture, but not a statistically valid picture of voters in Colorado.

Even the most scrupulous professional pollsters don’t always get the numbers exactly right. There often is a slight, but significant skew as a result of the specific methodology a pollster uses. For example,  failure to include a representative number of random sample calls to cell phone users could under-represent younger people, low-income families and minorities.

Nate Silver of FiveThirtyEight.com argues it is more reliable to look at groups of polls through the lens of a probability model.  He claims analyzing a pool of polls and weighting each one by their history of accuracy can burp out a more accurate polling results. Even then, Salvanto would say, it is not a prediction, just a reflection in time.

Then there are the polls that aren’t really polls. Push-polls ask questions, less to get an answer and more to deliver a message, often a negative one, about a political opponent. Cheap robopolls get lower than average response rates, which can skew results. Because they are prohibited by law from calling cell phone users randomly, they have a built-in bias.

The bottom line: Purchasers need to be smart consumers of research. Before looking at results, look at the sample so you know whose views are represented in the results. Understand the methodology being used and the statistical confidence it will yield. Know the benefits and limitations of different types of research, and certainly between qualitative and quantitative research. Collaborate with a pollster on the questions that need to be asked and let him or her advise you how to ask them fairly so you get usable responses, not just what you want to hear.

Salvanto’s book may be the place to start on your journey to understanding polling’s potential and limitations.


The Survey, the Sample and the Question

Political season drives home the point of how many decisions are made based on polling. It's a great time to reflect on what separates good polling from bad information.

The keys to getting accurate and usable research data are:

  • Using the right survey or combination of survey techniques;

  • Interviewing a representative sample of your target audience; and

  • Asking direct, fair and single-topic questions.

Research breaks down into quantitative and qualitative techniques. Quantitative research involves enough interviews to produce reliable statistical results. Qualitative research engages smaller groups of people who provide insight and context on advertising, new product features and pubic issues.

A lot goes into selecting the right technique or combination of techniques for a particular need. One of the most elemental aspects of the choice is getting a representative sample to interview.

Several years ago, Portland homebuilders wanted to embark on a marketing PR campaign to reach young families about buying their first home. The poll they commissioned wound up interviewing a large number of older respondents and relatively few young adults with new families. The "findings" were worthless to the task at hand.

Selecting the right kind of research technique also can be undermined by asking loaded questions. The whole idea of research is to find out what people think, not trying to conspire to get the "right answer." Straightforward, unbiased questions can elicit accurate responses that give you what you need to know, not necessarily what you want to hear.