New Book Says Polls Provide Indications, Not Predictions

Anthony Salvanto with CBS News has written a new book that explains what polling can and can’t do. It’s a good place to begin to become a savvy research consumer.

Anthony Salvanto with CBS News has written a new book that explains what polling can and can’t do. It’s a good place to begin to become a savvy research consumer.

Political polls give indications of voter attitudes, not predictions of election outcomes, says Anthony Salvanto in his new book, Where Did You Get This Number.

Salvanto, the director of elections and surveys for CBS News, says he wrote his book to explain how polling works after skepticism arose following the 2016 presidential election that polls suggested was a lock for Hillary Clinton. She did win the popular vote, but lost in states critical to a victory in the Electoral College. The polls were right and wrong at the same time.

In an interview on Face the Nation, Salvanto said he is often asked how national poll numbers are generated based on as few as 1,000 ten-minute telephone interviews. He explains representative samples can produce reliable results. Pollsters may not interview you, but they interview people like you.

A representative sample is just part of the best practices followed by professional pollsters. Clear, objective questions must be asked. Individual questions should test a single variable. Conclusions should be tempered by statistical validity. For example, a national poll with a 1,000-respondent sample may provide a valid national picture, but not a statistically valid picture of voters in Colorado.

Even the most scrupulous professional pollsters don’t always get the numbers exactly right. There often is a slight, but significant skew as a result of the specific methodology a pollster uses. For example,  failure to include a representative number of random sample calls to cell phone users could under-represent younger people, low-income families and minorities.

Nate Silver of FiveThirtyEight.com argues it is more reliable to look at groups of polls through the lens of a probability model.  He claims analyzing a pool of polls and weighting each one by their history of accuracy can burp out a more accurate polling results. Even then, Salvanto would say, it is not a prediction, just a reflection in time.

Then there are the polls that aren’t really polls. Push-polls ask questions, less to get an answer and more to deliver a message, often a negative one, about a political opponent. Cheap robopolls get lower than average response rates, which can skew results. Because they are prohibited by law from calling cell phone users randomly, they have a built-in bias.

The bottom line: Purchasers need to be smart consumers of research. Before looking at results, look at the sample so you know whose views are represented in the results. Understand the methodology being used and the statistical confidence it will yield. Know the benefits and limitations of different types of research, and certainly between qualitative and quantitative research. Collaborate with a pollster on the questions that need to be asked and let him or her advise you how to ask them fairly so you get usable responses, not just what you want to hear.

Salvanto’s book may be the place to start on your journey to understanding polling’s potential and limitations.


Getting to Your Audience – and to Your Point

Research is more than polling. Good research, for example, is knowing what it takes to open branded emails on the range of browsers and devices used by your target audience.

Sometimes research is using common sense. Such as, keeping it brief and to the point. People are busy and bombarded with "messages." You need to make yours stick out, be useful and be mercifully short.

Follow the example of the Princeton University admissions office in meeting your audience's expectations. It sent 1-word letters of acceptance. The letter said, "Yes."

Polling Lessons from 1936 Presidential Election

Political pollsters’ blood pressure always rises on Election Day as voters prove their surveys are accurate or not. For example, the Literary Digest predicted a landslide victory for the Republican presidential candidate based on its research, but the Democratic candidate won by a large margin. 

That was the 1936 race between GOP candidate Alf Landon and Democrat Franklin D. Roosevelt.

Not much has changed during the past 76 years. Numerous pollsters predicted a Mitt Romney win in 2012, but were off by wide margins. The outcome of both elections highlights how polling results can lead to incorrect projections if the survey methodology doesn’t adapt to change. 

Based on an analysis by NY Times polling guru Nate Silver, polling firms that used traditional telephone polls with landline-only samples had results skewed to the GOP. 

On the other hand, telephone polls that included cell phone numbers and random Internet-based surveys were more accurate in projecting the election’s outcome.

 Reasons why newer sampling techniques trump traditional methods include:

  • More than three in 10 households in the United States don’t have landlines. The demographics of cell-only households tend to be younger and urban and people of color — groups that voted heavily for President Obama, according to an MSNBC analysis.

Cell Phone-Only Households Increase

The percent of households with cell phone only phone service has increased substantially according to recent survey results from the Centers for Disease Control and Prevention.

Now, more than one in four U.S. households (26.6 percent) has only wireless telephone connections, an increase of 2.1 percent over 2009. During the past 36 months the share of U.S. households with cell phone only service has nearly doubled from 13.6 percent.

The results highlight that traditionally difficult to reach groups, young people, the low income and people of color, will be more difficult to reach by phone for market and public opinion researchers. The CDC study found:

A Race Still Too Close to Call

The campaign for Oregon governor is still a horse race -- some say it's the closest race for governor in the nation. Behind-the-scenes efforts, not polls, will determine whether Dudley or Kitzhaber wins the election. 

A series of polls conducted through mid-September show the race between Chris Dudley, Republican, and John Kitzhaber, Democrat, is razor close. The most recent public poll  shows Dudley leading Kitzhaber 49% to 43%. Others find the race a dead heat.