The New York Times

Four Different Pollsters, Four Different Results

Political polling can vary widely based on factors such as who is interviewed and the weighting pollsters give to likely voters, white voters, Hispanic voters and black voters.

Political polling can vary widely based on factors such as who is interviewed and the weighting pollsters give to likely voters, white voters, Hispanic voters and black voters.

Wariness of presidential political polls is warranted. The New York Times conducted an experiment that involved four different pollsters evaluating the same data set, which produced four different results.

Hillary Clinton received 42 percent support in two of the four polls and 39 percent and 40 percent in the other two. Donald Trump topped out at 41 percent in one poll, 39 percent in another and 38 percent in the other two. The Times “benchmark” poll had it 41 percent for Clinton and 40 percent for Trump.

The experiment highlights how polling, even by credible pollsters, can vary widely within the acceptable norms of polling. Critical variables include a representative sample, sampling error and basic assumptions. The latter accounted for the variance in the Times experiment that centered on the same 867 poll responses.

The most significant variables in the pollster analysis of response data: predicting the percentage of white, Hispanic and black likely voters in the November 8 general election.

When white voters reached 70 percent and Hispanic voters fell to 13 percent, Trump came out ahead by a percentage point.

When white voters were estimated at 68 percent and Hispanic voters at 15 percent, Clinton prevailed by 3 percentage points.

These choices weren’t random. Different pollsters relied on different models or sources of data. For example, the pollster who predicted the biggest lead for Clinton used self-reported intentions for likely voters, traditional weighting and Census data. The pollster who gave the nod to Trump relied on voter history to determine likely voters, a weighting model and voter files.

Their varying decisions on these questions add up to big differences in the result,” according to Nate Cohn in The Upshot report on polling. “In general, the pollsters who used vote history in the likely voter model showed a better result for Mr. Trump.”

Laid bare, the experiment shows “there really is a lot of flexibility for pollsters to make choices that generate a fundamentally different result. You can see why we say it’s best to average polls and to stop fretting so much about single polls.”

Tom Eiland is a CFM partner and the leader of the firm’s research practice. His work merges online research with client communications and engagement efforts, and he has a wide range of clients in the education, health care and transportation sectors. You can reach Tom at tome@cfmpdx.com.

Smartphone Rage Prompts New Research Tactics

The accelerating switch from landline to mobile phones is undermining the ability of traditional telephone surveys to capture accurate reflections of many target groups, from young shoppers to likely voters.

As anyone watching the just completed 2012 London Olympics could readily tell, mobile phones are omnipresent. Recent data indicates more than half of all Americans with mobile phones have smartphones, which opens an expanding world of smartphone apps. More than a quarter of all smartphone users say they would rather surrender their computer than their smartphone.

That doesn't bode well for landline phones, which don't have cameras and games, and has caused pollsters to scramble to adjust.

Because the demographics of smartphone users differ from the mix of landline phone users, pollsters are having to juggle their samples. In an article last week, The New York Times reported that veteran GOP and Democratic pollsters who conduct surveys for NBC News/Wall Street Journal decided to increase their percentage of exclusive mobile phone users to 30 percent of their overall sample.

The Times reported other polling firms supplement traditional telephone surveys with online surveys in an attempt to capture the same variable demographic of mobile phone users.

Mark Mellman, who polls for Democratic candidates, told the Times, "That group is not only younger, but also attitudinally different from other people of all ages." Mellman said they are "disproportionately urban, African-American, on either the high or low side of the economic ladder and Democratic."

While it is legal to make randomly selected polling calls to landline phones, regulations prevent such calling on cell phones.

"I have yet to see a standard that I believe is anything more than a guesstimate," Republican pollster Whit Ayres admitted to the Times. Ayres predicted the research industry would eventually shift more toward Web-based surveys.

CFM is already well-versed in this conversion. Yes, we still conduct telephone surveys. However, we counsel that web-based surveys using representative and large databases can produce equally reliable results, with the added benefit of enabling ongoing engagement with a panel of respondents.