public opinion research

Why Representative Samples Really Matter

If you want market research that matters, make sure the sample of people in your survey matches the audience you want to reach with your product or message.

If you want market research that matters, make sure the sample of people in your survey matches the audience you want to reach with your product or message.

A favorite story involves meeting with a client interested in promoting first-time homeownership. I mentioned the need for market research. No problem, the client said, we have that covered. I was handed the research summary and, as a matter of habit, jumped to the page about the telephone survey sample. It was very revealing. 

More than 50 percent of the respondents were 65 years or older. They were the majority of people who answered the phone and were willing to spend 15 or 20 minutes talking to a stranger about owning a home. Unfortunately, they weren’t the people the client had in mind as first-time homebuyers. 

Survey data is worthless unless the sample of who you interview reflects the audience you seek to reach. The sample in my client’s survey would have been terrific if the subject was reverse mortgages. It stunk as a reflection of who to address potential first-time homebuyers. 

Conversations between clients and research professionals must start with who to interview. If you have the wrong sample, the answers you get from the questions you pore over won’t matter a lick. 

Too often, the question of who to interview is glossed over. Sometimes the most obvious sample goes overlooked. When I was a lobbyist, a client hired me to “fix” his message that wasn’t gaining any traction with legislators. I started by interviewing about a third of the legislature, including virtually all of the lawmakers on the committees that were most engaged on my client’s issue. 

The interviews produced a wealth of insight. My client’s issue had latent support, but needed to be explained and demonstrated in a far different way. Lawmakers basically wrote the script my client and I used to lobby them. And it worked. 

Representative samples are harder to achieve for a mix of reasons. For example, increasing numbers of people don’t have landline phones and, if they do, they shield themselves from unsolicited calls with Caller ID. It takes a lot more calls, at greater expense, to collect a representative sample. Market research must cope with growing segmentation, which adds extra layers of complexity in selecting the right group of people to survey. 

The value of representative samples goes beyond quantitative research. Focus groups must be representative, too. And why would you do a customer satisfaction intercept survey for Nordstrom by interviewing people coming out of a rival department store? Representative samples matter in public opinion polling. A poll of New York voters wouldn’t be all that useful in projecting election results in Indiana. 

Despite the difficulty, solid research is grounded on good samples. Who you talk to matters if you want findings that mean something for your marketing.  

Gary Conkling is president and co-founder of CFM Strategic Communications, and he leads the firm's PR practice, specializing in crisis communications. He is a former journalist, who later worked on Capitol Hill and represented a major Oregon company. But most importantly, he’s a die-hard Ducks fan. You can reach Gary at and you can follow him on Twitter at @GaryConkling.

Political Polling Validity Becomes Shaky

Political polling is getting less reliable in predicting actual election outcomes. Reasons include the growing use of cell phones, reluctance to participate in telephone surveys and the rising cost of representative research samples.

Political polling is getting less reliable in predicting actual election outcomes. Reasons include the growing use of cell phones, reluctance to participate in telephone surveys and the rising cost of representative research samples.

Political polling doesn't seem to be as spot on as it used to be. Greater use of cell phones, wariness to participate in surveys and unrepresentative samples are among the reasons that political polls and election results turn out differently.

Cliff Zukin, a Rutgers political science professor and past president of the American Association for Public Opinion Research, writes in the New York Times that "polls and pollsters are going to be less reliable," so voters and the news media should beware.

"We are less rue how to conduct good survey research now than we were four years ago, and much less than eight years ago," says Zukin. "Don't look for too much help in what the polling aggregation sites may be offering. They, too, have been falling further off the track of late. It's not their fault. They are only as good as the raw material they have to work with."

Polling failures have been exposed in the most undetected 2014 mid-term election sweep in which Republicans captures both houses of Congress, Prime Minister Benjamin Netanyahu's solid victory in Israel and British Prime Minister David Cameron's relatively easy re-election win.

Cell phones are everywhere and increasingly have replaced landline telephones. Pollsters can find cell phone numbers, but federal law prevents calling them with automatic dialers. According to Zukin, "To complete a 1,000-person survey, it's not unusual to have to dial more than 20,000 random numbers, most of which do not go to working telephone numbers." That adds budget-busting cost to telephone surveys, which in turn lead to "compromises in sampling and interviewing."

Response rates to surveys have declined precipitously. In the 1970s, Zukin says an 80 percent response rate was considered acceptable. Now response rates have dipped below 10 percent. It is hard to draw a representative sample when large chunks of the population refuse to participate. Some cohorts, such as lower income household members, are more unlikely to participate than others, which can skew results. And it takes more calls to achieve a representative sample, which encourages corner-skipping.

Internet polling has emerged as a strong alternative. It is cheaper than telephone surveys and, at least or the moment, people seem more willing to participate, in part because they have more choice in when and how to respond.

But Internet use has built-in biases, too, Zukin notes. While 97 percent of people between the ages of 18 and 29 use the Internet, 40 percent of adults older than age 65 don't. "Almost all online election polling is done with non probability samples," Zukin says, which makes it impossible to calculate a margin of error. 

The most vexing polling problem is not a new one – determining who will actually vote. Public opinion polling is one thing; trying to predict the outcome of an actual election is another. Pollsters recognize that respondents will overstate their likelihood of actually voting, but have limited ability to identify who will and who won't cast ballots.

Non voting can occur for a mix of reasons – bad weather, lack of interest or political protest. Some registered voters simply forget to vote, especially in non-presidential elections. Less motivated voters vote in top-line races and leave the rest of their ballots blank, making it hard to predict the "turnout" for so-called down-ballot candidates and ballot measures.

Scott Keeter, who directs survey research at Pew Research, says the combination of these factors is shifting political polling "from science to art."

Political polls will continue to be magnets for media coverage, but readers should be aware that the results may not have as much validity as polling in the past.

Finding Messages That Are Persuasive and Believable

A message that is persuasive but not believable can undermine your investment in marketing or public affairs communications.

A message that is persuasive but not believable can undermine your investment in marketing or public affairs communications.

A message that is persuasive but not believable can undermine your investment in marketing or public affairs communications.Arguments can be persuasive without being believable. Good research will help you determine whether your argument is both.

The worst trap you can wander into is betting the farm on an argument that research shows is persuasive, but fails to probe deeper to see whether it is believable.

Several years ago, we conducted quantitative research to test the best arguments for a state transportation funding package. The argument that proved most persuasive was the list of transportation projects contained in the legislation to be funded. People liked knowing what the increased gas tax money would pay for.

However, probing deeper revealed that many of the people who liked the idea of a specific list of transportation projects believed that they never would be built. The list was persuasive, but they doubted the credibility of the state agency to follow through.

When the transportation funding bill went to the voters, it was soundly thrashed. Exit polling underscored the problem — a persuasive argument wasn't credible enough to carry the day.

The questions of persuasiveness and believability don't just apply to public policy issues and campaigns. They also are meaningful in a marketing context. A product feature may appeal to potential customers, but unless it convinces them to buy, it is just a nice feature — appealing, but not put-it-in-my-shopping-cart convincing.

For those who like to skimp on research, the persuasive-believable conundrum can become another excuse not to do any research. For people interested in getting a return on their investment in communications and marketing, more nuanced research that digs deeper than superficial appeal is a money-saver.

In reality, a deeper research dive isn't always a lot more expensive. It is more dependent on using a research instrument that enables more careful exploration of views. That is one of the built-in values for online research tools. You can ask more questions because people will answer them, if they get to choose the time and place to respond.

Sifting through rival messages to see which one has the most appeal is an important first step. To make sure it isn't a misstep, find out whether the appeal is real. You could be sorely disappointed if you don't.

Kitzhaber, Merkley Retain Double-Digit Leads

Senator Jeff Merkley and embattled Governor John Kitzhaber are still on track to win re-election over GOP challengers.

Senator Jeff Merkley and embattled Governor John Kitzhaber are still on track to win re-election over GOP challengers.

Senator Jeff Merkley and embattled Governor John Kitzhaber are still on track to win re-election over GOP challengers.The word "corruption" and Oregon politics don't usually go together and GOP gubernatorial candidate Dennis Richardson's efforts to couple them haven't appeared to narrow Governor John Kitzhaber's double-digit lead.

Survey USA conducted a statewide poll for KATU-TV that shows Kitzhaber clinging to a 51 percent to 38 percent lead over Richardson, with only 6 percent of the electorate still undecided. The survey was conducted between October 16-19 with 561 likely voters, interviewed by both landline and cell phones.

If only men voted, the race would be tighter, as Kitzhaber holds a narrow 48 percent to 46 percent lead. But the governor seeking an unprecedented fourth term wallops Richardson among women voters by 54 percent to 30 percent.

Kitzhaber tops Richardson in the 18-34, 50-64 and 65+ categories and ties him at 45 percent each in the 35-49 cohort.

People who blame Cover Oregon on Kitzhaber typically say they will vote for Richardson' those who don't signal a vote for the incumbent governor.

The survey indicates Democratic Senator Jeff Merkley retains a commanding lead over his GOP challenger, Monica Wehby, at 53 percent to 32 percent. The race is closer among men, but a romp among women who favor Merkley by 60 percent over Wehby at 24 percent.

Survey results show Measure 91 to legalize marijuana and Measure 92 to require labeling for GMO food both have a shot at passing, with nearly identical percentages heading down the stretch. Measure 91 enjoys a 48 percent to 37 percent lead, while Measure 92 maintains a slimmer margin at 44 percent to 37 percent. More than 15 percent of likely voters said they were undecided on the measures, which could tip the final results either way. A major factor for the positive margin is the overwhelming support for both measures among younger voters, which ranges from around 70 percent for to 15 percent against.

The Richardson and Wehby candidacies reflect two more wrong turns for GOP efforts to win statewide office in Oregon. Richardson has tried to downplay social issues, without success, and Wehby has played up her moderacy on those issues, also without success among likely women and younger voters.

GOP operatives are wringing their hands over the absence of a more commanding figure in the gubernatorial race who could have capitalized on scandals that have rocked Kitzhaber and his fiancé Cylvia Hayes just as ballots were heading to mailboxes.

While earlier polls suggested Kitzhaber was vulnerable or at least may have worn out his political welcome, his polling numbers still point to a relatively comfortable re-election margin.

More scandals could still affect the final outcome, but that prospect grows slimmer by the day as Oregonians, especially those who have already made up their minds, cast ballots early.

Scottish Attitudes on Independence Don't Seem Foreign

Scottish voters will cast ballots this September to decide whether they want to separate from the United Kingdom. Given that voting on national independence is a pretty big deal, a Scottish research firm says more than half of eligible voters remain undecided.

The Scottish Social Attitudes (SSA) survey is conducted annually and examines a wide range of questions, including ones about national identity and constitutional preference. The survey is more than a snapshot political poll. Conducted since 1999, it provides invaluable trend analysis about Scottish attitudes.

Political analysts have assumed that the strong showing of the Scottish National Party in 2011 elections signaled a shift in support for independence. However, the SSA survey indicates attitudes about breaking away from the United Kingdom have changed much in the last decade. The survey instead showed were pleased the SNP had done a good job in sticking up for Scotland inside the UK, which is something they wanted.

Research: Creativity's Best Friend

You are working as part of a creative team. Everyone has lots of bright ideas. So how do you decide what idea will work? In a word: research.

Survey research doesn't need to squash creativity, but research can direct creativity to its most useful ends. A concept that looks great on paper or sounds perfect in a brainstorming session may fall flat when it reaches its intended audience. It is better to know that earlier than later.

Some creative directors view research as the enemy, when in fact research can be a trusted guide to connect with the people you are trying to reach with a TV ad, an op-ed or a piece of direct mail.

Quality research is a must for strategic communications in both the marketing and public affairs spaces. Here's why:

  • People only can absorb so much information, so it makes sense to lead with your best feature or most compelling fact. Research can establish what sells best.
  • A lot of decision-making rests on trust. That puts a premium on messengers. Research can test different people or types of people to see who your target audience trusts the most.
  • Some facts or arguments prompt positive responses, but a positive response doesn't equate automatically to a sale. Research can probe whether a feature has enough value to earn a purchase or whether an argument is believable enough to earn support.