public opinion polls

Clear, Fair Questions Key to Reliable Survey Results

Reliable survey research depends on many factors, but it all starts with questions that are asked clearly and fairly.

Reliable survey research depends on many factors, but it all starts with questions that are asked clearly and fairly.

A lot of things need to be done right to deliver reliable and useful survey data. At the top of the list is asking relevant questions clearly and fairly.

Fuzzy questions produce fuzzy answers. Skewed questions produce skewed results. Fuzzy answers and skewed results aren’t a solid foundation for successful advertising and advocacy campaigns or for smart business decision-making.

Clarity in the wording of questions is essential to avoid confusing respondents. You want to make sure respondents have a common comprehension of the question so you can reliably measure their responses. A best practice in public opinion polling and market research is to test questions to ensure they are as clear and commonly understood as possible

Question clarity is becoming more important because of increased cultural diversity and audiences that include English-as-a-second-language speakers. Question testing needs to include this variable as well.

Along with clarity, questions should explore a single idea, issue or product feature. If you mix in multiple ideas, issues or features, you won’t necessarily know which one is the factor that fetches a  “like” or “dislike” answer. Sarah Taylor, in a blog post about survey questions, provides this example:

For example, asking "how valuable and organized was the Ebook?" is mixing two issues together: value and organization. Maybe the respondent really enjoyed the content in the Ebook, but thought it was organized poorly. If you ask them that question, the response may be for either issue and you won't really have a sense of what needs to be changed for your next Ebook. Instead, ask two questions: “How valuable was the e-book?" and “Rate the organization of the Ebook" to get better quality data.

The vocabulary of questions should be basic. Respondents will run the gamut of education and backgrounds. If your subject matter is technical, take special care to avoid jargon or phrases that may go over the head of some respondents. In some cases, you may need to preface a question so there is a common framework for all respondents. Special care is required to make sure any explanation is fair and factual.

Which brings us to skewed questions. If you are in the propaganda business, skewed questions and push polls may be tools of your trade. But for everyone else, skewed questions aren’t helpful and can be disastrous if you mistakenly base million-dollar campaigns on their findings.

Finding out what you need to know – as opposed to what you want to know – is the proper attitude to bring to any kind of research, whether  a quantitative poll or qualitative focus group. Curiosity, not a fixed mind set, is the right frame of mind.

Credible pollsters and market researchers can readily identify and will avoid loaded words or leading sequences of questions. But sometimes skew still slips in, as it did in this famous question, “Would you vote for a woman for president if she were qualified in every other way?” In addition to being confusing, the question implies being a woman is a qualification as opposed to a gender. Subtle, yes, but doesn’t mean it can’t have a significant effect.

The order of questions can influence responses. Pollsters and market researchers will often rotate the order of questions to minimize the possibility of skewing results.

Give respondents a full range of choices in evaluating an idea, issue or product feature. You also might give them a chance to comment through an open-ended question, which allows you to capture their words, not just their answers.

One of the most common forms of skewing is the omission of a key word or concept. If you were polling the congressional Republican proposals to replace Obamacare, it would be a mistake not to test the concept of lower health insurance premiums in connection with the cost of keeping premiums low for patients with pre-existing conditions. Who wouldn’t want to see lower health insurance premiums, but at which price and paid by whom? Installing residential rooftop solar panels will be enticing as a way to cut electricity bills, but the financial equation is incomplete with exploring how long it would take to break even after making the investment in solar panels.

Many other factors determine the accuracy and reliability of survey findings, such as representative samples, sample sizes, length of surveys, when surveys are conducted and the type of survey used. Surveys can generate rich, meaningful and actionable data if done right. A good place to start is to get the survey questions right.

Polling Misses Mark in Oregon Presidential Primary

A major public opinion poll conducted before the Oregon primary showed Hillary Clinton outpacing Bernie Sanders by double digits. Actual election results were almost the exact opposite.

A major public opinion poll conducted before the Oregon primary showed Hillary Clinton outpacing Bernie Sanders by double digits. Actual election results were almost the exact opposite.

A well publicized public opinion poll conducted between May 6 and 9 showed Democratic frontrunner Hillary Clinton leading challenger Bernie Sanders in the Oregon primary by a 48 to 33 percent margin. Actual election results were almost the opposite, with Sanders carving out a double-digit victory.

How could a poll be so out of whack? One frequent reason is failure to account for voter turnout. However, the Clinton-Sanders poll took higher-than-average turnout into account, which showed Clinton’s lead narrowing to 45 to 38 percent. Still wrong, by a wide margin.

The same poll, which interviewed 901 likely Oregon voters, under-predicted Donald Trump’s vote count. He received 45 percent of the GOP presidential vote in the poll, but almost 65 percent of the actual vote. Another big miss.

Telephone surveys have become somewhat less reliable if they don’t include a percentage of cell phone users, which ensures that younger and minority voices are heard. While that sampling flaw might understate the vote in Portland or college towns like Eugene, it doesn’t explain Sanders’ strong showing in rural Wallowa and Lake counties or his dominance in all but one of Oregon’s 36 counties.

Candidates usually do better in states where they campaign in person. Sanders appeared in Oregon four times before the primary. Clinton made no appearances, but did send Bill Clinton to campaign. That’s a hard factor to capture in a public opinion poll, but it is a question worth asking to see if being here breaks someone’s vote one way or another. 

The Los Angeles Times carried a story over the weekend about the intense Democratic push in Oregon to register new voters as Democrats before the April 26 deadline. Many of the new voters were automatically registered as a result of Oregon’s Motor Voter law, but not affiliated with any political party. Secretary of State Jeanne Atkins reported a larger than normal registration switch, which favored Democrats. These factors would have been hard to track in a poll, but they may have been worth asking about to gauge the velocity of a late shift toward Sanders, who predicted he would win if the turnout was large. He obviously knew what he was talking about. 

Polling is a tough business, and it is getting tougher. Fewer people are willing to be interviewed by phone, which means pollsters need to make more calls to achieve a representative sample, which is more costly. Respondent reticence means polls have to take less time and include fewer questions, sometimes the questions that would be useful in improving confidence in poll findings.

While Sanders’ success in Oregon is not a huge surprise, it may be more telling than at first glance. His victory points out the foibles of one-off polls and the political benefits of an intensive ground game. 

More significantly, the results in Oregon show Sanders’ message packs some punch, and not just where you would expect.