A lot of things need to be done right to deliver reliable and useful survey data. At the top of the list is asking relevant questions clearly and fairly.
Fuzzy questions produce fuzzy answers. Skewed questions produce skewed results. Fuzzy answers and skewed results aren’t a solid foundation for successful advertising and advocacy campaigns or for smart business decision-making.
Clarity in the wording of questions is essential to avoid confusing respondents. You want to make sure respondents have a common comprehension of the question so you can reliably measure their responses. A best practice in public opinion polling and market research is to test questions to ensure they are as clear and commonly understood as possible
Question clarity is becoming more important because of increased cultural diversity and audiences that include English-as-a-second-language speakers. Question testing needs to include this variable as well.
Along with clarity, questions should explore a single idea, issue or product feature. If you mix in multiple ideas, issues or features, you won’t necessarily know which one is the factor that fetches a “like” or “dislike” answer. Sarah Taylor, in a blog post about survey questions, provides this example:
For example, asking "how valuable and organized was the Ebook?" is mixing two issues together: value and organization. Maybe the respondent really enjoyed the content in the Ebook, but thought it was organized poorly. If you ask them that question, the response may be for either issue and you won't really have a sense of what needs to be changed for your next Ebook. Instead, ask two questions: “How valuable was the e-book?" and “Rate the organization of the Ebook" to get better quality data.
The vocabulary of questions should be basic. Respondents will run the gamut of education and backgrounds. If your subject matter is technical, take special care to avoid jargon or phrases that may go over the head of some respondents. In some cases, you may need to preface a question so there is a common framework for all respondents. Special care is required to make sure any explanation is fair and factual.
Which brings us to skewed questions. If you are in the propaganda business, skewed questions and push polls may be tools of your trade. But for everyone else, skewed questions aren’t helpful and can be disastrous if you mistakenly base million-dollar campaigns on their findings.
Finding out what you need to know – as opposed to what you want to know – is the proper attitude to bring to any kind of research, whether a quantitative poll or qualitative focus group. Curiosity, not a fixed mind set, is the right frame of mind.
Credible pollsters and market researchers can readily identify and will avoid loaded words or leading sequences of questions. But sometimes skew still slips in, as it did in this famous question, “Would you vote for a woman for president if she were qualified in every other way?” In addition to being confusing, the question implies being a woman is a qualification as opposed to a gender. Subtle, yes, but doesn’t mean it can’t have a significant effect.
The order of questions can influence responses. Pollsters and market researchers will often rotate the order of questions to minimize the possibility of skewing results.
Give respondents a full range of choices in evaluating an idea, issue or product feature. You also might give them a chance to comment through an open-ended question, which allows you to capture their words, not just their answers.
One of the most common forms of skewing is the omission of a key word or concept. If you were polling the congressional Republican proposals to replace Obamacare, it would be a mistake not to test the concept of lower health insurance premiums in connection with the cost of keeping premiums low for patients with pre-existing conditions. Who wouldn’t want to see lower health insurance premiums, but at which price and paid by whom? Installing residential rooftop solar panels will be enticing as a way to cut electricity bills, but the financial equation is incomplete with exploring how long it would take to break even after making the investment in solar panels.
Many other factors determine the accuracy and reliability of survey findings, such as representative samples, sample sizes, length of surveys, when surveys are conducted and the type of survey used. Surveys can generate rich, meaningful and actionable data if done right. A good place to start is to get the survey questions right.