Use of Margin of Error Can Be Misleading

As CFM increases the use of online research, more people are asking about the statistical reliability of Web-based surveys. Well, for a variety of reasons, the rules of statistical testing don’t apply to online surveys.

That said, the term “margin of error” is misleading and can build a false sense of confidence in telephone surveys.

There are multiple sources of errors that can occur in traditional survey research, including:

  1. Sampling error;
  2. Coverage error;
  3. Error associated with non–response;
  4. Questionnaire design;
  5. Error associated with question wording;
  6. Error associated with response options;
  7. Error associated with interview bias; and
  8. Post-survey weighting and adjustments.

Telephone survey research doesn’t occur in a scientific “clean room” devoid of outside influences. It occurs at home or at work, between two people. One of those people, the respondent, is multi-tasking – listening to survey questions, watching TV, surfing the Web, minding the kids, washing clothes and fixing dinner. As much as we try to neutralize interviewer bias, it occurs.

Harris Interactive, the world's 11th largest research firm, no longer uses the words “margin of error” in its surveys. Harris explains, “All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100 percent response rates. These are only theoretical because no published polls come close to this ideal.”

But we know traditional surveys are accurate because the track record in political and issues management efforts is very good. The same can be said for online research.

CFM and other research firms use a number of “tests” to ensure the reliability of online research.

  1. We compare results to telephone surveys. We have found attitudes and opinions found in large sample online surveys are the same as those in telephone surveys.
  2. We compare results between online and telephone surveys for behavioral questions. Again we have found differences are insignificant.
  3. We compare the demographic characteristics of survey respondents to census and other data sources. So far the only two differences have been: women are more likely to respond than men and the average age of respondents is higher than the census. We take these differences into account when analyzing results.

Research firms don’t sell a pig in a poke. It is in our best interest to provide reliable information. If our clients succeed, we succeed.

You can trust the findings of online research when it is done properly. All survey research is subject to multiple sources of error. Relying on margins of error to justify telephone over online methodologies can be misleading.