The Sample Matters, Not Just the Tool

Technology and user patterns are changing the way polls are conducted, but one thing remains constant — an accurate sample, otherwise the methodology and results don't matter.The Washington Post gave readers an informative glimpse into the debate raging among research professionals over the use of "opt-in Internet survey data," or what we call panel research.

Using databases, as opposed to randomly selected samples, is not new or even exceptional. Organizations that have collected email addresses from customers, patients and stakeholders use panel research on a routine basis to measure everything from customer satisfaction to reactions to advertising to introductions of new products and services.

The qualms over panel research center on its application to public-opinion polling. Traditional researchers believe telephone surveys remain the gold standard and web-based surveys remain untested. Like The Washington Post and New York Times, we respectfully disagree.

The national newspapers use YouGov polling, which relies on representative panels drawn from huge databases. Participants opt in to the panel through an online survey. But the pollsters in charge manage the panel's profile to match the "public" being measured, whether it is the national electorate or likely voters in an Ohio congressional district.

We conduct online research using panels for both market and public-opinion research. Ensuring a representative sample of participants is important whether you are relying on a proprietary database or drawing from databases that you lease. 

Soliciting panel participants from leased databases through email is not all that different than drawing a random sample of phone numbers to call on the telephone. Both generally work on the principle of probability. The skill and integrity of polling comes from making sure wherever your sample comes from, it accurately reflects the constituency you are trying to poll.

Telephone survey sample sizes necessary to achieve an acceptable confidence level are smaller than those used in panel research. Larger panel sizes have the serendipitous effect of enabling greater segmentation of the sample. If you take a typical telephone survey sample in Oregon, you will have 500 completed interviews, with approximately 100 per each of the state's five congressional districts. A typical statewide panel of 2,000 participants will yield 400 people per congressional district. 

The Post and Times have benchmarked the Internet surveys they have conducted with traditional telephone surveys and found only marginal differences when comparable samples are used. We have shown doubting clients the same results in telephone and online polls.

So, if the results are the same, why bother with Internet surveys? There are several good reasons:

  • More people only have cell phones and fewer people have landlines in their homes.

  • People who have traditional telephones also increasingly have Caller ID that allows them to screen out calls they don’t want.

  • Fewer people with landlines and even fewer willing to take calls from pollsters, it takes a lot more calls to get a representative sample, which translates into greater cost.

  • There are well-established biases in telephone polls, which tend to under-represent lower income and ethnic minorities.

The case for using web-based survey techniques is even stronger:

  • More people have access to the Internet and routinely go online.

  • Online surveys are less intrusive because participants can respond on their schedule, not just when someone calls.

  • People are more willing to spend time online answering questions, so you can ask more questions.

  • After an initial set-up cost, online surveys are cheaper and faster to conduct.

By far and away, the most superior quality of online surveys is the ability to follow up with respondents. For example, online or in-person focus groups can be organized based on how panel members answer specific questions. This allows researchers to probe deeper into the reasons why people hold particular views and what influenced them. Online focus groups erase the barrier of time and place that can limit the utility of in-person focus groups. It would be prohibitively expensive to gather 25 people from around the state in a focus group at a fixed location and time. It is a breeze to do it online and allow the conversation to continue over multiple days.

We agree with The Washington Post that there is no need to call one form of research good and the other bad. Telephone and web-based surveys are both valid methodologies. The question is choosing the best tool for the job. 

When you realize that what really matters is the sample, not the tool, you will understand the key ingredient to successful polling is hiring a pollster committed to accurate samples, regardless of the tool.