Judge a Survey by Its Sample

Before you look at top line poll or market research results, flip to the page describing the sample to make sure it matches your target audience so the findings have validity as the basis for your political or marketing campaigns.

Before you look at top line poll or market research results, flip to the page describing the sample to make sure it matches your target audience so the findings have validity as the basis for your political or marketing campaigns.

Critiques of surveys usually zero in on slanted questions that produce skewed answers. But these are easy pickings for critics. A greater research sin, but more difficult to identify, is conducting a survey with a bad sample that fails to reflect a target audience.
 
A favorite example is the telephone survey conducted for a homebuilder association interested in a marketing campaign aimed at first-time homebuyers. The results were meaningless because more than 50 percent of respondents said they were 65 years or older, an unlikely cohort to purchase their first home.
 
Well-crafted questions are certainly important to generate valid findings. But well-intentioned questions can’t overcome a faulty sample. You can ask all the right questions about teen fashion, but get pointless results by interviewing mostly middle-age women.
 
High-profile goofs by public opinion pollsters are often attributable poorly designed samples. Instead of using a sample of all registered voters, clients like to focus on likely voters. Why ignore 40 percent of registered voters when the strongest base of support is among those less likely to vote? Ignoring passive or disenfranchised voters can prove disastrous for candidates, as it did in the trend-disruptive 2016 presidential election..
 
Public opinion pollsters and market researchers understand that valuable survey research starts with sample design, not a questionnaire. Savvy consumers of research should expect a heavy dose of discussion about sample composition to reflect as fairly as possible the views of your intended audience.
 
No sample is ever perfect, so ask your pollster or market researcher how they will make it as close to perfect as possible. Will they conduct a telephone survey using both land and cell phones? What are the demographic characteristics they will use to create sample quotas to ensure a representative sample?  After the interviews are completed, will they use weighting to ensure data reflects opinions of hard to reach groups, such as those age 18 to 34 or people of color?
 
There are a lot of ways surveys can go haywire. But one of the most fundamental flaws is sample design. Just remember, using messages that work for retirees looking for condos in Cancun probably won’t be convincing to a young family seeking a low-cost bungalow close to a good school.