Research problem and design · Lesson 3

Match research design to the uncertainty

6 min read

Lesson objectives

  • Choose exploratory, descriptive, or causal work for a specific question and state what each design can and cannot establish.
Reading tools

Not every evidence gap calls for a survey

Choose a method by what the decision lacks. When the team does not understand how people describe a problem or which alternatives they consider, explore. When the categories are known and the team needs the frequency of a behavior or opinion in a defined population, use a descriptive design and suitable sample. When the choice depends on the effect of a specified change compared with what would have happened without it, use a causal test that attends to assignment and competing explanations. A project can move through these stages, but the evidence does not become one kind of certainty. A depth interview helps explain possibilities and develop measures; five interviews cannot become a market percentage. A large survey measures what it asks and does not by itself establish that a product change caused an answer.

For exploration, begin an interview guide with a concrete experience rather than a desired judgment:

  • 'Tell me about the last time you tried to return an order
  • what happened?' is stronger than 'Does a complicated return make you lose trust?' Follow up on context and alternatives
  • separate the participant's words from the researcher's interpretation

Include contrasting experiences, such as people who completed a return and people who started but stopped, rather than recruiting only those who agree with the team's theory. The output is a set of possible mechanisms, language customers use, and questions needing measurement. It is not prevalence. Even if a theme appears in six interviews, 'most of the market' does not follow; participant selection may explain the pattern.

For description, define population, unit, access frame, and event before calculating a share. 'How many buyers completed a return within seven days?' may be answered from operations records if events are complete. 'How many found the explanation clear?' needs a comprehensible measure and declared recruitment. An opt-in survey can describe its respondents honestly, but it does not automatically yield a probability margin of error or represent every customer. Even a probability sample can suffer nonresponse and coverage problems. Report sample size, exact question wording, selection method, and period; avoid decimal precision that the design cannot support. For subgroup comparisons, inspect each subgroup size rather than relying on the overall total.

For a causal question, imagine the two outcomes a person would have under the old and new versions; both cannot be observed at once. We approximate that comparison by assigning suitable people or units to concurrent versions, holding other conditions as stable as possible, and checking whether assignment was followed. Before launch, define the outcome, assignment unit, observation period, and harm guardrail. If the new page is shown only to fast-service cities and the old page to distant regions, behavior may differ because of the version or context. Calling a weak comparison a 'test' does not make it causal. A small trial may be inconclusive; the honest decision may be to run a larger one or retain the existing version, not to claim a zero effect.

Lesson summary

Choose exploratory, descriptive, or causal work for a specific question and state what each design can and cannot establish.