Research problem and design · Lesson 2

Audit existing evidence before new collection

6 min read

Lesson objectives

  • Build a source matrix that exposes definition, coverage, and currency differences, then identify the gap worth new research.
Reading tools

Start with what is actually known

Before commissioning another survey, ask which part of the business choice existing records can answer. An organization might have purchases, service complaints, visit analytics, earlier interviews, and published reports. Each source sees a different slice of behavior. A purchase log establishes an event recorded by the system but does not reveal the motive on its own. A complaint describes a problem for people who chose to complain, not its prevalence among all customers. An earlier survey may measure purchase intention rather than completed purchase. A useful desk review is more than a list of links: it separates direct evidence from clues and identifies the question that remains unanswered. This saves unnecessary collection and avoids building a new instrument on a wrong definition.

For each source, use a short matrix with six fields:

  • what event or opinion was measured
  • who was included and excluded
  • the period, unit, and denominator
  • how the data were collected
  • when they were produced or updated
  • and which choice they can inform

These details change interpretation. If a report says 70% prefer fast delivery, inspect the exact question, response choices, respondents, and recruitment before using the figure. If its denominator is site visitors while the decision concerns first-time purchasers, the measure may not fit. Two dashboards may use the same label, 'active customer,' while applying different rules. Record the source version and table or query location; similar names do not make series combinable.

Judge a source relative to the question, not only by the reputation of its publisher. An official methodological note may reliably define a survey yet say nothing about this service's local customers. A support note may precisely describe one incident while offering a weak prevalence estimate. Make three separate judgments: the trustworthiness of the underlying data, the fit of population and measure to the choice, and the currency of the period after product or market changes. Write 'unknown' when recruitment or update date is unavailable; do not turn a blank into confidence. For a changing source, check its current version and publication or effective date. An older foundational source may still explain a method, provided it is not passed off as a description of today's market.

When two numbers conflict, check definitions before assuming behavior changed. Is the event identical? Is the denominator identical? Are the periods comparable? Does one log include returns or cancellations? Was the region or channel redefined? Make a reconciliation table that states each definition, source, and difference blocking comparison. If raw data permit recalculation under a common rule, do so and record the formula. If they do not, present separate indicators and avoid a time trend. An overall average can also hide composition: rising total sales alongside a falling share of repeat purchasers may reflect more new customers. The first series does not answer a retention question.

Lesson summary

Build a source matrix that exposes definition, coverage, and currency differences, then identify the gap worth new research.