Collect and weigh evidence
6 min read
Lesson objectives
- Rank conflicting evidence by decision relevance and quality, then propose a test that separates two explanations instead of collecting data without purpose.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
'Arabic user satisfaction fell after localization, but task completion rose.' This apparent conflict cannot be solved by discarding one indicator. The survey question may have changed; one task may have improved while another remains confusing; the user mix may be different. Start with a specific decision: expand the release, stop it, or repair a point in the journey? For each piece of evidence ask what was measured, for whom, when and how. A satisfaction prompt shown only after interaction excludes people who left earlier. A completion rate can rise if difficult cases disappear from its denominator. Two numbers displayed together are not necessarily comparable. Check their definitions before constructing a story to reconcile them.
Create an evidence register with six columns: claim, source, collection period, population or sample, what it supports, and what it does not support. Add a qualitative confidence label rather than a false precision score: high when a record is close to the behaviour of interest and consistently defined; moderate when a sample is relevant but limited; low when it is one anecdote or has an unknown definition. Confidence does not judge a person's honesty. It describes the evidence's ability to answer a particular question. One interview can be highly valuable for discovering how a button misleads someone, yet weak for estimating how often the problem occurs across markets.
Seek sources that do not share the same selection problem. Support tickets and support-agent interviews can both reflect people who sought help; their agreement is not two independent prevalence estimates. A sample of user sessions including people who never contacted support adds another view. Nor does a linguist's judgement replace a task-completion test. The linguist can reveal a textual defect; behavioural observation can show its likely effect. Users may complete a task despite a wording fault that exposes them to a later risk, so linguistic review still has a role. Triangulation means comparing evidence collected differently and within stated scopes, not merely counting three documents in a folder.
Even with a stable denominator, rival explanations can remain. State two hypotheses that predict different observations. Hypothesis A: the ambiguous button causes hesitation, predicting a longer pause before clicking and a mistaken expectation of the next step in comprehension testing. Hypothesis B: card authorization causes the difficulty, predicting drop-off after the click and bank error messages, not hesitation before it. A brief user task that records where people stall, together with audited event logs, distinguishes these accounts better than a broad 'Did you like the Arabic?' survey. If both patterns appear, do not force the evidence into a single cause. Prioritize two repairs by their likely effect and risk, then test each after change.
Choose a sample from the decision, not from a magic number. If you need to know whether one dangerous error should block launch, deliberately inspect critical journeys and do not claim to estimate prevalence. If you need a frequency estimate across a large library, define the sampling frame and method, spread selection across content types and markets, and record exclusions. Inspect failures and successes. Looking only at complaints can overstate the problem; looking only at content chosen by a vendor may conceal it. Do not promise that a small group of users proves every edge case is safe. Put the limit of inference next to the finding in the decision memo, not in a distant footnote.
The advisory output is more than a confidence table. It is a revisable judgement: 'We cannot attribute the completion change to localization because the denominator changed. We will standardize measurement, then test button comprehension and authorization failure, and return with a decision in two days.' Name the measurement, content and defect owners because source ownership affects the team's ability to correct a signal. Under a tight deadline, recommend a limited release or monitoring gate rather than a final declaration of success or failure. Missing evidence is not evidence of no problem. Keep access to user data for this investigation within the minimum approved by the responsible owners; an analysis need does not confer unrestricted access.
Lesson summary
Weigh evidence by denominator and collection method, then test competing accounts.