Weigh conflicting evidence
5 min read
Lesson objectives
- The learner constructs an evidence ledger recording definitions, sample, bias, evidential weight and the next test for each source.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
Suppose an administrative registry says 80% of eligible households are enrolled, an independent survey estimates 58%, and community interviews describe many stalled applications. Do not average the three. Periods, populations and meanings of enrolment may differ, and purposive interviews do not yield a comparable rate. Begin an evidence ledger with one row per finding: question, unit, denominator, date, collection method, owner, and the bounded claim it can support. A registry can trace transactions that entered it, but is silent about households that never applied. A well-designed survey may reach non-applicants, but its estimate depends on sampling, non-response, recall and how eligibility is defined. A purposive interview can reveal a plausible access mechanism in a person's own terms; it cannot show how common that mechanism is in the population.
First test comparability. Does the registry count people or households? Does 'enrolled' mean an open file or confirmed eligibility? Does the survey include recent residents? Is its 58% among all eligible households or only those aware of the programme? Build a definitions table before interpreting the difference. If the registry carries files from previous years while the survey covers the current quarter, extract a matched snapshot or state that the figures cannot yet be compared. Do not discard a source because it conflicts with the sponsor's expectation; describe what it can and cannot say. The conflict may point to a systemic problem in eligibility estimation or identity matching, not a simple arithmetic error.
Next examine how each source produced its result. For the registry, test duplicates, incomplete records and suspended cases still counted as enrolled. For the survey, inspect the sampling frame and weights, response rate, wording of the benefit question and recall period. For interviews, note who was recruited through a service centre and who never approached one, and whether interpretation or accessible participation was available. These checks do not establish that surveys are always superior to records, or the reverse. They determine fit for the decision question. If the task is to estimate the number excluded, interviews at a centre cannot carry the weight of a credible population sample. If the task is to understand a form step that fails, interviews may reveal something the survey rate cannot.
Then write rival explanations that can be falsified. The registry may run high because it retains old files; the survey may run low because respondents do not recognise the formal benefit name; or the sources may refer to different populations. For each, specify a prediction: if period mismatch dominates, the rates should move closer when the registry is restricted to the survey quarter. If naming matters, a carefully tested description may change recognition without prompting a desired answer. Do not rewrite a survey question after seeing its results to obtain a preferred rate; document any test and its effect. If the available data cannot distinguish explanations, present clearly labelled scenarios or a range rather than a falsely precise point estimate.
Counting documents is not the same as independent corroboration. The agency report, public dashboard and press release may all reproduce the same registry number; they are three displays of one underlying dataset. Look for independent collection methods and harmonised definitions. Data linkage must respect permissions: a data steward can match a sample securely and return aggregate tables without handing individual files to the adviser. Give the ledger columns for bias risk, confidence judgement, the requested check, and its owner. Confidence is an explained judgement, not an automatic star score based on source type. Mark a finding as consistent, conflicting but testable, or presently incomparable, and say how each status affects the decision.
When a decision is near, prioritize a short test with decision value. Recalculate the registry for the survey quarter, review a declared sample of 40 records, and inspect the survey instrument. If the conflict remains, tell the agency what changes between 58% and 80% enrolment scenarios: help for applicants may still be justified, while the size and cost of broad expansion remain uncertain. A funding recommendation may therefore favour a conditional pilot rather than immediate rollout. This does not mean delaying every action until perfect certainty. It means separating a reversible step from a large commitment whose scale current evidence cannot support. Record the evidence that would change your recommendation so the data request has a clear purpose.
Lesson summary
The learner constructs an evidence ledger recording definitions, sample, bias, evidential weight and the next test for each source.