Weighing evidence and rival explanations
5 min read
Lesson objectives
- Build an evidence ledger that separates observation from causal inference, checks selection and measurement bias, and prioritizes a discriminating test.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
You may receive three apparently conflicting pieces of evidence: an official record reports stable decision time, interviews describe a longer wait, and small firms report abandoning applications before submission. Do not select a source merely by its title. Ask what each measures and who is absent. The record may start its clock only after completeness, while applicants describe the journey from assembling documents. The administrative record may cover all complete files, while interviews cover only reachable people. The discrepancy helps define the outcome; it does not by itself show that one party is unreliable. Advanced analysis records confidence, its reason and the limits of generalization for each claim, then chooses a test whose opposite result could change the decision.
Start an evidence ledger with claim, direct observation, collection method, alternative interpretation and next check. 'The new requirement reduced harm' is a causal claim. Fewer recorded reports are an observation, but recording may have changed or reporting become harder. Separate a number's source from the proposed mechanism: did the rule alter behaviour or the ability to observe it? Do not divide complaints by all affected people unless you know who could complain. For a time series, keep the harm definition, period and population stable or mark the series incomparable. Qualitative evidence is not automatically inferior to quantitative evidence; it may reveal an unrecorded mechanism, but participant selection and analysis limit what can be inferred.
Choose a discriminating test rather than simply requesting more data. If one explanation is too few staff and another is unclear requirements, headcount alone does not settle the issue. Separate the pre-completeness and post-completeness journey, inspect return reasons and measure staff workload. If clarity is the main cause, returns may fall after guidance improves while staffing stays constant. If capacity is the cause, complete-file backlogs will persist. The test does not establish causation with certainty when demand or triage policy changes concurrently, but it narrows speculation. Set a decision point in advance: which finding warrants an administrative trial and which calls for competent review of the rule?
Beware three false assurances. A large sample does not fix a poorly defined measure. Agreement in one meeting does not represent people unable to attend. A temporal association between a rule and an outcome does not reveal what would have happened without the rule. Conversely, limited data need not paralyse a reversible low-risk step that also improves learning. State what is known, plausible and still a legal or scientific dispute, then recommend proportionately. The World Bank's regulatory governance indicators describe aspects of transparency, participation and impact assessment for business-related rulemaking. They illustrate measurable process features, not a complete score for every law or proof about a country without inspecting the underlying data and method.
Lesson summary
Inputs: 90 complete files in the record average five processing days; 30 interviews, mostly 24 returned files, describe a 17-day journey. Solution: the record covers post-completeness time in a defined group and may omit abandonments. Interviews desc