Build a Falsifiable Theory of Change
6 min read
Lesson objectives
- The learner maps a service-policy causal chain, identifies its weakest assumption and chooses a test that separates intervention effects from a rival explanation.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A policy brief cannot stop at 'simplify the form and quality will rise'. That names a hope, not a mechanism. A theory of change links an authorized action to a specific behavior change and then to a public outcome, with assumptions that can be tested. For example, removing a document that an approved rule does not require may reduce requests for completion; fewer applicants need to return; more receive a valid first-attempt decision; applicant burden falls without more erroneous eligibility decisions. Each arrow needs different evidence. The document might be removed while the principal return reason is still an unclear service heading, so the hoped-for outcome fails. Specify the starting point and comparator: who uses the service, which rule version applies, how many cases are returned and over what period? State what the agency will actually do and who can authorize it. Separate an immediate output, such as published wording and staff training, from the outcome, a valid decision communicated to the applicant. Staff-training attendance and form downloads cannot stand in for decision quality. Add a possible unintended effect: fewer document requests might accompany more wrong approvals if a necessary check disappears. A useful theory of change reveals when to stop or revise, not a narrative that guarantees success in advance.
Write a specific assumption beside each arrow. 'Users understand the form' is broad; 'after revision, most tested applicants can identify the required evidence from the page without help' can be investigated. 'Returns fall' requires a unit: returned applications, unique people or repeated contacts for one application? Choose consistently before comparing. Build an evidence map with intervention, expected behavioral change, data source, rival explanation and condition for revising the decision. The team might attribute improved completion to new wording when an additional reviewer was hired in the same week. The rival explanation is review capacity, not merely 'other factors'. Investigate reviewer workload and processing time for the same periods or use a staged rollout with reasonably comparable groups. If a strong comparison is infeasible, say the result is observational; do not award it a causal interpretation. Distinguishing assumption from observed effect prevents premature scale without prohibiting a small, reversible trial when risks can be contained and reviewed. Record the relevant denominator before the trial, including people rescued by an assisted channel.
Find the weakest causal link by what could overturn the decision, not by what is easiest to count. A team can readily count views of new instructions, while the decisive uncertainty is whether people who no longer supply the document are correctly eligible. Ask for an independently checked sample of decisions and reasons. Review channel effects: if web applicants receive simplified wording but telephone staff still demand the document, the web rate might improve without a unified policy. Test matching content and rule versions with staff. Set a stop signal before the trial. If eligibility decisions corrected on review exceed a locally agreed boundary, measured among decisions actually checked, pause scale and investigate. The competent owner sets a threshold in light of potential harm; this course does not invent a universally binding number. Set a learning signal too, such as fewer returns while accuracy in comparable decision samples remains stable. If data are absent for people who abandon before submission, state that blind spot and create a proportionate capture plan. 'We cannot see a problem' is not evidence that none exists.
A decision maker can read the theory of change in six lines:
- problem and denominator
- authorized intervention
- immediate output
- desired outcome
- constraints and harms
- and evidence for the next decision
Put uncertainty beside the arrow on which funding depends instead of hiding it in a generic footnote. If the decisive assumption is that applicants understand the text, make comprehension research with varied user groups a condition for moving forward. If it is consistent rule application by reviewers, test agreement on otherwise identical cases. If it is that fewer documents speed service, measure the full journey rather than time on one screen. Keep an assumption register with owner and review date. A policy may save agency cost while increasing applicant work, so define the public outcome before counting activity. What level of result is sufficient is an institutional decision balancing access, accuracy and resources. The analyst's role is to show the evidence, competing explanations and their limits. This map will later inform evaluation design, but it is not itself an estimate of causal effect.
Lesson summary
Link intervention to testable behavior and outcome, specify a rival explanation and stop signal before trial, and avoid calling an observed improvement causal proof.