Frame the problem and protection gap · Lesson 4

Test explanations for an access gap

5 min read

Lesson objectives

  • The learner converts gap indicators into rival hypotheses and testable predictions before selecting an intervention.
Reading tools

A welfare dashboard shows that applications from 1,200 potentially eligible households were not completed, and a manager proposes an advertising campaign. Advertising is one possible response, but it assumes people do not know the programme. They may know it and leave at the document-upload step, complete the form but wait for a slow verification decision, or appear twice because a failed first attempt generated a new application. Before appraising interventions, draw the service chain: awareness, application start, completion, eligibility decision, account activation, and actual payment or service. Give each transition a numerator, denominator and date. The final count alone does not locate leakage and therefore cannot select the remedy. Advisory reasoning should state a causal story clearly enough that evidence could disconfirm it.

Construct a hypothesis matrix with three operational columns: explanation, prediction if true, and evidence that could refute it. Low awareness predicts many eligible households have not started an application and a measurable knowledge gap in an appropriate sample. Documentation burden predicts exits concentrated at upload and perhaps a larger gap for people with weak connectivity or assistance needs. Slow verification predicts completed files awaiting a decision, not necessarily fewer starts. Payment failure predicts accepted cases without receipt. A list of plausible causes is not enough. For each, specify an observation that distinguishes it from rivals. If the needed stage is not recorded, design a limited test rather than fill the cell with an invented value.

The metric itself may alter behaviour. If an agency counts applications instead of unique households, a person who restarts after a system failure increases the application count while access remains unchanged. If the system marks a file 'complete' as soon as one document is uploaded, that may not mean it is ready for a decision. Ask the operations owner for state definitions and whether software releases changed the series. Compare groups with reasonably similar access opportunities where possible, but do not treat a district difference as automatic evidence of one channel's effect: distance, income, language and age may also differ. Record what cannot be adjusted. Protect applicants as well: an interface test must not withhold a legal entitlement or prolong a group's wait merely for evaluation convenience.

At the first decision point, value the information that could change the intervention choice. If two days of service-stage extraction show that most leakage occurs after approval, a broad advertising campaign may waste funds. If enrolled recipients are paid reliably while many eligible households never start, checking awareness and assisted application becomes more relevant. Prioritize data that can be gathered soon and might change the ranking of options. Ask whether a narrow repair can be tested without a major programme change. Clearer instructions or a staffed help route may be piloted, but scale-up should measure reach, cost, distribution and error. Evidence that a mechanism exists is not an estimate of the size of its effect.

Use conditional scenarios rather than a falsely certain single recommendation. If 70% of incomplete applications stop at documentation, test a simpler requirement or alternative proof with the competent authority while watching eligibility error. If the main gap lies between awareness and starting, test clearer information through suitable channels, with an ethical comparison or limited before-and-after measurement and careful interpretation. If approval is common but payments are late, do not make more applications the target until delivery capacity is repaired. These figures are fictional exercise conditions, not universal legal thresholds. For each path, name a success indicator and a possible harm, such as more acceptance errors or exclusion complaints, and identify who can halt expansion.

Finally separate absence of evidence from evidence of absence. No recorded complaints from a group does not show that access is easy; people may not know or trust the complaint route. An inconclusive small sample does not establish that an effect is zero. State sample size, recruitment and follow-up period. When presenting the matrix, show which explanations were actually ruled out, which remain plausible, what small decision can be made now, and what larger commitment should wait. Diagnosis then becomes a decision plan rather than a list ending in 'more research is needed' without a task, deadline or accountable owner.

Lesson summary

The learner converts gap indicators into rival hypotheses and testable predictions before selecting an intervention.