From a broad request to an answerable advisory question
5 min read
Lesson objectives
- The learner turns a broad commission into a decision question with population, period, comparison, inference level and professional boundaries.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A sponsor asks, 'Prove that our benefit reduced household hardship.' The sentence sounds specific but demands a causal conclusion before the decision or available evidence has been defined. The pending decision might be renewed funding, a change in benefit value, or a repair to access. Each calls for different evidence. The adviser's first move is not to choose a statistical method. Meet the decision owner and data steward to establish which options are open, when a decision is due, and what finding would change it. If the agency cannot alter the benefit value this year but can repair disbursement, a study of the payment amount alone will miss the near-term choice. Record the original request and restate it as a question that the data and time can answer honestly.
Build the question from six linked elements. Decision: which feasible options are compared? Population: who is covered and who may be affected while missing from the registry? Period: which months or payment cycles? Outcome: enrolment, timely receipt, or hardship measured through a defined indicator? Comparison: against what is the outcome judged, and how do the groups differ? Inference level: description, a plausible contribution, or an estimated causal effect? Give each element an operational definition. 'Did the programme work?' has no denominator or time frame. A more useful question is, 'Among households estimated eligible in three districts during the last quarter, where does access fail between application and payment, and which assisted-access change can be tested in one cycle without increasing payment error?' This is a diagnostic decision question. It does not claim to isolate the programme's effect on general hardship.
Keep four layers of statements separate in the scope memo. Facts are observations from dated and defined records or surveys. Estimates derive from samples or models and carry uncertainty. Explanations are competing hypotheses linking a pattern to a mechanism. Recommendations are practical judgements conditioned on criteria, resources and risks. A rise in applications after an outreach campaign is an observation. The campaign may have improved awareness, but need or recording rules may also have changed; the rise alone does not establish its cause. If hardship fell at the same time, employment or prices may have shifted too. A defensible statement is that the decline coincided with the intervention while available data cannot isolate its effect from external changes. Propose a feasible comparison if the decision needs a causal claim. The OECD's evaluation criteria can help ask about relevance, coherence, effectiveness and impact; naming a criterion is no substitute for evidence designed to answer it.
Agree at the outset who may use the data and for which purpose. An individual claimant file may contain details unnecessary for policy analysis. Consent to receive a service does not automatically authorize publication of a person's story. Define aggregation, data-owner quality checks, and a path for legal or safeguarding questions to an authorized specialist. This programme does not empower an adviser to settle an individual's entitlement, make a clinical diagnosis or determine a child-protection case. If an urgent individual complaint appears, route it appropriately rather than treating it as prevalence evidence from a representative sample. Separating system advice from case decisions protects both claimants and the institutional decision.
Test answerability before accepting the commission. For each requested claim, list the possible source, what is missing, and the strongest statement the evidence permits. If the sponsor asks for proof of impact but there is no baseline, credible comparison or reliable exposure timing, say that the study can describe delivery and possibly a change in outcomes; it cannot confidently attribute the change to the intervention. Propose a baseline or future comparison compatible with ethical and operational constraints. Ask which decision can be deferred and which must be made conditionally under uncertainty. Record an assumption whose failure would change your advice. This turns future data into a reason to update a recommendation instead of defend an absolute claim.
The output of scoping is a one-page agreement containing the revised question, options, population and period, analytic products, inference limits, data sign-off owner, and the next decision point. Readers should understand why an appealing but unsupported question was narrowed and how the new question serves the actual choice. Do not erase people experiencing the access barrier: plan an appropriate way to hear them while stating that purposive interviews do not yield a population rate. Nor should the funder's preference silently stand in for all affected groups' priorities. These differences matter to the relevance of advice and to who bears the risk of a mistaken conclusion.
Lesson summary
The learner turns a broad commission into a decision question with population, period, comparison, inference level and professional boundaries.