Appraising evidence before transferring it to a local decision
5 min read
Lesson objectives
- Compare studies, distinguishing internal inference from conditions for transferring effects.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
Once a policy question is framed, an analyst may find one study reporting success for appointment coordination and another reporting a different effect. The task is not to choose the most attractive estimate. Ask what was measured, how confident we are that the intervention caused the difference, and what must hold for the result to apply here. WHO guidance on synthesizing health policy and systems research emphasizes question framing, context-sensitive evidence, equity and complexity. We turn that into a compact evidence matrix for a particular decision, not a ceremonial quality score. A well-conducted study does not give permission to copy its number into every city or patient group.
Start with the unit of comparison. If the decision owner asks about visits within 14 days among high-need referrals, a study measuring patient satisfaction after six months cannot answer it directly. Both outcomes matter, but they answer different questions. Use columns for population and selection method, current service, intervention and implementation, outcome and definition, follow-up, sample size, main threat to inference, and transfer conditions. Note whether the result is a proportion of referrals, a proportion of patients, or mean waiting days. A change in unit changes the denominator and can make superficially similar figures incomparable. State what is unknown instead of filling a cell from the article title.
Consider fictional study A in an urban clinic network. Six hundred referrals were randomly assigned: 300 received a dedicated appointment coordinator and 300 followed usual booking. Within 14 days, 210 in the coordination group and 180 in the comparison group had visited. Rates are 70% and 60%; the absolute difference is ten percentage points, or 30 additional early visits per 300 referrals in this study. Randomization helps reduce baseline differences, but we still ask whether follow-up records were missing, the outcome definition changed, or enough coordinators could be supplied for the proposed service. Do not call the difference ‘a 10% increase’ without clarification: the relative increase is 10/60, about 16.7%, whereas the finding here is ten percentage points.
Fictional study B compared 400 referrals before confirmation messages with 400 after implementation. Early visits rose from 220/400=55% to 280/400=70%, a 15-point difference. But the clinic also opened evening hours in the second period and changed its visit-record system. The before-and-after comparison does not isolate messages from these changes. Messages might help, but they have not been shown to cause all 15 points. Study B also tests a different intervention from coordination in A. Adding ten and fifteen or choosing the larger figure to predict the combined programme would be a mistake. B can inform implementation questions and possible costs while providing a weaker estimate of the causal message effect.
Separate internal inference from transferability. Internally, ask whether the comparison is fair within the study, and how much missing data, bias or chance could affect the result. For transfer, compare your population, baseline pathway, available staff, capacity and language with those in the study. Even if A's difference is fully valid in its setting, predicting 30 additional early visits per 300 local referrals assumes similar people and service capacity. If one group has less reliable phone access, messages in B might widen inequity while improving the overall average. Put each assumption beside the number so it remains visible to the decision owner instead of silently becoming a local fact.
A disciplined analytical decision is possible now. Treat A as initial evidence for the direction of coordination's effect under its study conditions, use a range rather than a certain local prediction, and treat B as an implementation signal rather than a credible causal estimate for messages. Request three local inputs: baseline visits by need and specialty, coordinator availability and hourly cost, and missing-visit-record rates. A perfect study need not precede every step; an authorized limited test can have a comparison and stopping rule, subject to patient and data protections. The matrix shows what supports each conclusion and what does not. If clinical safety is implicated, seek the relevant professional's assessment rather than treating an administrative comparison as a clinical evaluation.
When results differ, check whether the intervention or the measurement changed. Study A counts a documented visit within 14 days of referral. In B, confirm that the new system uses the visit date rather than the entry date. A software change could move cases across the threshold without changing patient experience. Request variable definitions and extraction dates. Examine withdrawal and missing follow-up by group: if people with the longest waits disappear disproportionately from the intervention group, the observed effect can be inflated even when 210/300 is calculated correctly.
“Transferable” should name a measurable condition, not merely a city that feels similar. For messages, measure the share of referrals with a usable phone number; for coordination, estimate staff hours per 100 referrals. If those inputs are unavailable locally, mark the effect assumption unverified rather than treating it as zero or copying the study value. Low, central and high scenarios can help planning, but label their assumptions and keep modelled scenarios separate from observed effects. Research tests a mechanism under study conditions; local analysis asks whether it can be delivered and whether it moves burden onto patients or staff.
Lesson summary
Reference solution: A compares 210/300=70% with 180/300=60%, a ten-point difference.