Screening and relevance
7 min read
Lesson objectives
- Turn a practical problem into a researchable question and documented evidence-search plan
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A record does not become relevant evidence merely because its title contains question terms. Screening for eligibility is separate from judging the trustworthiness of a result. Set rules before learning which direction the studies report, so selection does not become a search for confirmation. Link each rule to population, outcome, design and time period that matter to the decision. If a rule must change after viewing records, retain its previous version, reason and effect on the set of studies. Do not present a later decision as if it had been specified in advance.
Use a screening register with record identifier, title-and-abstract decision, full-text status, eligibility decision, principal exclusion reason and reviewer. If a short title is ambiguous, do not guess that the record is ineligible. Mark it unresolved and obtain the abstract or full text. Two genuinely independent reviewers may reduce dependence on one judgment where resources permit. Names in a register do not establish that independent screening occurred. Resolve disagreements against the published rule and record the resolution. A bounded scan with one reviewer should say so rather than claiming the procedures of a full systematic review.
After eligibility, ask whether the result supports the proposed claim. A study of service-centre staff may be relevant in population but measure satisfaction rather than departure. It can inform context but cannot estimate resignations. Another study may measure departure clearly in a different economy or period. Its relevance is conditional on explaining transfer to the present setting. Record “direct,” “indirect” or “context only,” with a reason; do not collapse these categories into one numerical average.
Keep relevance apart from risk of bias. For a randomized trial, the Cochrane Handbook's risk-of-bias chapter makes clear that assessment concerns a particular result; concerns may differ across outcomes within the same trial. Do not stamp a whole paper “good” and inherit that judgment for every reported number. Examine allocation, missing data, outcome measurement and selection among multiple possible analyses. Observational comparisons face further challenges, such as pre-existing differences between groups. Do not apply a randomized-trial risk tool to qualitative work simply because the tool is familiar.
Every claim in a decision memo needs a trace: question, eligibility rule, included record, particular result, credibility concern and limited conclusion. If studies define departure differently, do not calculate a pooled number before assessing comparability. Where evidence is insufficient, write what can be stated now and what further data would settle the issue. “No evidence was found in these sources through this date” is narrower than “there is no evidence.” Screening thereby prevents an inflated inference rather than merely filling a flow chart.
For an eligible study, separate three axes: relevance to the question, risk of bias within the study, and transferability to your decision. A trial may be carefully conducted in its population but far from the centre's workforce. A local observational report may be highly relevant yet measure departure poorly. Give each axis its own reason. “High-quality study” should not become a pass for every claim. If an important detail is missing from the report, mark it unclear rather than guessing favourable or unfavourable.
When reviewers disagree about eligibility, show the disputed wording and the rule applied, not only two yes/no votes. Perhaps retention means intention to stay on a survey rather than observed employment in a register. Repair the outcome dictionary, reapply the rule to similar records and count changed decisions. A consistent rule is better than ad hoc record-by-record resolutions. With only one reviewer, rechecking a sample of exclusions later can still reveal mistakes, but label it self-review, not independent screening.
Do not place an excluded study in the direct-evidence table because its finding is interesting. It may appear separately as context with that role made explicit. When full text is unavailable, do not infer detailed methods from its abstract. State what could be verified and what remains unknown; seek the text through a legitimate channel if the record matters to the decision. Screening is not a contest to minimize paper count. It makes the chain from question to answer traceable.
Lesson summary
Screening and relevance: apply prespecified inclusion rules and distinguish evidence relevance from certainty.