Decision framing and traceable search · Lesson 3

A bounded search protocol

8 min read

Lesson objectives

  • A bounded search protocol
Reading tools

You have five days to brief a decision about adding remote education support instead of offering only face-to-face sessions. A team enters one phrase in a general search engine, finds a favorable study, and writes that “the evidence favors” remote support. The problem is not merely the short deadline. No plan tells the reader where the team searched or what it would have done with an unfavorable result. A search protocol is a plan recorded before execution that connects the decision question to sources, terms, limits, dates, and a record trail. A bounded professional dossier does not claim systematic-review completeness, but the deadline does not excuse an irreproducible search. Put a specific aim at the top of the log: estimate the effect of remote support on task completion and access over one term, while noting operating costs and unwanted effects.

Choose sources for the kinds of evidence required. A subject-indexed education database may find studies, a trial register may reveal planned or unreported intervention outcomes, and methodologically described organizational reports may inform context. Internal operating data may help with local baseline and costs if its definitions can be checked. A general web search helps discover terms or documents, but by itself does not define a coverage boundary. Do not count a vendor report as an independent study until you inspect its sample, methods, and relationship to other reports of the same participants. Conversely, do not exclude unpublished results simply because they are less visible. Record both accessible material and inaccessible material. If only two sources can be searched before the deadline, state which two, why they were selected, and what they may miss. This makes the resulting decision appropriately conditional.

Test the query against two papers already known to be relevant. One calls the intervention virtual mentoring and the other synchronous support. If the first is returned and the second is absent, find the missing expression and add it to the synonym group. Keep both versions and the date of revision. A known-item check does not prove comprehensive retrieval; it exposes an obvious miss. Also inspect ten irrelevant results. If they mostly concern course retention rather than completing a task, refine the outcome concept or make the eligibility rule clearer. Search sensitivity is the proportion of relevant records retrieved; precision is the proportion of retrieved records that are relevant. A deadline puts pressure on screening capacity, but a neat-looking query that misses decision-changing evidence is a poor bargain.

In a worked fictional case, source A returns 84 records on October 3 and source B returns 31 on October 4. Removing 17 duplicate records leaves 98 unique records: 84 + 31 − 17. Title and abstract screening removes 72, leaving 26 records for full-text review. Do not call these 26 “studies.” Several records may report different outcomes or times from one study. Read the full texts and record specific exclusion reasons, such as no comparator or no outcome in the chosen period. Then link reports of the same sample to one study identifier. If nine reports describe seven studies, seven is the study count entering cross-study synthesis; the extra reports can still supply useful details. If three full texts cannot be obtained, do not mark them ineligible just because access failed. Record that unresolved status and its possible effect on the conclusion.

The output is a search log containing the question, each source, the exact query used there, run date, language and date limits, numbers retrieved, deduplication rule, and reasons for full-text exclusion. Record who screened the material and how disagreements were resolved. Under time pressure, the team may be unable to double-screen every title. State the actual independently checked sample and the disagreements found; do not claim complete duplicate screening that never occurred. If disagreement clusters around the meaning of “remote support,” revisit the eligibility rule before screening further. Do not change that rule merely to remove findings inconsistent with expectations. The log lets a reader reconstruct the path from question to included studies while seeing where coverage is limited.

Translating a query between databases is a reasoning task, not a paste operation. One interface may search title and abstract by default, another may search full text or require field tags for proximity. Put the intended concept beside the executed syntax in the log. If a database returns thousands of unrelated results, inspect its search field before adding arbitrary conditions. Raw result counts do not measure database quality; sources differ in scope, indexing, and duplicate records. In a bilingual search, the Arabic term for mentor may cover a different professional role from the English label. Inspect samples of results in both languages rather than trusting literal translation. If an information specialist revises the strategy, preserve the earlier query, their contribution, and the reason for the changed result count.

Date limits need a decision-relevant reason. If a support platform changed technology substantially five years ago, recent intervention studies may be more transferable to its direct effect. An older study of barriers to access might still inform the equity question. Keep those two purposes separate rather than excluding every older source with one rule. A language restriction can save time but may exclude settings close to the target population. If it is unavoidable, disclose the possible effect and at least examine available multilingual indexes or abstracts. Inability to translate a report does not make it irrelevant. Distinguish when the study took place from when its report was published; both dates may matter when technology and delivery models evolve.

Give the search a reviewable stopping point. After running the planned sources on the declared dates, inspect references of included studies and relevant nearby reviews for studies the query missed. If a missed eligible study changes the direction of an outcome, repair the query or source selection and update the log. Do not silently add it by hand. Finding no such study does not prove universal completeness, but it tests the stated boundary. Set an update route for evidence arriving after the brief: who receives it, which decision outcome it could change, and when the recommendation will be reopened. A bounded search needs both a cutoff date and a trigger for later reconsideration.

For a counterexample, suppose the vendor gives the team only its own bibliography. It may contain a relevant study, but it is not a balanced search frame. Treat it as one discovery route, then search independently for adverse and null findings using the same eligibility rules. Even if the known-item test succeeds, a source containing only the vendor's reports is not adequate coverage for a contested purchase. A careful conclusion says, “We searched sources A and B and the trial register through the stated date and checked references of included studies. Unindexed or inaccessible reports may remain.” That sentence does not make a professional decision useless. It tells the decision owner what would justify revisiting it.

Before handoff, reconcile the counts. The worked search retrieved 115 records, removed 17 duplicates, and left 98 unique records. Screening excluded 72, leaving 26 for full-text review. If a diagram says 25, find the missing record before presenting it. An inaccessible full text is not an exclusion based on content because its content was not examined. Reconcile study identifiers separately from report identifiers so two reports of one study do not gain double weight later. Accurate arithmetic supports traceability; it does not prove that the search was exhaustive.

Lesson summary

A bounded search protocol