Evidence decision and mandate
8 min read
Lesson objectives
- Evidence decision and mandate
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
Imagine the appointments director telling an evaluation team, “Prove evening appointments are better.” Before searching, ask better for whom, on which outcome, against what comparison, and over what period. The desired conclusion is embedded in the request. It may encourage selection of favorable evidence while obscuring costs and harms. This evidence dossier supports a bounded organizational choice whose limits can be inspected. It does not claim to identify every study or produce a publishable systematic review. Write the available choices first: keep the current schedule, run a limited evening pilot, or expand evening sessions. Identify the affected groups, decision owner, and deadline.
Set decision thresholds before seeing results. In this example the organization might expand if median waiting falls by at least three days, cost per completed appointment rises by no more than eight percent, and the no-show gap between the two groups does not widen. These are fictional management choices, not published benchmarks. Shorter waits at much higher cost may justify a smaller pilot. A better overall average alongside worse access for people unable to attend in the evening does not justify universal rollout. Define the numerator, denominator, period, and data source of every measure. A no-show proportion is booked appointments not completed divided by all booked appointments, with early cancellations handled consistently. Cost per completed appointment is operating cost for the period divided by completed appointments, not by offered slots.
In the worked answer the team rejects “prove superiority” and writes a comparative question that permits conflicting findings. It records the assumption that all four branches define an appointment in the same way, then checks booking records. If one branch counts an early cancellation as a no-show while another removes it from the denominator, comparison pauses until the definition is aligned. Branch allocation is not randomized; a post-pilot difference alone cannot establish causation. The output is one page with alternatives and thresholds and another with the question, evidence eligibility rules, last search date, and coverage limits. Someone absent from the meeting should be able to find which outcome would change the choice, where it came from, and which failed assumption would halt inference.
Before collecting evidence, turn the question into eligibility rules another reviewer can apply. One rule might include appointment services that compare access outside ordinary hours with usual scheduling and measure waiting or completed appointments over a defined period. A study only of staff working hours could inform a cost question, but it does not by itself answer the client-outcome question. Keep it under a separate question rather than treating every relevant document as an effect study. If two reports describe the same participants, link them under one study identifier; otherwise their outcomes may be counted twice. If a no-show report omits its denominator, request it or mark that result unavailable for numerical comparison. Do not invent a denominator to complete the table. Ideally, write eligibility rules before learning the direction of the results. Changing them after seeing an inconvenient finding creates an opportunity for selective inclusion.
Consider four fictional sources: a comparison study in an urban center measuring waits, a nonrandomized local pilot measuring no-shows, a staff report on shift costs, and a vendor blog claiming 95 percent satisfaction without explaining its sample. They do not deserve equal weight or answer the same question. The first may inform waiting time, subject to whether its setting transfers to the four branches. The second requires checking branch selection and simultaneous changes. The third is a cost input, not direct evidence of clients' outcomes. The blog does not supply a dependable satisfaction estimate until we know who was asked, how they were chosen, and what was asked. Record the claim and possible interest without calling it verified evidence. These are not final certainty grades. They map which source can address which decision question before later lessons appraise bias and certainty.
Keep a reproducible search log: source, run date, exact query, language and date limits, records before and after deduplication, and a reason for each full-text exclusion. An exploratory query might be ("evening appointment" OR "extended hours") AND (waiting OR no-show), with corresponding Arabic terms. It is a starting point, not a guarantee of coverage. It could miss “after-hours” or a paper described as flexible booking. Test it against a known relevant paper. If that paper is absent, find out why, revise the terms, and retain the old and new query with their dates. Zero results from one query do not justify “there is no evidence.” Say instead that no eligible source was located in the specified sources by the recorded date. The second claim reaches much further than the search trail permits.
The threshold itself merits sensitivity testing. Suppose the pilot reduces waiting by four days, exceeding the three-day threshold, and estimates a seven percent cost rise, below the eight percent ceiling. Before expanding, inspect uncertainty in the cost calculation. If the completed-appointment count includes cancellations, correcting that denominator might raise the cost increase to ten percent and reverse the decision. Show both calculations and explain the change; do not select the one favorable to expansion. If the measurement period includes an unusual holiday week, compare a like period or avoid attributing the whole difference to evening hours. “What would change the decision?” now has a precise check: would a corrected denominator or comparison window move the estimate across a prespecified threshold? A later lesson will formalize bias assessment; here the point is to keep the decision open to disconfirming evidence.
Separate three lines in the briefing: what was observed, what can be inferred under assumptions, and what is recommended. The observed fact may be a shorter wait at the two pilot branches. A causal interpretation is weaker if those branches had different demand or received extra staff at the same time. A recommendation may therefore be a better-controlled pilot, not immediate expansion. Attach a source and limit to each line. “The results are positive” hides differences among outcomes. At the decision meeting, invite one specific challenge from a client representative and another from operations. The former may identify an accessibility harm; the latter may identify shift costs missing from the estimate. Record each challenge, the response, and a way to check it. A defensible professional decision lets readers see the disagreement rather than asking them to trust the author's tone.
State what the dossier can and cannot conclude before the recommendation. It can summarize evidence found in named sources by a recorded date and explain whether results and settings are comparable. It cannot establish that no other studies exist, turn an uncontrolled association into a causal effect, or claim the status of a published systematic review. Those limits need not prevent action. Limited evidence may support a reversible pilot with baseline measurement and stopping conditions. Ask which missing piece of information is most likely to reverse the choice within the decision window. Correcting the cost denominator tomorrow may move the estimate across the eight percent ceiling; a broad attitude survey next year cannot resolve today's choice. Prioritize the former and place the latter in an update plan.
Lesson summary
Evidence decision and mandate