Decision question and experiment protocol · Lesson 1

A testable decision

6 min read

Lesson objectives

  • Frame an experiment that links the alternative, comparator, denominator and guardrail to a defined expansion decision.
Reading tools

“Test reminders” leaves the decision underspecified. The sponsor might want to buy a messaging service, change appointment times, reduce missed visits, or learn whether any attendance gain is worth the operating cost. Start with feasible choices: retain current confirmation, add a defined reminder, or improve the booking page instead. A comparison cannot answer the sponsor if the alternatives remain vague. State the decision rule before outcomes are visible: adopt reminders if attendance among eligible bookings rises by a useful amount without breaching a harm limit; otherwise revise or retain current service. The useful increase and harm limit belong to the decision owner and need an operational rationale. They are not universal constants.

Define who enters the comparison before drawing a sample. A booking is not always a person: one person can book twice, and a family may share a telephone number. If assignment is by booking and repeated bookings cross arms, the same person may receive both experiences. The protocol therefore records the eligibility unit, assignment unit and analysis unit, with a consistent rule for repeats. Define when measurement begins and ends. Attendance on the scheduled day is different from opening a message, and clicking a link is not equivalent to attending. Select one primary outcome tied to the decision and protective measures such as complaints or messages sent to a wrong number. Measure outcomes and guardrails comparably in both arms.

Last month alone is a weak comparator. Opening hours, case mix, weather or front-desk staffing might change between periods. If eligible bookings can be randomly assigned at the same time to reminder or current confirmation, the comparison better separates the reminder from those changes, provided assignment is implemented as planned and exposure and outcomes are logged. Randomization does not guarantee identical small groups, repair cross-arm contamination or make differential missingness disappear. If random allocation is infeasible or ethically inappropriate, a before-and-after comparison must not be labelled a randomized experiment or reported as a secure causal effect.

The memo also needs a practical boundary. A reminder may help people who agreed to this contact channel but say little about people without email or people who opt out. Permission to use contact data and the right to withdraw are governance questions in the relevant setting; attendance data cannot answer them. This educational case does not establish local legal requirements. If a design would withhold an essential service or disclose sensitive information, consider a less harmful design first. Name a decision owner and date. Even a precise effect estimate has little use if no feasible action follows it.

Draw the causal chain before assigning anyone: a reminder is sent, it reaches the intended person in time, the person remembers or reschedules, and attendance changes. Each link can fail for a different reason. If, for example, a quarter of addresses cannot receive messages, a small assigned-arm effect does not prove the wording itself is useless; it may show that the delivery policy as implemented is ineffective. Do not discard undelivered cases from the primary comparison to obtain a more attractive estimate. Log delivery and discuss it as a later explanation. More rescheduling might be an interesting secondary benefit, but it cannot replace the primary same-day attendance outcome after the team sees results. The chain shows exactly what this experiment tests and what would require a later test.

Lesson summary

A useful test starts with a defined decision and concurrent comparator, then fixes the assignment unit, outcome and guardrails before results appear.