Decision question and experiment protocol · Lesson 5

Freezing a protocol

6 min read

Lesson objectives

  • Freeze an executable protocol and log an operational correction without changing the decision rule after results.
Reading tools

A protocol is not a ceremonial document; it is a set of instructions another team can follow without guessing. Before first assignment, name the decision owner, effect question, eligibility, allocation unit, both arms, randomization method, primary outcome and window, guardrails, sample-size assumptions, denominator analysis, success rule and stop condition. Give the version a name, date and approver. If a new operator cannot tell whom to send each variant and when, or an analyst cannot tell whether a record enters the denominator, the protocol is incomplete. Test it against two synthetic records: an ordinary eligible unit and a duplicate or person who withdraws. This catches ambiguous eligibility faster than discovering it after hundreds of messages.

Define how much operational flexibility is allowed. A messaging provider may fail and a comparable channel be substituted after review, or an unnecessary data field might be removed to protect participants. Important changes must be recorded with their reason, time and affected units before the test resumes. Choosing a five-day window after the seven-day result looks weak, or promoting a successful secondary measure to primary, changes the question after seeing data. A new exploratory analysis is possible, but label it exploratory and propose a later test. Do not report it as the judgment promised by the original protocol. Freezing does not prohibit learning; it prevents hiding the history of the decision.

Maintain a change log with version, date, change, reason, whether arm results were known at decision time, and possible effects on eligibility, exposure or measurement. If a developer fixes a logging fault before unblinding results, the test may continue after documenting affected records and checking arms. If someone changes the success rule after results are disclosed, the log cannot erase that knowledge. Keep the original-protocol estimate separate from a new analysis and state each limit. Preserve both the original protocol and the working version, not just the latest file. This is a control for inference and governance, not an automatic guarantee that the design is valid.

Check consistency across participant notice, data plan and product version. If the notice promises one message a week, the intervention recipe must not send two daily. If the outcome is “correctly completed request,” do not collect button taps in its place. If an analyst sees arm labels before cleaning rules are fixed, assign responsibility for exclusions under documented rules and delay unmasking where feasible. A dry run on synthetic records does not prove the experiment will succeed, but it reveals conflicting names, windows and denominators before they harm people or erase the ability to measure.

Close the review with a small mapping from each promise in the decision memo to a measurement and an action. For example: reduce missed appointments; measure same-day attendance; require +5 points; expand in a limited way only if complaints stay under 2%. If the promise is to “improve trust” but no valid trust measure exists, remove the claim or add a defined test that does not overburden people. Do not set several competing primary goals without a rule for resolving them. A sound protocol lets the team say “inconclusive” plainly; it does not force a success story because the decision date is near.

Specify what happens if recruitment misses the planned sample. If 400 eligible people were expected but only 120 arrive, an observed difference near zero does not establish that the intervention has no useful effect; the estimate may be too imprecise to rule one out. Set a feasibility-review date and maximum extension before looking at arm differences, including whether an extension uses comparable operating conditions. If the sample plan changes after the direction of the result is known, disclose that and reassess inference; do not rename a data-dependent extension a technical repair. Include an “inconclusive” action: improve measurement, plan a new study, or make a cautious cost-and-harm decision without claiming that the test proved no effect.

Lesson summary

Dated protocol versions distinguish a measurement fix from selecting a favorable outcome after results appear.