Decision question and experiment protocol · Lesson 4

Ethical and privacy feasibility

6 min read

Lesson objectives

  • Review experimental burden, privacy and contact-channel permission in a pre-launch risk register.
Reading tools

The ability to randomize people is not permission to run every experiment. Begin with the hoped-for benefit, who bears the burden and what each arm will receive. If one arm deprives people of an essential service to which they are already entitled, that comparison may be unacceptable even if it would give a clear estimate. Look instead for two acceptable service versions, a safe sequence of delivery or a lower-risk question. Where there is genuine uncertainty between acceptable alternatives, explain why the test is being run to the decision owner and to participants as the context requires. Do not use “test” as a label that bypasses service duties or shifts risk onto people least able to object.

Separate two decisions: may the team contact or collect data from these people for this purpose, and can the design measure a valid difference between arms? Messaging might be permitted in a particular service while the comparison is contaminated. Conversely, allocation could be statistically sound while use of a sensitive address is inappropriate. Check where contact data came from, its stated purpose, who sees the record, how withdrawal works, retention and which fields the question actually requires. Do not put names, message content and a sensitive outcome in a broadly accessible analysis file if a pseudonymous ID and limited outcome suffice. Minimization and restricted access reduce potential harm, but do not by themselves authorize a use that has not been reviewed. Legal requirements differ by jurisdiction and service. Guidance from a UK regulator illustrates its UK setting, not a rule for every learner's location.

Participant burden is itself consequential. A survey may take ten minutes from someone with little spare time; a message may reveal a personal situation to someone sharing a device. Identify risks before finalizing metrics: information disclosure, pressure to participate, exclusion of people without a digital channel, uneven service quality and staff burden. For each risk state how it could occur, who is affected, how the team would notice it during delivery, and a mitigation the team can actually perform. “We protect privacy” is an intention. A checkable control is to omit the appointment reason from a notification, test its lock-screen display, restrict the ID-to-name linking table to a named role, and process withdrawals before the next batch.

Make a pre-launch gate with distinct decisions. Proceed if both alternatives are acceptable, the responsible function has reviewed the data and channel, and monitoring and mitigations are executable. Redesign if a risk can be removed without destroying the question, for example by replacing a sensitive notification with a generic one. Stop if participants cannot be protected or an arm creates unjustified deprivation. Do not reduce the gate to a total score that hides one severe hazard; a single serious risk may block launch even when its estimated likelihood is low. Record the gate owner, decision, date and reason for any exception so that a favorable effect cannot retrospectively excuse an ethical defect.

During the test, watch a guardrail that can trigger a pause, such as a complaint about disclosed message content or repeated delivery to a wrong address. Specify who reviews a report, how quickly, how messaging can be disabled without withdrawing the essential service, and how evidence is retained without further exposure. The ethical stop rule is distinct from the primary outcome's success rule; an apparently beneficial test can still be stopped for harm. Afterward, state the knowledge limit honestly: the estimate does not automatically represent people excluded to protect privacy. This balance is part of the professional decision, not a footnote added after analysis.

Lesson summary

Valid allocation is insufficient; an auditable risk register determines whether to launch, redesign or stop a test.