Choosing the comparator
6 min read
Lesson objectives
- Choose an implementable comparator that separates a component effect from a bundle and diagnoses contamination.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
The comparator is not loosely “whoever did not get the new thing.” If a manager must choose between buying a messaging service and improving the booking form, comparing messaging with doing nothing does not answer the investment choice. If the decision is to add a reminder to current service, the relevant comparator is current service in the same period under the same operating conditions. Write the feasible decision alternatives first, then ask what contrast between two implementable policies the manager needs. An active comparator, such as a new short message against the usual message, may be appropriate when communication cannot or should not be withheld. It estimates the incremental effect of a change, not the effect of any communication versus none. Keep that interpretation in the report.
Define each arm as a recipe that an independent operator could deliver: message text, timing, channel, handling of undelivered cases and accompanying staff actions. State the component intended to vary and hold the others fixed where feasible. If the new arm receives a message and a staff call while the control retains old confirmation without a call, the contrast concerns the package of message plus call. It cannot be attributed to the message alone. The package might itself be a legitimate decision if the manager intends to deploy both components together; then name and cost the whole package. If the manager needs to choose which component deserves funding, the design must isolate components or deliberately test their combinations.
Concurrent operation reduces time confusion but cannot repair implementation failures. Staff may learn the new script and apply it in the current-service arm; a customer may view both page versions on different devices. This is contamination: the experiences actually delivered become closer than the protocol promised. Before launch, inspect the exposure path and define how the displayed version, change permissions and repeat visits will be logged. If staff in one branch inevitably share the new script, team or branch assignment may be more defensible than individual assignment, while leaving fewer independent allocation units. A randomized label in a database is not sufficient proof that different experiences reached people.
Contamination appears in the delivery record as a gap between “assigned” and “shown.” Log the unit ID, assigned arm, version actually displayed, display time and reason for each exception. Comparing displayed versions after the test is no simple substitute for the assigned-arm analysis: people who request extra help may already have harder cases. Still, the display log can explain a muted contrast and inform whether the next test needs stable device assignment, grouped repeat visits or whole-branch allocation.
Select benefits and guardrails that can be measured in both arms. A simplified form might increase completed requests yet raise manual rework; the decision concerns the service as a whole, not completion alone. Use equal observation windows and measurement effort. Do not call the new arm to solicit satisfaction and then treat the control's lack of recorded complaints as a fair comparison. Plan a log of operational changes, such as another campaign, changed opening hours, a system outage or staff training. A change affecting both arms similarly during the same period may leave the difference informative; a change isolated to one arm weakens attribution to the named component.
When several alternatives are real, a two-arm experiment does not automatically answer them all. The team might compare the two highest-value options first and explain the cost of postponing a third, or design multiple arms if resources and power permit. With many pairwise contrasts, the primary comparison must be fixed before results are seen. If withholding a service would be ethically inappropriate, compare two acceptable versions or safe timings, but do not promise a strong causal conclusion unless assignment and exposure support it. The professional task is to align the manager's question with an observable contrast, not to select a fashionable design label.
Lesson summary
The comparator follows the actual decision alternatives; changing support and page together estimates a bundle, not the page alone.