Decision question and experiment protocol · Lesson 2

Defining the estimand

7 min read

Lesson objectives

  • Define a trial population, treatment, comparator, outcome window and contrast while retaining the assigned denominator.
Reading tools

“Did the new journey work?” can yield incompatible answers because work does not specify a population, outcome or time. Write the target effect in five slots before implementation: who enters the calculation; the offered intervention policy; the alternative policy; the outcome and its window; and how the difference will be summarized. For example: among all eligible enrollees in a two-week intake, what is the difference in the proportion completing lesson one within seven days between assignment to personalized onboarding and assignment to the current start message? The denominator is everyone assigned, the outcome has a fixed window, and the contrast is a percentage-point difference between two rates. “Increase engagement” is insufficient: opening a message, starting a lesson and finishing it are separate events with different decision implications.

For a difference in proportions, subtract the comparator outcome rate from the intervention outcome rate. If the rates are 72% and 66%, the absolute difference is +6 percentage points; the relative increase is 6/66, about 9.1% of the comparator rate. Both calculations can be valid, but silently switching between them misleads a sponsor who set a six-point threshold. State the beneficial direction. If the outcome is an error or missed appointment, a lower rate may be better, and the sign convention must agree with the decision rule. Hold the counting unit fixed. The raw number of completions favours a larger group even when its completion rate is lower. When one person creates several events, do not place event counts over a person denominator without defining what a single outcome means.

Assignment, exposure and use are different events. A person may be assigned a personalized message that never arrives; a worker may show the other arm's material. The first policy question asks what happens when the offer is assigned to each eligible unit, retaining undelivered cases in their originally assigned arm. Often called intention-to-treat analysis, this preserves the comparison created by allocation. Looking only at message openers selects more engaged people after assignment and no longer estimates the offer-policy effect. An effect of actual receipt or adherence can be investigated as another question with additional methods and assumptions; it must not be read directly from the assigned-arm estimate. Log the number assigned, reached, opened and completing the lesson so that the team can diagnose where an effect was lost without replacing its primary outcome.

Resolve events that occur during the trial. What if an enrollee withdraws, changes email, or completes a different lesson first? A protocol may retain every eligible assigned enrollee in the denominator and count no recorded lesson-one completion within seven days as no completion, while distinguishing a truly missing measurement from a recorded non-completion. This is not permission to invent a lost record's outcome; missing measurement needs its own investigation and sensitivity analysis. Specify permissible exclusions before assignment, such as test accounts or duplicates identified by a stated rule. Do not exclude “inactive subscribers” after assignment if inactivity might be changed by the intervention. In the result, name the population the effect concerns rather than extending it to ineligible people or a different operating period.

Keep the entry rule separate from later analysis. If bot or test accounts are identified before allocation, a stated exclusion rule can be applied equally to both arms. If delivery later fails because an address is invalid, removing only those cases from the intervention arm selects a different group from the one assigned. Record the failure in a flow table and retain the planned primary comparison. Then examine sensitivity to plausible explanations for a missing outcome record. Do not call a changed denominator mere “cleaning” without showing how it changes the question; the decision can move even when the completion count does not.

The estimand is a contract for the question, not a statistical estimate or proof. Choosing a rate contrast does not guarantee that a small sample can distinguish a useful effect from fluctuation. Person-level assignment does not guarantee that shared family mailboxes have no cross-arm interference. The desired answer and the ability of a design and its data to identify it are separate matters. As a protocol check, read the five slots as one complete question and ask whether every condition can be implemented and the outcome observed for all units with the same rule. If not, repair eligibility, allocation or logging before launch instead of polishing the description after collecting data.

Lesson summary

The target effect is a prespecified five-part question; opening or receiving a message does not replace assignment in the primary comparison.