Observation, uncertainty and causal claims
4 min read
Lesson objectives
- Calculate percentage-point and relative changes in a reminder test and bound causal interpretation.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A club sends ticket holders a reminder before the next match, then attendance rises. Did the reminder work? A change observed after an action does not answer that question by itself. Opponent, weather, start time, usable capacity and buyer mix may also change. Describing what happened differs from estimating what would have happened without the reminder. Start with the timeline and an outcome definition, then seek a credible comparison before calling the difference an effect.
Suppose valid-ticket usage rises from 72% to 78%. That is six percentage points, not a six-percent increase from the original rate; the relative rise is 6÷72≈8.3%. Even this arithmetic requires a stable denominator. Were voided tickets excluded in both periods? Did complimentary allocations change? Fix inconsistent measurement first. If the figures cover all known entitlements, “sampling error” is not necessarily the main uncertainty. Unmatched scans and external changes may matter more. A confidence interval designed for a random survey should not be used to conceal administrative-data quality problems.
This comparison improves on a simple before-and-after report because it tries to estimate the outcome for an eligible group without the message at the same time. Random allocation helps balance pre-existing factors on average, but cannot cure faulty execution, biased records or generalization beyond the tested group. If staff selected the most engaged fans for the message group instead, the difference would be confounded by that prior selection. Report group assignment, size and result alongside the limitations before recommending a monitored rollout.
Also decide whether the analysis estimates the effect of *assignment* to receive a message or of *actually reading* it. If some messages fail to arrive, removing those holders from the assigned group after seeing the result can break the randomized comparison: reachability may itself differ across fans. Keep people in their assigned groups for the primary report and show delivery rates separately. If no attendance difference appears, do not conclude that reminders are worthless everywhere. Check measurement, sample size and the limits of this event before deciding whether a controlled repeat is useful.
Lesson summary
An observed post-reminder change is not automatically causal. Random assignment improves comparison when groups stay intact and delivery failures and generalization limits are disclosed.