Inspection and adaptation when experiment conditions change
8 min read
Lesson objectives
Build a transparency-inspection-adaptation loop for an ambiguous Sprint signal
Reading tools
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
In Scrum the initial plan is not treated as a guaranteed prediction. A Sprint Goal gives the team direction; the team builds a Done Increment and uses events to inspect progress, the product and its way of working. Useful inspection needs a transparent signal. What exactly is the measured event? Who enters the denominator? Are the periods comparing the same kind of users? Which change actually reached the Done Increment? If the conditions differ, a raw comparison can conceal a problem or invent one. The Product Owner's job is not to reverse every decision at the first unpleasant number, but to make the hypothesis and its limits visible and order valuable learning.
Imagine an appointment booking service. In the first Sprint, 300 of 500 booking starts complete. In the second, 330 of 600 complete; 200 of those starts come from new visitors attracted by a different campaign. Overall completion changes from 60% to 55%. This alone cannot show that a booking interface change caused a five-percentage-point loss. We need separate results for returning and new users and assurance that the definitions of start and completion remained stable. Suppose 280 of 400 returning users completed in the second period, a 70% rate, while 50 of 200 new users completed, a 25% rate. Returning users currently have a higher rate than newcomers, but we do not know whether either segment improved over its past performance. The aggregate rate does not tell the team where to adapt.
Write a short decision protocol before inspecting the outcome. First, state the Product Goal and how the Increment relates to it; do not replace the value measure after seeing the figures merely to declare success. Second, verify Definition of Done and measurement integrity: an untested screen must not be presented as a Done Increment. Third, compare relevant segments and record their sizes and dates. Fourth, choose the smallest adaptation that answers the decision question. Does the campaign message need work, is the first step hard, or should the interface change be reversed? Fifth, set a review point and the user result the team wants to observe. These are supporting analytical practices, not a Scrum-mandated research procedure. Their purpose is to prevent a causal leap.
Different Scrum events serve different inspection purposes. At the Daily Scrum, Developers inspect progress toward the Sprint Goal and adapt their daily plan; it is not an individual status report to the Product Owner. At the Sprint Review, the Scrum Team and stakeholders inspect the Sprint's outcome and progress toward the Product Goal and may adapt the Product Backlog. At the Sprint Retrospective, the team inspects its work and interactions and plans improvements to its effectiveness. Confusing these purposes can leave every meeting on the calendar while the needed learning decision is absent. If risk threatens the Sprint Goal during a Sprint, the Product Owner and Developers can negotiate scope without compromising quality or treating the goal as a moving list of requests.
Imagine a trial whose decision changes whenever a convenient number appears. After one good day the team wants rollout; after a weak day it wants more observation. The review time itself can then become biased toward the desired result. Before starting, agree a suitable observation period, a learning question, and a harm signal that warrants early stopping. This working agreement supports empiricism; it is not an additional prescribed Scrum event. Specify what should remain stable for comparison. If the audience changes during the trial, record a new condition that affects interpretation. Ethical or operational reasons can justify adapting a plan, provided the reason is explicit. Choosing whichever moment produces the best number distorts learning.
The framework values help make inspection candid and safe. Commitment connects effort to goals and mutual support; focus directs attention toward work advancing those goals. Openness makes evidence and impediments visible. Respect treats contributors as capable professionals while their assumptions remain open to examination. Courage supports saying what the decision needs even when it challenges a comfortable expectation. Respecting a colleague is compatible with inspecting that colleague's view. When contrary evidence appears, ask about its source and context and discuss implications without turning disagreement into a judgment of the person.
Lesson summary
Set the learning question, review point and harm limit, then adapt when evidence reveals changed comparison conditions.