Framing a testable question
5 min read
Lesson objectives
- Apply framing a testable question to a case with defined evidence.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A service team receives a message: “The queue is growing, so the new system has failed.” The sentence blends a possible observation about the queue, a causal explanation that blames the system, and a practical judgment about failure. A specialist first separates these claims and asks which decision the analysis must inform. A manager may be considering an immediate rollback. The first testable question, however, might be whether comparable cases waited longer after the change than before it. Ticket volume alone does not answer a waiting-time question, and a waiting-time change alone does not identify its cause.
A useful question specifies the population, period, measure, and comparison. “Was the average time to first reply for incoming email cases higher during the two weeks after launch than during the preceding two weeks?” is more informative than “Did service get worse?” Even that average can hide a long tail. If the decision concerns a one-day response promise, also inspect the share exceeding 24 hours and perhaps the median and distribution. The measure follows the decision. Before collecting data, identify the actions that a meaningful difference would prompt and the size of difference that matters. Otherwise, a team can gather numbers after it has already chosen its conclusion.
Define a case as carefully as the measure. The new system might combine three customer messages into one ticket while the former system counted three tickets. A direct ticket-count comparison would then mix units. A second team might have entered the same channel during the later period. For a late-response rate, specify that the numerator is email cases whose first human reply took more than 24 hours and the denominator is all email cases with a complete observation window in that period. A newly opened case that has not yet had 24 hours should not automatically be counted as on time. Explicit counting rules protect the comparison from a change in recording practice masquerading as a performance change.
A testable causal claim allows observations that would count against it. The mere coincidence of delay and a software launch is insufficient. If delay appeared in a team using the new system but not in a similar team still using the old one, under roughly stable workload, that would strengthen the hypothesis. It would not settle the matter because the teams might differ in other ways. If the delay started before launch or appeared equally in an unchanged channel, the hypothesis would weaken. Write down such disconfirming observations before seeing the results; doing so makes it harder to move the test after the evidence arrives.
Keep three questions apart. A descriptive question asks how much waiting changed and needs consistently defined measurements. An explanatory question asks why and requires rival explanations and further comparison. A decision question asks whether to pause or reverse the rollout and includes the cost of error, delay, and alternatives. A team can find a real increase without knowing its cause and still take a limited reversible step, such as closer monitoring or temporary staffing. Uncertainty about cause should neither become a false claim of certainty nor prevent a proportionate interim action.
Lesson summary
Framing a testable question: the lesson links the check to a worked case and practice.