Decision question and experiment protocol · Lesson 6
Separating learning questions from investment choices
17 min read
Lesson objectives
Choose between a quick test and immediate service improvement when the learning budget is limited.
Reading tools
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
The service manager brings two ideas to the morning meeting. One adds a staff member during peak hours. The other tests a message that helps users choose a less crowded appointment. The manager wants an answer this week. Before selecting a test, ask which decision can actually be implemented. Funding for the extra staff member might last one month, while approval for a new message takes two months. The decision available now therefore differs from the question that merely attracts interest. Write the feasible alternatives and the date on which each expires. Alongside them, record which information could change the choice. Research then becomes part of management work. Delay has a cost, an owner and a deadline, rather than becoming an unlimited request for more data.
Separate the learning question from the spending decision. You may want to understand why users abandon a registration form. That broad question could involve interviews, observation and measurement of several stages. Purchasing a reminder service for a year is a specific decision with a price, contract and deadline. Connect the two by explaining how the answer would change the purchase. If investigation shows that most departing users lack a required document, a document reminder may help. If the appointment time is unsuitable, the service schedule may need to change. Ask the team to write at least two paths from a possible result to an action. When every answer leads to the same action, reconsider why the test should consume time.
Start with a short list of missing information. For each entry, state why it matters, how it can be learned and what the method cannot resolve. Available seats can be checked in an operating record. User annoyance may require a brief question or an objection log. The reminder's effect on attendance needs an appropriate comparison. These tasks differ in strength, timing and cost. A randomized experiment is unnecessary for every information gap. Equally, asking three staff members to predict attendance does not measure the message effect. Match the method to the evidence required by each gap. Then state what will remain unknown even after using that method. This final line helps the sponsor understand the limits of the answer from the outset.
Once alternatives are clear, examine reversibility. Testing message wording for a week differs from signing a contract that is difficult to terminate. A reversible change may permit gradual learning with explicit exposure and harm limits. A long commitment requires closer examination of assumptions before signing. Reversibility also has operational dimensions. Removing a message may be technically easy, while correcting a promise already received by a thousand users may be difficult. List what can be reversed immediately and what leaves a continuing effect. Match test size to the team's ability to contain that effect. This grounds risk assessment in the actual service rather than simply describing the intervention as small. Spending size alone does not determine possible harm to a user.
Draw a three-stage decision sequence. The first stage checks that the problem exists and that measurement is sound. The second tests whether the team can deliver the alternative clearly and consistently. The third compares the outcome the sponsor values. Each stage needs a question, cost and condition for proceeding. If the attendance system counts a reservation as attendance, repair the definition before estimating an effect. If messages reach only half the eligible population, investigate delivery before judging a policy intended for everyone. Preparation may happen in parallel, but inference still follows this order. Record each stage's output on a short worksheet. The team reviews that sheet before committing resources to the next stage. This can uncover an inexpensive obstacle before purchasing evidence that cannot answer the intended decision.
Express delay cost in units the decision owner understands. It may include deferred benefit, extra work that continues during the study and staff commitments that displace another task. State the period and denominator. If a centre loses twenty otherwise valid bookings each week, three weeks of waiting concerns sixty potentially affected bookings. That is not an estimate of the alternative's effect. It describes the size of the unresolved problem during learning. Converting it to value requires an additional assumption. Write that assumption next to the result so the sponsor can change it. Then consider whether the learning could change the decision enough to justify its cost. A longer study may be worthwhile for a broad, hard-to-reverse commitment, while costing more than the benefit of a short-lived change.
Ranges can help rank information needs better than one forecast. Write a conservative and an optimistic benefit, explaining the basis for each. Bounds might reflect service capacity or a relevant earlier study. If they come only from team judgment, label them planning assumptions. Ask whether the preferred alternative changes between the bounds. If the same option remains better throughout, another information gap may matter more than narrowing this range. If the ranking reverses, measuring the effect may become valuable. This is sensitivity analysis for organizing a decision; it assigns no probability to either bound. Identify the assumption with the greatest influence on action, then seek a reasonable way to examine it. The output is a specific learning priority whose justification a colleague can review.
Cheap information can still arrive too late. An analyst may review a complete record in one day, but outcomes may mature a week after the last visit. An immediate dashboard can instead provide a quick number that omits the required outcome window. Put the date when decisive information becomes available in the plan. Distinguish it from the date message sending ends, then compare it with the last useful decision date. If the purchase deadline precedes mature outcomes, consider extending the deadline, narrowing the decision or acknowledging that the test informs the next cycle. Continuing may remain educationally valuable, but its value for the present purchase is limited. Recording this limit early prevents later conflict over why the team did not provide a firm answer before follow-up could possibly be complete.
Check that test cost and decision value use compatible periods. Comparing annual value with a one-month study cost can be appropriate when the study is purchased once to inform an annual choice. One month's value, however, does not automatically represent a year with seasonal changes. State the assumed stability of effects and volume, and examine what happens when either changes. When allocating the cost of a shared tool between two experiments, identify the portion that your decision can actually change. A subscription may already be paid and nonrefundable. Additional staff time then represents an avoidable future resource, while the subscription describes past expenditure. This distinction prevents charging an alternative for the same past cost twice. Privacy and harm requirements remain separate from the monetary comparison.
A further distinction helps narrow the learning question: implementation feasibility differs from benefit size. A small operational rehearsal may establish that staff can send the correct message on time. That is a procedural achievement. The participant count may still be too small for attendance comparisons to resolve an economic benefit. Assign each knowledge type a suitable criterion. For procedure, inspect recipient identity, sending time, objection handling and stable outcome recording. For benefit, inspect the comparison, independent allocation units and uncertainty. Using procedural success to justify a long contract jumps between questions. Explain that jump to the sponsor through explicit outputs from both stages. A team record might state that ten synthetic messages were delivered without leakage while the attendance effect still needs a comparison. This preserves the value of the small achievement without attaching a larger promise than the evidence can support.
Before approving the learning budget, ask a colleague who missed the meeting to review the decision. Give them the alternatives, deadlines and assumptions without the proposed tool's name. Ask which missing fact should change the action first. If they choose a different next step, examine why before adding analysis. They may have treated user value as realized cash income or assumed that a short extension was unavailable. Clarify the worksheet and discuss the choice again. This review tests the clarity of the relationship between evidence and action. It can happen before recruiting a single participant or sending any message. Retain a dated version of agreed assumptions and record later changes with reasons. When results arrive, return to that version to distinguish what the study taught from changes in surrounding circumstances.
Lesson summary
Separating learning questions from investment choices.