Decision question and experiment protocol · Lesson 8

A pilot that tests the procedure

16 min read

Lesson objectives

  • Decide procedural readiness from a feasibility sample, distinguishing it from a definitive efficacy test.
Reading tools

A small test ends and the team lead says, “Seven out of ten attended; let us launch.” Before agreeing, return to the test's original purpose. Perhaps it asked whether staff could identify eligible users and send the correct invitation. Attendance in ten cases does not, by itself, estimate improvement over current service. In this lesson you construct a pilot that examines procedure before an effect study. You name operational success clearly and report what the attempt revealed about recruitment, delivery and measurement. Then choose whether to proceed to the main study, repair a component or stop. The small exercise is valuable when it discovers a threat to the larger study. Recording this purpose beforehand protects the team against pressure to turn every positive number into an efficacy announcement.

Write the pilot question as one sentence tied to a procedure. Examples include confirming eligibility before revealing the assigned version or retrieving appointment outcomes within two days of occurrence. These differ from the hypothesis that a reminder raises attendance. Several questions may be needed, but each should have a criterion and record source. If usability is the concern, specify the task users attempt and the point at which difficulty is observed. If recruitment flow is the concern, count eligible users by channel over a defined period. Choosing the question first prevents the pilot from expanding into an unplanned study of everything. It also helps determine who needs to participate and which information is sufficient to examine the procedure with reasonable burden.

Set progression criteria before viewing pilot results. They might include a critical condition such as never sending to an ineligible person and an operational condition such as retrieving most outcomes within the required deadline. These conditions differ in kind. One eligibility error might require pausing delivery for repair. Extraction delays might instead improve through staff scheduling. Write the next action and decision owner beside each criterion. Base it on service risk and team capability, labeling it a project-specific criterion. Borrowing a familiar percentage from another study does not establish suitability here. Where uncertainty remains, describe errors and consequences rather than compressing the pilot into one color or overall success mark that conceals different risks.

Inspect the denominator of every operational percentage. Delivery may be calculated among eligible users, people actually sent a message or records with valid contact details. Each denominator answers a different question. If the intended policy serves all eligible users, reporting success only among valid addresses hides the work needed to update the remainder. Preserve a flow from eligible users through permitted contacts, attempted sends and confirmed deliveries. Record the source of every transition and the reason for exclusions. The sponsor can then see whether the obstacle concerns channel eligibility, software or data quality. This makes repair specific and prevents buying a new tool to solve a problem outside its function.

Use the pilot to estimate the work itself. Ask staff to record time spent screening, sending and resolving exceptions through the project's approved privacy arrangements. Sending may take one minute while correcting invalid numbers takes half an hour. This difference determines capacity when volume doubles. Separate fixed work per batch from work repeated for each case. Examine peak times and competing duties. A task can be inexpensive in total minutes yet critical at a particular time. With only one operator, an extraction problem may displace direct user service. Make that tradeoff explicit in the resource plan so the sponsor can consider a smaller test, extra staff or more suitable timing.

Deliberately test difficult cases rather than waiting for them to occur by chance in a small sample. Consider an outdated number, canceled appointment, person with multiple bookings and a time close to the sending boundary. Use synthetic data where sufficient for the software check. Record expected result, observation and a replayable receipt for each case. A failure-revealing case may be more valuable than ten similar easy cases when procedural safety is the objective. It does not estimate population defect prevalence, since you selected it for difficulty. Preserve that distinction in the report. Add a small natural workflow sample if ordinary workload matters, explaining that it answers a different question from testing edge cases.

When a defect appears, define what must be repeated after repair. If a canceled booking receives a message, replay that same canceled case after fixing eligibility. Also try a similar valid booking to ensure the repair has not blocked legitimate service. Both tests matter. Rejecting every send may hide the defect while disabling the intervention. Record procedure version, repair time and replay cases together. If eligibility changes, revisit which population the main study will describe. If only linkage is faulty, distinguish record repair from changing the participant population. Repeating a pilot should have a specific reason and reviewable output rather than cycling through the same procedure in search of a more attractive percentage.

Pilot percentages require restraint appropriate to the small count. Delivery to nine of ten cases may tempt someone to label the system ninety-percent reliable. That is the observed fraction, and one additional case changes it substantially. Ask what kind of case failed and whether important conditions were represented. An obsolete address points toward data updating; failures concentrated in a busy hour point toward capacity. A small sample can reveal a cause without precisely estimating its frequency. Always show counts beside percentages and match statements about precision to available information. One serious defect can justify immediate repair even if seen only once. A broad reliability claim requires more coverage and measurement than an attractive percentage from ten cases.

A sponsor may ask whether pilot data can enter the main study. First document differences in eligibility, intervention wording, allocation, outcome definition and measurement. If material elements changed, simple pooling can combine different questions. Even with similar procedures, inspecting early outcomes may have influenced the chosen plan. Inclusion therefore needs prospective handling and a suitable method, rather than adding counts after seeing a favorable direction. Pilot records can be retained as separate implementation evidence, with the main comparison starting under a newly specified plan. For this training project, that is a clear approach when pooling cannot be justified. The records remain useful for understanding defects and cost, but serve a different purpose from the primary hypothesis test.

When a pilot leads to repair, convert that result into a closable task. State the defect, its consequence, a reproducing case, corrective action and owner. Add a target replay date and retain the actual observation rather than a promise that the team will fix things. Repair may reveal that the cause lies outside the proposed tool, such as conflicting booking definitions across branches. The sponsor then needs an agreement on definitions before recruiting more people. A pilot may also justify stopping an alternative because operating cost or risk is unsuitable. That is a legitimate learning result rather than a loss to conceal. The output to review is the quality of the next decision and its connection to evidence, not the number of participants moved into a larger phase.

Keep the progression report to one page with links to detailed records. Include the pilot question, procedure version, case count and tested conditions. Show observations against each criterion, remaining defects and the person deciding whether to proceed. Explain how learning changed the main-study plan. Additional training time, an independent outcome recorder or revised sample planning for greater loss may be needed. State the new estimate's basis and limits rather than adopting one small-sample proportion without review. Attach explicit conditions for reopening the decision when the tool, channel or team changes. A colleague can then use the report during handover, knowing which procedure component needs fresh checking and which evidence remains applicable to the unchanged version.

Lesson summary

A pilot that tests the procedure.