Decision question and experiment protocol · Lesson 7

Involving users in defining the outcome

17 min read

Lesson objectives

  • Review a digital-service outcome that staff call success while users experience additional burden.
Reading tools

The service report looks healthy: nine out of ten requests are marked complete. Yet callers say they visit repeatedly to obtain a useful answer. These two results may concern different stages. The system flag might mean that a response was sent, while users mean that their need was resolved without restarting the request. A useful experiment begins by examining this distinction before choosing its metric. Ask the service owner which action can improve, and ask users what change they would notice. Place both descriptions side by side. They may agree once time and denominator are clear. Alternatively, the administrative metric may describe staff activity more than user benefit. Your task is to build a measurable definition that represents the decision without reducing user experience to an easily counted click.

Use a short interview to define success, then distinguish that work from estimating an intervention effect. Ask users about the last time they attempted the task and what they regarded as its end. Request a concrete example rather than a general satisfaction judgment. Someone may have received confirmation but still needed a call to discover a missing document. Record a journey with stages rather than concluding that the message fails for everyone. A few interviews can reveal possible explanations and the words people use. Measuring how widespread the problem is requires appropriate sampling and measurement. Comparing a new message version requires a suitable design to estimate its effect. Giving interviews their proper role improves the outcome definition while keeping generalization limits visible.

Prepare four prompts that support discussion without steering people toward your preferred answer. What were you trying to achieve? How did you know it was achieved? What did you do after receiving the response? Which part required unexpected effort? Ask, then allow the participant to describe the event. If they use an ambiguous word such as fast, ask them to connect it with an incident or duration. Five minutes and five days are different meanings of speed. Record the statement as given, placing your interpretation in another field. Separating testimony from interpretation makes the team's inference easier to review later. Check that asking questions and retaining notes fit the project's approved privacy arrangements. Synthetic cases can help rehearse the interview guide before meeting real participants.

Organize the descriptions into three layers. First is an operational output the institution can count, such as responding on time. Second is a near-term user outcome, such as understanding the next step. Third is a more distant benefit, such as resolving the need on the first visit. Draw possible links between the layers and examine where they can break. A reply can be quick but unclear. Instructions may be understood while the available appointment remains far away. Choose the primary outcome in light of the decision, intervention and follow-up period. Retain some other layers to investigate the mechanism or detect harm. This structure prevents the promise from changing during presentation. A trial that rewrites a response does not automatically establish improvements throughout everything that follows in the user journey.

Choose whom to hear from using diversity relevant to the decision. Active users are easy to reach, but they may conceal difficulties experienced by people who leave early. Map groups whose journeys actually differ: new and returning users, for example, or people needing language support versus those reading instructions independently. Explain how each can be reached without collecting more sensitive information than necessary. Track who responds and who cannot be contacted. Response differences limit what you understand. You can report what interviews with returning users revealed while keeping new-user experiences unresolved. Do not invent population weights for the group table. Interviews selected for variety help discover problems; they do not automatically represent population proportions.

Review the question itself because the intervention may change willingness to answer. If the new-message arm receives a prominent satisfaction link while current-service users receive a late phone call, the comparison may reflect different collection methods. Align question timing, channel and length across arms, preserving the option not to answer. Separate response rate from satisfaction among respondents. If twenty of one hundred respond in one arm and seventy in the other, the satisfaction contrast may reflect who answered. Begin by showing these counts, then discuss what is identifiable. An independent operational record can serve as the primary outcome, with the survey providing supplementary explanation. Document why you make this tradeoff: easier analysis alone does not make a metric representative of user benefit.

Build a practical agreement between user needs and operating requirements. Users may want an immediate reply, while staff need to inspect a document before giving a correct commitment. A composite success definition, such as a correct response within a suitable period, may help. Adding conditions changes the success rate and makes interpretation more demanding. Preserve component records so you can identify what changed. When composite success rises, examine whether timeliness, correctness or both improved. A decline may call for repair of one stage rather than rejection of the whole idea. Set benefit priorities or weights before results appear. A mandatory safety limit differs from a small formatting preference. Resolving this distinction early prevents each stakeholder from selecting whichever metric makes their work look best after the experiment.

Protect user input from becoming a decorative quotation. Link each outcome-definition change to the observation that motivated it. A definition log can contain a short original statement, the team's interpretation and the resulting measurement choice. Repeated visits, for example, might motivate counting visits within a week rather than measuring only first-response time. Ask a colleague to propose another explanation. A second visit may be necessary to collect an original document rather than repair a failed response. Inspect that reason before classifying every return as a negative event. This review protects meaning and reduces the risk of a metric that penalizes normal service behavior. It also lets you explain metric selection to a reviewer without exposing the participant's identity or personal details.

An identically named outcome may mean different things to different user groups. In a learning-material reservation service, success may mean collecting the entire set for some users, while one urgently needed item suffices for others. Define the policy decision first: are you testing complete collection or the option of partial collection? Record preferences before allocation if they define eligibility or an intended subgroup. Dividing results by a choice made after seeing the version can reintroduce selection. Keep each reported outcome within its question. You can estimate the effect of offering partial collection to everyone, but that differs from the effect of actually collecting partially among people who chose it. This preserves user input without undermining random comparison or attributing an unmeasured preference to participants.

Before using an outcome question at scale, test its wording with users. Ask someone to explain it in their own words, then classify a short example. If “your request was completed” means receiving a message to them while you mean resolving their need, name the intended action more precisely. Try simpler wording and check understanding again. Record the change and its reason before freezing the comparison survey. Wording changes during the trial can create two periods with different measures. If a serious ambiguity requires repair, document timing and review the comparative implications rather than merging answers as if the question were unchanged. This modest preparation can uncover a measurement error whose consequences outweigh the benefit of recruiting more participants.

Finally, express the definition in one sentence the team and users can understand without reading the analysis plan. Name the population, the action that counts as success and the end of the allowed window. Provide examples of success, an incomplete result and an unknown case requiring review. These examples make the definition trainable and testable. A new colleague can classify them before seeing the trial arm or aggregate results. When disagreement appears, revisit the task's meaning and information source before adding another exception. Numerous exceptions can signal that one number is trying to represent different decisions. Separate those questions while preserving a clear primary priority. The benefit is a team able to identify what actually changed and a sponsor able to act on the service users really experience.

Lesson summary

Involving users in defining the outcome.