A value hypothesis with release thresholds

Start with an outcome, not suggestion acceptance After choosing a routing opportunity, a support organization considers an assistant that drafts replies for agents. The vendor proposes “suggestion acceptance rate” as the success measure. An agent may accept text because it saves clicks, or reject accurate text because a different tone is needed. Accepting a misleading answer may create customer complaints later. Separate product usage from the outcome the feature is meant to improve. Usage tells us whether the feature entered the workflow. Outcome tells us whether it helped the agent and customer. Without that distinction, a team can reward a system that produces many apparently useful drafts while increasing rework. Write a specific value hypothesis: “If agents see a draft grounded in a current knowledge base, with a source link and an obvious edit control, the share of requests correctly resolved within 24 hours will rise from 60% to at least 68%, without an increase in later corrections or misinformation complaints.” These are decision thresholds for a fictional case, not published results. “Correct resolution” needs a review of closed tickets against a criterion such as the required action being completed and the issue not reopening within a declared follow-up period. Closure alone is a weak proxy; pressure might encourage premature closure. The 24-hour clock starts when the request arrives, not when an agent opens the draft, or earlier waiting time vanishes from the measure.

Map each measure to a denominator The primary measure is eligible tickets correctly resolved within 24 hours divided by all eligible tickets entering the test. Fix eligibility before seeing results: digital requests in a supported language and current-policy topic, excluding those that require an individual legal or financial exception. Exclusion defines the product's initial scope. It must not be adjusted after a failed test to improve a percentage. Record how many requests are excluded and why, since the reachable opportunity might be too small to justify investment. Reply preparation time is a secondary measure, but reducing it is not a win if resolution quality falls. An adoption measure is tickets where the agent opened the draft divided by tickets where the feature was presented; opening a draft does not mean it was used well. Guardrails prevent a good average from being mistaken for a safe launch. Choose the rate of corrections after sending, complaints about incorrect information and escalation in sensitive cases. Define a denominator and period for each. For example, corrections within seven days of sending divided by all replies in the arm; confirmed complaints divided by eligible tickets, shown with raw counts as well because the count may be small. In small language or topic slices, inspect the cases rather than projecting statistical certainty from an unstable percentage. Do not create ten nearly identical guardrails. Cover the actual harms generated text might cause.

Design evidence before running the test Randomize eligible tickets or agent teams according to how easily one arm's behavior could spill into the other. Assignment by agent may be better if agents learn from drafts and later apply them to control tickets. State why the chosen unit limits contamination. The comparison arm uses the usual workflow; the treatment arm receives an editable draft. Do not change the knowledge base or closure policy in only one arm during the test. If a policy changes for everyone, record the date and consider restarting or analyzing periods separately. Human reviewers of resolution can read the answer and request context without knowing the arm where practical. This reduces bias; it is not a claim of fully independent research. Record where blinding is impossible. Suppose each arm has 400 eligible tickets. In control, 240/400 are correctly resolved within 24 hours, or 60%. In treatment, 272/400 are, or 68%. The difference is eight percentage points, not an 8% relative increase; the relative increase is 8/60, about 13.3%. Yet treatment records 64 replies corrected within a week, 64/400=16%, against 36/400=9% in control. If the team set a correction guardrail of no more than 10% in advance, a full launch fails. Do not select the resolution measure and hide the guardrail. Inspect the cause of corrections. An outdated knowledge base, an omitted policy exception and an agent sending unreviewed text require different repairs. The figures do not prove a permanent effect. Four hundred tickets per arm is a limited sample; topic or agent mix can still differ after randomization. Show differences with counts, review high-severity slices and retest when operating context changes. “Statistically significant” should not become a generic substitute for uncertainty analysis. Size the next test around the smallest effect that would change the decision and the error rate the team can tolerate. If the eight-point resolution gain occurs almost entirely in easy requests, it may not justify cost or risk in the rest.

When a proxy changes behavior The observed difference also has uncertainty. If tickets were independent, an approximate standard error for 60% versus 68% with 400 tickets per arm is the square root of [0.60×0.40/400 + 0.68×0.32/400], about 0.034. A rough interval around the eight-point difference runs from about 1.4 to 14.6 percentage points. This makes a precise benefit promise inappropriate. If assignment is by team, tickets within each team may be correlated; the ticket-level approximation can then be too narrow. Analyze using the actual assignment unit and record it instead of borrowing false precision from an independence assumption. If the team is rewarded for accepted drafts, the assistant can raise that figure by showing very short drafts that require the agent to add the crucial information later. An adoption dashboard improves while the real work does not shrink. In a review sample, compare the proposed and sent replies: how much was changed, and were missing conditions or sources supplied? Do not turn edited character count into a lone target. An edit might merely adjust tone, while one small edit could avert a serious error. Use revision data to explain preparation time and resolution quality, not replace them. Harm can also disappear from the dashboard when corrected tickets move to another channel. Link ticket identity across sending, follow-up and correction with a data-retention rule appropriate to the case and access restricted to those who need it. If some channels cannot be linked, report the fraction with known outcomes. A result of 68% on 400 cases is incomplete if the fate of 60 cases is missing. Were they included in the denominator, and is missingness balanced across arms? Do not silently label the unknown cases successful or delete them. Compute a conservative bound: what is the rate if all unresolved cases failed? The next decision might be to repair measurement before broadening the feature. The decision record can predefine three branches. If correct resolution improves and every guardrail passes, expand to a larger segment while monitoring. If it improves but a guardrail fails, as in this example, stop the causative scope, repair it and retest. If resolution does not improve but guardrails pass, ask whether drafts are poor, agents do not adopt them, or downstream approvals determine resolution time. These are diagnoses of a product chain, not verdicts on a model alone. “We do not yet know” is legitimate when evidence is incomplete or unstable, provided it leads to a specific test rather than indefinite postponement.