Framing the job before choosing AI

From complaint to an improvable job A support director says, “Our team is slow; we need AI to route messages.” That sentence can start discovery, but it is not a product definition. The delay may occur before a request reaches an appropriate agent, while an agent searches for an answer, or while a required approval is pending. A classifier can become impressively accurate at routing while customers still wait just as long if approval is the real bottleneck. Product strategy starts by defining the job that needs to improve and measuring its current performance before choosing a technology. Define the job as “get a correct answer from a team that can act, with the first handoff.” The direct user may be the support agent, while the requester experiences the outcome. A team manager wants fewer transfers and shorter resolution time but might call the project successful when more messages are routed automatically. These goals do not necessarily align. An opportunity brief asks who starts the job, what observable event means it has finished, and what happens when a decision is wrong. The word “classification” answers none of those questions. Map the path from intake to topic identification, team assignment and resolution. Put waiting time and transfer errors at each step so the team can find the constraint instead of moving work around. The baseline is the performance of the current process against which a change will be compared. Suppose the center receives 1,000 digital requests per week. During a calibration week, 720 requests reach the team that resolves them on the first handoff, 180 need one transfer, and 100 need two or more. First-handoff correctness is 720/1,000, or 72%. The remaining 28% is not automatically “agent error.” The original message may lack information; the boundaries between teams may be ambiguous; or the escalation policy may be unclear. Review a sample from each segment before designing a solution. The denominator is all incoming digital requests, not only completed cases. Excluding unresolved cases would make the difficult work disappear from the measure. If phone requests are absent from the data, state that the measure covers digital messages only.

Alternatives worth testing One alternative improves intake: a clear topic field with examples and a routing rule for billing. A second applies explicit rules to a selected topic or phrase, with an “unclear” queue for human review. A third uses a learned classifier to predict a team from free text. The model might help when wording varies and there are many categories, but it needs representative examples, a defined cost of error, and a route for low-confidence predictions. Do not choose the third option because it sounds sophisticated. Test whether repairing the intake field or clarifying ownership removes a large part of the problem at lower cost and risk. Google's Rules of Machine Learning advises teams to start with simple baselines and sound infrastructure before adding model complexity. That is engineering guidance, not proof that rules will win this particular comparison. Imagine a review of 100 transferred requests finds 40 whose topic was explicit but the customer selected an obsolete menu option, 35 too ambiguous to route without a follow-up question, and 25 whose text was clear but not covered by existing rules. Repairing the menu could address some of the 40. A targeted follow-up question might address some of the 35. A learned classifier could be tested on some of the 25. Do not add these numbers and claim 100 rescues. This is a sample conditional on having been transferred, and the effects of changes may overlap. The next test compares separate alternatives on new requests before projecting a weekly effect. This prevents a team from building a model for an interface or policy problem.

A decision card that can be falsified Write a falsifiable hypothesis: “If we clarify the topic choice and add an unclear-case route, first-handoff correctness will rise from 72% to at least 80%, with no more than 30 seconds added to median submission time and no increase in sensitive-case misroutes.” Eighty percent is a proposed decision threshold for this fictional team, not a published result. Measure it on new requests labeled after resolution by reviewers who did not see the tool's recommendation. Keep the category definitions, channel and measurement period stable. An improved average does not justify a broad launch if sensitive requests still fail. After the simple repair is tested, a model can be tested on the remainder rather than on all requests; that changes both the build cost and the volume of training data needed. The card has four lines. Customer outcome: reach a team that can solve the issue sooner. Product measure: first-handoff correctness, with resolution time and escalations as guardrails. Alternatives: intake repair, rules and prediction. First evidence: a causal breakdown of transfers, then a bounded field test. “Correct” should be grounded in the team that actually resolved the case, not in a historical routing label alone. If two teams can solve it, agree on a functional definition such as fewer handoffs at equal solution quality instead of forcing the dataset into an arbitrary single label. The strategic objective is not to imitate past routing decisions. It is to improve the customer's result under organizational constraints.

Check the definition before investing An aggregate 72% can hide a material difference. Suppose 800 routine requests achieve 75% correct first handoffs while 200 sensitive requests achieve 60%. The total is still 600 plus 120, or 720, but a single policy for both groups is hard to defend. Sending a sensitive refund issue to the wrong team may cost more than moving an ordinary inquiry. Define decision slices before the test: request type, language, channel and sensitivity when there is a legitimate reason to use it. Do not create so many slices that every estimate becomes unstable. Choose slices tied to a failure mechanism or consequence. When counts are small, show raw counts alongside percentages and avoid false precision. There is also a delayed-label problem. The team that closed a ticket may not have been the best team to receive it first; closure may have followed several teams collaborating. In a small audit, two reviewers read the original request and action history without seeing a model prediction. They ask whether another first handoff would have changed transfers or resolution time. Disagreement is informative: it may expose an ambiguous ownership policy. If both reviewers conclude that two destinations are acceptable, revise the job and measure before gathering training examples. Historical data records how the organization used to work; it does not automatically define future success. That review clarifies the next investment decision. If most errors can plausibly be repaired through an interface or rule, testing the simpler change also has information value. It shows whether customers use the new fields, whether agent behavior changes, and what problem remains for a model. A failed test does not automatically make AI the answer. The cause codes may be wrong, customers may ignore the field, or approvals may remain the true constraint. Return to the job map and test a specific competing explanation. This is a decision cycle that can be checked, not a promise that every request deserves automation.