Distinguish a task opportunity from tool hype
6 min read
Lesson objectives
- Frame a comparable work problem and test an AI use case against a non-AI improvement while protecting quality.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
A customer service director says, “We need an AI assistant immediately; our competitors have one.” That statement names a means, not the work to be changed or the outcome worth paying for. Start with a unit of work: one customer request from receipt to a correct closure on which the customer can rely. If the delay is an approval queue rather than drafting a reply, a writing assistant will not remove the bottleneck. Describe the current task before choosing a tool. Who receives the request? Which information must the agent find? Where does the case wait? Who may close it? Which exceptions require a human decision? The workflow map is useful only if it locates a changeable step and a way to observe its effect.
Keep three kinds of measure separate. An activity measure records tool use, such as sessions. A process measure records a changed step, such as minutes spent searching for the current policy. An outcome measure records what matters to the customer and the organization, such as a correct resolution at first contact. Usage can rise while outcomes stay flat if review or rework rises too. State the denominator of every rate. “Correct resolution rate” can mean correct closures divided by closures or by all incoming requests; switching denominators can conceal a backlog. Specify the comparison period and case mix. A week dominated by simple requests is a poor comparison for a week dominated by difficult complaints. Add at least one guardrail measure, such as a consequential error or improper disclosure, so that a faster process is not purchased by reducing trust.
Ask what an effective option without AI would be. A cleaned knowledge index or removal of a duplicated approval step may cost less and have clearer ownership. Compare feasible options with known resources, rather than a new prototype against a deliberately neglected process. Use two decision gates. Eligibility comes first: may the data be used for this purpose, is there a review and escalation owner, and can plausible harms be controlled inside the pilot? A failed non-negotiable condition cannot be repaired by assigning it more points in a weighted score. Then compare eligible options on outcome value, effort, measurability, and impact on people. These gates are an analytical design for the fictional organization, not a standard scoring system issued by NIST. The NIST AI Risk Management Framework uses MAP to understand purpose, context, and affected parties, and MANAGE to consider benefits, risks, and whether AI is appropriate at all. Those functions inform our reasoning without dictating a single universal workflow.
The interpretation matters as much as the arithmetic. We did not reject AI because it is always riskier, nor favor the index because it is always simpler. We defined a task, an alternative, and a common measurement point; calculated net rather than generation time; and checked the customer result. A small sample frames the next experiment, not a causal verdict. If difficult requests cause most of the service delay, success on simple policy queries cannot be presented as a service-wide gain. Record the important unknowns: whether policies are consistent, whether agent training changes the result, how often documents change, and the cost of maintaining the index. These are assumptions to test in the baseline and pilot, not reasons to avoid a bounded, revisable decision.
Before testing either option, set an observable stop rule instead of saying that the team will “monitor quality.” In the support case, no generated reply should reach a customer without agent approval. Review every consequential error found in the first fifty cases, and suspend expansion if a financial condition is misstated again or personal data is disclosed. A quantitative error threshold may also be set after examining the baseline and sample size; it must not be selected for convenience after results are visible. Measure full time from receipt to correct resolution, not drafting time alone. If faster searching merely shifts work to an overloaded review team, inspect its queue and the distribution of effort across workers. If the revised page needs weekly maintenance, include that work in operating cost. The comparison then concerns the whole work system rather than one tool’s apparent speed.
Lesson summary
Frame a comparable work problem and test an AI use case against a non-AI improvement while protecting quality.