Diagnosing adoption opportunity and work context · Lesson 3

Establish a baseline and variation

6 min read

Lesson objectives

  • Build a baseline with stable denominators, periods and segments, separating case-mix change from an AI intervention.
Reading tools

A management dashboard can make an average resolution time before and after AI adoption look decisive. The figures may answer a different question. A baseline is not merely a number saved before buying a tool. It is a definition of how a particular job will be observed so that a later comparison is meaningful. Set the clock start: arrival in the system or opening by an employee? Set the end: first reply or a correct closure that stays closed for a defined period? A change of definition between periods makes decimal precision pointless. State the denominator too: all incoming requests, closed requests only, or a comparable complexity group. Excluding open cases from time calculations can make performance look faster merely because difficult cases have not yet closed.

Segmenting cases exposes variation instead of hiding it. Imagine two classes: a simple policy question and a complaint requiring investigation. If the simple class falls from twelve to eight minutes while complaints remain at seventy, a defined piece of work improved. Yet the organization-wide mean could rise if complaints take a larger share. Conversely, the mean could fall because simple questions become more common, without any improvement inside either class. Fix category definitions before the test and show each group’s count and outcome. Do not invent narrow categories that leave a huge unexplained “other” group. Inspect who falls outside and why. If the assistant itself changes case classification, retain a sample that can be coded again under a stable definition.

Choose a summary statistic for the decision. The mean helps estimate total work burden if unusual values are understood, but one case taking days can move it sharply. The median describes the middle case and resists extremes, yet can conceal a slow tail that some customers experience. The 90th percentile is the time at or below which ninety percent of ranked cases fall; it can expose that tail. Avoid treating every metric as a simultaneous target. Name one primary outcome and another measure that protects quality or fairness. If the aim is to shorten delay for most people, a median with a 90th-percentile guardrail may help. If rare critical cases cause the main harm, inspect those separately rather than waiting for an average to change.

Worked fictional case: before piloting an assistant, a team handles 100 simple requests averaging ten minutes and twenty complex ones averaging fifty. Total work is 100×10+20×50=2,000 minutes, yielding a weighted mean of 2,000÷120=16.67 minutes. After the pilot it handles sixty simple requests averaging eight minutes and sixty complex ones averaging 48. Total work is 60×8+60×48=3,360 minutes and the mean is 3,360÷120=28 minutes. It would be wrong to say the tool slowed every kind of work because the overall mean rose from 16.67 to 28; each category became faster. It would also be wrong to claim a universal two-minute saving: the complex-case share changed from 20/120 to 60/120. Hold the original mix fixed for a descriptive comparison: 100×8+20×48=1,760, and 1,760÷120=14.67 minutes. That is two minutes below the original mean on the original mix, a conditional descriptive estimate rather than proven causal impact.

Even after fixing the mix, causal questions remain. Did staffing, the knowledge base, or seasonality change? Did the pilot select only the fastest employees? Did “complex” acquire a new meaning after the assistant proposed categories? A useful baseline records concurrent changes and their dates. Where feasible, observe a comparison group without the assistant but similar in request type, channel, and experience, over the same period. Do not attribute every difference to the tool if management chose its most prepared branch for the pilot; discuss selection. Uncertainty need not cause paralysis. Make a bounded decision and specify which additional observation would justify expansion.

Quality rates also require an explicit denominator. Four consequential errors among eighty closed requests is five percent of closures, not of one hundred arrivals. The twenty still open may differ. A seven-day reopening rate after closure can add useful information; if the reopening rule changes, the figures are not directly comparable. Agree on the review sample before launch. Will it be every tenth case or a stratified selection including complex cases? A convenient set of the team’s best responses will not represent quality. Measurement discipline prevents both a mistaken scale-up and the rejection of a real improvement.

The MEASURE function of the NIST AI RMF concerns analyzing and monitoring system risks and impacts, including whether metrics remain effective for their context after deployment. That supports reviewing the suitability of measures as work changes. The mean, median and fixed-mix calculations here are transparent analytical applications, not a scoring formula mandated by NIST. The product of this lesson is a short baseline card: task definition, period, denominator, case groups, data source, initial value, guardrail, and changes that could explain the difference. Ask someone unfamiliar with the project to reproduce one value. If they cannot, the definition or source is incomplete.

Lesson summary

Build a baseline with stable denominators, periods and segments, separating case-mix change from an AI intervention.