Performance Diagnosis and Experience Hypothesis · Lesson 1

Is This Really a Learning Gap?

9 min read

Lesson objectives

  • Distinguish a skill gap from workflow and incentive faults using task evidence, then frame a testable performance problem and the boundary of a learning intervention.
Reading tools

A customer support director says, ‘Escalations are up; we need another triage course.’ She may be right, but naming a solution does not identify a cause. A justified escalation protects a customer when the first-line agent lacks authority to resolve a case. An unnecessary escalation adds waiting and rework. Before designing a course, define the error operationally: an eligible request that the first-line team could resolve under the current policy, with the needed information available at decision time, but that is escalated without a valid reason. This differs from an out-of-scope request or a case for which no approved procedure exists. Count eligible requests as the denominator, rather than messages or employees. If that denominator changes between weeks, a rate may appear to improve without any change in judgment.

In this fictional case, 24 agents work across two shifts. Over four weeks the system recorded 400 requests eligible for triage; 96 were escalated. Two supervisors reviewed a stratified sample of 80 requests across shifts and request types and marked 18 escalations unnecessary. The descriptive rate in the reviewed sample is 18 divided by 80, or 22.5%. It is not automatically the rate for all four weeks: the sample is limited, and classification depends on reviewer agreement. Before comparing teams or committing budget, document the classification rule, how disagreements were resolved, and how often the record lacked information needed for a judgment. Treating every escalation as an error would penalize correct compliance with the escalation policy and corrupt the design target.

Look at observable task performance before relying on stated knowledge. Three agents recite the new triage rule accurately in an interview but escalate during a simulation when the request screen omits a contract detail. That suggests difficulty applying the rule under a particular condition, not definitive proof that the rule itself is unknown. Work records show that 11 of the 18 unnecessary escalations followed a registration-screen change: the field identifying contract type moved behind another tab. Five errors occurred even with the field visible, and two records cannot support a diagnosis. We now have at least three plausible hypotheses: a weak interface cue, insufficient practice interpreting contracts, or time pressure that rewards escalation. A uniform content module for all agents would ignore these distinct mechanisms.

Build a cause-and-evidence matrix rather than collecting opinions. For each hypothesis, state what you would expect if it were true and what observation could disconfirm it. The interface hypothesis predicts errors concentrated when contract type is hidden and fewer errors when it is displayed, even without training. A skill hypothesis predicts continued misreading of contract exceptions in a simulation with a clear interface, followed by improvement after guided practice and feedback. An incentive hypothesis predicts escalations near shift end or under a speed-of-closure target, even when information and skill are available. Association alone does not establish causation, but these predictions make the next observation discriminating. Check the authority policy too: a case that agents are prohibited from closing must be escalated, however easy it appears.

How would we know whether this combination helped? Define the rate of unnecessary escalation among reviewed eligible requests before intervention. Also monitor dangerous failures to escalate, handling time, and outcomes by shift. A decline in unnecessary escalation alone could mean that agents stopped escalating even when they should, so the safety measure matters. Compare a similar time window and record changes in policy, interface and case volume. If the interface and practice launch together, do not attribute the subsequent change to practice alone. An interface pilot on one shift and a later exception-practice pilot can help separate mechanisms if operations permit; a pre/post simulation can test the specific skill. Neither by itself establishes a complete causal effect on customer service.

The review needs a repeatable observation protocol. The reviewer should see the request as the agent saw it at decision time, not as the record looked after resolution. First mark whether the case was within first-line authority; next whether the contract cue was visible soon enough; then whether the decision matched policy. If cue visibility is unknown, classify the case as indeterminate rather than inventing the agent's intention. Have two reviewers classify a small shared set before dividing the rest. When they disagree, consult a written decision rule or leave the case unresolved; averaging their opinions does not create a valid label. This work costs time but prevents a missing record from being counted as an employee error. Report the number of indeterminate cases. Concealing them can make accuracy look better merely because the hardest cases were excluded.

Separate interventions when assigning design responsibility. Moving a field is a product and interface change; a course cannot permanently repair a hidden cue. Exception practice is learning work, but it needs borderline examples where two authority rules nearly meet, rather than another slide that displays the policy. Changing a speed-of-closure target is a performance-management decision; it can alter behavior without changing knowledge. Put three columns in the design brief: plausible cause, fitting intervention, and evidence that would make the team reconsider. If the screen cannot change immediately, a temporary decision card may be reasonable if it is tested during real work. Do not call it a permanent cure or infer mastery just because employees consult it. An advanced designer defines these boundaries so that learning is not credited or blamed for a problem outside its mechanism.

Set a decision before starting and another before expanding. Here the start decision permits a short simulation only after confirming authority policy, because teaching a decision outside an agent's permission is more harmful than delaying the activity. Expansion should not rely on learners liking the exercise. It calls for fewer contract-exception errors on fresh cases that were not shown during practice, without a rise in failures to escalate required cases at work. If simulation improves but workplace performance does not, examine the environment: did the mix of contracts change, did time pressure increase, or does the field remain hidden on some devices? This difference between learning evidence and transfer evidence protects the decision from a convenient but unsupported inference. If both fail, revise the rule explanation and examples before simply adding more attempts.

Lesson summary

Distinguish a skill gap from workflow and incentive faults using task evidence, then frame a testable performance problem and the boundary of a learning intervention.