Crisis diagnosis and mandate boundaries · Lesson 2

Evidence ledger and competing explanations

7 min read

Lesson objectives

  • Build a time-stamped evidence ledger and test competing explanations.
Reading tools

Imagine a booking platform fails unexpectedly. Engineering says the server is up, customer service sees dozens of complaints, and an external monitor reports poor responses. If an adviser immediately says “the problem is on users’ devices” because that was the first explanation heard, a working hypothesis has become a public fact. The skill is not collecting the largest pile of screenshots. It is creating a ledger that lets people trace each statement and understand what it can and cannot prove. Call it an evidence ledger, not a facts list: it will contain early signals, limited negative findings, and inferences still needing a test.

A useful entry has six fields:

  • the exact claim
  • original source
  • event time and logging time
  • coverage or denominator
  • verification status
  • and possible effect on the decision

An internal ticket number may help, but personal data need not be broadcast to enable review. “At 10:02, 99 of 100 loading tests from data centre A succeeded” does not mean “the service works for everyone.” It covers one centre, one request type, and a short window. “Thirty-five complaints since 09:50” does not yet mean 35 independent users until duplicates are removed. It still signals an audience needing explanation and action. Include the denominator, request type, and location whenever they change what a number means.

Use working labels: verified, likely, unresolved, and contradicted. Verified means an authorized source or repeatable measurement supports the exact claim within its stated scope. Likely means evidence leans in one direction but a discriminating test is unfinished. Unresolved means information is missing or conflicts. Contradicted means a reliable test refutes the precise wording; it does not refute every neighboring explanation. Avoid numerical confidence percentages without a calibrated method. “Measurements support an effect on users in region B” is more honest than “we are 87% certain” when 87 merely represents a team member’s intuition.

Test competing explanations before designing a message. The booking problem might be a general outage, a mobile-only fault, regional slowness, or load from automated requests. Find tests that distinguish them: compare success on mobile and web, in two regions and across networks; inspect response time as well as success code; align complaints with the journey and time. A company dashboard measuring the home page may coexist with genuine complaints about checkout. The apparent conflict disappears when scope is made explicit. Signals are not necessarily independent: ten posts repeating one picture are not ten distinct reports, and two sensors sharing the same data source can fail together.

State what each hypothesis predicts before seeing the next result. A general outage should affect several journeys and regions; a mobile-checkout fault should leave the home page and desktop closer to their normal baseline. The forecast need not be perfect, but it exposes an explanation quietly rewritten after the evidence arrives. Compare against a relevant baseline too: 98% success may sound reassuring until the normal figure and the more consequential payment failures are known.

When you favor an explanation, ask what evidence would make you change your mind. If mobile tests succeed from several places while desktop payment fails, your hypothesis must change. This falsification point resists attachment to the first story. Keep inconvenient evidence in the ledger too: a working home page is real but does not negate a broken checkout. If an external monitor conflicts with your tests, check whether it measures the same page, from which location, and how often. Do not dismiss it because it is external, or automatically equate it with a reliable transaction log.

Assign an owner and decision-linked due time to each unknown. Engineering owns the technical cause of mobile failure. Whether customers should be warned against retrying a payment is a joint decision involving engineering, payments, and service. If the discriminating test takes three hours, that need not block all communication for three hours. State the verified effect and current action, with a scheduled update before the investigation ends. If the unknown changes protective instructions, escalate it immediately to the qualified decision owner. The ledger thereby manages uncertainty rather than becoming an archive compiled after harm.

Before the ledger informs an advisory recommendation, ask three questions. Does the source support the whole sentence or only a narrower claim? Is the signal independent or recirculated? What decision would change if contrary evidence arrived? These questions support a scoped update without overwhelming audiences with internal data or projecting false certainty. Preserve the time, editor, and reason for material changes. Overwriting an earlier line erases the learning path and can make an earlier message seem inexplicable later.

Lesson summary

Apply evidence ledger and competing explanations to a case with explicit evidence, decision, and inference limits.