Frame the advisory decision · Lesson 3

Evidence ladder and confidence limits

4 min read

Lesson objectives

  • Link consequential claims to source, coverage, date and limits, then expose conflicting evidence rather than concealing it in a misleading average.
Reading tools

Many citations do not strengthen evidence if all repeat the same statistic; an ‘official’ label does not make a number relevant to a different question. Maintain an evidence ledger for every claim capable of changing the decision. Its columns are the exact claim, who created the data, when it was collected rather than merely republished, the population or unit measured, collection method, coverage, conflicting sources, and what inference the observation permits. Link the edition and URL or document identifier to the precise claim; a link in a general reading list is insufficient. If the question is future service demand, a count at one station establishes an observation at one place and time, not demand in every district five years hence. If three charts repeat the same underlying table, they are three displays of one source, not three independent confirmations.

Judge confidence descriptively with reasons rather than assigning a precise number without a model. Ask about direct relevance to the decision, collection quality, recency, source independence, and likely bias. A recent operator interview may be strong at revealing a delivery constraint but weak at estimating the proportion of the population affected. A long administrative series may count transactions well yet say little about service experience, especially if the definition of a transaction changed. Do not resolve disagreement by automatically averaging two sources. Check whether they measure the same construct, period and denominator. If one counts unique users and the other counts visits, their difference is methodological. Show a range only if the definitions are sufficiently aligned; otherwise present the separate estimates and the unresolved gap. State what each interpretation would do to the choice among alternatives.

Separate observation, explanation and decision rule in the advice. ‘The observed week recorded 240 passenger boardings’ is an observation. ‘Demand rose because fares fell’ is a causal explanation requiring comparison or testing. ‘Expand the route if average demand passes an agreed threshold in two seasons and operating cost is affordable’ is a conditional decision rule. Each transition introduces assumptions; disclose those that matter. Do not write ‘high confidence’ merely because a figure has many decimal places. Write an auditable limit such as ‘confidence is limited because the count covers one week and no seasonal comparison’. Confidence differs by claim: a valid vehicle purchase quote may support acquisition cost but says nothing about expected use. The adviser can recommend a trial designed to reduce uncertainty, or say the evidence does not support an irreversible commitment. That is a professional decision, not a failure to produce a number.

Prevent evidence from being reused beyond its scope by adding a ‘does not support’ column to each ledger row. The order record does not support a unique-buyer count until deduplicated; the open survey does not support a market percentage without a sampling frame; the partner forecast does not support the new price without recalculation. Give each item a review date. This does not imply that facts expire automatically; it forces the decision team to check whether circumstances have changed. If the definition of an order changes between months, do not compute growth from the raw pair until definitions are harmonised or a break in series is reported. These limits make the ledger an updatable decision tool rather than a decorative reading list.

Lesson summary

Separate observation, explanation and decision rule in the advice.