Challenge assumptions using a relevant reference class
17 min read
Lesson objectives
- Choose a reference class consistent with estimate maturity and scope, and calculate transparent summaries.
- Reconcile an estimate with prior experience without hiding extreme cases or duplicating contingency.
Please sign in to use all the tools
Once you sign in, the reading tools are yours: larger or smaller text, typeface, line spacing, focus mode, highlighting and bold, and notes saved to your account and gathered in My study.
An initiative owner says the estimate is precise because every task was examined. A reviewer counters with a list of over-budget projects and asks for one contingency percentage for all proposals. Both can be wrong: detailed estimates may share an untested optimistic assumption, while a generic uplift may come from unrelated work. This lesson uses an appropriate reference class to challenge an estimate from outside the team's narrative, then returns to its detail to explain the difference. The output is a forecast with a stated basis and transfer limits, informing what needs evidence, contingency or redesign. Comparison does not automatically establish the cause of every overrun or make past averages an inevitable future for each new project.
Define the forecast target and the estimate's decision stage first. A concept-stage cost differs from a contract price after completed design; duration from investment approval differs from duration after site handover. Reference estimates should have been made with information resembling what is known now. Comparing an initial estimate with old budgets revised a month before closure makes past errors look small by shifting their starting point. Ask precisely: what was actual cost divided by the approved estimate at our present stage, for work with similar scope and delivery method? Specify whether cost includes operations, investment or both, and the dates and price basis used for consistency.
Design inclusion rules before viewing results where possible. They may cover service type, integration complexity, procurement model, site scale and estimate maturity at approval. Explain why each rule relates to the risk under review instead of creating a long list that excludes uncomfortable cases. Distinguish decisive criteria from characteristics analysable within the class. Requiring exactly the same technology might leave only two cases, whereas similar interface counts and migration requirements could provide a broader, more relevant comparison. The selection requires reviewable professional judgement, not merely matching project names or departmental labels.
Create a verifiable row for each case: a case name, estimate date and scope, cost on a known price basis, completion date, actual cost for the same scope, and source and data status. These fields trace figures to their records and distinguish final outcomes from provisional forecasts before comparison. Retain scope changes and their effects rather than overwriting versions. Forecast error for original work and cost of the programme after changes answer different questions. Show uncompleted cases separately with spending and outcomes, revealing risk of never reaching a usable service rather than only the cost of successful completions.
Choose the summary to match the decision. For a central forecast, a median with a clear range may be useful. To show a level not exceeded by a given proportion of previous cases, state an empirical ranking and its method. For the eightieth percentile here, use the nearest-rank method: multiply sample size by 80% and round upward to a whole rank, giving rank eight. Its ratio is 1.50, producing a test cost of 150 from an original estimate of one hundred. Other interpolation methods exist, so an unlabeled method can create apparent disagreement between analysts performing different valid calculations. Retain the method with the decision so the result can be reproduced.
Investigate extreme cases without hiding them. A ratio of 2.00 may reflect an inherent project risk, a data error or incomparable scope. Examine the cause and document the treatment. If valid and relevant, excluding it merely for being large erases part of the risk being investigated. Show results with and without it as sensitivity analysis while keeping the main class governed by its declared rule. If it is genuinely erroneous or out of scope, correct or exclude it for a reason that would also apply to a small value. Selectively penalising high figures turns reference analysis into justification for a preferred estimate.
Use the outside perspective to challenge specific internal assumptions. If reference cases show migration usually takes longer, check whether the current estimate includes data cleaning, failed trials, revalidation and parallel running. Missing work can be added quantitatively; alternatively, an evidenced difference such as completed data cleaning may justify a different forecast. State how that difference changes transferability: saying the team is better provides no measurement. If the current estimate remains far below the class without a testable explanation, identify it as optimistic and requiring an informed choice, or revise it with contingency tied to remaining risk. The Green Book approach encourages examination of optimism, but does not prescribe one local uplift for this case.
Separate expected work cost from contingency for uncertainty. An estimate may start at one hundred, then reveal twenty of omitted data-cleaning work. Applying another 20% merely because the class median is 1.20, without reconsidering the starting point, may count the same omission twice. Reassess whether the estimate remains at the same maturity and scope as the reference. One approach retains a base of 120 with a separate remaining-risk analysis; another uses an inclusive reference forecast and explains its components. The board needs to know what buys known work and what protects against unresolved variation, with a reconciliation rather than automatic successive percentages.
Watch for completed-case selection bias. Two projects stopped before service may disappear from completion-cost ratios. Their spending can be below budget precisely because they stopped early; describing that low ratio as cost efficiency reverses its meaning. Retain two separate measures: cost outcomes of completed work and the fate of all committed initiatives. The second distinguishes completion, stopping and ongoing work with evidence status and cutoff date. The committee can then understand that the completion-cost class is conditional on completion and that failure to reach a deliverable needs separate evaluation. This shifts review from success stories to commitments and their outcomes.
Check transferability through time. Improved migration tools, supplier terms or licensing requirements can separate older work from current conditions. Recent and older groups may be shown separately with reasons, but a small recent group produces less stable summaries. Use a broader class to test the range and a closer one to explain differences, stating how each influenced judgement. Prices also need a suitable common base: nominal increases from general inflation do not all represent forecasting error in scale or performance. Smaller projects can cost more per site because of fixed setup costs; dividing totals by sites does not establish linear transferability.
Define duration independently from cost. It may end at first service availability, final-site migration or contract closure; identify which milestone answers the portfolio question. When combining cost and duration limits, check joint achievement rather than assuming one protects the other. High-cost cases might also be late, or extra teams might trade more cost for less time. Use matching case rows to see the relationship. One multiplier raising money and time equally can conceal a real decision: buy capacity to shorten duration, or protect budget with a later date?
Present an explicit reconciliation between detailed estimates and the reference class. Start with the gap, then missing work corrected, evidenced local differences and remaining uncertainty. If the committee selects a forecast below the reference, identify the risk owner, monitoring and review condition. If it chooses more contingency, specify drawdown conditions so reserve does not automatically become task funding without need. The class guides questions and confidence, while commitment size still depends on need, impact, capacity and risk limits. Retain the reason for divergence rather than only the approved number.
Keep the reference record alive after approval. Retain original and revised forecasts with change reasons, then add actual outcomes when mature. Overwriting old forecasts and declaring perfect accuracy removes the evidence of when the organisation knew what. Use the next project to test one reference-driven assumption, such as cleaning time or validation cycles. If evidence supports a different local forecast, document the difference and outcome. If it contradicts the estimate, update estimation practice rather than hide the case. Previous work then becomes a reviewable decision memory instead of rigid percentages that obstruct improvements whose effect has been demonstrated.
Check cost-record quality before interpreting differences economically. One case's actual cost may include paid invoices only, another unpaid commitments, and a third internal staff time. Harmonise definitions and disclose components that cannot be recovered. If a closing invoice remains outstanding, mark the amount provisional and identify when it will mature. A range or deferred judgement is more useful than silently inserting an incomplete figure into an apparently uniform class. Retain the case in the source register while temporarily excluding it from a particular calculation for a repairable data gap. Temporary exclusions need follow-up dates so difficult cases do not disappear permanently.
Small samples make upper ranks sensitive to each case. With ten observations, one new record can change the position called the eightieth percentile, and the interpolation method can change its value. Show sample size beside each summary. Compare logical groups defined in advance rather than searching many partitions and reporting the most optimistic one. Very limited organisational data can still raise qualitative questions about missing work, provided quantitative forecasts remain explicitly conditional. Weak evidence does not make decision-making impossible: it may justify a small information purchase, staged commitment or an openly approved contingency within committee authority.
Send the reference worksheet to someone who understands the work but is not defending the estimate, asking them to challenge class selection before examining the final ratio. They should reapply inclusion criteria to two excluded cases and one extreme, asking whether lower numbers would change the treatment. Then trace two figures to their sources and recalculate one summary. These steps test reproducibility and fair case selection while decision responsibility remains with the authorised owner. In actual decisions, retain comments, corrections and reasons for legitimate remaining disagreement. Review should test selection methods rather than reward consensus around a comfortable number.
Lesson summary
The lesson uses relevant prior work to challenge estimating assumptions. It explains inclusion rules, medians, means, empirical ranks and scope or price changes. Completion costs are separated from stopped initiatives, and missing work is reconciled with contingency and evidence limits.