From flow data to measurable decisions · Lesson 4

Diagnosing performance without mistaking association for cause

6 min read

Lesson objectives

  • Standardize work-mix weights and interpret a supplier-ranking reversal without an unsupported causal claim.
  • Calculate a difference in changes and identify a specific competing explanation.
Reading tools

A service improvement after changing carriers makes it tempting to credit the new carrier immediately. Yet routes, parcel sizes or the season may have changed simultaneously. Association means that two changes appeared together; a causal claim needs a comparison that helps estimate what would have happened without the intervention. Begin with an auditable diagnosis rather than a ready-made causal conclusion. Check cohort composition, denominators and measurement boundaries, then construct a relevant comparison. This can prevent an expensive decision based on an aggregate hiding the allocation of work, such as removing a capable supplier simply because it handled the most difficult orders.

ComparisonAB
Easy90/100=90%19/20=95%
Difficult8/10=80%81/100=81%
Actual delivered work mix98/110≈89.09%100/120≈83.33%
Hypothetical half-easy mix0.5×90+0.5×80=85%0.5×95+0.5×81=88%

Stratification does not resolve every explanation. B might receive lighter parcels even within difficult routes, while A handles urgent orders. Examine assignment factors known before execution: distance, delivery window, shipment size and customer priority. Do not casually adjust for an outcome caused by supplier selection itself; that can remove part of the effect being investigated. If one supplier never serves a route category, the data cannot supply its performance there without additional assumptions. The supported comparison is consequently narrower than a recommendation to replace one supplier throughout the network. Diagnose where a comparison is supported and where a pilot or additional operational evidence is required.

A second design compares changes over time. In another fictional case, service at a warehouse adopting a packaging system rises from 80% to 90%. A comparison warehouse rises from 85% to 88%. The treated site's raw improvement is ten percentage points; the comparison improves by three; the difference in changes is seven points. World Bank guidance explains difference-in-differences and its parallel-trends assumption: the comparison's change must represent the treated group's change without intervention. Two observations cannot establish that condition. Correctly subtracting three from ten is therefore not, by itself, proof that packaging caused a seven-point improvement.

Strengthen interpretation by inspecting several pre-intervention periods and checking that service definitions and work mixes did not change differently across the two sites. Look for a concurrent change unique to the treated warehouse, such as extra staffing or a faster carrier. Similar earlier movements support the comparison's plausibility but do not prove the unobserved post-intervention path without treatment. For a future trial, choose an assignment unit that limits interference. Whole shifts or lines may be needed when workers share tools. Thousands of parcels processed under the same shift or carrier do not automatically provide thousands of independent experimental units.

Keep sample size and cost next to the service result. B's five-point easy-route advantage rests on only 20 observed orders and deserves follow-up before a large volume transfer. Also check whether service was purchased through expensive expedites or accompanied by damage appearing later. Establish comparable baselines for cost, damage and complaints and define when the intervention would no longer be worthwhile. A bounded pilot may be justified without proof of the final effect. Match the commitment to the strength of evidence and state the unresolved question that the next test should address, rather than using uncertainty as either a blanket veto or a reason to ignore the comparison.

Failure diagnosis also requires overlap accounting. Among 100 fictional orders, 90 arrive on time, 80 are complete and 75 meet both conditions. Ten are late and 20 incomplete, but there are not 30 failed orders: five are both late and incomplete. Distinct failures equal 10+20−5=25, matching 100−75. Multiplying 90% by 80% to obtain 72% joint success would assume independence that is not established and disagrees with the observed intersection. Build the joint table before selecting an intervention. Fixing delay on five still-incomplete orders need not create any new successful orders. Operational problems resolved and customer promises converted to success are therefore different counts.

Name numerical differences precisely. Moving from 80% to 85% is five percentage points, while the relative increase is 5/80=6.25%. Both descriptions are valid when labeled. For a planned 1,000 orders, a standardized three-point advantage projects to about 30 additional successful orders if mix and within-category rates remain unchanged. It is not a promise of exactly 30 outcomes: capacity and shared disruptions may change. Use the projection to size a pilot or explore potential value, then compare it with observed execution using consistent denominators.

Lesson summary

Fair comparison starts with definitions, mix and design; correct arithmetic alone does not isolate a cause.