Designing an analysis plan and bounding conclusions

An analysis plan is not a list of tools. “Clean the data and build a chart” does not specify what is being investigated or which result would change the judgment. A useful plan links the decision question to explicit explanations, identifies necessary data and calculations, and describes the result expected under each explanation. It also states what happens if evidence is insufficient and what the decision maker will receive. A one-page plan can prevent wandering through charts in search of an attractive story.

Start with four columns: possible explanation, expected pattern, required data, and a check that would weaken the explanation. Longer delivery times might reflect higher demand, a changed regional mix or a problem at one processing stage. A demand-pressure explanation predicts delays during high-volume periods. It becomes weaker if delays increase at stable volume within the same region. This does not automatically establish another explanation; eliminating one possibility does not prove an alternative. Retain competing explanations rather than combining them into a single narrative before investigation.

Specify quality checks before examining hypotheses. Check record counts, dates, uniqueness, missing values and consistent definitions across periods. In a delivery-duration analysis, arrival times preceding dispatch may reflect time-zone differences or recording errors, not exceptionally fast service. Use a meaning-based treatment rule and preserve the number of unresolved cases. Do not convert them to zero merely to keep a formula running: doing so can reduce the average and conceal the data problem.

After quality checks, move from simple descriptions to more explanatory work. Calculate overall volume and outcomes, inspect distributions and relevant segments, then investigate a defined explanation. This order provides reconciliation points. Segment totals should match the overall total when categories are exhaustive and do not overlap. A mismatch may reveal missing categories, overlapping membership or inconsistent exclusions. Resolve reconciliation before interpreting differences. A more sophisticated model cannot repair an incorrect aggregation rule.

Distinguish planned analysis from exploration. Unexpected patterns can be valuable, but record them as discoveries requiring suitable confirmation. Searching dozens of segments and highlighting the largest difference increases the chance of emphasizing a temporary fluctuation. Do not present a choice made after inspecting results as the only hypothesis specified beforehand. Examine another period, separate sample or appropriate experiment where possible. Retain limitations when further verification is unavailable. Exploration generates better questions; it does not create additional certainty by itself.

Set a practical stopping rule. The question may be whether a problem is concentrated in one shift enough to justify reviewing its procedures. If delays are evenly distributed and the data do not record processing stages, further shift charts cannot resolve the missing information. An appropriate output identifies the gap and proposes collecting stage timestamps in a defined follow-up sample. Sometimes the useful finding is that current data cannot distinguish the alternatives, provided the report explains what is missing and how it can be obtained.

Include sensitivity analysis: recalculate under another reasonable choice to see how strongly the result depends on that choice. If unknown outcomes affect a success rate, calculate one bound assuming every unknown case succeeded and another assuming none did. Show the proportion of unknown cases. These bounds are not a probability interval; they are arithmetic limits under explicit assumptions. If the decision differs between the bounds, missing outcomes matter to the decision and should be resolved or acknowledged before commitment.

State conclusions in three layers: what was observed, what the analysis suggests, and what remains unestablished. “Late delivery rose from 8% to 12% among eligible orders” is descriptive. “Most of the increase occurred in the evening shift” identifies its location. “Adding one employee will eliminate the problem” is an intervention claim that does not follow automatically from those descriptions. A limited trial with outcome monitoring may be preferable to a definitive promise. The process owner chooses the intervention within their authority; the analyst specifies evidence that would support or weaken the interpretation.

Finally, record the conditions needed to reproduce the analysis: sources, extraction dates, definitions, transformations, calculation order, exclusions and output version. A report does not need to contain all sensitive source records. An authorized workspace can preserve a version identifier or file fingerprint instead. The purpose is to enable a permitted reviewer to repeat the calculation and understand why it changes. When late-arriving data or revised definitions appear, these records help distinguish a legitimate revision from an accidental execution error.