Sampling evidence and confidence limits

Project data can be plentiful without representing the decision at hand. Fifty defects from launch week do not necessarily describe a full year of quality; ten interviews with team managers may say little about users at a remote site. Define the population about which the conclusion is sought: every request in a month, every scope change since baseline, or all support incidents across two sites. Then define the sampling frame actually accessible. The gap between population and frame exposes bias before the first case is chosen. If only closed tickets are available, difficult open cases disappear and resolution time looks artificially good.

Select cases according to risk and data type. For invoices, examine all high-value items and a sample of smaller ones rather than the first ten files. For interviews, deliberately include a rare but consequential role such as the safety acceptance owner; this is not a statistical representation of all employees. For service time, include busy and quiet periods and request categories, and record missing timestamps. Do not call a purposive sample random or assign it an uncalculated numeric margin of error. Where critical cases are few, inspect all of them if feasible; a general average can conceal a single unacceptable failure.

Qualitative investigation needs a stopping rule. Interviews may continue until the main mechanisms recur and no material new hypothesis appears, but this does not justify ignoring an unheard group. Record why another participant was added and what changed in the cause map after the interview. If three suppliers say specifications changed, inspect document versions; repeated testimony from people relying on one source is not independent triangulation. If twenty employees call a tool slow, obtain measured response time at known load before buying servers. Interviews identify a test question, not replace the test.

Keep a selection record with list extraction date, sort fields, accessible population size, exclusion reasons, selected IDs, and reviewer. If the definition of a report changed mid-period, separate versions or recode against one documented rule. A department refusing to provide its reports is an access gap affecting confidence, not automatic proof of poor performance. If a large funding decision is irreversible and peak records are unavailable, recommend future measurement or defer the irreversible choice rather than make a firm claim from only easy data.

Test how missing information affects the choice. In the example, if all 25 open reports are still within deadlines, they are not yet failures; their absence still makes closure-time analysis incomplete. If ten are overdue, compliance judgment changes. Bound the result while their status is unknown and seek due dates as the nearest decisive evidence. Do not impute missing cases as either zero or success. The outcome can also change if high-impact reports are fewer but more important. Show their compliance separately rather than weighting solely by paper count.

Confidence should match decision strength, not writing style. “The sample represents closed reports in a quiet month and does not test peak capacity” prevents false generalisation while preserving a useful observation. It might support a low-cost reminder improvement but cannot alone justify rebuilding the project office or terminating contracts. Set a criterion in advance: what difference between busy and quiet periods, or what number of late critical reports, would change the recommendation? Apply the same rule when new data arrive even if they contradict the first view. Verification then becomes a decision loop rather than endless research.