Evidence architecture and baseline
11 min read
Lesson objectives
- Diagnose evidence traceability, compare alternatives, and produce a traceable recommendation.
For evidence traceability, the consultant faces this condition: the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. The decision is not a technology selection; it is which explanation deserves investment. Inputs are transaction logs, cycle time, service errors, user interviews, and operating cost. The work record separates each observation from its hypothesis and asks for disconfirming evidence before scope is fixed. The executable output is an evidence register linking each conclusion to its source, period, limits, and confidence, accepted only when another sponsor can trace evidence to criterion and then to decision.
| Evidence | Validity test | Conflict test | Use rule |
|---|---|---|---|
| transaction logs | Adds an independent view; state what it supports and what remains outside its scope. | Compare it with cycle time; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| cycle time | Shows a performance pattern when definition, period, and case mix are stable. | Compare it with service errors; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| service errors | Adds an independent view; state what it supports and what remains outside its scope. | Compare it with user interviews; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| user interviews | Explains context and motivation, but needs an operating record to estimate prevalence. | Compare it with operating cost; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| operating cost | Shows a performance pattern when definition, period, and case mix are stable. | Compare it with exception records; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| exception records | Shows a performance pattern when definition, period, and case mix are stable. | Compare it with stakeholder observations; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| stakeholder observations | Explains context and motivation, but needs an operating record to estimate prevalence. | Compare it with transaction logs; explain agreement or conflict. | Use it in an evidence register linking each conclusion to its source, period, limits, and confidence only to the extent that it supports number of indicators with a documented definition, owner, source, and baseline period. |
| Alternative explanation | What it explains | Discriminating test | What weakens it |
|---|---|---|---|
| Technology | Explains change in transaction logs without change in cycle time. | Isolate the component and observe number of indicators with a documented definition, owner, source, and baseline period. | Weakened if the management dashboard shows high satisfaction while the contact centre records rising rework and complaints after isolation. |
| Process | Explains waiting between cycle time and service errors. | Measure value time and exceptions for evidence traceability. | Weakened when no handoff or rework exists. |
| Data | Explains conflict between transaction logs and stakeholder observations. | Align definition, sample, and refresh before an evidence register linking each conclusion to its source, period, limits, and confidence. | Weakened when lineage agrees while the effect persists. |
| Incentives | Explains behaviour that reduces number of indicators with a documented definition, owner, source, and baseline period despite available tools. | Compare team targets with resource decisions in: the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Weakened across differently rewarded teams with similar behaviour. |
| Judgment | Prefer the evidence traceability explanation with the fewest hidden assumptions. | Record confidence, disconfirming evidence, and the next test. | Does not justify an irreversible commitment. |
| Option | Decisive criterion | Required evidence | Decision rule |
|---|---|---|---|
| accepting the published average | validity | Test its effect on number of indicators with a documented definition, owner, source, and baseline period with transaction logs. | Continue only when it supports an evidence register linking each conclusion to its source, period, limits, and confidence without breaching risk tolerance. |
| rejecting all data | recency | Test its effect on number of indicators with a documented definition, owner, source, and baseline period with cycle time. | Continue only when it supports an evidence register linking each conclusion to its source, period, limits, and confidence without breaching risk tolerance. |
| building a segmented baseline that tests the conflict | coverage | Test its effect on number of indicators with a documented definition, owner, source, and baseline period with service errors. | Continue only when it supports an evidence register linking each conclusion to its source, period, limits, and confidence without breaching risk tolerance. |
| a bounded reversible test | bias | Test its effect on number of indicators with a documented definition, owner, source, and baseline period with user interviews. | Continue only when it supports an evidence register linking each conclusion to its source, period, limits, and confidence without breaching risk tolerance. |
| Criterion | Operational definition | Rating source | Sensitivity test |
|---|---|---|---|
| validity | Define it through evidence traceability, not a promotional adjective. | Rate from transaction logs in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| recency | Define it through evidence traceability, not a promotional adjective. | Rate from cycle time in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| coverage | Define it through evidence traceability, not a promotional adjective. | Rate from service errors in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| bias | Define it through evidence traceability, not a promotional adjective. | Rate from user interviews in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| reproducibility | Define it through evidence traceability, not a promotional adjective. | Rate from operating cost in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| measurability | Define it through evidence traceability, not a promotional adjective. | Rate from exception records in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| transition impact | Define it through evidence traceability, not a promotional adjective. | Rate from stakeholder observations in the condition where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints. | Vary the weight and test whether an evidence register linking each conclusion to its source, period, limits, and confidence remains preferred. |
| Stage | Specific work | Owner | Decision receipt |
|---|---|---|---|
| Fix the question | Link evidence traceability to an owner and number of indicators with a documented definition, owner, source, and baseline period. | Engagement owner | Scope that can accept or reject an evidence register linking each conclusion to its source, period, limits, and confidence |
| Test the cause | Compare transaction logs and cycle time using service errors. | Discovery lead | One explanation strengthened and another weakened |
| Option gate | Compare accepting the published average and rejecting all data on validity and recency. | Sponsor | Continue, change, or stop |
| Bounded wave | Implement an evidence register linking each conclusion to its source, period, limits, and confidence with rollback and a benefit owner. | Outcome owner | Observable change in number of indicators with a documented definition, owner, source, and baseline period |
| Review | Separate the effect where the management dashboard shows high satisfaction while the contact centre records rising rework and complaints from demand or mix change. | Benefit owner | Benefit with attribution limits |
| Handoff | Transfer the definition of number of indicators with a documented definition, owner, source, and baseline period and the evidence traceability decision record. | Service owner | Local ability to repeat measurement |
Boundary for evidence traceability: escalate specialist legal, security, or financial questions and keep their effect visible in an evidence register linking each conclusion to its source, period, limits, and confidence.
Lesson summary
The lesson links evidence traceability to evidence, trade-off, execution, and measurement while exposing assumptions and limits.