Core data concepts · Lesson 6

Data quality and lineage across roles

7 min read

Lesson objectives

  • Trace a record from source to report and assign quality checks to the responsible roles
Reading tools

“The data is correct” is incomplete without a question and a processing stage. A value of 12 may be valid in a payment system but appear twice in a sales report because the analytics pipeline replayed a batch. A date can be syntactically valid yet represent file arrival rather than sale time. First define the grain: does one record mean a payment, an invoice line, a change event or a daily snapshot? Then state the measure, numerator, denominator and time window. Check quality at capture, transformation and aggregation, not only after the dashboard appears. At each step retain an identifier that leads to the source or to a particular processing version. This is the minimum needed to explain two apparently contradictory figures.

Uniqueness, completeness, consistency and timeliness ask different questions. Testing payment_id uniqueness may reveal a replayed payment, but it cannot prove that two invoices for one customer are duplicates. customer_id completeness matters if the report joins payments to customers; another question may not need it. Consistent currency precedes adding amounts, and consistent time zones precede branch-level daily comparisons. Accuracy is not implied by the absence of nulls: a branch code may be populated but wrong. Tie each check to a decision its failure could change, and define a threshold and response. Do not silently replace a rejected amount with zero when zero is a legitimate value; quarantine it and disclose the effect.

Lineage records how a number reached a screen. Record the source file or table, batch extraction time, transformation rule version, input and rejected counts and deduplication key. When a currency rule changes or an old week is reloaded, one must know which report and output version were affected. Lineage does not require copying every sensitive record into a narrative document; run identifiers and appropriate metadata can connect the stages. Without a route from report row to source, one cannot tell an entry error from a transformation error or a correct aggregation at a different grain. This trace supports reproducibility and confidence, while access control remains a separate responsibility.

Roles share the workflow without losing their distinct decisions. A database administrator maintains operation, access, backup and performance of the data store they manage. A data engineer orchestrates ingestion, transformation and movement and monitors batch failures. A data analyst defines measures, checks dimension meaning and communicates the conclusion and limits. A small team may put all three roles in one person, but the decisions remain separate. If an analyst spots an unexpected figure, hiding a branch on a chart is not a repair. They inspect the measure and lineage with the engineer and escalate a source-system defect to its owner. A managed cloud service does not assign business meaning to a measure automatically.

Measure quality at the proper grain. If 1,000 payment records arrive with 20 duplicated entries and 10 missing currencies, do not automatically assert that 970 are usable: the failing sets may overlap. Count the union of rejects after declaring rule precedence, and establish whether repeated IDs are real replays or distinct transactions with a bad identifier. Observe rate trends across batches instead of making one successful batch a universal guarantee. If “successful payment” changes from every paid status to paid and not refunded, version the measure and record the change date. Comparing old and new series without that note suggests business growth or decline that may have never occurred.

A manager challenges a number changed after yesterday’s data correction. Trace the report through transformation back to source. Each correction needs a description of the error, discovery time and affected record set. Determine whether the problem belongs to raw data or the rule deriving the value. If the source is correct but transformation is wrong, retain the source and reprocess the appropriate version. If the source itself changes, record its new version and justification. Give the report a version linked to its inputs. Equal totals before and after repair can hide changed branch allocations, so compare a suitable distribution too. This distinguishes data repair from metric redefinition and helps the report owner explain the impact.

Lesson summary

Trace a record from source to report and assign quality checks to the responsible roles.