Data Quality and Documented Cleaning Before Analysis
4 min read
Lesson objectives
- Assess data quality as fitness for purpose using distinct quality dimensions.
- Calculate simple completeness, uniqueness and validity indicators without mixing dimensions.
- Document cleaning rules, affected records, action, rationale and effect while preserving traceability to source.
Data quality means fitness for purpose
There is no useful notion of “quality” detached from purpose. Data can be current enough for annual planning but unsuitable for daily operational monitoring. The Government Data Quality Framework treats quality as fitness for purpose and emphasizes that it is broader than one-off cleaning; quality should be understood and monitored through the data lifecycle. For the analyst, the practical question is: which defect could change this decision or make this metric misleading?
| Dimension | Simplified check question |
|---|---|
| Completeness | Are required values or records present? |
| Uniqueness | Is the same record unintentionally duplicated? |
| Consistency | Are definitions/codes aligned across sources and periods? |
| Timeliness | Are the data current enough for the decision? |
| Validity | Does the value follow allowed type, range and rules? |
| Accuracy | Does the value represent the intended reality well enough? |
Clean data, but leave an audit trail of what changed
Before cleaning, preserve an unchanged raw copy or another way to trace back to source. Then use a cleaning log that states the rule, number of affected records, action, rationale and expected analytical effect. Example: “3 records identical across all fields and load key; verified as reload duplicates; retained one copy of each; row count changed from 100 to 97.” This is more transparent than writing “data cleaned.”
| Issue | Do not automatically | Better action |
|---|---|---|
| Missing value | turn it into zero | determine what missing means and its effect, then justify deletion/imputation/separate category |
| Possible duplicate | delete every repeated key | verify unit of observation and whether repetition is legitimate or a load error |
| Outlier | delete it because it is far away | inspect source and context; it may be valid and important |
| Different date format | guess day and month | use a documented source rule or flag for review |
Quality dimensions can trade off. Waiting for every record to be fully verified may improve accuracy but make a report less timely. Publishing a rapid report may improve timeliness but requires provisional data to be labeled and completeness to be stated. There is no single answer in every case; the choice depends on purpose, the consequence of error and whether results can be updated later.
After cleaning, recompute basic checks: row count, unique keys, missingness, numeric ranges and key metrics that should remain stable or whose interpretation should clearly change. If cleaning materially changes a result, disclose that fact rather than hiding it because it shows how much the decision depends on processing rules.
Lesson summary
Data quality depends on intended use. Keep completeness, uniqueness, consistency, timeliness, validity and accuracy distinct, and do not automatically “fix” missing values, duplicates or outliers without understanding them. Good cleaning is traceable and makes its effect on metrics and decisions visible.