Data cleaning and deduplication
Mechanism: raw record → quality rule → versioned accepted/rejected set
rows = ["a", "a", "b", ""]
print(sorted({r.strip() for r in rows if r.strip()}))
Run with python3; expected output is ['a', 'b']. Preserve raw data, rules, rejected records, and versions; deduplication can erase rare but legitimate examples or hide train/test leakage.
Sources
- Datasheets for Datasets — collection and maintenance documentation.
- Data Cleaning: Problems and Current Approaches — data-quality mechanisms.
Data cleaning is not cosmetic. Invalid rows, duplicated examples, contradictory labels, and stale records can dominate training gradients, inflate eval scores, or make failures impossible to diagnose.
Cleaning checks
- Schema validity: types, enums, ranges, required fields.
- Text quality: encoding problems, boilerplate, broken markup, language mismatch.
- Label integrity: impossible labels, contradictory labels, missing adjudication.
- Entity consistency: IDs, timestamps, joins, and references agree.
- Freshness: examples are within the intended time window.
- Authorization: examples can legally and ethically be used for the purpose.
Deduplication levels
| Level | Example |
|---|---|
| Exact duplicate | identical row or document |
| Near duplicate | same article with minor formatting changes |
| Semantic duplicate | same question or answer phrased differently |
| Entity duplicate | same user/account/item appears across splits |
| Benchmark duplicate | eval item appears in training or prompt examples |
Practical workflow
- Run automatic validation before manual review.
- Remove exact duplicates and obvious invalid records.
- Cluster near-duplicates for review.
- Check duplicates across training, validation, test, and holdout.
- Record every cleaning rule in the dataset version.
Pitfall
Over-cleaning can erase hard cases. Do not remove examples merely because they are messy if messy inputs are part of the real product distribution.
Connects to: data quality · split leakage · benchmark leakage