Data for LLMs

Mechanism: corpus policy → tokenized examples → objective → evaluation boundary

documents = {"licensed": 12, "unknown_rights": 2}
print("quarantine", documents["unknown_rights"])

Run with python3; expected output identifies material requiring policy review. LLM data needs provenance, rights, quality, deduplication, PII controls, contamination checks, and versioned mixtures.

Sources

• The Pile — large-scale corpus construction.
• Data Provenance Initiative — dataset provenance for foundation models.

LLMs are shaped by enormous data mixtures: web text, books, code, dialogue, academic text, synthetic examples, instruction data, preference data, and domain corpora. At that scale, data curation becomes model design.

Pretraining corpus decisions

• Source selection: web, code, books, papers, forums, documentation, domain corpora.
• Quality filtering: remove spam, boilerplate, low-information pages, and broken text.
• Deduplication: reduce memorization and prevent repeated low-quality documents from dominating.
• Mixture weights: decide how much code, math, multilingual, domain, and conversational data to include.
• Licensing and consent: ensure data can be used for the intended purpose.
• Safety filtering: reduce harmful, private, or policy-violating content where required.

LLM data stages

Scale changes the failure modes

At internet scale, tiny percentages become millions of examples. Deduplication, contamination checks, privacy filtering, and provenance tracking matter because manual inspection can only sample the corpus.

Pitfall

More tokens are not automatically better. Low-quality, duplicated, stale, or contaminated tokens can waste compute and teach the wrong distribution.

Connects to: next-token pretraining · supervised fine-tuning · DPO