Training Datasets: Quality, Splits, and Preparation

1. What Is Training Data?

Training data is the collection of examples a machine learning model learns from. Each example typically consists of one or more input features and, in supervised settings, a corresponding target or label. During training, an optimization algorithm—most commonly stochastic gradient descent (SGD) or a variant such as AdamW—adjusts the model's internal parameters to minimize a loss function that measures the discrepancy between predictions and targets.

For large language models (LLMs), training data is predominantly unstructured natural language text. A single training example might be a document, a web page, a code file, or a conversation transcript. The model learns to predict the next token in a sequence, which implicitly teaches grammar, facts, reasoning patterns, and style. The breadth and depth of this data determines what the model can know and do.

The Three Roles of Data in ML

Dataset	Purpose	Used During	Exposed to Model?
Training set	Learn parameters (weights, biases)	Gradient descent / backpropagation	Yes — directly
Validation set	Tune hyperparameters, early stopping	Periodic evaluation between epochs	No — gradients not computed
Test set	Estimate generalization performance	Final evaluation only	No — touched once

Analogy: The training set is the homework a student practices on. The validation set is the quiz the teacher uses to adjust study strategy. The test set is the final exam—seen only once, graded for real. If the student memorizes the homework answers instead of learning the concepts, they fail the exam. That is overfitting.

Why Data Defines the Model

A model can only learn patterns that exist in its training data. If the data contains biases, factual errors, or narrow perspectives, the model will inherit all of them. The architecture and optimization procedure determine how efficiently patterns are extracted, but the data determines what patterns are available to extract. This is why dataset engineering is often the highest-leverage activity in an ML project.

2. Why Quality Matters More Than Quantity

A persistent myth in machine learning holds that more data is always better. In practice, the relationship between dataset size and model quality is highly non-linear, and the quality of each example dominates the marginal value of additional examples once a corpus reaches a certain scale.

What “Quality” Means

Accuracy: Labels are correct; text is factual and coherent.
Diversity: The data covers a wide range of topics, styles, domains, and difficulty levels.
Relevance: Examples are pertinent to the task or downstream use case.
Consistency: Formatting, annotation guidelines, and labeling conventions are uniform.
Cleanliness: Duplicates, boilerplate, spam, and encoding artifacts are removed.
Freshness: Information is current enough for the intended application.

Evidence From LLM Training

Multiple lines of evidence support the primacy of quality:

Evidence	Source / Study	Implication
Phi-series models trained on “textbook quality” synthetic data punch far above their parameter count	Microsoft Research (Phi-1, Phi-1.5, Phi-2)	Curated high-quality data can substitute for raw scale
FineWeb-Edu filtering by educational content score dramatically improves benchmark results	HuggingFace FineWeb (2024)	Quality classifiers are a powerful filtering tool
Deduplication alone yields measurable perplexity reductions	Lee et al., “Deduplicating Training Data Makes Language Models Better” (2022)	Even accidental repetition degrades learning
Stable cascade of quality filters (language ID, perplexity, heuristics) used in RefinedWeb	Penedo et al. (2023)	Systematic pipeline filtering outperforms raw dumps

Rule of thumb: Before adding more data, audit what you already have. Removing 20% of a dataset that is low-quality, duplicated, or off-topic often improves downstream performance more than doubling the corpus with unfiltered scrapes.

When Quantity Does Matter

Quantity is not irrelevant. For models with billions or trillions of parameters, a minimum volume of tokens is required to avoid underfitting and to expose the model to enough linguistic diversity to generalize. The Chinchilla scaling laws (discussed in Section 12) quantify this relationship precisely. The point is that quantity without quality is wasted compute—and quality with insufficient quantity starves the model.

3. Train / Validation / Test Splits Explained

Partitioning data into separate sets is the single most important practice for honest model evaluation. Without it, you cannot distinguish between a model that has learned generalizable patterns and one that has memorized specific examples.

Common Split Ratios

Ratio	Train	Validation	Test	Best For
60 / 20 / 20	60%	20%	20%	Small datasets where test set needs statistical power
70 / 15 / 15	70%	15%	15%	Medium datasets; balanced compromise
80 / 10 / 10	80%	10%	10%	Large datasets where 10% is still many examples
90 / 5 / 5	90%	5%	5%	Very large datasets (millions+ of examples)
98 / 1 / 1	98%	1%	1%	Web-scale LLM pretraining (billions of tokens)

How to Choose

The validation and test sets must be large enough to produce a statistically stable estimate of performance. If your test set contains only 50 examples, a single misclassified sample swings accuracy by 2%. As a heuristic, aim for at least 1,000 examples in each evaluation set for classification tasks. For LLM evaluation, held-out validation loss on a few hundred million tokens is usually sufficient because the metric is an average over many predictions.

Stratified vs Random Splits

A random split assigns each example to a partition by drawing from a uniform distribution. This works well when classes are balanced. When classes are imbalanced, a stratified split preserves the class distribution in each partition, ensuring that rare classes appear in both validation and test sets.

# Stratified split with scikit-learn
from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.4, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42
)
# Result: 60% train, 20% val, 20% test with preserved class ratios

Time-Series Splits

For temporal data, random splitting causes leakage because future information can leak into the training set. Use a chronological split: train on the oldest data, validate on the next period, and test on the most recent period. Walk-forward validation (expanding or rolling window) is the gold standard for time-series evaluation.

K-Fold Cross-Validation

When data is scarce, k-fold cross-validation extracts maximum value. The data is divided into k equally sized folds. The model is trained k times, each time holding out a different fold as the validation set and training on the remaining k−1 folds. Performance is averaged across folds. Common values are k=5 and k=10. For very small datasets, leave-one-out cross-validation (k = n) is used.

Practical note: For LLM pretraining, explicit train/validation splits are sometimes replaced by a held-out shard of documents. The validation loss on this shard is monitored during training to detect overfitting and trigger early stopping or learning-rate decay.

4. Overfitting, Underfitting, and the Generalization Gap

Overfitting

Overfitting occurs when a model learns patterns specific to the training data that do not generalize to unseen data. The model effectively memorizes noise, outliers, and idiosyncratic features rather than discovering the underlying signal. Symptoms include training loss continuing to decrease while validation loss plateaus or increases.

Symptom	Overfitted Model	Underfitted Model
Training loss	Very low	High
Validation loss	Higher than training loss (gap)	High, close to training loss
Training accuracy	Near 100%	Below target
Validation accuracy	Significantly lower	Below target
Model behavior	Memorizes specifics	Fails to capture patterns

Causes of Overfitting

Model too complex relative to data (too many parameters, too deep)
Training too long without regularization or early stopping
Insufficient training data for the model capacity
Noisy or mislabeled training examples
Training and test distributions mismatched

Mitigation Strategies

Early stopping: Halt training when validation loss stops improving.
Regularization: L1 (Lasso), L2 (Ridge), or weight decay penalizes large weights.
Dropout: Randomly zero out activations during training to prevent co-adaptation.
Data augmentation: Artificially expand the training set with transformed examples.
Reduce model capacity: Use fewer layers, smaller hidden dimensions.
Batch normalization / layer normalization: Stabilize training and add mild regularization.
More data: The most reliable cure if obtainable.

Underfitting

Underfitting is the opposite failure mode: the model is too simple or trained too briefly to capture the patterns in the data. Both training and validation loss remain high. The model has high bias and low variance. Remedies include increasing model capacity, training longer, reducing regularization, or improving feature engineering.

Warning: Underfitting is often misdiagnosed as a data problem when it is actually an optimization problem. If the learning rate is too high, the model may fail to converge regardless of data quality. Always check loss curves and gradient norms before concluding the data is insufficient.

5. The Bias-Variance Tradeoff

The expected prediction error of a model can be decomposed into three components: bias, variance, and irreducible error. Understanding this decomposition clarifies why overfitting and underfitting happen and how to balance them.

Component	Definition	High Value Means	Remedy
Bias	Systematic error from wrong assumptions (e.g., linear model for nonlinear data)	Underfitting	Increase model complexity
Variance	Sensitivity to fluctuations in the training set	Overfitting	More data, regularization, simpler model
Irreducible error	Inherent noise in the data that no model can eliminate	Bayes-optimal error floor	Clean data; cannot be reduced by modeling

The total expected error is approximately:

Error = Bias² + Variance + Irreducible Error

As model complexity increases, bias tends to decrease (the model can represent more nuanced functions) while variance tends to increase (the model becomes more sensitive to the specific training sample). The optimal complexity minimizes the sum of bias and variance. This is the bias-variance tradeoff.

Modern context: In the era of massively over-parameterized neural networks and LLMs, the classical bias-variance tradeoff is supplemented by the phenomenon of double descent. Test error first decreases, then increases (classical regime), then decreases again as the model becomes vastly over-parameterized (interpolating regime). In practice, very large models trained on very large datasets can achieve low bias and low variance simultaneously.

6. Data Leakage and How to Prevent It

Data leakage occurs when information from outside the training set—typically from the validation or test set—unintentionally influences the training process. The result is an optimistically biased evaluation that collapses when the model is deployed on truly unseen data. Leakage is one of the most common and insidious errors in applied machine learning.

Common Sources of Leakage

Leakage Type	Description	Example
Feature leakage	A feature in the training data encodes the target	“Days until churn” as a feature for churn prediction
Train-test contamination	Same or near-duplicate examples appear in both train and test	Duplicate documents across web crawl shards
Temporal leakage	Future data is used to train a model that should only see past data	Random shuffle of time-series data before splitting
Preprocessing leakage	Normalization or feature selection is fit on the full dataset before splitting	Computing mean/std on all data, then splitting
Group leakage	Examples from the same group (patient, user, session) span train and test	Splitting by row instead of by patient ID in medical data
Benchmark leakage	Test benchmark questions are included in pretraining data	MMLU questions scraped into Common Crawl

Prevention Checklist

Split before preprocessing. Fit all transformations (scaling, imputation, tokenization vocabulary) on the training set only, then apply them to validation and test.
Use chronological splits for time-series. Never shuffle temporal data before partitioning.
Group-aware splitting. When examples share a group identity (user ID, patient ID, document source), ensure all examples from a group stay in one partition.
Deduplicate across all partitions. Run near-duplicate detection (e.g., MinHash, SimHash) across the entire dataset before splitting.
Audit features for target proxies. Any feature that would not be available at inference time should be removed.
Decontaminate against benchmarks. Filter training data against held-out evaluation sets using n-gram overlap detection.
Document the split protocol. Record the random seed, split ratios, and stratification strategy for reproducibility.

# Correct: fit scaler on train only, then transform val/test
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)    # fit + transform
X_val_scaled   = scaler.transform(X_val)           # transform only
X_test_scaled  = scaler.transform(X_test)          # transform only

# WRONG: fitting on the full dataset causes leakage
# scaler.fit_transform(X)  # NEVER do this before splitting

Critical: In LLM training, benchmark contamination is a widespread form of leakage. If a model has seen MMLU, HellaSwag, or GSM8K examples during pretraining, its scores on those benchmarks are inflated and do not reflect genuine reasoning ability. Every serious evaluation should report decontamination procedures.

7. Data Preprocessing: Cleaning, Normalization, Tokenization

Raw data is almost never ready for training. Preprocessing transforms messy, heterogeneous inputs into a clean, consistent, numeric representation that a model can consume efficiently. The preprocessing pipeline is a first-class engineering artifact—it must be reproducible, versioned, and applied identically at training and inference time.

7.1 Cleaning

Cleaning removes or repairs content that would mislead the model. For text corpora, typical cleaning steps include:

Strip HTML tags, boilerplate (headers, footers, navigation menus)
Remove or repair broken Unicode (mojibake, encoding mismatches)
Filter out very short documents (< 50 tokens) or very long outliers
Remove pages with excessive repetition (e.g., n-gram repetition ratio above threshold)
Filter profanity, spam, and low-quality content using classifiers
Language identification and filtering to target languages
Deduplication at the document, paragraph, and line level
PII removal (email addresses, phone numbers, SSNs, API keys)

7.2 Normalization

Normalization standardizes the numeric or textual representation so that the model sees consistent inputs. For tabular features, this means scaling. For text, it includes case folding, punctuation normalization, and whitespace cleanup.

Technique	Formula / Action	When to Use
Standardization (Z-score)	(x − mean) / std	Features with Gaussian-ish distribution
Min-Max scaling	(x − min) / (max − min)	Bounded features; neural network inputs
Robust scaling	(x − median) / IQR	Features with outliers
Log transformation	log(x + 1)	Heavy-tailed distributions (counts, prices)
Case folding (text)	lowercase all characters	When case is not semantically meaningful
Unicode normalization (text)	NFKC or NFC	Unify equivalent character representations

7.3 Tokenization

Tokenization converts raw text into a sequence of discrete tokens, each mapped to an integer ID in the model's vocabulary. The choice of tokenizer affects vocabulary size, sequence lengths, handling of rare words, and multilingual coverage.

Tokenizer	Type	Used By	Vocab Size (typical)
Byte-Pair Encoding (BPE)	Subword	GPT-2, GPT-3, LLaMA	32,000 – 50,000
SentencePiece (Unigram)	Subword	T5, ALBERT, XLNet	32,000 – 128,000
WordPiece	Subword	BERT, DistilBERT	30,000 – 100,000
Byte-level	Character/byte	GPT-2 (byte-level BPE)	256 base + merges
Tiktoken	Byte-level BPE (fast impl)	GPT-4, GPT-4o	100,000 – 200,000

# Tokenization with HuggingFace tokenizers (BPE)
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="<unk>"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(vocab_size=32000, special_tokens=["<pad>", "<unk>", "<s>", "</s>"])
tokenizer.train(files=["corpus.txt"], trainer=trainer)

encoding = tokenizer.encode("Training data quality matters.")
print(encoding.tokens)  # ['Training', 'data', 'quality', 'matter', 's', '.']
print(encoding.ids)     # [412, 89, 2301, 567, 5, 3]

Vocabulary size tradeoff: A larger vocabulary means fewer tokens per document (shorter sequences, faster training) but a bigger embedding matrix (more parameters). A smaller vocabulary means longer sequences but a leaner model. The 32K–128K range is the sweet spot for most modern LLMs.

8. Data Augmentation Techniques

Data augmentation artificially increases the effective size and diversity of a training set by applying label-preserving transformations to existing examples. It is one of the most effective regularizers, particularly when labeled data is scarce.

Augmentation for Text and LLMs

Technique	Description	Preserves Label?
Synonym replacement	Replace words with WordNet or embedding-based synonyms	Usually
Random word deletion	Drop a fraction of words randomly	Yes (for classification)
Back-translation	Translate to another language and back; yields paraphrase	Yes
Random word swap	Swap adjacent word pairs	Usually
Masked language modeling	Mask random tokens; model predicts them (self-supervised)	N/A (pretraining)
Instruction paraphrasing	Use an LLM to rewrite instructions in different styles	Yes (for instruction tuning)
Response augmentation	Generate additional correct responses for the same prompt	Yes

Augmentation for Images and Vision

Random crop, resize, and flip
Color jitter (brightness, contrast, saturation, hue)
Random rotation and affine transforms
Cutout / random erasing
Mixup (linear interpolation of two images and their labels)
CutMix (paste a patch from one image onto another)
RandAugment, AutoAugment (learned augmentation policies)

Augmentation for Audio

Time-stretching and pitch-shifting
Adding background noise (Gaussian, environmental)
Room impulse response simulation (reverberation)
SpecAugment (masking time and frequency bands on spectrograms)

Caution: Aggressive augmentation can change the semantics of an example. If a synonym replacement turns “The movie was not good” into “The movie was good,” the label is no longer correct. Always validate augmentation pipelines by inspecting transformed examples manually before training at scale.

9. Dataset Formats: JSONL, CSV, Parquet, HuggingFace

The storage format of a dataset affects disk usage, I/O throughput, random access speed, and interoperability with training frameworks. The right choice depends on dataset size, access patterns, and the toolchain.

Format	Human-Readable	Compression	Columnar	Best For
JSONL (JSON Lines)	Yes	None (can gzip)	No (row-based)	NLP examples, instruction tuning, debugging
CSV	Yes	None (can gzip)	No (row-based)	Tabular data, small datasets, interchange
Parquet	No	Snappy / Zstd (built-in)	Yes	Large-scale LLM pretraining, analytics
HuggingFace Datasets (Arrow)	No	Memory-mapped	Yes	Integration with transformers, streaming
TFRecord	No	Built-in	No (record-based)	TensorFlow pipelines
WebDataset / Sharded tar	No	Per-file	No	Distributed training over object stores

JSONL Example

JSONL (JSON Lines) stores one JSON object per line. It is the de facto standard for instruction tuning datasets because each line is a self-contained example that can be streamed, shuffled, or sampled independently.

{"instruction": "Explain what a neural network is.", "input": "", "output": "A neural network is ..."}
{"instruction": "Translate to French.", "input": "Good morning.", "output": "Bonjour."}
{"instruction": "Write a Python function to reverse a list.", "input": "", "output": "def reverse(lst): ..."}

Parquet Advantages

Columnar storage: Read only the columns you need, skipping irrelevant data.
Built-in compression: Snappy or Zstd reduce disk usage 3–10x compared to JSON.
Predicate pushdown: Filter rows at the file level before reading data into memory.
Schema embedding: Type information is stored in the file metadata.
Ecosystem support: Pandas, Polars, DuckDB, PyArrow, HuggingFace, and Spark all read Parquet natively.

HuggingFace Datasets

HuggingFace datasets library wraps Arrow as its underlying storage format and provides a unified API for loading, streaming, transforming, and pushing datasets to the Hub. It supports lazy loading via memory-mapping, which allows datasets larger than RAM to be processed efficiently.

from datasets import load_dataset

# Load from the Hub
ds = load_dataset("HuggingFaceFW/fineweb-edu", split="train", streaming=True)

# Iterate in streaming mode (no full download needed)
for example in ds:
    print(example["text"][:100])
    break

# Convert JSONL to Parquet
ds_jsonl = load_dataset("json", data_files="data.jsonl")
ds_jsonl["train"].to_parquet("data.parquet")

10. Common Public Pretraining Datasets

The following corpora are the backbone of most open-source and many proprietary LLM pretraining runs. Each has distinct characteristics in terms of scale, composition, and licensing.

Dataset	Approx. Size (tokens)	Composition	Notes
Common Crawl	~100T+ (raw)	Web pages scraped from the internet since 2007	Raw, noisy; requires heavy filtering and dedup
Wikipedia	~4B (English)	Encyclopedia articles, high quality, multilingual	Gold-standard quality; small but dense
RedPajama	~1.2T (v2)	Reproduction of LLaMA training mix: CC, C4, GitHub, Books, Wiki, ArXiv, StackExchange	Open replication of LLaMA's data composition
FineWeb	~15T (English)	Filtered Common Crawl (2013–2024)	State-of-the-art web filtering pipeline by HuggingFace
FineWeb-Edu	~1.3T	FineWeb subset filtered for educational content	Quality classifier trained on LLM-rated educational value
OpenWebText	~8B	Open-source replication of WebText (Reddit outbound links, karma >= 3)	Used for GPT-2 reproduction; now largely superseded
C4 (Colossal Clean Crawled Corpus)	~750B	Filtered Common Crawl (April 2019 snapshot)	Used for T5; aggressive heuristic filtering
The Stack v2	~67B (code)	Permissively licensed source code from GitHub	Used for Code Llama, StarCoder2; per-file dedup
Books3 / Books1	~100B	Book-length text (novels, nonfiction)	Copyright concerns; Books3 was taken down
ArXiv	~100B	Scientific papers (math, physics, CS)	High-quality academic prose; LaTeX source

Typical Pretraining Mix

LLM pretraining corpora are rarely a single source. They are a carefully weighted mixture. A representative mix (inspired by LLaMA and RedPajama) might look like:

Source	Share	Rationale
Filtered web crawl	~67%	Breadth and diversity of topics and styles
Books / long-form	~4.5%	Coherent long-context reasoning, narrative
Wikipedia	~4.5%	Factual, encyclopedic knowledge
GitHub code	~4.5%	Programming ability, logical reasoning
ArXiv / scientific	~2.5%	Technical and mathematical content
StackExchange	~2%	Q&A format, practical reasoning
News / other	~15%	Current events, formal writing

Domain weighting matters: The proportion of code, math, and multilingual data in the pretraining mix strongly influences downstream capabilities. Even a small fraction of high-quality code data (5–10%) can dramatically improve a model's ability to reason structured and algorithmic problems.

11. Instruction Tuning Datasets

Pretraining on raw text produces a base model that can predict the next token but does not follow instructions. Instruction tuning (also called supervised fine-tuning, or SFT) teaches the model to respond to human prompts in a helpful, structured way. The datasets for this phase are smaller, higher-cost, and more carefully curated than pretraining corpora.

Dataset	Size	Source	Format	Notes
Alpaca (52K)	52,000 examples	Self-instruct with text-davinci-003	instruction / input / output	Stanford; open-sourced March 2023
Dolly 15K	15,000 examples	Human-written by Databricks employees	instruction / response / category	Truly human-generated; open license
OpenAssistant (OASST1)	161,000 messages	Crowdsourced conversation trees	Multi-turn conversation	LAION; 35+ languages; human-ranked
ShareGPT	~90,000 conversations	User-shared ChatGPT conversations	Multi-turn, user/assistant roles	Collected from sharegpt.com; license uncertain
FLAN Collection	~1.8M examples	1,800+ tasks reformatted as instructions	Mixed task formats	Google; largest open instruction-tuning set
OpenOrca	~4.2M examples	GPT-3.5/GPT-4 responses to FLAN prompts	instruction / response	Distilled from proprietary models
Lima	1,000 examples	Curated community Q&A and wikiHow	instruction / response	Shows 1K high-quality examples suffice
WizardLM Evol-Instruct	~250K examples	Evolutionary instruction generation	instruction / response	Progressively more complex instructions

The LIMA Lesson

The LIMA paper (Zhou et al., 2023) demonstrated that fine-tuning on just 1,000 carefully curated, high-quality instruction-response pairs can produce a model that rivals chatbots trained on millions of examples. This reinforced the central theme: quality dominates quantity in instruction tuning.

Best practice: For instruction tuning, invest in a small set (1K–50K) of extremely high-quality, diverse, human-reviewed examples. If more volume is needed, use synthetic data generated by a strong teacher model, but always filter the output for correctness, coherence, and diversity.

Preference / RLHF Datasets

Beyond SFT, modern alignment uses preference data for RLHF (Reinforcement Learning from Human Feedback) or DPO (Direct Preference Optimization). Each example contains a prompt and two responses, labeled by a human as “chosen” and “rejected.”

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I think it might be Lyon or Paris, not sure."
}

12. Chinchilla Scaling Laws: Data Size vs Model Size

The Chinchilla paper (Hoffmann et al., DeepMind, 2022) established that previous large models were “under-trained”—they had too many parameters for the amount of data they were trained on. The optimal allocation of compute balances model size (N, parameters) and data size (D, tokens) so that both scale proportionally.

The Key Finding

For compute-optimal training, the number of training tokens should scale approximately linearly with the number of parameters:

D_optimal ≈ 20 × N

That is, a 10-billion-parameter model should be trained on roughly 200 billion tokens, and a 70-billion-parameter model should be trained on roughly 1.4 trillion tokens. The earlier GPT-3 (175B parameters, 300B tokens) was significantly under-trained by this criterion; Chinchilla (70B parameters, 1.4T tokens) matched or exceeded GPT-3's performance with far fewer parameters but more data.

Model Size (params)	Optimal Tokens (≈20N)	Optimal Tokens (newer ≈30–40N)	Example Models
1B	20B	30–40B	TinyLlama, Phi-1
7B	140B	210–280B	LLaMA-1 7B (trained on 1T, over-trained)
13B	260B	390–520B	LLaMA-1 13B
70B	1.4T	2.1–2.8T	Chinchilla, LLaMA-2 70B
175B	3.5T	5.25–7T	GPT-3 (under-trained at 300B)
540B	10.8T	16–21T	PaLM (trained on 780B, under-trained)

Post-Chinchilla practice: Many open-source models (LLaMA-2, LLaMA-3) deliberately over-train beyond the compute-optimal point. This uses more tokens than Chinchilla prescribes, which costs more at training time but produces a stronger model for inference. Since inference cost dominates over the model's lifetime, over-training on more data is economically rational. LLaMA-3 8B was trained on 15T tokens, far exceeding the 160B compute-optimal estimate.

Implications for Dataset Planning

Do not build a model bigger than your data can support. A 70B model needs ~1.4T+ tokens of quality data. If you only have 200B tokens, a 10B model is more compute-efficient.
Plan data collection before architecture. The dataset size should drive the model size decision, not the reverse.
Epoch count matters. Training on the same data for multiple epochs has diminishing returns and increases overfitting risk. Chinchilla assumed single-epoch training; multi-epoch modifies the scaling law.
Quality modifies the constant. The 20N ratio assumes “average” data quality. With exceptionally high-quality data, fewer tokens may suffice (as Phi models suggest). With low-quality data, more tokens are needed to achieve the same loss.

13. Data Contamination and Benchmark Leakage

Data contamination occurs when a model's training data contains examples from an evaluation benchmark. This inflates benchmark scores and makes reported performance unreliable. As LLM pretraining corpora grow to tens of trillions of tokens scraped from the web, contamination has become a systemic concern.

Types of Contamination

Type	Description	Detection Difficulty
Exact match	Benchmark questions appear verbatim in training data	Easy (string matching)
Near-duplicate	Questions with minor paraphrasing or formatting changes	Medium (n-gram overlap)
Semantic overlap	Same concept or fact is present, rephrased substantially	Hard (requires semantic comparison)
Output leakage	Answer choices or gold answers appear in training data independently	Hard

Why It Happens

Benchmark datasets are often published on GitHub, arXiv, or blog posts—all of which are scraped into Common Crawl.
Benchmark questions are discussed on forums (Reddit, StackExchange) and those discussions are crawled.
Aggregated “evaluation” repositories end up in training data.
Model developers may inadvertently include benchmark source files in code training data.

Decontamination Procedures

n-gram overlap detection: For each benchmark question, compute a set of n-grams (typically 8- to 13-grams). Search the training corpus for any document containing those n-grams. Remove matches.
Fuzzy matching: Use MinHash, SimHash, or suffix-array-based approaches for near-duplicate detection.
Semantic decontamination: Embed benchmark questions with a sentence encoder and retrieve nearest neighbors from the training corpus using cosine similarity. Review and remove high-similarity hits.
Exclusion lists: Maintain a blocklist of URLs and domains known to host benchmark content (e.g., specific GitHub repos, HuggingFace dataset pages).
Temporal cutoff: Use only web crawl snapshots from before the benchmark's publication date.

# Simple n-gram decontamination (8-gram overlap)
from collections import defaultdict

def ngrams(text, n=8):
    tokens = text.lower().split()
    return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))

# Build index of benchmark n-grams
benchmark_ngrams = set()
for q in benchmark_questions:
    benchmark_ngrams |= ngrams(q)

# Scan training data and flag contaminated documents
for doc in training_corpus:
    doc_ngrams = ngrams(doc)
    overlap = doc_ngrams & benchmark_ngrams
    if overlap:
        print(f"Contaminated: {len(overlap)} overlapping n-grams")
        # Remove or quarantine this document

Industry concern: Many published benchmark scores for both open and closed models may be contaminated. When comparing models, look for: (1) decontamination procedures reported in the paper, (2) evaluation on held-out or newly created benchmarks, (3) performance consistency across multiple independent benchmarks. A model that excels on one benchmark but is mediocre on similar ones is a red flag for contamination.

14. CLI Examples: Inspecting, Splitting, Converting Datasets

The following command-line examples use common tools for dataset inspection, splitting, and format conversion. They assume a Linux environment with Python and standard ML libraries installed.

14.1 Inspecting a JSONL Dataset

# Count lines (examples) in a JSONL file
wc -l data.jsonl

# Preview the first 3 examples, pretty-printed
head -n 3 data.jsonl | python -m json.tool

# Check for malformed JSON lines
python -c "import json,sys; [json.loads(l) for i,l in enumerate(open(sys.argv[1])) if l.strip()] or print('All valid')" data.jsonl

# Extract a single field and compute basic stats
python -c "
import json
lengths = [len(json.loads(l)['text']) for l in open('data.jsonl') if l.strip()]
print(f'Count: {len(lengths)}')
print(f'Mean chars: {sum(lengths)/len(lengths):.0f}')
print(f'Min: {min(lengths)}, Max: {max(lengths)}')
"

14.2 Splitting Data with scikit-learn

# Split a JSONL file into 80/10/10 train/val/test
python -c "
import json, random
random.seed(42)

lines = [l for l in open('data.jsonl') if l.strip()]
random.shuffle(lines)

n = len(lines)
n_train = int(n * 0.8)
n_val   = int(n * 0.1)

train = lines[:n_train]
val   = lines[n_train:n_train+n_val]
test  = lines[n_train+n_val:]

for name, split in [('train', train), ('val', val), ('test', test)]:
    with open(f'{name}.jsonl', 'w') as f:
        f.writelines(split)
    print(f'{name}: {len(split)} examples')
"

14.3 Splitting with HuggingFace Datasets

# Load and split in one step
python -c "
from datasets import load_dataset

ds = load_dataset('json', data_files='data.jsonl')
split = ds['train'].train_test_split(test_size=0.2, seed=42)
test_val = split['test'].train_test_split(test_size=0.5, seed=42)

print(f\"Train: {len(split['train'])}\")
print(f\"Val:   {len(test_val['train'])}\")
print(f\"Test:  {len(test_val['test'])}\")
"

14.4 Converting JSONL to Parquet

# Convert JSONL to Parquet with PyArrow
python -c "
import pyarrow as pa
import pyarrow.parquet as pq
import json

records = [json.loads(l) for l in open('data.jsonl') if l.strip()]
table = pa.Table.from_pylist(records)
pq.write_table(table, 'data.parquet', compression='zstd')
print(f'Wrote {len(records)} records to data.parquet')
"

# Convert with Pandas
python -c "
import pandas as pd
df = pd.read_json('data.jsonl', lines=True)
df.to_parquet('data.parquet', engine='pyarrow', compression='snappy')
print(f'Wrote {len(df)} rows to data.parquet')
"

14.5 Converting CSV to JSONL

# Convert CSV to JSONL using csv module
python -c "
import csv, json
with open('data.csv') as fin, open('data.jsonl', 'w') as fout:
    reader = csv.DictReader(fin)
    for row in reader:
        fout.write(json.dumps(row) + '\n')
print('Conversion complete')
"

14.6 Deduplicating a Dataset

# Exact deduplication by text field
python -c "
import json

seen = set()
kept = 0
removed = 0

with open('data.jsonl') as fin, open('deduped.jsonl', 'w') as fout:
    for line in fin:
        if not line.strip():
            continue
        obj = json.loads(line)
        text = obj.get('text', '')
        h = hash(text)
        if h not in seen:
            seen.add(h)
            fout.write(line)
            kept += 1
        else:
            removed += 1

print(f'Kept: {kept}, Removed (duplicates): {removed}')
"

14.7 Streaming a Large Dataset from HuggingFace Hub

# Stream FineWeb-Edu without downloading the full dataset
python -c "
from datasets import load_dataset

ds = load_dataset('HuggingFaceFW/fineweb-edu', name='sample-10BT', split='train', streaming=True)

count = 0
for example in ds:
    if count < 3:
        print(f'--- Example {count+1} ---')
        print(example['text'][:200])
        print(f'Score: {example[\"score\"]}')
        print()
    count += 1

print(f'Streamed through {count} examples')
"

14.8 Tokenizing a Corpus for Pretraining

# Count tokens in a text file using tiktoken (GPT-4 tokenizer)
python -c "
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')

total = 0
with open('corpus.txt', 'r', encoding='utf-8', errors='ignore') as f:
    for line in f:
        total += len(enc.encode(line))

print(f'Total tokens: {total:,}')
print(f'Total tokens (billions): {total / 1e9:.2f}B')
"

14.9 Checking for Benchmark Contamination

# Check 10-gram overlap between training data and a benchmark
python -c "
import json

def ngrams(text, n=10):
    toks = text.lower().split()
    return set(tuple(toks[i:i+n]) for i in range(len(toks)-n+1))

# Load benchmark questions
bench_grams = set()
for q in json.load(open('benchmark.json'))['questions']:
    bench_grams |= ngrams(q)

# Scan training data
hits = 0
for line in open('train.jsonl'):
    if not line.strip():
        continue
    doc = json.loads(line).get('text', '')
    if ngrams(doc) & bench_grams:
        hits += 1

print(f'Contaminated documents found: {hits}')
"

15. Checklist for Dataset Quality

Use this checklist before committing a dataset to a training run. Each item prevents a common failure mode that can silently degrade model quality or invalidate evaluation results.

Composition & Coverage

☐ Dataset size is appropriate for model capacity (reference Chinchilla scaling laws)
☐ Domain coverage matches the intended downstream use case
☐ Language distribution is intentional and documented
☐ Temporal range is known and appropriate (no future leakage)
☐ Sources and licensing are documented for every subset

Cleaning & Deduplication

☐ HTML, boilerplate, and navigation text stripped
☐ Encoding issues (mojibake, broken Unicode) repaired
☐ Exact duplicates removed (document level)
☐ Near-duplicates removed (MinHash / SimHash, paragraph level)
☐ Line-level deduplication performed
☐ Very short or very long documents filtered
☐ Repetitive content (high n-gram repetition ratio) filtered
☐ PII (emails, phone numbers, API keys) removed or masked

Quality Filtering

☐ Language identification run and non-target languages filtered
☐ Perplexity filtering applied (e.g., using a reference LM)
☐ Quality classifier applied (educational score, spam filter, etc.)
☐ Profanity / toxicity filtering applied where appropriate
☐ Manual spot-check of 100+ random samples performed

Splits & Leakage Prevention

☐ Train / validation / test splits created with documented seed
☐ No duplicate or near-duplicate examples across splits
☐ Group-aware splitting used where applicable (patient, user, source)
☐ Temporal splitting used for time-series data
☐ Preprocessing (scaling, vocabulary) fit on train set only
☐ Features that proxy the target removed

Contamination & Evaluation Integrity

☐ n-gram decontamination run against all evaluation benchmarks
☐ Known benchmark-hosting URLs excluded from crawl
☐ Temporal cutoff applied if benchmarks postdate crawl date
☐ Held-out evaluation set reserved and never used for tuning

Format & Reproducibility

☐ Storage format chosen (JSONL / Parquet / Arrow) and documented
☐ Schema documented (field names, types, descriptions)
☐ Tokenizer version recorded and frozen
☐ Data processing pipeline is versioned (git / DVC)
☐ Random seeds documented for all stochastic steps
☐ Dataset card / README created with statistics and provenance

Final Validation

☐ Total token count verified against Chinchilla targets
☐ Token distribution per domain verified (no unexpected skew)
☐ A small model (1M–10M params) trained as a smoke test
☐ Validation loss curve looks reasonable (no NaNs, no collapse)
☐ Dataset can be reloaded deterministically from the pipeline

Remember: A dataset is never “done.” As models improve and use cases evolve, you will discover new failure modes, new sources of bias, and new quality dimensions to filter on. Treat the dataset as a living artifact with version history, continuous auditing, and iterative improvement.

Summary

Datasets are the most consequential component of any machine learning system. The best models are built on data that is clean, diverse, well-split, properly preprocessed, and rigorously decontaminated. The key takeaways from this reference:

Quality over quantity. Curated, filtered, deduplicated data consistently outperforms raw dumps at every scale.
Splits are sacred. Train, validation, and test sets must be disjoint, with no leakage across partitions.
Preprocessing is a first-class pipeline. It must be fit on training data only and applied identically at inference time.
Chinchilla scaling laws guide the data-model balance. Match your dataset size to your model size; do not starve or waste.
Contamination undermines all evaluation. Decontaminate against benchmarks and report the procedure.
Instruction tuning rewards quality. A few thousand excellent examples can outperform millions of mediocre ones.
Format matters for scale. Parquet and Arrow enable efficient processing of terabyte-scale corpora.

Mastering dataset engineering is the highest-leverage skill in the modern AI stack. The model architecture can be copied from a paper; the dataset cannot.

Training, Validation, and Inference Datasets