Cross-validation done right
A single train/validation split gives a noisy estimate — get unlucky and you draw the wrong conclusion. Cross-validation (CV) rotates the validation role across the data for a more stable estimate, and it uses scarce data efficiently.
k-fold CV
Split the data into k folds. Train on k−1, validate on the held-out fold,
rotate so every fold is validated once, then average. You get k scores — their
mean estimates performance and their spread estimates how reliable that
number is. k = 5 or 10 are standard.
Pick the variant that matches your data
| Variant | Use when |
|---|---|
| Stratified k-fold | classification — preserves class balance in each fold (essential for imbalanced data) |
| Grouped k-fold | repeated entities (user, patient) — keep a group entirely in one fold to avoid group leakage |
| Time-series split | temporal data — always train on the past, validate on the future; never shuffle |
| Leave-one-out | very small datasets (expensive, high variance) |
Nested CV: the part people skip
If you tune hyperparameters using your CV score and then report that same score, it's optimistic — you've fit to the validation folds. Nested CV uses an inner loop to tune and an outer loop to estimate honestly. At minimum, keep a final test set you never touch during tuning.
Pitfall
All preprocessing must happen inside the CV loop (via a pipeline). Fit a scaler or do resampling before splitting and every fold is contaminated — the most common way CV lies to you.
Connects to: splits & leakage · tuning · pipelines