Training dynamics: schedules, warmup & debugging
Two networks with identical architecture can succeed or fail purely on how they were trained. These are the practical levers — and the skill of reading a loss curve.
Learning-rate schedules
A fixed learning rate is rarely best. Standard recipe:
- Warmup — start tiny and ramp up over the first few hundred/thousand steps. Early on, weights are random and a big step can blow up; warmup avoids the early divergence that plagues transformers.
- Decay — gradually lower the rate (cosine or linear) so the model takes big steps early and fine, careful steps later as it nears a good minimum.
The other key knobs
- Gradient clipping — cap the gradient norm so a rare huge gradient can't throw the weights off a cliff. Near-mandatory for RNNs and transformers.
- Batch size — larger batches give smoother gradients and faster hardware throughput but use more memory and can generalize slightly worse; often scaled together with the learning rate.
- Mixed precision (fp16/bf16) — train in lower precision for speed and memory, keeping a few things in fp32 for stability. Standard for large models.
Reading the loss curve
| Symptom | Likely cause |
|---|---|
| Loss → NaN / explodes | LR too high, no warmup, no grad clipping, unscaled inputs |
| Loss flat from step 0 | LR too low, bad init, broken data pipeline |
| Train ↓ but val ↑ | overfitting → regularize / more data |
| Loss spikes then recovers | usually fine; persistent spikes → lower LR or clip harder |
Debugging order of operations
- Overfit a single batch to ~zero loss — proves the model + loss + backprop wiring works.
- Then scale up data and tune the LR (the highest-leverage hyperparameter).
- Only then touch architecture.
If a model won't learn, suspect the data pipeline and learning rate long before the architecture.
Connects to: optimizers · init & norm · learning curves