Deep Learning
Deep learning is what happens when you stack many simple, differentiable layers and let gradient descent discover the features instead of hand-engineering them. That single shift — learned representations over crafted features — is why it took over vision, speech, and language, and it's the substrate every LLM is built on.
A neural network is a stack of linear models separated by nonlinearities, trained end to end by backprop. Everything else is making that train stably at scale.
Mental model
Deep learning composes differentiable transformations so useful representations can be learned with the task. Architecture determines information paths; the objective supplies pressure; backpropagation assigns local credit; optimization and numerical systems make the process viable at scale.
Roadmap: how a network learns
- Neural networks & backpropagation
- Activation functions & why nonlinearity matters
- Loss functions in deep learning
Making training work
- Initialization & normalization
- Optimizers: from SGD to AdamW
- Regularization: dropout, weight decay & augmentation
- Training dynamics: schedules, warmup & debugging
Architectures
Representations & scale
Paradigms & strategy
- Reinforcement learning, the essentials covers reward, policy, and the paradigm behind RLHF and reasoning models.
- The bitter lesson explains why general, compute-hungry methods keep beating hand-crafted structure.
Connects to: Computation and Autodiff · Model Architectures · Language and Foundation Models
Core sources
- Deep Learning — canonical reference for feed-forward networks, optimization, and regularization.
- Dive into Deep Learning — executable implementations with equations and tensor shapes.
- Neural Networks: Zero to Hero — mechanism-first implementations from scalar autodiff to GPT.
- Stanford CS231n — convolutional networks, optimization, and practical training diagnostics.