Model Architectures
An architecture specifies a parameterized computation and its inductive biases: which interactions are cheap, which information paths exist, which symmetries are encoded, and how cost scales with input size.
Mental model
CNNs reuse local filters across space. RNNs compress history into recurrent state. Attention creates content-dependent interactions. Transformers compose attention and per-token transformations with residual pathways. State-space models propagate structured latent state. Diffusion models learn iterative denoising. No architecture is universally best; each moves cost and bias.
Current foundation note
Candidate note roadmap
convolutions-cnns-and-spatial-bias— kernels, receptive fields, equivariance, and hierarchy.rnns-lstms-and-sequence-state— recurrence, gating, gradient paths, and serial cost.transformer-block-from-first-principles— attention, MLP, residual streams, normalization, and shapes.encoder-decoder-and-cross-attention— representation, conditioning, and sequence transduction.mixture-of-experts— sparse routing, load balancing, capacity, and distributed cost.state-space-models— recurrence, convolution views, selective state, and tradeoffs.autoregressive-and-energy-based-perspectives— factorization, scoring, normalization, and sampling.diffusion-architecture-and-denoising-objectives— U-Nets/DiTs, schedules, conditioning, and sampling.foundation-models-as-platforms— pretraining breadth, adaptation surfaces, and operational implications.
Comparison rule
Compare architectures by information path, inductive bias, asymptotic and measured cost, memory traffic, trainability, data regime, and the task's evaluation—not by release recency.
Connects to: Deep Learning · The Decoder Transformer · Vision, Audio and Multimodal AI
Core sources
- Deep Learning — canonical foundations for neural architectures and optimization.
- Dive into Deep Learning — executable architecture implementations with shape-level explanations.
- Attention Is All You Need — original Transformer architecture.
- An Image is Worth 16x16 Words — Vision Transformer and its data/scale result.