LLMs
A large language model is a transformer trained on one deceptively simple objective — predict the next token — at enormous scale. Everything people find magical or maddening about LLMs (in-context learning, hallucination, sensitivity to phrasing) falls out of that objective and the architecture around it. This branch builds the mental model from the inside out.
An LLM is a next-token predictor. It is astonishingly capable and it has no built-in notion of truth — both facts come from the same training objective.
Mental model
A language model factorizes sequence probability into repeated next-token predictions. A decoder transformer converts the current prefix into logits; decoding commits one token; the new prefix becomes the next input. Training, post-training, context construction, and serving each change a different part of that system.
Roadmap: architecture to behavior
- From prompt to generated token
- The decoder transformer
- Transformer attention map
- Tokenization: why models see tokens, not words
- Positional encodings & RoPE
Training & adaptation
- Pretraining: next-token prediction
- Base vs instruct vs chat models
- Emergent abilities & in-context learning
Generation & context
- Decoding & sampling
- Context window & the KV cache
- Long context & lost in the middle
- Reasoning & test-time compute
Behavior & deployment
Connects to: Model Architectures · Training and Adaptation · Inference Systems
Core sources
- Attention Is All You Need — the original transformer architecture.
- Language Models are Few-Shot Learners — autoregressive scaling and in-context learning evidence.
- Speech and Language Processing — current reference for tokenization, language modeling, and transformers.
- Hugging Face LLM Course — executable tokenizer, transformer, fine-tuning, and inference material.