Inference & Optimization
You train a model once but serve it forever, so inference is where most of the cost and latency of an AI product live. This branch is the systems side of running models: how to make them fast, cheap, and scalable without retraining.
Mental model
Inference is a scheduled flow of tensor kernels and memory movement under a latency objective. Prompt processing and token-by-token decoding have different bottlenecks; batching, cache policy, precision, model shape, and hardware determine the feasible quality-cost envelope.
Roadmap: measure the workload
- Why inference is the real cost explains train-once, serve-forever economics.
- Latency vs throughput separates TTFT, prefill, decode, tokens/sec, and p95.
- Cost modeling for LLM serving turns model, traffic, tokens, hardware, and cache behavior into unit economics.
Make serving faster
- The KV cache and memory covers why memory capacity and bandwidth dominate LLM serving.
- Batching for LLM serving compares static, dynamic, and continuous batching.
- Speculative decoding uses a small draft model to accelerate larger-model decoding.
- FlashAttention and efficient attention reduces memory movement in attention kernels.
Make serving cheaper
- Quantization for inference covers int8, 4-bit, GPTQ, AWQ, and what can degrade.
- Prefix and semantic caching avoids repeated work at the prompt and product layers.
- Right-sizing models compares smaller models, distillation, routing, and task decomposition.
Run it in production
- Serving engines maps vLLM, TGI, TensorRT-LLM, and framework-level tradeoffs.
- GPU and hardware basics explains VRAM, bandwidth, compute, interconnects, and utilization.
Connects to: From Prompt to Generated Token · MLOps · Latency, Cost, and Quality
Core sources
- vLLM paper — PagedAttention and serving-throughput evaluation.
- FlashAttention — IO-aware exact attention.
- Fast Inference from Transformers via Speculative Decoding — exact speculative decoding and speed analysis.
- GPTQ — post-training quantization for generative transformers.
- Hugging Face KV cache strategies — current cache implementations and trade-offs.