Context window & the KV cache

The context window is everything the model can "see" at once — system prompt, history, retrieved docs, the user's message, and the tokens it has generated so far. It is the model's entire working memory; nothing outside it exists to the model.

The window is a hard budget

Measured in tokens, the window is finite (a few K to ~1M depending on model). Everything competes for it: instructions, few-shot examples, retrieved context, and the running conversation. When it fills, something must be dropped or summarized — which is the whole job of context engineering. The model has no memory across calls; persistence is something you engineer by putting the right things back in the window.

The KV cache: why generation is fast

Generation is autoregressive — one token at a time, each conditioned on all previous ones. Naively, every new token would re-process the entire sequence. The KV cache stores the attention Keys and Values already computed for prior tokens, so each new step only computes attention for the new token against the cached past. Without it, long generations would be unbearably slow.

The cost: the cache grows with sequence length and consumes GPU memory — often the real limit on how long a context you can serve and how many requests fit on a GPU. This is why long contexts are expensive in both latency and money.

Practical implications

• Prompt caching — providers can cache the KV for a stable prefix (a long system prompt or document), so repeated calls skip recomputing it: cheaper and faster. Put the stable stuff first.
• More context ≠ better — beyond cost, quality degrades (lost in the middle). Curate, don't dump.
• The quadratic cost of attention is the reason all of this matters.

Connects to: token budget · long context limits · context engineering