Managing the context window
Every token in the context window competes for space, cost, and the model's attention. Context management is the discipline of deciding what earns its place — and it's most of the work in RAG and agent systems.
The budget and its claimants
System prompt · instructions · few-shot examples · retrieved chunks · tool outputs · conversation history · the user's message · room to generate. They all draw from the same finite pool. When it fills, quality drops or requests fail — so you must actively curate.
Levers
- Select, don't dump — retrieve and include only what's relevant; more context often means worse answers (lost in the middle), not just costlier ones.
- Compress — summarize old turns and verbose material; replace raw logs with distilled facts.
- Trim history — keep recent turns verbatim, summarize or drop older ones (memory).
- Rank and place — put the most important content at the start/end, not the middle.
- Offload — keep large/stable knowledge in retrieval or tools, not stuffed into every prompt.
Why "just use a bigger window" isn't the answer
Bigger windows help but don't dissolve the problem: attention still favors the edges, cost and latency rise with length, and irrelevant context actively distracts the model. Capacity is not curation.
Treat context like a tight budget you're spending on the model's behalf. The goal is the smallest context that makes the task succeed.
Connects to: context window & cost · assembling context · retrieval