Context windows are a budget, not a feature

Production agents fail on context bloat long before they hit the provider's hard token limit. The symptom looks like confusion: the agent repeats a question answered ten turns ago, ignores a constraint buried in a 40-page paste, or hallucinates a file path that existed only in a truncated tool output. The fix is rarely a larger model window. The fix is a context window budget — a per-turn allocation plan that decides what enters inference, what stays in external memory, and what gets evicted before the next step. Context is not unlimited memory. It is a scarce resource with latency, cost, and attention trade-offs attached to every token.

Frontier models in 2026 advertise context windows from 200K to over 1M tokens. That capacity is real. It is also a trap. Filling the window does not make the agent smarter — it makes inference slower, more expensive, and more vulnerable to the lost-in-the-middle effect, where critical facts in the center of a long prompt get deprioritized by attention. The practitioners shipping reliable agents treat the advertised limit as a ceiling, not a target. Most production turns aim for 60–75% utilization, reserving headroom for tool replies and model output that arrive after the initial call.

Context window budget failures show up as agent amnesia

Context failures are silent. Unlike a rate-limit error or a tool timeout, a bloated context window produces plausible wrong answers. The agent continues running. Logs look clean. Cost rises gradually. Only when a human replays the session does the pattern emerge: the agent had the information somewhere in the transcript, but that information lost priority against newer, noisier content.

Three failure modes dominate production incidents traced back to context management.

Dump-everything prompts. Teams paste entire repositories, full ticket backlogs, or multi-megabyte JSON exports into the first turn because the window "fits it." The model receives the data. It does not reliably attend to all of it. Critical constraints at turn one compete with 200 tool results by turn fifteen.

Unbounded tool output. Each agent loop re-sends prior tool inputs and outputs. A single grep across a monorepo or a database query returning ten thousand rows can consume more tokens than the entire conversation history. Tool traces grow faster than chat messages because they accumulate verbatim.

Blind truncation. When context approaches the limit, naive sliding windows drop the oldest messages — often the system instructions, the user's original goal, or the constraint list established at session start. Summarization without structure preserves narrative but drops entity identifiers: ticket numbers, file paths, API error codes, dollar amounts.

A bigger context window invites slower, more expensive, less focused agents.

The cost is not only inference spend. Latency compounds with input size. A turn that processes 400K tokens may add seconds of prefill time before the first output token — unacceptable for interactive workflows. Context window budget design is how teams cap that growth before it becomes an architecture problem.

A context window budget allocates capacity across four layers

A context window budget is not the provider's maximum. It is a product decision: how many tokens this turn may spend, and on which layers. Treat allocation like a capacity plan, not an afterthought.

These ranges shift by workflow. A coding agent with forty MCP tools needs a larger tool layer and smaller retrieval. A support agent with a narrow toolset spends more on retrieved knowledge base articles. The point is explicit reservation — not accidental filling.

Define two thresholds per session. A soft threshold at ~70% of the budget triggers compaction: summarize older turns into a structured state block, compress tool traces into a bullet ledger, evict stale RAG chunks. A hard threshold at ~90% halts new retrieval until compaction completes. Waiting for a 400 error from the API means the session already lost continuity.

context_budget:
  max_input_tokens: 120000        # product cap, not provider max
  soft_threshold: 0.70
  hard_threshold: 0.90
  layers:
    system:
      pinned: true
      max_tokens: 18000
    tools:
      max_tokens: 24000
      cache_stable: true
    working_memory:
      max_tokens: 42000
      verbatim_tail_turns: 6
    retrieval:
      max_tokens: 36000
      strategy: just_in_time

Instrument each layer. If tool output consistently exceeds its allocation, the problem is tool design — not model choice. The MCP servers treated as production infrastructure article covers why verbose tool responses are an access-control and API-design problem, not merely a prompt problem.

Stable content belongs in cache; volatile content belongs outside

Not everything worth knowing belongs in the active context window. Production agents separate hot state (needed for this turn's decision) from cold state (durable but retrievable on demand).

Hot state stays in-context: the current user message, the last six verbatim turns, active constraints, the open subtask, and tool results from the current reasoning chain. Cold state lives outside: full conversation archives, vector stores, file systems, databases, and the structured notes an agent writes to itself between turns.

Prompt caching amortizes stable hot content. System instructions, safety policies, and tool schemas that do not change between turns should be marked cacheable where the provider supports it — cached input tokens typically bill at a fraction of fresh input cost. The win requires discipline: mutating the system prompt every turn defeats caching entirely.

Structured note-taking extends cold state without bloating hot state. The agent writes a compact progress artifact — open todos, pinned file paths, failed strategies to avoid, decisions already made — to external storage. The next turn injects only that artifact plus the recent tail. This pattern mirrors how specs function as execution contracts: the durable artifact is small, versioned, and authoritative; the generated chatter around it is disposable.

Compaction must preserve entities, not vibes. A summary that says "the user had billing questions" is useless. A summary that says "invoice #8842, status overdue, refund capped at $240 per policy REF-12, prior attempt failed on gateway timeout" survives the next ten turns. Validate compaction output with spot checks: regex for IDs, amounts, dates, and file paths before accepting the summary into working memory.

Tool output dominates context growth in agent loops

ReAct-style agents re-send the full trace on every iteration. Each tool call adds input tokens (the call) and output tokens (the response). Ten iterations with verbose outputs can exceed the entire system prompt, tool schemas, and user history combined.

Compress tool output at the boundary before injection. A database query returning 10,000 rows becomes a count, a sample of three rows, and column names. A log search returning 500 lines becomes the five lines surrounding the error signature plus the query used. A file read returns the relevant function, not the entire module — unless the task explicitly requires whole-file context.

Assign eviction priority when over budget:

• User message and pinned system policy: priority 100 — never evict
• Active subtask state and open constraints: priority 90
• Failed tool results (errors inform retry strategy): priority 80
• Successful tool results from the current chain: priority 60
• Successful tool results from completed subtasks: priority 40 — compress to ledger
• Stale RAG chunks from prior subtasks: priority 20 — evict first

Reset working context at subgoal boundaries. When an agent finishes "diagnose payment latency" and begins "draft remediation PR," the diagnostic trace compresses to a five-line ledger. State persists in external memory; the next subgoal starts with a clean working layer. The runbooks that govern production agents encode these boundaries explicitly — each runbook step defines what evidence enters context and what gets archived.

How should teams design a context window budget?

Context window budget design is a first-class engineering task for any agent shipping beyond a demo. The questions below cover the decisions that separate reliable production agents from expensive chatbots.

What belongs inside the context window versus external memory?

Inside the window: anything required to choose the next action — current user intent, active constraints, recent verbatim turns, tool results for the open reasoning chain, and retrieval chunks directly relevant to this step. Outside the window: full histories, large documents, raw logs, complete datasets, and anything retrievable by ID or query. The test is retrieval cost versus attention cost. If fetching a 2K-token excerpt on demand beats keeping 50K tokens loaded "just in case," keep it external.

When should an agent trigger context compaction?

Trigger compaction at the soft budget threshold (~70%), not at the provider hard limit. Additional triggers: every K turns in long sessions, on subgoal completion, when tool trace tokens exceed working-memory allocation, and before injecting a large retrieval payload. Compaction should produce structured state — todos, pinned entities, failed strategies — not narrative prose. Run compaction with a smaller model if latency matters; validate that entity identifiers survived.

Does a larger context window remove the need for budgeting?

No. Larger windows increase cost per turn, prefill latency, and lost-in-the-middle risk. Models with 1M-token capacity still perform best when the prompt contains high-signal tokens. Budgeting is how teams enforce signal density. The window size sets the ceiling; the budget sets the target.

A common argument runs the other way

The opposing view holds that context window limits are a temporary infrastructure constraint — that models with 2M or 10M tokens will eventually make budgeting obsolete, and teams should simply load everything and let attention mechanisms sort it out.

That argument works for batch analysis jobs with a single pass over static documents. It fails for interactive agents running dozens of turns with accumulating tool traces, shifting subgoals, and real-time cost constraints. Research on long-context retrieval consistently shows degraded recall for information placed in the middle of very long prompts — window size does not eliminate attention economics.

Larger windows also raise the stakes of silent failure. An agent that "forgets" a constraint at 128K tokens is obviously broken. An agent that loses the same constraint inside 800K tokens of noise looks like it never had the information. Budgeting forces explicit decisions about what matters now — which is the actual job of context engineering.

Key takeaways

• A context window budget is a per-turn allocation plan — not the provider's advertised maximum.
• Target 60–75% utilization; reserve headroom for tool replies and model output on every turn.
• Allocate explicitly across system, tools, working memory, and retrieval layers with eviction rules.
• Tool output accumulates faster than chat history — compress at the boundary before injection.
• Compaction at 70% preserves entities and constraints; blind truncation destroys session continuity.
• Stable content (system prompts, tool schemas) belongs in cache; volatile state belongs in external memory.
• Larger context windows increase cost and latency — they do not remove the need for budgeting.

Conclusion

The shift from prompt engineering to context engineering is really a shift from wording to allocation. The question is no longer "what should the system prompt say?" but "what earns a token on this turn, and what stays retrievable until needed?" Teams that answer that question with a written budget — layer caps, compaction triggers, eviction priorities, instrumentation — ship agents that stay coherent across long sessions without linear cost growth. Teams that treat the context window as a feature checkbox ship demos that work for five turns and degrade silently after twenty.

The next design decision is not which model has the biggest window. It is which artifacts are small enough to pin, which tool outputs are summarized by default, and which state survives a compaction pass. Get those decisions explicit before the agent touches production — because by the time context failure shows up in metrics, the session has already lost the thread.