Context window management

Deep dive

How it works

In: message list, window size, reserve, trigger and keep settings · Out: a message list that fits the budget · State: the message history, including any earlier summary · Cost: O(1) per check with a running token sum; one summarization call per compaction

1. Count tokens with the provider's counter (chars/4 estimates are worse on code and non-English text; see BPE). Budget = window − reserved output − system prompt/tool definitions.
2. Cap on arrival: a tool result over a per-result limit is cut to head + tail with an omission marker. Cheapest rung, no cache effect since the message is new.
3. Clear when used tokens pass a trigger (e.g. 80 % of budget): replace all but the newest N tool results with a short placeholder. Calls and assistant text stay, so the model still sees what it did and can re-run a tool.
4. Compact if still over: pick a cut at a user-turn boundary leaving about keep_tail tokens of recent turns verbatim, ask a model to summarize everything before it (goal, decisions, files touched, open items, errors), and replace that prefix with one message.
5. Persist what must survive outside the window (notes file, memory) before it is cleared.

[system][tools] | u c r a | u c r a | u c r a | u c r a   ← u=user c=tool call r=result a=assistant
after clear:      u c ∅ a | u c ∅ a | u c r a | u c r a   ← ∅ = placeholder
after compact:    [summary] | u c r a | u c r a           ← cut only before a 'u'

Code

#[derive(Clone, Copy, PartialEq, Debug)]
pub enum Kind { User, Assistant, ToolCall, ToolResult }

#[derive(Clone, Debug)]
pub struct Msg { pub kind: Kind, pub text: String }

/// Rough estimate: ~4 bytes per token for English/code. Use the provider's counter for real budgets.
pub fn tokens(s: &str) -> usize { s.len() / 4 + 4 }

/// Keep head and tail of an oversized tool output; errors and summaries sit at the ends.
pub fn truncate_middle(s: &str, max_tokens: usize) -> String {
    let max = max_tokens * 4;
    if s.len() <= max { return s.to_string(); }
    let cut = |i: usize, up: bool| { // snap to a char boundary
        let mut i = i;
        while !s.is_char_boundary(i) { if up { i += 1 } else { i -= 1 } }
        i
    };
    let (h, t) = (cut(max * 2 / 3, false), cut(s.len() - max / 3, true));
    format!("{}\n[... {} bytes omitted ...]\n{}", &s[..h], t - h, &s[t..])
}

pub struct Context {
    pub msgs: Vec<Msg>,
    pub window: usize,       // model limit in tokens
    pub reserve: usize,      // room for the reply
    pub keep_results: usize, // newest tool results left intact
    pub clear_at_least: usize, // skip clearing if it would free less (cache write is not free)
    pub keep_tail: usize,    // tokens of recent turns kept verbatim by compaction
}

const CLEARED: &str = "[tool result cleared to save context; call the tool again if needed]";

impl Context {
    pub fn used(&self, system_tokens: usize) -> usize {
        system_tokens + self.msgs.iter().map(|m| tokens(&m.text)).sum::<usize>()
    }
    fn limit(&self) -> usize { self.window - self.reserve }

    /// Step 1 (cheap, lossy, no model call): blank all but the newest `keep_results` tool results.
    pub fn clear_old_results(&mut self) -> usize {
        let idx: Vec<usize> = (0..self.msgs.len()).filter(|&i| self.msgs[i].kind == Kind::ToolResult).collect();
        let old = &idx[..idx.len().saturating_sub(self.keep_results)];
        let freed: usize = old.iter().map(|&i| tokens(&self.msgs[i].text).saturating_sub(tokens(CLEARED))).sum();
        if freed < self.clear_at_least { return 0 }
        for &i in old { self.msgs[i].text = CLEARED.to_string(); }
        freed
    }

    /// Step 2 (expensive, lossy): replace everything before a turn boundary with a summary.
    /// The cut always lands on a User message, so a ToolCall is never separated from its ToolResult.
    pub fn compact(&mut self, summarize: impl FnOnce(&[Msg]) -> String) -> bool {
        let mut acc = 0;
        let mut cut = self.msgs.len();
        while cut > 0 && acc < self.keep_tail { cut -= 1; acc += tokens(&self.msgs[cut].text); }
        while cut > 0 && self.msgs[cut].kind != Kind::User { cut -= 1; }
        if cut == 0 { return false } // nothing older than the kept tail
        let summary = summarize(&self.msgs[..cut]);
        let head = Msg { kind: Kind::User, text: format!("[summary of earlier conversation]\n{summary}") };
        self.msgs.splice(..cut, [head]);
        true
    }

    /// Run before each model call. Ladder: truncate on insert (not here), clear, then compact.
    pub fn fit(&mut self, system_tokens: usize, summarize: impl FnOnce(&[Msg]) -> String) {
        if self.used(system_tokens) <= self.limit() * 8 / 10 { return } // trigger at 80 %: headroom for next turn
        self.clear_old_results();
        if self.used(system_tokens) > self.limit() * 8 / 10 { self.compact(summarize); }
    }
}

The summarizer is a closure so it can be a model call, a cheaper model, or a test stub. A pinned system prompt and tool definitions live outside msgs and are never edited.

Complexity and constants

• Counting: O(messages) per check; keep a per-message token count and a running sum to make it O(1).
• Chars/4 is a rough English heuristic only; code, JSON and non-Latin scripts tokenize worse. Count with the provider's API for decisions near the limit.
• Cost drivers: input tokens per call (every call re-reads the history), cache writes after each edit, and one summarization call that reads the whole prefix and writes a few thousand output tokens.
• Compaction frequency: with a window W, trigger t·W and a post-compaction size s·W, one compaction happens about every (t−s)·W new tokens; a small s compacts less often but loses more.

Efficiency

• Cost: input tokens per call dominate: n turns adding k tokens each process k·n(n+1)/2 input tokens unmanaged. Cached reads cost 2.5-10 % of base input price depending on model, cache writes 25 % more (5-minute TTL) (Anthropic docs). Bound: prefill cost and latency; quality also degrades as stale context accumulates (Anthropic).
• Levers:
  • cap tool results on arrival — a 50 KB log is never re-read on later turns — large outputs
  • clear in batches (clear_at_least) — one cache rewrite instead of one per turn — prompt-cached agents
  • stable system/tools first, append-only history — cache reads instead of full-price input — every call
  • rare compaction, smaller model for the summary — one long read, then cheaper turns — long runs
  • send less: paged reads, grep before read (code retrieval), narrow tool sets (tool retrieval), subagent digests — no tokens at all — noisy subtasks
• Naive → standard → state of the art:
  • resend everything · O(turns^2) input tokens
  • cap, clear old results, compact at a budget trigger · bounded by the window per call
  • the same ladder with cache-aware batched edits and server-side clearing/compaction · bounded, with cache rewrites rare

Implementation notes

• Never split a tool call from its result: most APIs reject a call without a matching result. Cut at user turns, or move the cut to include both.
• Keep the placeholder one identical short string: it costs few tokens and never varies.
• Anthropic's API offers this server-side (clear_tool_uses_20250919, beta header context-management-2025-06-27; options trigger, keep, clear_at_least, exclude_tools, clear_tool_inputs) and server-side compaction; your client history stays unmodified.
• Summary prompt: ask for state, not narrative: task and constraints, decisions with reasons, files and identifiers, failed approaches, next step. Verbatim identifiers (paths, IDs, error strings) are what summaries lose.
• Keep the user's original request and any standing instructions verbatim; do not rely on the summary to carry them.
• Trigger below the hard limit (about 80 %): the next turn's tool result and reply must still fit.
• Thinking blocks are history too; the provider may require them intact when tool calls continue, so follow its rules before dropping them.

Pitfalls

• Summary of a summary of a summary drifts: facts about early decisions decay. Keep the first user message and key facts outside the summary.
• Clearing a result the model still depends on makes it loop re-running the tool; exclude tools whose output is reference material.
• Clearing every turn defeats prompt caching and costs more than it saves.
• Estimating tokens by characters and overflowing on code/CJK: leave headroom or use an exact counter.
• Cutting mid-turn leaves orphan tool results; the API rejects the request.
• Summarizing with the same long context at the limit: the summarization call itself must fit (summarize in pieces or compact earlier).

Variants and alternatives

In production

Claude Code compacts conversations automatically near the limit; Anthropic's API provides tool-result clearing and server-side compaction; aider summarizes older chat history when it exceeds a token budget; Letta (MemGPT) evicts and recursively summarizes messages while keeping memory blocks pinned. The research prototype MeMento applies the same idea to vision-language embodied agents.

Implementations

Sources

• Anthropic, Effective context engineering for AI agents, 2025. post
• Anthropic, Context editing and Compaction, Claude Platform docs. context editing · compaction
• Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024. paper · code
• Charles Packer et al., MemGPT: Towards LLMs as Operating Systems, 2023. paper · code
• Long-Horizon Embodied Decision-Making via Multimodal Memory Compression (MeMento, DunphyBench), 2026. paper