CodeSmith Context Engineering: What to Do After the Context Window Fills Up

Context Engineering: After the Window Fills Up

Source version of CodeSmith: v0.5.0 (commit 3a74c82f). All paths are relative to the repo root; line numbers refer to this version.
Intended audience: readers who have watched a context window fill to bursting and want to know exactly what compaction touches.

Article 8 was about "don't put it in to begin with": big outputs become handles, paged in on demand. But every line of defense has a limit — once a session runs a few hundred rounds, the handle mechanism cannot stop the history itself from growing. What to do after the window fills up is the last question context engineering cannot dodge.

And this question carries an awkwardness none of the three previous installments had: compaction must rewrite history. Article 9 just declared that "every byte written into history stays unchanged for life"; this installment is about the one exception — and about how many firewalls the engine built around that exception to press the cost of breaking the law to a minimum.

The Textbook splits "why compact" into three motives, and length-and-cost is only the first. The second is subtler: knowledge in summarized form is easier to use than in its raw form — attention is good at "lookup" and bad at "aggregation".

A concrete example: the context holds a hundred records — "cage 1 black cat, cage 2 white cat…" — and when asked "how many black cats and how many white cats", the model has to walk the list and count on the spot every time it answers. Write the conclusion into the context first — "current tally: black cats 90, white cats 10" — and the answer turns from a reasoning problem into a retrieval problem.

The third motive is easing the model's "context anxiety": when the window is nearly full, the model may rush to a sloppy finish before the task is done. Compact while the window is not yet full, and decision quality is actually higher.

Pair this with a conceptual distinction: context overflow means it no longer fits; context rot means it fits but can no longer be found. The latter is stealthier — the agent appears to keep working normally while decision quality quietly slides. The DeepSeek figures cited later in the cycle section (retrieval precision 0.90 at 8K, only 0.59 left at 1M) measure precisely the slope of that rot.

The Trigger: 967,000 Is Not a Number Pulled From Thin Air

Automatic triggering has one formula (crates/agent/src/models.rs:432-444): take the model's window, subtract a 20K summary reserve, then subtract a 13K safety buffer — the buffer being headroom for turn metadata, tool schemas, and estimator drift. Small windows (<200K) get multiplied by 2/3 on top of that, because the old chars/3 estimator's 3/2 amplification correction is still in force (Article 8 covered it), and a small window cannot absorb that error.

pub fn compaction_threshold_for_model(model: &str) -> usize {
    let Some(window) = context_window_for_model(model) else {
        return DEFAULT_COMPACTION_TOKEN_THRESHOLD;
    };
    let effective_window = auto_compact_effective_window(window, model);
    let base = usize::try_from(effective_window.saturating_sub(AUTO_COMPACT_BUFFER_TOKENS))
        .unwrap_or(DEFAULT_COMPACTION_TOKEN_THRESHOLD);
    if window < SMALL_CONTEXT_WINDOW_TOKENS {
        base * CONSERVATIVE_ESTIMATE_DENOMINATOR / CONSERVATIVE_ESTIMATE_NUMERATOR
    } else {
        base
    }
}

This code is doing exactly one thing: window minus reserve and buffer yields the threshold, and small windows take a further haircut at the conservative estimation ratio. Fit real numbers on it — DeepSeek V4 Pro (1M window): 1,000,000 − 20,000 − 13,000 = 967,000; a 128K model: (128,000 − 33,000) × 2/3 ≈ 63,333; and a model it cannot put a name to gets a conservative 95,000.

More worth studying than the formula is the choice of trigger signal — tokens only (crates/agent-runtime/src/compaction/compact.rs:576-587):

// Token-only trigger: the prior message-count branch was a
// 128K-era heuristic that fired compaction on long chats of small
// messages — exactly the case where rewriting the V4 prefix cache is
// most wasteful. Token budget is the only signal that maps to actual
// model context pressure.

On a 1M window, a hundred "okay" and "got it" messages can still add up to one full-price rewrite. The trigger condition is itself part of cache economics.

should_compact (compact.rs:528-588) counts tokens only for the "summable portion" — messages pinned at the tail do not count; the pinned messages' tokens are also deducted from the threshold, because they are still there after compaction.

For example: threshold 967,000, total session water level 950K, of which 40K is pinned messages. The summable portion is only 910K, and the threshold has been docked to 927K — not there yet. The trigger fires only once the summable portion climbs past 927K. The more you pin, the earlier it fires.

After the trigger comes a pair of circuit breakers (compaction/circuit_breaker.rs). The failure breaker guards against "cannot compact": three consecutive compaction failures trip it, with a half-open probe 5 minutes later; a manual /compact can bypass it — "explicit user agency".

The rapid-refill detector (circuit_breaker.rs:246-253) guards against "compacted for nothing": compaction succeeded, but within fewer than 3 tool rounds the water level is back at the threshold — three times in a row, and it stops compacting. The classic victim: an agent repeatedly reading a 5MB log file — each time you compact, the next tool result fills the window again; the history is rewritten over and over while the pressure does not move an inch.

When it trips you get not an error but an actionable prompt (RAPID_REFILL_MESSAGE, verbatim):

"Context refilled within a few tool rounds of the last compaction, 3 times in a row — a file or tool output is probably too large. Read it back in chunks (read_file start_line/end_line) or retrieve a slice (retrieve_tool_result mode=lines), or start a new session; auto-compaction is paused for this turn."

The diagnosis is crisp: the problem is not that the history is too long — it is that the intake is too large.

This ledger books only genuinely wasted work: compactions vetoed by an extension (SessionBeforeCompact) and compactions whose summary call failed do not count — those are the compaction pipeline's own illnesses and belong to the failure breaker; the three compaction paths — auto-compaction, targeted refresh, and emergency recovery — share one ledger, so wasted work pouring in from any path still trips the breaker.

The ruler that takes the trigger reading has an anchor of its own: TokenUsageAnchor (compaction/mod.rs:190), which after every successful assistant commit records the message count at that moment and the true input_tokens reported by the provider — a full-context reading, so the constants an estimator never sees, such as the system prompt and tool schemas, are naturally absorbed into the anchor.

Estimation afterwards applies only to messages added after the anchor. Say the anchor was recorded at "message 200, input_tokens 412,000"; the model then adds 10 messages, the estimator counts only those 10 messages' 8K, and the water level is 420K — the absolute level is pinned by the real bill. The estimator owns the slope; the bill owns the intercept.

Anchor invalidation does not rely on inference — it relies on events: any action that rewrites history in place — micro-compact clearing tool results (count unchanged, bytes reduced; a count-based check would miss it), full-summary replacement, hard truncation — invalidates the anchor, and the next successful stream re-anchors.

Six Lines of Defense: Don't Touch the LLM If You Can Help It

compact_messages_safe (compact.rs:883-1056) is not one compaction; it is a cascade running from free to expensive. Each layer is tried first, and if it is enough, it exits early:

Layer	Action	Cost
0	Intake truncation of tool results (over 12,000 characters squeezed down to summary + excerpt)	Free; happens at write time
1	micro-compact: stale tool results cleared into placeholders	Free
2	prune: mechanical dedup of duplicate/verbose tool results	Free
3	session-memory-first: MEMORY.md serves as the summary	Free
4	Full LLM summary	One model call
5	Emergency recovery: forced compaction + hard truncation	Cost is no object

micro-compact (compaction/micro_compact.rs) does the cheapest cleanup of all, as its module docs attest:

//! Micro-compaction: lightweight tool-result clearing without API calls.
//!
//! Clears content from old tool results (file reads, shell output, grep, etc.)
//! while preserving message structure. Triggered by time gaps (>=60 min since
//! last assistant message) or byte thresholds. Not triggered in subagents.

Without a single API call, it swaps the content of old tool results for a one-line placeholder: [tool result cleared for context economy]. The trigger is one of two conditions: more than 3,600 seconds since the last assistant message, or cumulative tool results over 32 KB. Subagents never trigger it — they do not live long enough to be worth the surgery.

The effect is like tidying a desk: the 40KB output of that grep from three hours ago, the full text of yesterday's file read — all of it still sits in the drawer called "messages", but the drawer now holds nothing but a note saying "contents cleared". Message structure and call pairing survive intact.

prune (compact.rs:666-759) does mechanical dedup: among repeated results of the same tool with the same arguments, only the newest full copy stays; older copies become a single line, [file_read] tool result pruned (N bytes; args: ...). The archetypal scene is the "edit a round, read it back" loop — the same file_read runs 5 times, the first 4 full texts each occupy 30KB, and now each is one line of text.

The direction of pruning is deliberate (compact.rs:716-720): prune from newest to oldest, stopping as soon as the caller has saved enough bytes — what gets preserved is the head of the request prefix; in the comment's own words, "preserving the earlier JSON request prefix for byte-level KV caches". Every 16 KB pruned, the threshold is rechecked; the moment it passes, work stops. Very often the story ends right here, without one cent of model calls.

session-memory-first (compaction/session_memory_compact.rs): when a persistent memory file (MEMORY.md) exists and the history's token count falls in the 10K–40K retention band, the memory file steps in directly as the summary — history is cut down to the retention point, at zero cost.

The summary announces itself in the system prompt: "The following context was loaded from your persistent session memory." (session_memory_compact.rs:92-103). Picture a 28K-token long Q&A whose conclusions MEMORY.md recorded long ago — this time even the summary call is spared; memory is traded for history outright.

For an industry comparison, Claude Code's compaction is five layers: bulky tool outputs are written to disk with the model seeing only a summary preview; low-value noise is simply deleted, not summarized — summarizing noise is just wasted tokens; micro-compaction at the API layer; archival summaries (kept round by round like a git log, not merged into one entry like a git squash); and finally full compaction as the backstop, likewise fitted with a circuit breaker.

CodeSmith's six-layer cascade is structurally the same — layer 0's intake truncation, layer 1's micro-compact, and layer 3's memory file are all variants of the one article of faith: don't touch the LLM if you can help it.

Where it truly diverges from the Textbook recipe is where the summary lands: in the Textbook, the system prompt and tool definitions never move, and compaction only touches tool results within the history; CodeSmith merges the summary straight into the system prompt — which means firing the bullet of "the one prefix violation" as well (the "A Violation That Gets Booked" section below walks the ledger in detail).

Four design principles apply to both sides: information value is non-uniform (decision points > evidence > noise); semantic integrity ("Sutskever left OpenAI in May 2024" must not be compressed into "left OpenAI"); task relevance (keep breadth when checking a roster, depth when checking background); and compression is understanding (the call that does the compressing itself needs comprehension close to the mainline model's, or its output will be neither reviewable nor reusable).

Material Selection: Not Deleting the Old — Rescuing the Useful

When the LLM summary finally has to be called, the first decision is who gets summarized and who gets pinned. The pinning rules of plan_compaction (compact.rs:351-425) are a checklist of "what matters most for continuing the work":

1. The last 4 messages are always kept (KEEP_RECENT_MESSAGES = 4) — the most recent context is the ground the model is standing on right now;
2. Messages mentioning working-set paths are pinned — repo-relative paths are regex-extracted from messages, the most recent 12 are scanned, with a cap of 24. If you are editing src/auth.rs, every message that mentions it stays out of the summary;
3. Error markers are pinned: error:, failed, panic, traceback — the scene of the failure is evidence for the repair task that follows;
4. Patch markers are pinned: diff --git, +++ b/ — the record of changes must not be diluted by the summary;
5. The tool-call pairing invariant: a pinned ToolResult drags its ToolUse into the pin set (and vice versa) — a post-summary history must not hold orphaned calls with no results;
6. At least one plain-text user message is kept (compact.rs:400-412) — some OpenAI-compatible chat templates require at least one user text; an all-tools tail gets rejected outright by the backend.

Read the checklist through and its worldview shows: compaction is not "delete the old"; it is "rescue the still-useful". Whether something is old is decided by time; whether it is useful is decided by the working set, the errors, and the patches.

The Summary's Two Paths: Let the Summary Request Itself Get the Discount

An LLM summary is itself a model call — how to send that request is another caching question. The design comment at compact.rs:1375-1436 spells out both paths:

• Cache-aligned replays the original messages verbatim with system: None and appends the summary instruction as the final user turn. The model sees the conversation as if it were its own history. This is what lets the V4 prefix cache hit on the bulk of the request (#572).
• Fallback reformats the conversation into a flat User:/Assistant: transcript inside a single user message and adds a "You are a helpful assistant that creates concise conversation summaries." system prompt. The model sees a transcript of someone else's conversation.

The cache-aligned path replays the original conversation verbatim (messages.to_vec()), sets system to None, and appends the summary instruction as the final user message — what the model sees is precisely its own history, so the bulk of this summary request hits the V4 prefix cache.

The comment's closing contrast is the vivid part: in the first, the model sees "its own conversation"; in the second, "a transcript of someone else's conversation". It is the difference between summarizing your own diary and summarizing a printout of a stranger's chat log — in the former, even the pages are familiar.

Admission bar: window ≥500K, and original conversation + 512 tokens of instruction ≤ 85% of the window (CACHE_ALIGNED_SUMMARY_CONTEXT_BUDGET_PERCENT). A small window cannot fit a verbatim replay, so it goes fallback — the conversation is flattened into a User: ... / Assistant: ... transcript packed into a single user message, plus one dedicated system prompt.

The summary instruction itself (compact.rs:1429-1437) is plain English: "Preserve key information, decisions made, exact file paths, commands, errors..." — key information, decisions, exact paths, commands, errors: everything the ongoing work needs. The limits are 900 words / 2,048 output tokens, temperature: 0.3.

If the summary request itself hits the window cap (prompt too long), there is self-healing too: strip 20% of the oldest removable messages and retry, at most 4 times (compact.rs:48-49).

The World After Compaction: A Violation That Gets Booked

Once the summary is generated, history is replaced wholesale with "the pinned messages", and the summary block is merged into the system prompt. The merge function's comment says plainly what grade of violation this is (engine/mod.rs:2884-2888):

/// **Zone affiliation (#2264)**: this mutates the system prompt, which is
/// part of the `PinnedPrefix` zone in the three-zone contract. Compaction
/// is the one intentional mid-session prefix mutation — the engine
/// intentionally accepts the cache-invalidation cost because the
/// context-reduction benefit outweighs it.

Then comes the attachment re-injection that Article 10 foreshadowed: four kinds of <system-reminder> user messages — plan and TODO, skills, running subagents, previews of the last 12 file reads — appended in sequence, each first deduplicated at the byte level, then fitted against the budget (engine/mod.rs:2913-2989).

Finally, post_compact_cleanup (compaction/post_compact_cleanup.rs:21-40) resets everything: the working set is force-rebuilt (deleted messages may have referenced stale paths), the system-prompt hash is cleared, the circuit breakers are reset. One compaction, one violation — and the ledger records it in full detail.

For the extreme cases there is emergency recovery (engine/host_executor.rs:2412-2428): when the preflight estimate before a request exceeds the budget — micro-compact → forced full compaction (auto_floor_tokens = 0 — at the hard ceiling, the cache floor no longer applies) → and if that is still not enough, messages are hard-truncated from the very front, with a floor of 4 plus the paired cascade of deletions. At most 2 recoveries per turn; once exhausted, the turn fails hard.

A Philosophical Critique: Do Not Build a Frankenstein

All of the above is engineering. But the repo's deepest thinking about compaction is a comment that indicts compaction itself. The module docs at cycle_manager.rs:1-41 first table the retrieval-degradation data from the DeepSeek paper: retrieval precision 0.90 at 8K context, dropping to 0.76 at 256K, only 0.59 left at 1M. Then, the verdict:

Lossy summarization compaction creates a "Frankenstein" context — half verbatim, half paraphrased — that the model cannot tell apart, so it treats the summary as if it were verbatim and confabulates around the gaps.

Make the "confabulates" concrete: the summary says "the user fixed the login-timeout bug" but failed to keep the original stack trace. The user asks, "what exactly was the error back then"; the model cannot tell which half is verbatim and which half paraphrase, treats the summary as the original, and improvises an error message on the spot.

The remedy is called checkpoint-restart (rolling periodic restarts): rather than patching the old context, start wholly anew on schedule (active input 768K, roughly 75% of 1M). The old cycle is archived as JSONL on disk, paired with a recall_archive tool for retrieval on demand — the old ledger is not thrown away, just moved out of the window.

The new cycle starts with three things: the original system prompt, structured state (TODO/working set/subagent snapshots — the [CYCLE STATE] of Article 10), and a handoff brief the model writes in its own hand to the next self — the <carry_forward> block, capped at 3,000 tokens.

The "What NOT to put" section of the brief template (prompts/cycle_handoff.md) is just as good: do not put tool-output bytes (already archived); do not put the contents of files you have read (the next cycle can re-read them — "cheaper than a wrong assumption"); do not put a blow-by-blow log. What belongs there are decisions, constraints, failed routes, open questions — the only things that cannot be rebuilt from anywhere else.

Compaction and restart are two worldviews on the same problem: the former restores the old as old, at the price of mixing the genuine with the paraphrased; the latter tears everything down and starts over, at the price of relying entirely on the brief. This repo built both, and lets the threshold and the user decide which to use.

The Manual Pole: Let the Agent Operate on Itself

Finally, a mechanism that hands the choice to the model itself: /purge (crates/agent-runtime/src/purge.rs). One sentence in the module header fixes the division of labor: compaction is the LLM batch-summarizing old messages; purge "lets the agent analyze the history and surgically remove or rewrite the individual messages it no longer needs".

The engine hands the model a message list with stable numbering and injects a temporary purge_context tool; the operations are exactly two: remove deletes a whole message (example: {"op": "remove", "msg": 3}), and replace rewrites one content block by regex (example: shrinking "read 12 files" to "read files"). Pairing rules mop up automatically — delete a tool call, and its paired result follows it out.

When to strike is listed clearly in the instructions: what should be trimmed is verbose tool output whose information has been fully consumed, redundant acknowledgments like "done" and "ok", and old file reads superseded by later writes; what should be kept is important decisions, architectural choices, and file paths still in use.

The final line of the operating instructions (purge.rs:26-65):

Be conservative. When in doubt, keep the message.

The model is handed the power to delete history — but only under a signed pledge of caution.

Conclusion: Every Compaction Is a Trade

To gather this installment together: the trigger listens only to tokens (967,000 is not a number pulled from thin air); the defenses line up six layers from free to expensive, never touching the LLM when avoidable; selection pins the tail, the errors, the patches, and the working set; the summary request stays cache-aligned where it can, so it too gets the discount; compaction is "the one prefix violation that gets booked"; and the ultimate indictment of lossy summarization is the cycle's wholesale blood transfusion.

And before all of that there is, in fact, one cheaper line of defense — isolation over compaction, as Article 6 covered: if bulky intermediate artifacts can be kept out of the main context in the first place (subagents, handles, the private kitchen), compaction never gets its turn. Compaction is the remedy for after isolation has fallen on every front, and the six layers of defense are its honest admission of that ordering.

With this, the context storyline is complete: Article 8 capped the bulk, Article 9 held the prefix, Article 10 managed the tail, and this installment dealt with the full window. With the window's books balanced, the bottleneck moves back onto the model itself — cheap brains cannot do deterministic problems, cannot read rules precisely, cannot repair themselves, and know how to smooth things over in their reports. The next act's solution has only one direction: let code think for the model.

Comments

[Alex Shev]

The distinction between overflow and context rot is especially useful because it turns compaction into a quality-control decision, not merely a token-limit reaction. Preserving stable handles to the source records alongside the summary would make the tally example auditable: a later turn can retrieve the underlying rows when the summary becomes insufficient or contested.