DEV Community

Linford Reyes
Linford Reyes

Posted on

Context compaction is silently destroying your LLM agent's memory

#llm #python #ai #opensource

Context compaction is silently destroying your LLM agent's memory

Tl;dr: Long-session LLM agents lose governing rules, todos, and decisions when context compaction runs — and the loss is usually silent. I built a zero-dependency Python library (memory-anchor) that snapshots that state verbatim before compaction and re-injects it after, plus a CLI to audit how much a compaction actually destroyed.

The problem nobody measures

Run any agent long enough and it hits the context window. The fix everyone reaches for is compaction: old turns get summarized into a paragraph and the summary replaces them. Cheap summarizers — especially the fast flash models everyone uses to keep costs down — flatten detail. In my own agent's logs:

  • a governing rule got paraphrased (the agent started behaving differently and nobody noticed why)
  • a pending todo vanished ("what was I doing?")
  • a decision's rationale was rewritten (settled questions got re-litigated)
  • a verification path disappeared (half-verified work got reported as done)

That last one is the scary one. After a 35% compaction pass, my agent forgot it had delegated a subtask and never collected the result. Not a hallucination — a silent state loss.

Why existing memory systems don't cover this

The current memory stack (mem0, Letta, Hindsight, and friends) remembers facts — entities, preferences, retrieval chunks. None of them guarantee that the rules and work state governing this session survive a compaction byte-for-byte. Research-backed systems like MemGPT treat memory as a tiered store, and Claude's own compaction is a black box — you don't get to see what it threw away.

There are components everywhere, but no off-the-shelf, framework-agnostic answer. So I built one.

What it does 

preserve(ctx) ─► manifest (4 lists, verbatim) ─► [summarizer runs] ─► recover(ctx) ─► recovery block injected at head of messages
Enter fullscreen mode Exit fullscreen mode

Four small pieces, pure stdlib, zero dependencies:

  • StateManifest — rules / todos / decisions / progress, serialized to JSON, with incremental merge(): done todos never resurrect, superseded decisions never reappear.
  • MemoryStore — atomic (tmp+rename) local JSON persistence, per-session indexed.
  • RecoveryInjector — rebuilds the recovery block; L1 (immutable rules) is never trimmed.
  • CompactableMemory — a two-line facade: 
from pathlib import Path
from memory_anchor import CompactableMemory

mem = CompactableMemory(base_dir=Path(".memory"))
mem.preserve(ctx)                              # before compaction
messages = mem.recover(ctx, messages, summary) # after compaction
Enter fullscreen mode Exit fullscreen mode

The part I'm most proud of: auditing compaction damage

v0.3 added cam judge — point it at your pre-compaction manifest and the summary that replaced it, and it classifies every item as verbatim (survived), paraphrased (semantics preserved), or lost. It's a CI gate, not a vibes check:

cam judge --before manifest.json --after summary.txt --min-verbatim 90
# exit code 1 if fewer than 90% of items survived byte-for-byte
Enter fullscreen mode Exit fullscreen mode

The numbers from my own system are not flattering to the status quo. Running the same 8KB briefing through two compressors:

compressor verbatim survival outcome
extractive (built-in) ~60% 2 governing rules silently lost
caveman (aggressive) 100% all items survived byte-for-byte

The judge didn't just measure the damage — it exposed it. That extractive pass was silently eating rules every single run, and nothing in the pipeline was looking.

Try it 

git clone https://github.com/44334433/memory-anchor
cd memory-anchor && pip install -e . && pytest   # 31 tests
Enter fullscreen mode Exit fullscreen mode

Or script it into your cron/CI without Python:

cam before my-session --rule "R1|never paraphrase governing rules|100" \
  --todo "ship v0.2|pending|run the drill"
# ...compaction happens...
cam after my-session --messages messages.json --budget 2000
Enter fullscreen mode Exit fullscreen mode

Honest limitations

  • This is not a semantic memory system. It preserves work state you explicitly declare — it won't infer what matters on your behalf (yet).
  • judge uses diff-based matching, not semantics. A truly reworded rule may count as "paraphrased" rather than "lost" — the threshold (45%) is a documented heuristic.
  • It's framework-agnostic by design: bring your own summarizer, hook the two calls wherever your pipeline compacts.

If your agent has ever "forgotten" something after compaction, you already know this pain. Anchor the state that must survive — and start measuring what your summarizer is destroying. Star/watch the repo if you want to see the framework adapters (LangChain, Claude Code, OpenHands) land — I'm holding off until real integrations ask for them.

Top comments (2)

Collapse
 
alexshev profile image
Alex Shev •

Compaction gets dangerous when it preserves conclusions but drops provenance. For coding agents, "we decided X" is weaker than "we decided X because file Y behaved this way on date Z." Without the why and the source, the compressed memory can steer future code while becoming impossible to challenge.

Collapse
 
linfordr profile image
Linford Reyes •

Great point — that's exactly the failure mode I've been circling. Just pushed v0.3.1: decisions now carry source + evidence fields alongside rationale, and cam judge grades provenance as part of the decision. The matched text is decision + rationale + source + evidence, so a summary that keeps "we decided X" but drops "because file Y behaved this way on date Z" scores below verbatim and fails the audit gate. The asymmetry you describe is real: a decision without its provenance isn't just weaker information, it becomes unquestionable — there's nothing left to challenge it with. Thanks for the feedback, this landed as a feature within hours.