If you've run a long agent session in any framework, you've hit the wall: context fills up, the framework decides it's time to compact, and everything stops while a summarizer chews through the whole transcript. In hermes-agent that batch compaction fires around 80% of the context threshold, and on a long session the pause is measured in minutes. The agent isn't stuck, it's doing necessary work, but from the outside it looks dead. And the longer you wait to compact, the bigger the lump you eventually pay.
Micro-compaction is a different way to pay the same bill. Instead of one big batch pass when you're nearly full, you fold the oldest un-absorbed exchange into a rolling summary after every turn. The total summarization work is roughly the same. What changes is the shape of the cost: small increments spread across the session instead of one long stall at the worst possible moment.
Origin note, since it's unusual: the first draft of this feature was written by Hermes itself, for its own codebase. What follows, the measurement, the bugs, and the upstreaming, is what it took to get that draft into a state worth merging.
How it works
After each turn, the finalizer looks for the oldest agent turn that hasn't been absorbed yet, summarizes it into a cumulative rolling summary, and splices a single summary marker into the transcript where that material used to be. There's exactly one marker at any time. Each pass replaces it rather than adding another. A protected head (the system prompt and early setup) and a protected tail (the most recent turns, 16% of the window by default) never get touched, so the model always has verbatim access to what it's currently working on.
One design decision worth calling out: user turns are never compacted. Assistant output compresses well because it boils down to "it did it this way," and the details can usually be reconstructed from the work that followed. User intent can't. If you summarize away what the user actually asked for, no amount of downstream context recovers it. The cost is a floor on how much the middle of a conversation can shrink, since user messages accumulate verbatim. I think that's the right trade. If a user is dropping 20K-token prompts, that's on them.
The feature is opt-in:
compression:
micro_compact: true
There's also micro_compact_every_n_turns if you want to thin out how often it runs.
What the numbers say
The claim isn't "saves tokens." Early on I framed it that way and it was the wrong headline. The real value is two things: the long pause gets amortized away, and context lasts longer before you hit the threshold at all.
The validation run was a real 3.5-hour session, a whole-project code review, with telemetry on every pass. Two results:
Zero batch compactions across roughly 75K tokens of transcript. The multi-minute pause never fired once.
Occupancy (context used as a share of the compaction threshold) went 8.7% → 15.0% → 17.5% → 21.8% → 22.0%, then flat. In the last stretch the conversation added 4,841 tokens and micro-compaction reclaimed 4,395. That's equilibrium at 22% instead of a steady climb toward 80%. The session could have run much longer than it did.
One gotcha if you instrument this yourself: the first pass costs tokens. The summary marker carries about 411 tokens of fixed scaffolding, and on pass one that's charged against a single absorbed exchange, so the delta is typically positive, around +330 in my measurements. From pass two the marker is replaced instead of added and each exchange is near-pure saving. Break-even lands around pass two or three. Judge the trajectory, never a single telemetry line.
Getting to trustworthy numbers was most of the work, honestly. The original draft had a bug where every pass appended a new marker and left the old ones in place. The rolling summary is cumulative, so each stale marker was a near-duplicate prefix of the next, and the transcript grew every turn. Measured over six turns: 4,104 tokens ballooned to 4,797. With the supersede fix, the same conversation went 4,104 to 2,572. A second bug had the resume cursor pointing at pre-splice indices, so on tool-heavy conversations it silently skipped about half the exchanges it should have absorbed. Neither bug is visible in a single-pass unit test. Both fell out of a randomized stress harness that runs 480 conversation shapes through 25 passes each and asserts invariants after every pass: one marker max, transcript stays API-valid, message count never grows, user turns intact. If you're building anything in this class, compaction bugs are accumulating drift, and simulating many turns is the only test shape that finds them.
The honest costs
Micro-compaction rewrites already-sent history every time it commits a pass, and that breaks the provider's prompt-cache prefix. Batch compaction breaks it too, but once, not every turn. hermes-agent's own config treats per-conversation caching as close to sacred, and its proactive-prune logic exists specifically to avoid "a tiny break every tool iteration." Micro-compaction as originally shipped was doing exactly that, which is why review flipped it to default-off. If your provider bills cache reads at a steep discount, per-turn invalidation is real money, and my 3.5-hour validation measured occupancy and pause counts but never priced the cache side. That tradeoff is not settled by my own numbers, and I'd rather say so than pretend the headline result covers it.
The passes themselves aren't free either. Median pass duration in the validation run was about 31 seconds, synchronous in turn finalization. The response has already streamed by then, so the user isn't staring at a frozen answer, but the turn doesn't close until the pass does. Total was around two minutes of summarization spread over 3.5 hours, which still beats one multi-minute lump, but 31-second increments strain the "small increments" framing. In my setup the bottleneck is the summarizer model, a 4-bit Qwen 7B on a shared box, not the input size, so a faster auxiliary model would probably shrink this a lot. Probably.
There's also no reclaim-size gate yet. A pass commits whatever one exchange saved, even if that's a few hundred tokens, and each commit is a cache break. The obvious follow-up is buffering absorbed exchanges and only splicing once the reclaim crosses a threshold, mirroring what the prune logic already does. That doesn't exist today.
Where it landed
The feature merged into hermes-agent and ships in v0.19.1, opt-in via the config flag above. The original PR was #74522; it landed through a maintainer salvage PR (#75345) that kept the commits and authorship and then reworked some internals with a design I'd call better than mine in places. The marker is now an assistant-role message, an exchange is a full agent turn bounded by user messages, and markers are only ever superseded when their content is provably contained in the rolling summary, which makes stale-marker bugs structurally hard to reintroduce. The docs in the repo cover the mechanics in more depth.
If you maintain an agent loop and batch compaction pauses are hurting you, the shape of this idea transfers even if the code doesn't: pay the summarization bill per-turn, protect the head and tail, never touch user intent, and instrument it before you trust it. Especially that last one. Every real bug in this feature was found by measurement, not by reading the code.
Top comments (0)