How I Cut Long-Chat Token Bills by 85–95% With Semantic Context Compression
A practical engineering post on semantic context compression for LLM agents — why "sliding window" truncation breaks agents, how a router detects the threshold, and what a good restore point looks like.
If you've built an agent that runs for hours or spans hundreds of turns, you know the bill: every turn you re-send the whole conversation to the model. Token cost grows linearly with session length, and eventually you hit the context window and the whole thing breaks.
The usual fix — sliding-window truncation ("keep the last N messages") — is terrible for agents. You throw away the plan you made in turn 3, the user's stated constraints from turn 10, and the decision you made at turn 47. The model starts repeating itself, contradicting earlier work, and the user notices immediately.
Semantic context compression is the fix. And it's one of the features I care about most in SentioGate, the BYOK LLM router I'm building.
What "semantic compression" means (vs. naive truncation)
| Approach | What's kept | What's lost | Cost at turn 200 |
|---|---|---|---|
| Sliding window (last 20 msgs) | Recent chatter | Plan, constraints, decisions | High — model re-derives context every turn |
| Naive "summarize older msgs" | A text blob | Structured state (files, tools, user intent) | Medium — summary drifts, loses key facts |
| Semantic compression + restore point | Structured state: goals, files, tool calls, user intent, open threads | Verbose intermediate chatter | Low — the compact state carries 90%+ of the useful signal |
The key insight: a long conversation isn't a bag of text. It's state — goals, constraints, decisions, open tasks, files being edited, tools being called. If you compress the state rather than the prose, you get 85–95% token reduction with near-zero loss of continuity.
How the router detects the moment to compact
The router watches each session's message tree. Two signals trigger compaction:
- Capacity threshold — when the session's context reaches a configured fraction of the model's context window (e.g., 75%), compact before the next turn overflows.
- Token threshold — when the cumulative input tokens for the session cross a budget, compact proactively even if you're far from the window.
The compaction preserves the most recent N messages verbatim (default: 4) and collapses everything older into a structured restore point — a compact, machine-readable summary of the session's state.
What the restore point looks like
The restore point isn't a free-text summary. It's a structured document with slots for:
- Goals — what the agent is trying to accomplish
- User intent & constraints — anything the user explicitly asked for or ruled out
- Files / resources — files being edited, configs, data sources
- Tool state — which tools have been called, what's pending
- Decisions made — choices that affect downstream turns
- Open threads — things not yet resolved
When a new model (or a fresh session) picks up the conversation, it reads the restore point instead of the full history. Input tokens drop from, say, 60,000 to 4,000–8,000, and the model has everything it needs to continue without re-deriving context.
Why this matters for model-hopping
A big unlock: because the restore point is model-agnostic, you can switch models mid-conversation — finish a planning turn with a frontier model, then hand the execution to a cheaper / faster model, then back to the frontier model for a final review. Each handoff reads the same compact state. No re-reading of full history.
That's the "seamless model-hopping" I mentioned in my intro post to SentioGate. The compaction engine is what makes it possible.
What to watch for (and what I'm still iterating on)
- Loss of rare-but-critical facts. The rare user message that's important later ("no, don't touch the prod DB") can get lost in compression if it's buried. The preserve-recent-N window helps, but I'm working on fact-extraction heuristics that flag "constraint" messages regardless of recency.
- Cross-session continuity. Today, compaction is per-session. Multi-session agents (a "planner" session feeding an "executor" session) need a shared compact state. That's on the roadmap.
- Quality scoring of the compact output. The restore point is only as good as the model that wrote it. I'm adding a lightweight QA pass that checks the compact against the original for the most critical facts before it's used.
The number I'm most proud of
85–95% input-token reduction on long sessions, measured across real agent workloads, with no measurable drop in task-completion quality. That's the entire value proposition of the compression engine in one sentence.
Links
If you're building long-running agents and your token bill is a problem, drop a comment — I'll follow up with a post on the exact scoring algorithm and how to tune the compaction thresholds for your workloads.
Top comments (0)