I ran one subject through a context assembler at three different token budgets and wrote down what came back. 200 tokens returned 2 memories. 500 returned 4. 800 returned 6.
That is a 4x spend for 3x the facts. It is the most useful number I have for sizing an agent's memory budget, and I almost never see it published.
| Budget requested | Tokens returned | Utilization | Memories returned | Marginal cost per new memory |
|---|---|---|---|---|
| 200 | 178 | 89.0% | 2 | 89 tokens |
| 500 | 443 | 88.6% | 4 | 133 tokens |
| 800 | 761 | 95.1% | 6 | 159 tokens |
Method, so you can repeat it: these are the published outputs of the budget command in the statewave-personal-assistant reference app, which calls the context API three times at three budgets against the same subject. I read them on 3 August 2026 and derived the marginal column myself.
The cheap facts get spent first
Marginal cost per fact rose 79% across that range, from 89 tokens to 159.
The assembler is not distributing effort evenly. It sorts by score and fills from the top, so the first two items are the densest, highest-priority things it holds. By the fifth and sixth it has worked down into longer, lower-scoring material.
So if you are choosing between 400 and 800 tokens, you are not choosing between four and eight facts. You are deciding whether to pay roughly double for two more items that already scored worse than everything in the bundle.
Utilization is the other column worth reading. It landed at 89.0%, 88.6% and 95.1%, and it never crossed 100%, because items are admitted whole. A memory that would overshoot the ceiling by one token gets dropped rather than truncated mid-sentence. Headroom across those runs was 11.0%, 11.4% and 4.9%. Plan for 5 to 11% of slack instead of assuming you get the number you asked for.
The stop rule is three lines, and that is the point
selected, used = [], 0
for memory in sorted(candidates, key=score, reverse=True):
if used + memory.token_count > budget:
break
selected.append(memory)
used += memory.token_count
A greedy knapsack fill. RCR-Router, a multi-agent routing paper, publishes essentially this as its Algorithm 1: sort by importance, accumulate, break on overflow. Nothing in it is stochastic, which is the property everything below depends on.
Compare it against the two patterns it replaces. Full history replay re-sends everything each turn, so cost grows with session lifetime and a throwaway comment carries the same weight as a production incident. Top-K vector retrieval returns the K nearest chunks, but K is a count and not a token budget, so your actual prompt size drifts with chunk length.
And no, a bigger window does not make this go away. Liu et al. tested position directly in Lost in the Middle and found accuracy follows a U-shaped curve against where the relevant passage sits: best at the start and end, worst in the middle, and that held even for models built for long context. Padding a 128K window with 100K tokens of history hands the model a large middle.
Scoring is where your opinions live
The fill is trivial. Ranking is where the design decisions sit. Four signals carry most of the ordering in the implementation those numbers came from:
| Signal | Range | What it encodes |
|---|---|---|
| Kind priority | 3 to 10 |
profile_fact 10, procedure 8, episode_summary 5, raw_episode 3 |
| Recency | 0 to 5 | Linear, the newest memory takes the maximum |
| Task relevance | 0 to 8 | Word overlap contributes up to 5, cosine similarity up to 8 |
| Temporal validity | -4 to +3 | Currently valid adds 3, expired subtracts 4 |
Read that as a set of opinions rather than a spec. A stable profile fact outranks a raw conversation turn by 7 points on kind alone, before any other signal applies, which is why identity survives budget pressure and chatter does not. An expired fact carries a 7-point swing against it. Cosine similarity is in there, but as one input among four rather than the whole ranking.
What makes small budgets viable at all is compilation: a pass over raw episodes producing typed facts with confidence scores and provenance back to the source events. Two hundred conversation turns become one profile_fact. Rank raw turns instead and you are sorting a bad unit, low density and high token count with no confidence signal attached.
Determinism matters more than the compression
A 73% token reduction is the headline here, from roughly 2,800 tokens of raw history down to 761. I care more that it is the same 761 tokens every time.
Researchers at Penn State measured the alternative. Their work on parallel context compaction reports that as context grows, both the volume a model produces and the information it retains vary run to run. Prompt instructions asking for a specific summary length get largely ignored.
Which means: if your compaction step is an LLM call, your retrieval layer is non-deterministic, and every eval you run afterwards measures two things at once.
Deterministic assembly means same subject, same task string, same budget, same point in time, same bytes. When an answer changes, you know the change came from the model or the prompt, because the context did not move.
There is a real cost attached. Compile-then-use bundles are denser per answer than a plain fact-store lookup. If your queries are mostly single-hop and cost-sensitive, a lighter store may be the better call.
Six steps for setting your own number
- Instrument before you tune. Log the returned token estimate and memory count on every context call. You want the utilization ratio. Running consistently below 85% means the budget is not your constraint and raising it will change nothing.
- Start at 800. Aim for 5 to 8 compiled facts, which for typical fact and procedure sizes lands between 600 and 1,200 tokens.
- Add the fixed costs before deciding it is too small. System prompt and tool schemas run 2,000 to 4,000 tokens on their own and get billed every call. Memory is usually the smallest line item in a well-built agent.
- Watch the marginal curve, not the total. Run your own version of the three-budget test above and find where the next 200 tokens stop changing answers.
- Set per-role budgets for multi-agent runs. A base budget plus a role offset. A planner needs structured plans; an executor needs less.
- Keep compaction off the hot path. Compile on a schedule, every N episodes or nightly, using an async mode that hands you a job ID to poll. Raise the budget only when a named eval fails at the current one.
Two ways this still goes wrong
Bad extraction upstream. Ranking garbage precisely and fitting it exactly into 800 tokens gives you 800 tokens of well-organized garbage. The compiler sits upstream of everything here.
Prefix cache invalidation downstream. TokenPilot's authors found that mutating the prompt sequence to save tokens invalidates the KV prefix cache, and the misses can cost more than the tokens saved. Assemble a stable prefix and vary the tail.
One related trap: this is not token-level compression. The AGORA authors tested extractive token-level compressors across 17 agent configurations and reported that all 17 collapsed despite achieving real compression, because the compression broke action grammar. Dropping tokens inside a well-formed fact is a different operation from dropping a whole low-ranking fact.
Where to start
Log the token estimate and memory count on the context calls you already make. Then run one subject at three budgets and find where the answers stop changing. If your curve flattens where mine did, you have your number in an afternoon.
If you would rather not build the compilation and ranking layer, the runtime those numbers came from is Apache-2.0 and on GitHub. It boots against your own Postgres with one command. The longer version of this post carries the full signal list and the receipt model I skipped.
Top comments (2)
The marginal-cost-per-fact column is the number nobody publishes and it is the only one that matters for sizing. I hit the same wall from the other side: my agent reads a busy message board, and past a certain window size everything extra it pulled was the low-score tail - more tokens, flatter decisions. Your 79% rise in marginal cost is a cleaner way to say it than I had. And if the assembler fills by score, the fix is rarely a bigger budget; it is a better ranker.
"More tokens, flatter decisions" is the whole failure in four words, and it's worth saying out loud, because the fix people reach for first is the budget.
Agreed on the ranker, with one addition. The 79% is two effects stacked: items further down rank lower, and they're also longer. The first two in that run were the densest, highest-priority items; by the fifth and sixth the fill had reached longer, lower-scoring material. A better ranker decides what ends up in the tail. What makes the tail cheaper is a smaller unit, and that's decided upstream, when raw turns get compiled into facts, not at ranking time.
What did you end up ranking the message board on besides similarity? Recency and thread position are the obvious ones; I'm curious whether author or reply count earned a place.