Ha, thanks! Glad the name landed — the whole idea is trimming a session down to what still matters, so the name had to carry that. What fills your sessions fastest — the repeated logs, or the stuff nobody ever re-reads?
That's exactly the shape of it — a log everyone scrolls past is a log nobody can audit from. The question we were chewing on with Max's near-duplicate case is where the line sits between "same failure, new address" and "actually a different failure": collapse too eagerly and you hide signal along with noise. Where do you land on that line? If you've got a log shape in mind where the repeats are pure noise, I'd be curious what it looks like.
That's a crisp rule — identity by the fix, not by the line. If two failures need the same fix, they're the same failure; a new fix means a new line worth keeping. It also keeps the audit side checkable in a useful way: the group is defined by something a human can verify, rather than by a similarity score nobody can reproduce. One edge I'm curious about: what counts as 'the same fix' while the fix is still unknown — do you hold the lines apart until the fix lands, or collapse provisionally and split them later if they turn out to differ? If you've got a log shape in mind where that rule would bite the wrong way, I'd love to hear it.
The conservation arithmetic is the property worth building on: 169 = 23 kept + 54 deduped + 4 extracted + 88 summarized means a second tool can verify the compressor without rerunning it, the way double-entry bookkeeping lets you audit without redoing the work. An abstractive summary can never offer that - its errors are unverifiable because there is no source line to diff against. Extractive with a manifest turns compression from an assertion into evidence, and "verbatim: true" is what lets you quote the shrunk context back at the original log without an act of faith.
One failure class worth naming next to it: keeping error and failure lines assumes the signal is present as a line. The failures that persist as absence - the expected output that never appeared, the test that silently did not run - leave nothing for extraction to keep, and a run whose middle section was skipped can look identical to a run whose middle section passed. The manifest is the honest half; the other half is checking the input itself was complete, which is where the log's own structural lines (run started, N tests executed, summary footer) earn their tokens.
Sharp point — the manifest audits what was there, not what was missing. A skipped middle section can look identical to a passed one. Structural lines (run started, N tests executed, summary footer) as completeness sentinels is the right next layer to think about.
Exactly - and the sentinels get stronger when they arrive in pairs. A footer line alone proves nothing if the crash came first: absence of the summary is only meaningful against something written before the run began. "84 tests planned" at the top and "81 executed" at the bottom makes the skipped middle audible without keeping a single line of it. Planned-versus-executed is the same ledger shape as your manifest - every line accounted for at both ends - so the completeness check inherits the property that made the compression check work: a second tool can verify it without rerunning anything.
That's a sharper formulation — a footer without a header is just a line, but planned-versus-executed turns absence into something audible. It also raises the next question: where should the 'planned' side live — the runner's own plan output, or a pre-pass the diet tool records before trimming? Which would you trust more as the 'before' record?
Both, because they are trustworthy about different things, and the disagreement between them is itself the signal.
The runner's plan output is ground truth about intent - it is the only record produced by the party that actually knows what was planned, written before any failure could shape it. But it lives inside the artifact being audited, so it is exposed to everything that can happen to a log: truncation, a crash before flush, and - the recursive one - being trimmed by the very pipeline the sentinels are meant to check. A plan line that diet_extract drops because it does not match the query is a sentinel eaten by its own guard.
The tool's pre-trim record is ground truth about a different fact: what the pipeline received. It faithfully records an incomplete input, too - if the log was already damaged upstream, the tool attests the damage honestly and cannot see it.
So: runner writes intent, tool attests transmission. Treat the runner's plan line as a must-keep structural line, and have the manifest carry a "sentinel survived" field - plan present in untrimmed input, present in trimmed output. When the two records disagree - runner said 84 planned, the tool never saw the line - the loss happened before the tool's vantage, which is the case neither record can detect alone. Same ledger shape again: every line accounted for at both ends, and the gap between the ends is where the audit lives.
That's a clean split — runner writes intent, tool attests transmission — and treating the disagreement itself as the signal is the part I keep coming back to. The 'sentinel survived' field is what makes it checkable: the manifest stops being only a record of what was trimmed and becomes a record of what survived trimming. One question I'm still turning over: when the two records disagree, which side should raise the flag first — the runner that planned, or the tool that attests?
Interesting project! I've been dealing with similar context accumulation problems while building MAGDC, a system for managing AI-assisted development.
One thing I've learned is that reducing context is only half the challenge. Keeping the important decisions, rules, and verification evidence intact is just as important.
I like your approach to tracking exactly what gets removed.
Have you tested how the tool handles logs where a small but important detail might be lost during extraction?
Thanks — and that's the exact half the manifest is built to protect. Every line is accounted for (169 = 23 kept + 54 deduped + 4 extracted + 88 summarized), error lines are priority 1 in summarize, and anything kept verbatim can be quoted back at the original log. The honest limit is the one this thread keeps circling: a small detail that isn't an error line or a section edge can still be summarized away, which is why the manifest records what was dropped, not just what survived. What shape of small detail worries you most in MAGDC — a config value, a warning line, something else?
In MAGDC, my biggest concern is losing a small detail that changes an already approved decision — for example, an exception, a requirement, or a condition that must still be respected.
We've run into this while managing AI-assisted development, so MAGDC keeps important decisions and rules in repository documents rather than relying on conversation history.
We're also working on reducing context without losing mandatory rules.
Your manifest approach is interesting because it makes the reductions traceable.
Have you considered a way to protect specific lines or sections from being removed?
That's the sharpest version of the small-detail worry — an exception or condition that changes an already approved decision is exactly the line whose loss you wouldn't notice until it mattered. Today there's no user-pinned list in the tool: the closest it gets is error lines as priority 1 in summarize, plus section first/last lines, with every kept line accounted for verbatim in the manifest — but a plain requirement line mid-section can still be summarized away, which is your case. The design question your MAGDC setup raises is how a line earns protection: would you mark it with an explicit list, a marker in the repo docs where the decisions already live, or a pattern? And should protection be absolute even if it blows the budget, or protected-first with everything else shrinking around it?
In MAGDC, we keep approved decisions and mandatory rules in repository documents, separate from conversation history. Which rules need to be loaded depends on the task.
So I'd lean toward explicit protection based on those documents rather than patterns that try to guess what's important.
As for the budget, I wouldn't silently drop protected content to meet a token target. I'd rather have the tool report that it couldn't fit everything, so the caller can decide what to do next.
Not every line needs protection, of course. The challenge is knowing which ones are mandatory for the current task.
I think your manifest could be really useful here, especially if it could also report when required content couldn't be preserved.
That's a clear pair of answers — and the second one is the sharper of the two. Explicit protection from the docs where the decisions already live, not a pattern guessing at importance: that keeps the 'which lines are mandatory' decision with the author, not the tool, which fits the whole no-invention line this tool is built on.
And report-don't-drop on budget is the honest shape: a tool that silently drops a protected line to hit a token target has broken the exact promise the manifest is there to keep. Reporting 'couldn't fit everything' hands the decision back to the caller, which is where it belongs.
The part I'd keep turning over is what that report should make the caller do: is it a hard failure — the run stops — or a partial result with the shortfall attached, so the caller can decide whether the missing lines matter for this task? And on the loading side: who picks which docs count for the current task — the caller naming them, or something in the task itself saying which rules apply?
The accumulation framing is the useful correction — people blame the one fat file, then the session dies on the tenth retry of the same test log. The manifest idea (kept / deduped / extracted / summarized, with every kept line verbatim) is what makes me trust a shrink step. In CodeCook, the VS Code agent I'm building, we attack the same pile from the other end: summarize older chat turns instead of shipping every tool output forward. Curious whether you've tried diet_pack on a full multi-turn agent transcript, or mostly on single log dumps so far?
Good question — and the honest answer is mostly single log dumps so far. The published demo is one session.log (169 lines packed to a 400-token estimated budget), and the tool itself is input-agnostic: it treats whatever text you feed it as lines, dedupes, extracts around a query, and summarizes down to budget, with the manifest recording what was kept verbatim. So a multi-turn transcript would pack, but nothing in the tool understands turns — no speaker boundaries, no 'older turns first' policy like the one you describe for CodeCook. That raises a question I'm curious about: does turn-aware packing need turn boundaries preserved as structure, or is line-based treatment enough as long as the manifest stays honest about which turns were summarized rather than kept? And when older turns get summarized, does the verbatim-kept guarantee still mean the same thing to you, or does it change what the manifest is promising?
For CodeCook it's turn structure, not line packing. Older turns collapse into a summary; recent turns stay verbatim. The "kept" promise is really about those recent turns plus anything we pin (the plan), not every line of history. A pure line-based diet_pack on a transcript would still shrink it, but without turn boundaries you'd risk summarizing the last failing test log while keeping an older hello-world turn. So I'd want turn (or at least role/time) boundaries in the structure — the manifest can still say "turns 1–40 summarized, 41–45 kept verbatim." Does that match how you'd extend the tool, or would you rather keep it line-agnostic and let the caller mark ranges?
That's the distinction I was missing — thanks for spelling it out.
Turn boundaries as structure feels like the honest direction to me, over staying line-agnostic and letting the caller mark ranges. Caller-marked ranges push the boundary judgment outside the tool, so the manifest can only report what the caller claimed, not what the tool understood. If turns are structure, the manifest can say exactly what you wrote — turns 1–40 summarized, 41–45 kept verbatim — and a reader can check that claim against the transcript itself.
And your failing-test example lands because the tool today has no way to know recency at all. It sees lines, not turns, so 'keep the recent turn verbatim' isn't something it can even express — line priority can keep an error line, but it can't promise a whole recent turn stays intact while an older hello-world turn goes. That's the gap your shape closes.
Two things I'm curious about, since you've actually built this in CodeCook: what counts as a turn boundary for you — a user message, a tool-call round, something else? And where does the pinned plan sit in that shape — does pin override turn summarization wherever the plan line appears, or does the plan live as its own kept block outside the turn sequence? If role/time boundaries were the fallback instead of full turn parsing, would that be enough to avoid the same risk, or would it just move it?
In CodeCook a turn is the user message plus the assistant response that follows it, including any tool-call rounds nested in that response — so summarizing older turns means whole exchange blocks, not single messages. The pinned plan lives as its own kept block outside the turn sequence: it isn't aged out with old turns, and it stays verbatim even when the turns around it collapse. Role/time alone would catch a lot of the failing-test case (you'd keep the newest chunks), but without turn grouping you can still split a tool-call mid-flight — keep the error line, drop the user ask that caused it — so I'd still want real turn boundaries if the tool is going to make promises a reader can check against the transcript.
The mid-flight split is the sharper version of the failing-test example I keep reaching for. Keep the error line, drop the user ask that caused it, and the summary still looks honest line by line — but nobody can check it against the transcript anymore. That's the point where the manifest's claim stops being a receipt.
So the turn boundary isn't really about recency for you; it's what makes "summarized" a checkable claim. A role/time fallback can keep the newest chunks, but without the grouping it can certify a split.
And the pinned plan answers the question I left hanging — it doesn't override the summarizer, it just never enters the turn sequence, so it never competes with the recency policy. Out of curiosity, how does CodeCook decide something is the plan — is it marked by the user, or does the tool recognize plan-shaped content on its own?
The manifest is what sold me. "169 = 23 + 54 + 4 + 88" is the kind of line you can actually review.
Two things from our own sessions that might be worth a look.
1. Verbatim cuts both ways.
Because every kept line is copied as is, a secret in the input survives as is too.
And the lines diet_summarize keeps on purpose (errors, failures) are exactly where connection strings and Authorization: Bearer … headers tend to show up.
Secret redaction is on your roadmap. When it lands, it would be great to see it as its own manifest category, something like "3 lines masked as credentials". That way the receipt stays complete.
2. Fetched web pages are another big accumulation source.
Every time an agent reads docs, the raw HTML lands in context. Most of it is navigation, scripts and markup.
Disclosure: we run Distill, a URL-to-Markdown API that keeps the main content and doesn't rewrite it. With the same chars/4 estimate you use, from real calls today:
Wikipedia's "Model Context Protocol" page: ~66,400 tokens of HTML, ~5,600 as Markdown.
The nodejs.org intro to Node.js: ~21,200 → ~1,100.
Python's asyncio task docs: ~44,300 → ~14,700.
Dense reference pages shrink much less, which feels right.
Curious whether you've seen fetched pages in your session logs, or whether they're mostly test and build output.
Thanks — the manifest line is doing the work you describe, and both points press on where it could break. On verbatim: you're right that kept-as-is cuts both ways. A secret that arrives in the input survives in the kept lines, and error/failure lines are exactly where credentials tend to sit. A masked-as-credentials manifest category — whether that masking happens in the tool or in a caller-side pass before it — would keep the receipt complete, the count still adding up, while the content itself doesn't travel. The question I'd turn over is what the masked line still proves: if the content is hidden, the auditor can verify the count but not the line itself, so is the receipt still evidence, or does it become a claim about evidence? On fetched pages: Context Diet only ever sees the text it's handed, so HTML-to-Markdown would sit before the tool, not inside it — the caller decides what text arrives. Your numbers are a useful shape for that pre-pass though: the shrink varies a lot by page density, which suggests the honest unit isn't a single ratio but the same conservation line, tokens in vs tokens kept, applied at that stage too. Curious: in your own sessions, are fetched pages a large share of what accumulates, or are test and build logs still the bulk?
One option: put a hash of the masked line in the manifest. The content doesn't travel, but anyone who has the original can recompute it and check that the receipt matches. We do the same with keys: we compare them by length and a SHA-256 prefix, never by value.
So it stays evidence, just evidence you can only verify from the source side.
On fetched pages: honestly, we haven't measured our own sessions. Our impression is that CLI and CI output (JSON from APIs, build logs) is still the bulk. Fetched pages come in bursts, when we're researching something.
Your "conservation line at every stage" framing is a nice way to put it.
That hash answer settles the question I left open, in the direction that matters. A masked line plus its hash keeps the receipt checkable: anyone holding the original can recompute it and confirm the count refers to a real line, without the content itself traveling. The receipt stops being a bare claim and becomes evidence again — evidence with a deliberately narrow verification path, since only the source side can check it.
One wrinkle worth keeping in view: a hash of a short, low-entropy secret is itself guessable, so the masked category would still want care about what it hashes. But as a shape for the category, hash-plus-count is cleaner than count alone.
And thanks for the honest answer on fetched pages — 'we haven't measured' plus the burst pattern is more useful than a guessed share. Bursts when researching, bulk still CLI and CI output: that fits what the tool was built for, the accumulation that repeats every run rather than the one-off research spike.
If you ever do measure that split over a full week, I'd be curious whether the bursts or the steady output dominate — that's the number that would decide whether a pre-pass like the one you described earns its place.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Accumulation has a cousin on the input side: the tool definitions themselves. I counted 20 MCP tools this week and they ranged from roughly 2K to 32K tokens each before a single call, and in the worst one nearly all of it was an output schema nobody wrote by hand. Might be worth a line in the manifest if you ever count tools too.
That's a sharp cousin to the accumulation problem — and it's one the tool can't see from inside a run. Context Diet only ever sees the text it's handed, so it has no way to count the definitions sitting loaded on the input side; your 2K–32K range is exactly the kind of silent pre-call cost the manifest arithmetic was built to make visible, just on the other side of the call.
If that counting ever happened, the honest shape would probably be the same conservation line — N tools loaded, X tokens of definitions before the first call — rather than a claim about what any of it means. Curious: in your count, was the 32K worst case mostly auto-generated schema, and does the number that matters to you sit per session (everything loaded) or per call (only what the model actually reaches for)?
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Yes, 32,136 of the 32,737 was the output schema the hosted server inferred from sample rows. Per session is the number I track: once a definition is loaded it rides along on every turn, mostly as cache reads. Claude Code defers loading until the agent looks the tool up, which moves the cost to first use rather than removing it.
That split is the useful part — 32,136 of 32,737 being inferred schema says the cost isn't the tools you chose, it's the shape the server hands you. And per session makes sense on your reasoning: once it's loaded it rides every turn, even if mostly as cache reads, so per call would understate it.
The Claude Code deferral is the interesting wrinkle: moving the cost to first use rather than removing it. When it lands on first lookup, does it land in full, or only the slice the agent actually reaches for? That decides whether deferral is a real saving or just a bill that arrives later.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
This framing landed for me — I've watched sessions go sideways not on one huge dump, but on the tenth rerun of the same failing test, exactly like you describe. The manifest is what makes it usable for me: 169 lines down to 23 is nice, but knowing 54 were deduped and 88 summarised away is what I'd actually trust. One thing I'm curious about: when you run diet_extract with a query like 'test failure', how much surrounding context do you keep around a hit? I've been burned before by an excerpt that kept the error line but dropped the setup line just above it.
Good catch — and your exact failure mode is real. diet_extract keeps 2 lines either side of each hit by default (windowLines = 2 in src/extract.ts, overlapping windows merged, top 10 chunks in original order). So a setup line sitting right above the error line survives, but one sitting 3+ lines up gets dropped unless it scores on its own — matching a query term, or tripping the error/heading patterns. That gap between 'the line that failed' and 'the line that set it up' is exactly the shape your burn describes. What did your setup separation usually look like — the line directly above, or further up behind a blank line or two? Curious where you'd draw the window if you were tuning it.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
The manifest is what makes this trustworthy, but I would add one rule for test logs: dedupe by failure signature rather than by exact line. In serving and test logs, the same failure often reappears with a new timestamp, request id, or port, so exact matching keeps noise while still hiding the pattern. Keeping the first occurrence verbatim, with its surrounding lines and a count of how many times that signature recurred, would preserve the diagnostic signal without letting repeated runs eat the budget.
That's a sharp extension of the near-duplicate case from this thread — the address-varying repeats shipped in v0.2.0 as masked-key dedupe, first occurrence verbatim with a count, but a timestamp or request id changing every run is a wider net than an address. Would your signature be the normalized line itself — strip the volatile fields and compare what's left — or something coarser, like the failing assertion plus its surrounding lines? The surrounding-lines part matters too: the first occurrence alone can lose the setup line above the error, which was exactly the failure mode in the extract discussion. Curious which half you'd weight heavier, the signature or the context around it.
The key insight here is that context management is about preserving useful information, not just reducing token count. I’d add that any compression layer should protect critical requirements and failure details while making every removal auditable. At IT Path Solutions, this is an important consideration when building AI-powered workflows: losing one small constraint can lead to a completely different result. A useful next step would be allowing developers to mark specific rules or sections as protected so they’re never silently dropped when the context budget gets tight.
Thanks, Glen — that's exactly the shape this landed in. The protect-lines idea shipped in v0.3.0 as protectLines: the caller supplies the lines verbatim, they're held at priority 0, ahead of even error lines, and if they don't fit the budget the default is to fail rather than silently drop them — there's a partial mode that keeps everything and flags the shortfall, but it's for diagnosis only, never reported as a successful compression. The manifest reports which protected lines were kept and which couldn't fit, so nothing protected disappears quietly. The failure-detail half you mention is already there in the other direction: error lines are priority 1 in summarize. The question I'd still put back to you, from your IT Path Solutions side: when you say 'mark specific rules or sections', do you picture the marking living in the repo docs where the decision already lives, or in a list the caller passes to the tool? The two point at different owners for which lines count.
The manifest is the part that makes this trustworthy. Showing that 169 items became 23 kept, 54 deduped and the rest summarized means someone can check what was dropped instead of hoping nothing important went missing. Keeping the process deterministic and extractive helps for the same reason.
Paul Irolla's observation matches what we see on client projects. The large file is rarely the problem. It is forty small tool results, and test logs in particular, piling up across a session until the useful instructions are buried.
Launch Gate's comparison with double entry bookkeeping is a good way to explain planned versus executed checks to people outside engineering. If every planned step has a matching record, a missing one is visible at once.
Odinaka's line about the same fix meaning the same failure is worth turning into a dedupe rule. Have you tried grouping repeated failures by the fix applied rather than by the error text?
Thanks for pulling those three threads together — that synthesis is exactly the shape of the discussion so far, and the fix question is the right one to press on.
The honest answer is no, not as a grouping key in the tool — and the reason is where the fix lives. A log line carries the error text, not the fix applied afterwards, so the tool today has nothing to group by except what is in the lines themselves. What shipped in v0.2.0 is the closest neighbour, not the same thing: traces that differ only by a 0x address collapse, first occurrence kept verbatim, with the near count kept separate — and it deliberately does not claim 'same fix', because address-equality is not fix-equality.
Grouping by fix would need the fix to arrive as an annotation from outside the log — a caller tagging each failure with the fix it got — and then the grouping itself is easy, but the judgment has moved outside the tool, which is the same tension this thread has hit with caller-marked ranges. That raises the question I'm curious about: in practice, where would that fix tag come from for you — a post-run annotation, a link to the commit that fixed it, something the runner records at the time? And the boundary case that follows: two failures that share a fix but come from different call sites — group them, or keep them apart because the location still matters?
The manifest-as-diff framing is the part I'd bet on — most compression tooling fails not because it cuts the wrong lines but because you can't prove it didn't invent one. Question on the boundary cases: the 88 lines summarized away keep first/last lines per section and error lines, but what about repeated-but-slightly-different lines, e.g. stack traces that differ only by a memory address? Exact-duplicate dedupe misses those and they're usually most of a long retry log. Is that something the block-repeat detection catches, or is it on the roadmap?
Good catch — and the honest answer is: today, neither stage catches them.
diet_dedupe is exact-only: it removes exact duplicate lines and repeated blocks of 3–8 lines (see src/dedupe.ts). A stack trace that differs only by a memory address is a different string, so it survives dedupe as a distinct line, and a repeated block with one address changed survives as a distinct block too.
Summarize doesn't rescue that case either: error lines are priority 1, so within budget they're all kept verbatim — the right instinct for a single failure, but the wrong shape for a retry log where the same failure repeats with a new address each time.
The direction that seems honest to me would be a normalization step before dedupe keying — mask volatile tokens (hex addresses, timestamps) only for the comparison key, keep the first occurrence verbatim, and record in the manifest that those lines were collapsed as near-duplicates rather than exact duplicates, so the 'no invention' claim stays auditable. But the masking list is where it gets delicate: mask too much and two genuinely different failures collapse into one.
What would you put on the mask list — addresses only, or timestamps and PIDs too? And would first-occurrence-verbatim plus a count satisfy the audit side for you, or would you want the masked variants listed somewhere?
Closing the loop on your question: this shipped in v0.2.0. diet_dedupe now normalizes before keying — 0x-prefixed hex addresses are masked for the comparison key only, the first occurrence stays verbatim, and near-duplicate collapses are counted separately in the manifest so they don't masquerade as exact matches. There's an --exact-only opt-out if you want the old behavior. Honest limit: bare hex without 0x, timestamps, and PIDs stay exact — nobody confirmed those for the mask list, and they can carry identity or ordering meaning. Change: github.com/techdevfun/context-diet... — if you run it against your traces, I'd love to know whether the first-occurrence-plus-count shape is enough for your audit side, or what else you'd want recorded.
@techdevfun Test logs, I'd say. A green run repeats the same lines every time, and a red one buries the failing assertion in the middle. Greps and listings are small one at a time, but they come far more often.
Test logs taking that vote makes sense — a green run is pure repetition, and a red run buries the one line that matters in the middle. That split is why dedupe does the heavy lifting before anything gets summarised: the repeated green lines collapse, and the failing assertion is exactly the line that has to survive. Which end do you find harder in practice — the red run, where the assertion is buried, or the green run, where everything looks identical until it suddenly isn't?
Usually it isn't one big file. It's small tool results nobody trims: a grep, a test log, a directory listing. Each one stays in the history until the session ends, so the window fills with output nobody looked at.
Exactly — death by a thousand small outputs, not one big file. A grep here, a directory listing there, and none of them ever get thrown away. Which small output fills your sessions fastest — test logs, or the greps and listings?
The manifest accounting is what sold me: "169 = 23 kept + 54 deduped + 4 extracted + 88 summarized" is something you can actually diff, unlike the compression percentages from the rivals you list.
One thing I kept wondering: exact-duplicate dedupe rarely fires on real logs, because every line carries a timestamp or run id. Does diet_dedupe normalize those before comparing, or does that fall to diet_extract? In my own agent session logs the accumulation is mostly near-duplicates, the same failing test re-pasted across ten retries with only the timestamp changing, and exact matching would miss almost all of it.
Good catch, and the honest answer is: today it would miss your case. diet_dedupe normalizes only 0x-prefixed memory addresses before comparing, so repeated stack traces differing only by address collapse as near-duplicates, with the first occurrence kept verbatim and the removal counted separately as near, not exact. Timestamps and PIDs are deliberately left exact in the code, on the grounds they can carry ordering or identity meaning, and no commenter had confirmed them for the mask list until now. diet_extract doesn't normalize anything either, it just scores lines and takes windows around matches, so it wouldn't rescue the timestamp-only repeats. So ten retries of the same failing test differing only by timestamp would survive dedupe as ten lines today. Is timestamp-masking the right call here, or would stripping the one field that orders the retries lose something you'd want kept?
Building in public with local AI agents. I hand my workday to an agent that runs on my own machine - screen, shell, browser, phone over ADB. https://agent-home.ai
The manifest-as-receipt approach is the part most "token optimizer" tools skip, and it's the right call — an unauditable rewrite of a build log is worse than a slightly bigger prompt. One thing worth stress-testing: your chars/4 estimate tends to undercount logs and stack traces, where short tokens (paths, symbols, hex) dominate, so a 400-token budget can land noticeably larger on a real tokenizer. Also, how does diet_dedupe treat stack frames that differ only by line number — collapsing those could erase the exact offset that points at the failure.
Thanks — and on the dedupe question, the current boundary already protects the offset you're worried about. The comparison key masks only 0x-prefixed addresses; line numbers aren't masked, so two frames differing only by line number have different keys and both lines survive — the kept text is always the first occurrence verbatim, so the exact offset stays in it. That's the same deliberate boundary as timestamps and PIDs, which stay exact because they can carry identity or ordering meaning. On the token estimate, the honest position is the same: chars/4 is a labeled estimate, not a measured count, and no measured comparison has been published — and your undercount direction for logs sounds right. The case I'm curious about: is there a log shape where the line number itself is the volatile part — a rebuilt binary shifting lines, say — where collapsing across line numbers would be the right call rather than a loss?
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The first thing I'd check is the chars/4 estimate on logs. Hex addresses, file paths and stack frames usually split into more tokens than prose, so a 400-token budget could land well over 400 on a real tokenizer. Have you compared the two on the session.log example?
Good challenge, and the honest answer is no — I haven't published a measured comparison on the session.log example. The tool is explicit about this in code: token counts are a chars/4 estimate, always labeled estimated, never quoted as measured. Your direction of bias sounds right to me: logs tokenize denser than prose, with short tokens from paths, symbols, and hex, so a 400-token estimated budget could land noticeably over 400 on a real tokenizer. What would settle it for you — a measured-token check reported alongside the estimate for the demo log, or a stated calibration range for logs versus prose? I'm curious which one you'd actually trust in practice.
Building in public with local AI agents. I hand my workday to an agent that runs on my own machine - screen, shell, browser, phone over ADB. https://agent-home.ai
The manifest is the right instinct: most context shavers report a percentage and then you cannot tell whether the summary dropped the one assertion that mattered, and "kept 23 / dropped 54 as duplicates" is a claim you can diff. One thing I would push on is the dedupe layer — exact-line dedupe is cheap, but the accumulation I actually see is near-duplicate re-reads, where a file comes back with two lines changed and the whole body is stored again as a new copy, so tracking last-read-per-path and returning a delta would spend fewer tokens than any summariser.
For the extractive pass, the rule that worries me is "first and last line of each section": on a stack trace that keeps the header and the tail while dropping the middle frames, which is exactly the causal chain an agent needs. Productising a protected-pattern list (keep lines matching /^\s*at |Caused by:|AssertionError/) would let callers declare what must survive rather than trusting a fixed policy. Or is the budget pressure low enough that you would rather keep the policy fixed and auditable?
That's a sharper version of the re-read problem than exact-line dedupe was built for. v0.2.0's masked-key dedupe catches the same trace with a different address, but a file that comes back with two lines changed is a new body to a line-based tool — the whole thing gets stored again, which is exactly the accumulation shape the article is about. A last-read-per-path delta would spend fewer tokens than any summariser; the honest tension is that the tool today is stateless, it sees lines, not paths or previous runs, so the delta needs somewhere to live. Would you put that state in the tool, or with the caller who already knows what it read last?
On the extractive side, the middle-frames worry lands the same way: keeping the header and the tail preserves the frame that names the failure and the one that ends it, but the causal chain in between is what an agent actually needs to walk. The pattern list you suggest — keep lines matching at / Caused by: / AssertionError — is caller-declared protection in the same family as protectLines in v0.3.0, but by pattern instead of verbatim lines, and that's a real difference: verbatim lines can't over-keep, while a pattern like 'at ' could match every frame in a long trace. Would you bound the list to a small caller-supplied set, or does that over-keeping risk not worry you when the alternative is dropping the chain?
Curious which half you'd weight heavier — the delta for re-reads, or the pattern list for the extractive pass — and whether you've seen a re-read shape where the delta alone would have beaten the summariser.
@techdevfun A single assertion buried mid-log. Any rule that keeps the tail gets a red run right when the failure sits at the end, so the mid-log case is the one that actually tests the summary.
That's the sharper pick — and the reason it bites is exactly what you said: a tail rule passes the easy case for free. A mid-log assertion is the one place where a summarizer has to prove it kept the right line, not just the last ones. It also draws a useful line for this tool: error lines are priority 1 in summarize, so a failing assertion phrased as an error should survive — but a plain expect line buried mid-log is the harder case, and the one that would actually test whether the summary is doing selection rather than truncation. Which shape is your canary — an error line, or a plain assertion that has to be recognized as the one that matters? If you've got a real red log where that distinction shows up, I'd love to see what it looks like.
@techdevfun Red, and it isn't close. A green run collapses to one line once the repeats go. A red run is where a summary can lose the one assertion that matters, so that's the end I'd test first.
Red first makes sense — and it's the sharper test for exactly the reason you give. A green run collapsing down is the shape dedupe already handles. A red run is where the cost of getting it wrong is real, because the one assertion that matters is the line a summary could drop.
That also frames the test nicely: a red log where the failing assertion survives trimming, while everything around it collapses. If you were picking the canary for that test, what would it be — a single assertion buried mid-log, or a red run where the failure only shows up in the last few lines?
@arcwrightai Exactly — that's why every cut is accounted for in the manifest. The answer is only half the work; being able to verify what was kept and what was dropped is the other half.
We're a place where coders share, stay up-to-date and grow their careers.
Context Diet is an elite name lol
Ha, thanks! Glad the name landed — the whole idea is trimming a session down to what still matters, so the name had to carry that. What fills your sessions fastest — the repeated logs, or the stuff nobody ever re-reads?
repeated logs, and nobody re-reads them.
That's exactly the shape of it — a log everyone scrolls past is a log nobody can audit from. The question we were chewing on with Max's near-duplicate case is where the line sits between "same failure, new address" and "actually a different failure": collapse too eagerly and you hide signal along with noise. Where do you land on that line? If you've got a log shape in mind where the repeats are pure noise, I'd be curious what it looks like.
if the fix is the same it's the same failure to me. new fix, new line.
That's a crisp rule — identity by the fix, not by the line. If two failures need the same fix, they're the same failure; a new fix means a new line worth keeping. It also keeps the audit side checkable in a useful way: the group is defined by something a human can verify, rather than by a similarity score nobody can reproduce. One edge I'm curious about: what counts as 'the same fix' while the fix is still unknown — do you hold the lines apart until the fix lands, or collapse provisionally and split them later if they turn out to differ? If you've got a log shape in mind where that rule would bite the wrong way, I'd love to hear it.
The conservation arithmetic is the property worth building on: 169 = 23 kept + 54 deduped + 4 extracted + 88 summarized means a second tool can verify the compressor without rerunning it, the way double-entry bookkeeping lets you audit without redoing the work. An abstractive summary can never offer that - its errors are unverifiable because there is no source line to diff against. Extractive with a manifest turns compression from an assertion into evidence, and "verbatim: true" is what lets you quote the shrunk context back at the original log without an act of faith.
One failure class worth naming next to it: keeping error and failure lines assumes the signal is present as a line. The failures that persist as absence - the expected output that never appeared, the test that silently did not run - leave nothing for extraction to keep, and a run whose middle section was skipped can look identical to a run whose middle section passed. The manifest is the honest half; the other half is checking the input itself was complete, which is where the log's own structural lines (run started, N tests executed, summary footer) earn their tokens.
Sharp point — the manifest audits what was there, not what was missing. A skipped middle section can look identical to a passed one. Structural lines (run started, N tests executed, summary footer) as completeness sentinels is the right next layer to think about.
Exactly - and the sentinels get stronger when they arrive in pairs. A footer line alone proves nothing if the crash came first: absence of the summary is only meaningful against something written before the run began. "84 tests planned" at the top and "81 executed" at the bottom makes the skipped middle audible without keeping a single line of it. Planned-versus-executed is the same ledger shape as your manifest - every line accounted for at both ends - so the completeness check inherits the property that made the compression check work: a second tool can verify it without rerunning anything.
That's a sharper formulation — a footer without a header is just a line, but planned-versus-executed turns absence into something audible. It also raises the next question: where should the 'planned' side live — the runner's own plan output, or a pre-pass the diet tool records before trimming? Which would you trust more as the 'before' record?
Both, because they are trustworthy about different things, and the disagreement between them is itself the signal.
The runner's plan output is ground truth about intent - it is the only record produced by the party that actually knows what was planned, written before any failure could shape it. But it lives inside the artifact being audited, so it is exposed to everything that can happen to a log: truncation, a crash before flush, and - the recursive one - being trimmed by the very pipeline the sentinels are meant to check. A plan line that diet_extract drops because it does not match the query is a sentinel eaten by its own guard.
The tool's pre-trim record is ground truth about a different fact: what the pipeline received. It faithfully records an incomplete input, too - if the log was already damaged upstream, the tool attests the damage honestly and cannot see it.
So: runner writes intent, tool attests transmission. Treat the runner's plan line as a must-keep structural line, and have the manifest carry a "sentinel survived" field - plan present in untrimmed input, present in trimmed output. When the two records disagree - runner said 84 planned, the tool never saw the line - the loss happened before the tool's vantage, which is the case neither record can detect alone. Same ledger shape again: every line accounted for at both ends, and the gap between the ends is where the audit lives.
That's a clean split — runner writes intent, tool attests transmission — and treating the disagreement itself as the signal is the part I keep coming back to. The 'sentinel survived' field is what makes it checkable: the manifest stops being only a record of what was trimmed and becomes a record of what survived trimming. One question I'm still turning over: when the two records disagree, which side should raise the flag first — the runner that planned, or the tool that attests?
Interesting project! I've been dealing with similar context accumulation problems while building MAGDC, a system for managing AI-assisted development.
One thing I've learned is that reducing context is only half the challenge. Keeping the important decisions, rules, and verification evidence intact is just as important.
I like your approach to tracking exactly what gets removed.
Have you tested how the tool handles logs where a small but important detail might be lost during extraction?
Thanks — and that's the exact half the manifest is built to protect. Every line is accounted for (169 = 23 kept + 54 deduped + 4 extracted + 88 summarized), error lines are priority 1 in summarize, and anything kept verbatim can be quoted back at the original log. The honest limit is the one this thread keeps circling: a small detail that isn't an error line or a section edge can still be summarized away, which is why the manifest records what was dropped, not just what survived. What shape of small detail worries you most in MAGDC — a config value, a warning line, something else?
Thanks for explaining!
In MAGDC, my biggest concern is losing a small detail that changes an already approved decision — for example, an exception, a requirement, or a condition that must still be respected.
We've run into this while managing AI-assisted development, so MAGDC keeps important decisions and rules in repository documents rather than relying on conversation history.
We're also working on reducing context without losing mandatory rules.
Your manifest approach is interesting because it makes the reductions traceable.
Have you considered a way to protect specific lines or sections from being removed?
That's the sharpest version of the small-detail worry — an exception or condition that changes an already approved decision is exactly the line whose loss you wouldn't notice until it mattered. Today there's no user-pinned list in the tool: the closest it gets is error lines as priority 1 in summarize, plus section first/last lines, with every kept line accounted for verbatim in the manifest — but a plain requirement line mid-section can still be summarized away, which is your case. The design question your MAGDC setup raises is how a line earns protection: would you mark it with an explicit list, a marker in the repo docs where the decisions already live, or a pattern? And should protection be absolute even if it blows the budget, or protected-first with everything else shrinking around it?
Good questions!
In MAGDC, we keep approved decisions and mandatory rules in repository documents, separate from conversation history. Which rules need to be loaded depends on the task.
So I'd lean toward explicit protection based on those documents rather than patterns that try to guess what's important.
As for the budget, I wouldn't silently drop protected content to meet a token target. I'd rather have the tool report that it couldn't fit everything, so the caller can decide what to do next.
Not every line needs protection, of course. The challenge is knowing which ones are mandatory for the current task.
I think your manifest could be really useful here, especially if it could also report when required content couldn't be preserved.
That's a clear pair of answers — and the second one is the sharper of the two. Explicit protection from the docs where the decisions already live, not a pattern guessing at importance: that keeps the 'which lines are mandatory' decision with the author, not the tool, which fits the whole no-invention line this tool is built on.
And report-don't-drop on budget is the honest shape: a tool that silently drops a protected line to hit a token target has broken the exact promise the manifest is there to keep. Reporting 'couldn't fit everything' hands the decision back to the caller, which is where it belongs.
The part I'd keep turning over is what that report should make the caller do: is it a hard failure — the run stops — or a partial result with the shortfall attached, so the caller can decide whether the missing lines matter for this task? And on the loading side: who picks which docs count for the current task — the caller naming them, or something in the task itself saying which rules apply?
The accumulation framing is the useful correction — people blame the one fat file, then the session dies on the tenth retry of the same test log. The manifest idea (kept / deduped / extracted / summarized, with every kept line verbatim) is what makes me trust a shrink step. In CodeCook, the VS Code agent I'm building, we attack the same pile from the other end: summarize older chat turns instead of shipping every tool output forward. Curious whether you've tried diet_pack on a full multi-turn agent transcript, or mostly on single log dumps so far?
Good question — and the honest answer is mostly single log dumps so far. The published demo is one session.log (169 lines packed to a 400-token estimated budget), and the tool itself is input-agnostic: it treats whatever text you feed it as lines, dedupes, extracts around a query, and summarizes down to budget, with the manifest recording what was kept verbatim. So a multi-turn transcript would pack, but nothing in the tool understands turns — no speaker boundaries, no 'older turns first' policy like the one you describe for CodeCook. That raises a question I'm curious about: does turn-aware packing need turn boundaries preserved as structure, or is line-based treatment enough as long as the manifest stays honest about which turns were summarized rather than kept? And when older turns get summarized, does the verbatim-kept guarantee still mean the same thing to you, or does it change what the manifest is promising?
For CodeCook it's turn structure, not line packing. Older turns collapse into a summary; recent turns stay verbatim. The "kept" promise is really about those recent turns plus anything we pin (the plan), not every line of history. A pure line-based diet_pack on a transcript would still shrink it, but without turn boundaries you'd risk summarizing the last failing test log while keeping an older hello-world turn. So I'd want turn (or at least role/time) boundaries in the structure — the manifest can still say "turns 1–40 summarized, 41–45 kept verbatim." Does that match how you'd extend the tool, or would you rather keep it line-agnostic and let the caller mark ranges?
That's the distinction I was missing — thanks for spelling it out.
Turn boundaries as structure feels like the honest direction to me, over staying line-agnostic and letting the caller mark ranges. Caller-marked ranges push the boundary judgment outside the tool, so the manifest can only report what the caller claimed, not what the tool understood. If turns are structure, the manifest can say exactly what you wrote — turns 1–40 summarized, 41–45 kept verbatim — and a reader can check that claim against the transcript itself.
And your failing-test example lands because the tool today has no way to know recency at all. It sees lines, not turns, so 'keep the recent turn verbatim' isn't something it can even express — line priority can keep an error line, but it can't promise a whole recent turn stays intact while an older hello-world turn goes. That's the gap your shape closes.
Two things I'm curious about, since you've actually built this in CodeCook: what counts as a turn boundary for you — a user message, a tool-call round, something else? And where does the pinned plan sit in that shape — does pin override turn summarization wherever the plan line appears, or does the plan live as its own kept block outside the turn sequence? If role/time boundaries were the fallback instead of full turn parsing, would that be enough to avoid the same risk, or would it just move it?
In CodeCook a turn is the user message plus the assistant response that follows it, including any tool-call rounds nested in that response — so summarizing older turns means whole exchange blocks, not single messages. The pinned plan lives as its own kept block outside the turn sequence: it isn't aged out with old turns, and it stays verbatim even when the turns around it collapse. Role/time alone would catch a lot of the failing-test case (you'd keep the newest chunks), but without turn grouping you can still split a tool-call mid-flight — keep the error line, drop the user ask that caused it — so I'd still want real turn boundaries if the tool is going to make promises a reader can check against the transcript.
The mid-flight split is the sharper version of the failing-test example I keep reaching for. Keep the error line, drop the user ask that caused it, and the summary still looks honest line by line — but nobody can check it against the transcript anymore. That's the point where the manifest's claim stops being a receipt.
So the turn boundary isn't really about recency for you; it's what makes "summarized" a checkable claim. A role/time fallback can keep the newest chunks, but without the grouping it can certify a split.
And the pinned plan answers the question I left hanging — it doesn't override the summarizer, it just never enters the turn sequence, so it never competes with the recency policy. Out of curiosity, how does CodeCook decide something is the plan — is it marked by the user, or does the tool recognize plan-shaped content on its own?
The manifest is what sold me. "169 = 23 + 54 + 4 + 88" is the kind of line you can actually review.
Two things from our own sessions that might be worth a look.
1. Verbatim cuts both ways.
Because every kept line is copied as is, a secret in the input survives as is too.
And the lines
diet_summarizekeeps on purpose (errors, failures) are exactly where connection strings andAuthorization: Bearer …headers tend to show up.Secret redaction is on your roadmap. When it lands, it would be great to see it as its own manifest category, something like "3 lines masked as credentials". That way the receipt stays complete.
2. Fetched web pages are another big accumulation source.
Every time an agent reads docs, the raw HTML lands in context. Most of it is navigation, scripts and markup.
Disclosure: we run Distill, a URL-to-Markdown API that keeps the main content and doesn't rewrite it. With the same chars/4 estimate you use, from real calls today:
Dense reference pages shrink much less, which feels right.
Curious whether you've seen fetched pages in your session logs, or whether they're mostly test and build output.
Thanks — the manifest line is doing the work you describe, and both points press on where it could break. On verbatim: you're right that kept-as-is cuts both ways. A secret that arrives in the input survives in the kept lines, and error/failure lines are exactly where credentials tend to sit. A masked-as-credentials manifest category — whether that masking happens in the tool or in a caller-side pass before it — would keep the receipt complete, the count still adding up, while the content itself doesn't travel. The question I'd turn over is what the masked line still proves: if the content is hidden, the auditor can verify the count but not the line itself, so is the receipt still evidence, or does it become a claim about evidence? On fetched pages: Context Diet only ever sees the text it's handed, so HTML-to-Markdown would sit before the tool, not inside it — the caller decides what text arrives. Your numbers are a useful shape for that pre-pass though: the shrink varies a lot by page density, which suggests the honest unit isn't a single ratio but the same conservation line, tokens in vs tokens kept, applied at that stage too. Curious: in your own sessions, are fetched pages a large share of what accumulates, or are test and build logs still the bulk?
Good question on the masked line.
One option: put a hash of the masked line in the manifest. The content doesn't travel, but anyone who has the original can recompute it and check that the receipt matches. We do the same with keys: we compare them by length and a SHA-256 prefix, never by value.
So it stays evidence, just evidence you can only verify from the source side.
On fetched pages: honestly, we haven't measured our own sessions. Our impression is that CLI and CI output (JSON from APIs, build logs) is still the bulk. Fetched pages come in bursts, when we're researching something.
Your "conservation line at every stage" framing is a nice way to put it.
That hash answer settles the question I left open, in the direction that matters. A masked line plus its hash keeps the receipt checkable: anyone holding the original can recompute it and confirm the count refers to a real line, without the content itself traveling. The receipt stops being a bare claim and becomes evidence again — evidence with a deliberately narrow verification path, since only the source side can check it.
One wrinkle worth keeping in view: a hash of a short, low-entropy secret is itself guessable, so the masked category would still want care about what it hashes. But as a shape for the category, hash-plus-count is cleaner than count alone.
And thanks for the honest answer on fetched pages — 'we haven't measured' plus the burst pattern is more useful than a guessed share. Bursts when researching, bulk still CLI and CI output: that fits what the tool was built for, the accumulation that repeats every run rather than the one-off research spike.
If you ever do measure that split over a full week, I'd be curious whether the bursts or the steady output dominate — that's the number that would decide whether a pre-pass like the one you described earns its place.
Accumulation has a cousin on the input side: the tool definitions themselves. I counted 20 MCP tools this week and they ranged from roughly 2K to 32K tokens each before a single call, and in the worst one nearly all of it was an output schema nobody wrote by hand. Might be worth a line in the manifest if you ever count tools too.
That's a sharp cousin to the accumulation problem — and it's one the tool can't see from inside a run. Context Diet only ever sees the text it's handed, so it has no way to count the definitions sitting loaded on the input side; your 2K–32K range is exactly the kind of silent pre-call cost the manifest arithmetic was built to make visible, just on the other side of the call.
If that counting ever happened, the honest shape would probably be the same conservation line — N tools loaded, X tokens of definitions before the first call — rather than a claim about what any of it means. Curious: in your count, was the 32K worst case mostly auto-generated schema, and does the number that matters to you sit per session (everything loaded) or per call (only what the model actually reaches for)?
Yes, 32,136 of the 32,737 was the output schema the hosted server inferred from sample rows. Per session is the number I track: once a definition is loaded it rides along on every turn, mostly as cache reads. Claude Code defers loading until the agent looks the tool up, which moves the cost to first use rather than removing it.
That split is the useful part — 32,136 of 32,737 being inferred schema says the cost isn't the tools you chose, it's the shape the server hands you. And per session makes sense on your reasoning: once it's loaded it rides every turn, even if mostly as cache reads, so per call would understate it.
The Claude Code deferral is the interesting wrinkle: moving the cost to first use rather than removing it. When it lands on first lookup, does it land in full, or only the slice the agent actually reaches for? That decides whether deferral is a real saving or just a bill that arrives later.
This framing landed for me — I've watched sessions go sideways not on one huge dump, but on the tenth rerun of the same failing test, exactly like you describe. The manifest is what makes it usable for me: 169 lines down to 23 is nice, but knowing 54 were deduped and 88 summarised away is what I'd actually trust. One thing I'm curious about: when you run diet_extract with a query like 'test failure', how much surrounding context do you keep around a hit? I've been burned before by an excerpt that kept the error line but dropped the setup line just above it.
Good catch — and your exact failure mode is real. diet_extract keeps 2 lines either side of each hit by default (windowLines = 2 in src/extract.ts, overlapping windows merged, top 10 chunks in original order). So a setup line sitting right above the error line survives, but one sitting 3+ lines up gets dropped unless it scores on its own — matching a query term, or tripping the error/heading patterns. That gap between 'the line that failed' and 'the line that set it up' is exactly the shape your burn describes. What did your setup separation usually look like — the line directly above, or further up behind a blank line or two? Curious where you'd draw the window if you were tuning it.
The manifest is what makes this trustworthy, but I would add one rule for test logs: dedupe by failure signature rather than by exact line. In serving and test logs, the same failure often reappears with a new timestamp, request id, or port, so exact matching keeps noise while still hiding the pattern. Keeping the first occurrence verbatim, with its surrounding lines and a count of how many times that signature recurred, would preserve the diagnostic signal without letting repeated runs eat the budget.
That's a sharp extension of the near-duplicate case from this thread — the address-varying repeats shipped in v0.2.0 as masked-key dedupe, first occurrence verbatim with a count, but a timestamp or request id changing every run is a wider net than an address. Would your signature be the normalized line itself — strip the volatile fields and compare what's left — or something coarser, like the failing assertion plus its surrounding lines? The surrounding-lines part matters too: the first occurrence alone can lose the setup line above the error, which was exactly the failure mode in the extract discussion. Curious which half you'd weight heavier, the signature or the context around it.
The key insight here is that context management is about preserving useful information, not just reducing token count. I’d add that any compression layer should protect critical requirements and failure details while making every removal auditable. At IT Path Solutions, this is an important consideration when building AI-powered workflows: losing one small constraint can lead to a completely different result. A useful next step would be allowing developers to mark specific rules or sections as protected so they’re never silently dropped when the context budget gets tight.
Thanks, Glen — that's exactly the shape this landed in. The protect-lines idea shipped in v0.3.0 as protectLines: the caller supplies the lines verbatim, they're held at priority 0, ahead of even error lines, and if they don't fit the budget the default is to fail rather than silently drop them — there's a partial mode that keeps everything and flags the shortfall, but it's for diagnosis only, never reported as a successful compression. The manifest reports which protected lines were kept and which couldn't fit, so nothing protected disappears quietly. The failure-detail half you mention is already there in the other direction: error lines are priority 1 in summarize. The question I'd still put back to you, from your IT Path Solutions side: when you say 'mark specific rules or sections', do you picture the marking living in the repo docs where the decision already lives, or in a list the caller passes to the tool? The two point at different owners for which lines count.
The manifest is the part that makes this trustworthy. Showing that 169 items became 23 kept, 54 deduped and the rest summarized means someone can check what was dropped instead of hoping nothing important went missing. Keeping the process deterministic and extractive helps for the same reason.
Paul Irolla's observation matches what we see on client projects. The large file is rarely the problem. It is forty small tool results, and test logs in particular, piling up across a session until the useful instructions are buried.
Launch Gate's comparison with double entry bookkeeping is a good way to explain planned versus executed checks to people outside engineering. If every planned step has a matching record, a missing one is visible at once.
Odinaka's line about the same fix meaning the same failure is worth turning into a dedupe rule. Have you tried grouping repeated failures by the fix applied rather than by the error text?
Thanks for pulling those three threads together — that synthesis is exactly the shape of the discussion so far, and the fix question is the right one to press on.
The honest answer is no, not as a grouping key in the tool — and the reason is where the fix lives. A log line carries the error text, not the fix applied afterwards, so the tool today has nothing to group by except what is in the lines themselves. What shipped in v0.2.0 is the closest neighbour, not the same thing: traces that differ only by a 0x address collapse, first occurrence kept verbatim, with the near count kept separate — and it deliberately does not claim 'same fix', because address-equality is not fix-equality.
Grouping by fix would need the fix to arrive as an annotation from outside the log — a caller tagging each failure with the fix it got — and then the grouping itself is easy, but the judgment has moved outside the tool, which is the same tension this thread has hit with caller-marked ranges. That raises the question I'm curious about: in practice, where would that fix tag come from for you — a post-run annotation, a link to the commit that fixed it, something the runner records at the time? And the boundary case that follows: two failures that share a fix but come from different call sites — group them, or keep them apart because the location still matters?
The manifest-as-diff framing is the part I'd bet on — most compression tooling fails not because it cuts the wrong lines but because you can't prove it didn't invent one. Question on the boundary cases: the 88 lines summarized away keep first/last lines per section and error lines, but what about repeated-but-slightly-different lines, e.g. stack traces that differ only by a memory address? Exact-duplicate dedupe misses those and they're usually most of a long retry log. Is that something the block-repeat detection catches, or is it on the roadmap?
Good catch — and the honest answer is: today, neither stage catches them.
diet_dedupe is exact-only: it removes exact duplicate lines and repeated blocks of 3–8 lines (see src/dedupe.ts). A stack trace that differs only by a memory address is a different string, so it survives dedupe as a distinct line, and a repeated block with one address changed survives as a distinct block too.
Summarize doesn't rescue that case either: error lines are priority 1, so within budget they're all kept verbatim — the right instinct for a single failure, but the wrong shape for a retry log where the same failure repeats with a new address each time.
The direction that seems honest to me would be a normalization step before dedupe keying — mask volatile tokens (hex addresses, timestamps) only for the comparison key, keep the first occurrence verbatim, and record in the manifest that those lines were collapsed as near-duplicates rather than exact duplicates, so the 'no invention' claim stays auditable. But the masking list is where it gets delicate: mask too much and two genuinely different failures collapse into one.
What would you put on the mask list — addresses only, or timestamps and PIDs too? And would first-occurrence-verbatim plus a count satisfy the audit side for you, or would you want the masked variants listed somewhere?
Closing the loop on your question: this shipped in v0.2.0. diet_dedupe now normalizes before keying — 0x-prefixed hex addresses are masked for the comparison key only, the first occurrence stays verbatim, and near-duplicate collapses are counted separately in the manifest so they don't masquerade as exact matches. There's an --exact-only opt-out if you want the old behavior. Honest limit: bare hex without 0x, timestamps, and PIDs stay exact — nobody confirmed those for the mask list, and they can carry identity or ordering meaning. Change: github.com/techdevfun/context-diet... — if you run it against your traces, I'd love to know whether the first-occurrence-plus-count shape is enough for your audit side, or what else you'd want recorded.
@techdevfun Test logs, I'd say. A green run repeats the same lines every time, and a red one buries the failing assertion in the middle. Greps and listings are small one at a time, but they come far more often.
Test logs taking that vote makes sense — a green run is pure repetition, and a red run buries the one line that matters in the middle. That split is why dedupe does the heavy lifting before anything gets summarised: the repeated green lines collapse, and the failing assertion is exactly the line that has to survive. Which end do you find harder in practice — the red run, where the assertion is buried, or the green run, where everything looks identical until it suddenly isn't?
Usually it isn't one big file. It's small tool results nobody trims: a grep, a test log, a directory listing. Each one stays in the history until the session ends, so the window fills with output nobody looked at.
Exactly — death by a thousand small outputs, not one big file. A grep here, a directory listing there, and none of them ever get thrown away. Which small output fills your sessions fastest — test logs, or the greps and listings?
The manifest accounting is what sold me: "169 = 23 kept + 54 deduped + 4 extracted + 88 summarized" is something you can actually diff, unlike the compression percentages from the rivals you list.
One thing I kept wondering: exact-duplicate dedupe rarely fires on real logs, because every line carries a timestamp or run id. Does diet_dedupe normalize those before comparing, or does that fall to diet_extract? In my own agent session logs the accumulation is mostly near-duplicates, the same failing test re-pasted across ten retries with only the timestamp changing, and exact matching would miss almost all of it.
Good catch, and the honest answer is: today it would miss your case. diet_dedupe normalizes only 0x-prefixed memory addresses before comparing, so repeated stack traces differing only by address collapse as near-duplicates, with the first occurrence kept verbatim and the removal counted separately as near, not exact. Timestamps and PIDs are deliberately left exact in the code, on the grounds they can carry ordering or identity meaning, and no commenter had confirmed them for the mask list until now. diet_extract doesn't normalize anything either, it just scores lines and takes windows around matches, so it wouldn't rescue the timestamp-only repeats. So ten retries of the same failing test differing only by timestamp would survive dedupe as ten lines today. Is timestamp-masking the right call here, or would stripping the one field that orders the retries lose something you'd want kept?
The manifest-as-receipt approach is the part most "token optimizer" tools skip, and it's the right call — an unauditable rewrite of a build log is worse than a slightly bigger prompt. One thing worth stress-testing: your chars/4 estimate tends to undercount logs and stack traces, where short tokens (paths, symbols, hex) dominate, so a 400-token budget can land noticeably larger on a real tokenizer. Also, how does diet_dedupe treat stack frames that differ only by line number — collapsing those could erase the exact offset that points at the failure.
Thanks — and on the dedupe question, the current boundary already protects the offset you're worried about. The comparison key masks only 0x-prefixed addresses; line numbers aren't masked, so two frames differing only by line number have different keys and both lines survive — the kept text is always the first occurrence verbatim, so the exact offset stays in it. That's the same deliberate boundary as timestamps and PIDs, which stay exact because they can carry identity or ordering meaning. On the token estimate, the honest position is the same: chars/4 is a labeled estimate, not a measured count, and no measured comparison has been published — and your undercount direction for logs sounds right. The case I'm curious about: is there a log shape where the line number itself is the volatile part — a rebuilt binary shifting lines, say — where collapsing across line numbers would be the right call rather than a loss?
The first thing I'd check is the chars/4 estimate on logs. Hex addresses, file paths and stack frames usually split into more tokens than prose, so a 400-token budget could land well over 400 on a real tokenizer. Have you compared the two on the session.log example?
Good challenge, and the honest answer is no — I haven't published a measured comparison on the session.log example. The tool is explicit about this in code: token counts are a chars/4 estimate, always labeled estimated, never quoted as measured. Your direction of bias sounds right to me: logs tokenize denser than prose, with short tokens from paths, symbols, and hex, so a 400-token estimated budget could land noticeably over 400 on a real tokenizer. What would settle it for you — a measured-token check reported alongside the estimate for the demo log, or a stated calibration range for logs versus prose? I'm curious which one you'd actually trust in practice.
The manifest is the right instinct: most context shavers report a percentage and then you cannot tell whether the summary dropped the one assertion that mattered, and "kept 23 / dropped 54 as duplicates" is a claim you can diff. One thing I would push on is the dedupe layer — exact-line dedupe is cheap, but the accumulation I actually see is near-duplicate re-reads, where a file comes back with two lines changed and the whole body is stored again as a new copy, so tracking last-read-per-path and returning a delta would spend fewer tokens than any summariser.
For the extractive pass, the rule that worries me is "first and last line of each section": on a stack trace that keeps the header and the tail while dropping the middle frames, which is exactly the causal chain an agent needs. Productising a protected-pattern list (keep lines matching /^\s*at |Caused by:|AssertionError/) would let callers declare what must survive rather than trusting a fixed policy. Or is the budget pressure low enough that you would rather keep the policy fixed and auditable?
That's a sharper version of the re-read problem than exact-line dedupe was built for. v0.2.0's masked-key dedupe catches the same trace with a different address, but a file that comes back with two lines changed is a new body to a line-based tool — the whole thing gets stored again, which is exactly the accumulation shape the article is about. A last-read-per-path delta would spend fewer tokens than any summariser; the honest tension is that the tool today is stateless, it sees lines, not paths or previous runs, so the delta needs somewhere to live. Would you put that state in the tool, or with the caller who already knows what it read last?
On the extractive side, the middle-frames worry lands the same way: keeping the header and the tail preserves the frame that names the failure and the one that ends it, but the causal chain in between is what an agent actually needs to walk. The pattern list you suggest — keep lines matching at / Caused by: / AssertionError — is caller-declared protection in the same family as protectLines in v0.3.0, but by pattern instead of verbatim lines, and that's a real difference: verbatim lines can't over-keep, while a pattern like 'at ' could match every frame in a long trace. Would you bound the list to a small caller-supplied set, or does that over-keeping risk not worry you when the alternative is dropping the chain?
Curious which half you'd weight heavier — the delta for re-reads, or the pattern list for the extractive pass — and whether you've seen a re-read shape where the delta alone would have beaten the summariser.
@techdevfun A single assertion buried mid-log. Any rule that keeps the tail gets a red run right when the failure sits at the end, so the mid-log case is the one that actually tests the summary.
That's the sharper pick — and the reason it bites is exactly what you said: a tail rule passes the easy case for free. A mid-log assertion is the one place where a summarizer has to prove it kept the right line, not just the last ones. It also draws a useful line for this tool: error lines are priority 1 in summarize, so a failing assertion phrased as an error should survive — but a plain expect line buried mid-log is the harder case, and the one that would actually test whether the summary is doing selection rather than truncation. Which shape is your canary — an error line, or a plain assertion that has to be recognized as the one that matters? If you've got a real red log where that distinction shows up, I'd love to see what it looks like.
@techdevfun Red, and it isn't close. A green run collapses to one line once the repeats go. A red run is where a summary can lose the one assertion that matters, so that's the end I'd test first.
Red first makes sense — and it's the sharper test for exactly the reason you give. A green run collapsing down is the shape dedupe already handles. A red run is where the cost of getting it wrong is real, because the one assertion that matters is the line a summary could drop.
That also frames the test nicely: a red log where the failing assertion survives trimming, while everything around it collapses. If you were picking the canary for that test, what would it be — a single assertion buried mid-log, or a red run where the failure only shows up in the last few lines?
A very insightful essay thanks!
Thanks, Nikita — glad it landed. Which part stuck with you most — the manifest-as-evidence idea, or the near-duplicate edge cases in the thread?
Prompting is the easy layer. The interesting work is in what happens after the model answers.
@arcwrightai Exactly — that's why every cut is accounted for in the manifest. The answer is only half the work; being able to verify what was kept and what was dropped is the other half.