A 1.6 MB screenshot costs fewer than 1,000 tokens — my trimmer was deleting it anyway
Ankita is a personal AI companion for the desktop — "a little companion, a lot more possible" — and companions live in long sessions. That is where context trimming fails quietly: nothing crashes, and the model just answers a question about a screenshot it never saw.
The bug is a unit error. A pasted screenshot travels as a base64 data URL, and the naive way to budget history is to measure that string. A 1.6 MB PNG then looks like 1.6 MB of context, so the trimmer deletes it first — before the model ever sees it. But vision attachments are billed by pixels, not by base64 length. That same image costs fewer than a thousand tokens. Measuring it as text guarantees the trimmer throws away the cheapest, most informative part of the turn.
Budget the image the way the provider bills it
The fix lives in src/core/history.mjs, in trimMessages(). Instead of weighing the data URL, Ankita reads just the image header and prices the pixels:
- PNG dimensions come from the
IHDRchunk, JPEG from its start-of-frame markers, GIF from its 10-byte header, WebP from itsVP8/VP8L/VP8Xchunks. Nothing is fully decoded. - The estimate follows the common high-detail recipe: fit the longest side inside 2048 px, resize the short side to 768 px, then count 512 px tiles at 85 + 170 tokens per tile. It is deliberately an upper bound across OpenAI-compatible vision endpoints, converted back to bytes at 4 bytes per token.
- An image that cannot be decoded falls back to the old behaviour — its URL length — so an unreadable image never gets a free pass.
The repo's own test makes the difference concrete: a 1700 × 2200 PNG padded with 100,000 filler bytes has a data URL longer than 100 KB, but its vision estimate is under 5,000 bytes. Worked through: 2048 / 2200 scales it to about 1583 × 2048, the short side resizes to 768 px giving roughly 768 × 994, which is 2 × 2 = 4 tiles, so 85 + (170 × 4) = 765 tokens, and 765 × 4 = 3,060 bytes. The trimmer now keeps that image; the old one deleted it.
Keep the goal, keep the pair, drop whole groups
Pixel pricing alone does not fix trimming — the order of what survives matters just as much. Three rules govern trimMessages():
Anchor the request. The newest user message is always kept, plus up to two earlier user messages (USER_REQUEST_ANCHORS = 3), with those earlier requests capped at 4,096 bytes combined. The point is that a long tool loop should not erase the goal or a recent correction.
Never split a tool exchange. An assistant tool call and its tool replies are one atomic group (TOOL_EXCHANGE_MESSAGES = 2 minimum). Dropping the reply but keeping the call — or vice versa — hands the model a conversation that never happened. Old groups are dropped whole, newest first, and chronology around the anchored request is preserved.
Shed in a fixed order, and fail loudly at the bottom. Under byte pressure, the biggest text units shrink first, and an attached file body is droppable only as a last resort — losing the file body beats failing the whole turn. Images are dropped only when their vision tokens genuinely do not fit. But if even the current request plus the core tool schemas exceed the window, trimming throws instead of sending a lobotomised turn: "The current request/tool schema exceeds the model context budget." The agent side (Agent.trimHistory() in src/core/agent.mjs) reserves output tokens, 1,024 bytes of overhead, the tool schemas, and memory context first, and refuses to run at all below 512 bytes of history.
Restoring a saved session gets the same treatment. Orphan tool messages from an interrupted session are discarded, malformed tool calls are replaced with a visible "[Incomplete tool calls removed from restored session.]" note, and — the bug that motivated part of this — user turns stored as an array of content parts are no longer silently deleted for not being strings. That one used to drop the attachment and leave the model re-answering stale context.
What this does not solve
The estimate is an upper bound, not any provider's exact count, so a tight window may still drop an image that would have fit. Worse, when an image is dropped to save the turn, nothing tells the user the model never saw it — the turn succeeds, blind. And the 4,096-byte prior-request cap means an older correction eventually vanishes no matter how important it was. Those are real trade-offs, not footnotes.
So here is the design question I keep coming back to: when the budget runs out, is silently dropping the image to save the turn the right call, or should the turn fail loudly so you know the model never saw it? I chose "save the turn" — a companion that refuses to answer is worse than one that answers from text — but I am not sure that holds when the whole question is about the image. Which failure mode would you ship?
Ankita is open-source (MIT) at https://github.com/akyourowngames/A.N.K.I.T.A — the trimming code is src/core/history.mjs, and I would genuinely like to hear how you budget multimodal history, or where you think this ordering is wrong.
Top comments (4)
The trimmer bug makes sense once you look at how it was estimating cost. If it was treating encoded base64 bytes as the text budget, it would flag a screenshot as oversized well before the model's own vision billing would, since the encoding itself adds roughly a third on top of the raw file size. Estimating from pixel dimensions like you did here is the right fix, the two numbers were never going to agree otherwise.
Exactly — that was the mismatch. The trimmer was weighing the base64 string, so a 1.6 MB PNG looked like 1.6 MB of context when the vision billing is pixels, not characters, and base64 alone inflates it by about a third before you even get to the token math.
The fix in trimMessages() in src/core/history.mjs reads just the image header — IHDR for PNG, start-of-frame for JPEG, VP8/VP8L/VP8X for WebP — and prices the pixels with the 2048/768 fit and 85 + 170 per 512px tile, instead of ever weighing the data URL. An image it can't decode still falls back to the old URL-length behaviour, so nothing gets a free pass.
Honest caveat: it's deliberately an upper bound across OpenAI-compatible endpoints, not any provider's exact count, so a tight window can still drop an image that would have fit. Thanks for reading it that closely.
One catch on the upper bound: it only holds for OpenAI-style tile pricing. Claude bills by pixels, roughly width x height / 750, and Claude 4.7 and later read up to a 2576px long edge. A 2560x1440 screenshot comes out at 1105 tokens with the 85 + 170 tile math, but 4784 on Claude, so behind an OpenAI-compatible proxy to Claude the trimmer could still keep more than fits.
The numbers for each Claude version are here: slimsnap.ai/blog/claude-2576-image....
You're right, and this is a real hole in the "upper bound" claim, not a rounding difference.
estimateImageTokens() in src/core/history.mjs takes only the image URL — there is no provider or model argument anywhere in that path, so there's exactly one pricing formula: fit inside 2048px, resize the short side to 768px, then 85 + 170 per 512px tile. Your 2560x1440 example reproduces exactly through it: 2048x1152 after the fit, 1365x768 after the short-side resize, 3x2 tiles, 1105 tokens.
So behind an OpenAI-compatible proxy to Claude, the estimate lands on the wrong side: it undercounts, and the trimmer keeps images that don't actually fit — worse than the original bug's direction, which dropped images that would have fit. The doc comment does scope the formula to "OpenAI-compatible vision endpoints", but that scoping doesn't save the proxy case, because the trimmer can't tell what's behind the endpoint.
Honest caveat: I haven't pinned Claude's exact count from the code side — there's no Claude branch in history.mjs at all, so whatever the exact per-version number is, the trimmer is blind to it. The real fix is a provider-aware estimate (or pricing at the max across known schemes), and that's still a TODO, not something shipped. Thanks for catching it — this one genuinely changes the fix.