DEV Community

Krish Verma
Krish Verma

Posted on

How I built model-aware recaps for my desktop AI companion: 85% pressure, a 1,048,576-token window, and 42 messages down to 12

How I built model-aware recaps for my desktop AI companion: 85% pressure, a 1,048,576-token window, and 42 messages down to 12

Ankita AI is a desktop AI assistant by Krish Verma — a little companion that lives on your desktop and gets things done while you're away. It holds long conversations: hours of planning, tool calls, and decisions. Last week I wrote about how it compacts those conversations at 36 messages into one 2,000-byte recap. That article was honest about the trigger, but the trigger was only half the story. The recap counted messages. A chat with five enormous turns could hit the model's real limit long before the 36th message arrived — and the trimmer would quietly drop the oldest turns before any recap ran. So the recap now reads the budget, not the count.

The budget comes from the model, not a constant

The fix lives in src/core/model-capabilities.mjs, in a function called modelContextWindow(). It resolves the window in three steps, in order:

  1. Your explicit choice wins. If CONTEXT_WINDOW is set in the environment, that number is the window, full stop. A deliberate setting must never lose to a catalogue.
  2. The model's advertised window. The provider's model catalogue reports context_length, context_window, or capabilities.limits.max_context_window_tokens. declaredContext() takes the first positive integer among them. Baseten's Muse Spark and the Modal DeepSeek deployment both advertise 1,048,576 tokens.
  3. A fallback of 32,768 tokens. If the catalogue says nothing usable — or says something malformed — the fallback stands. The comment in the code is blunt: malformed provider metadata must not erase the fallback.

Switching models refreshes the limit everywhere it matters: the chat agent, the tool planner, and the voice session all re-resolve the window when the model changes (src/core/bootstrap.mjs, src/core/voice-session.mjs). A small model and a million-token model in the same session get different budgets, because they are different budgets.

85% is when the recap fires

The trigger is COMPACTION_PRESSURE_RATIO = 0.85 in src/core/agent.mjs. The recap runs when history reaches 85% of the available budget — available meaning after everything else has taken its share first:

  • The reply itself. outputReserve() holds back room for the model's answer. An explicit MAX_TOKENS wins; otherwise the provider's own request default (requestDefaults.max_tokens) and its enforced minimum (minMaxTokens) are counted — the tokens the provider will actually send, not the ones I guessed. With neither, the reserve is capped at a quarter of the window, never less than 256 tokens, so a small window is never starved by a 4,096-token default.
  • Tool schemas, personal memory, the system prompt. toolBudgetBytes() subtracts those before history gets a byte.
  • 1,024 bytes of protocol overhead (CONTEXT_OVERHEAD_BYTES), for the framing nobody remembers to count.

Only then does the pressure check run: cost >= available * 0.85. The old message-count trigger (36 messages) still exists as a backstop, but the budget check is what fires first on a chat full of large turns. A recap that used to arrive too late now arrives while there is still room to write it.

The recap also learned manners. It appears immediately in the chat and in the companion island, with a separate label, instead of only showing up after a reload. And an independent review caught the recap counter climbing 1 → 2 → 3 on unchanged history; after the fix it stays 1 → 1 → 1, with an 8-turn cooldown between recaps.

The check that convinced me

The verification record (docs/verification/context-and-subagents.md) has a hosted run that is the whole point in one paragraph. A real Agent.send against an anonymous free Kilo model — reflection/beam-free, advertising a context of 1,048,576 — compacted 42 messages down to 12, stored the recap, and then correctly recalled the synthetic detail "Juniper Harbour" from the compacted history. The recap call returned HTTP 200 with max_tokens=400, prompt usage 724 tokens, completion usage 338 including 224 reasoning tokens, and a reported cost of 0. The reply that followed used max_tokens=1024.

The short-history case is the other half of the proof: with a small configured history it first failed with zero recap calls, then passed with exactly one.

What I still don't trust

The recap is written by the same model that later reads it as fact. The summariser prompt says "Do not invent facts" — but that is an instruction, not a verification pass, and the recap wears the user's role in history. If the model hallucinates a decision into the recap, every later turn treats it as something you said.

And 0.85 is a guess, not a measurement. It leaves 15% of the budget for the next response to grow into — but on a 32,768-token window that is under 5,000 tokens, and on a million-token window it is a small city. The ratio is the same; the slack is not. The window itself also depends on the provider telling the truth in its catalogue, and nothing in this code checks that claim against reality.

The code is in Ankita on GitHub, and the project lives at ankita-sable.vercel.app.

Tell me I'm wrong: is a pressure ratio the right trigger, or should the recap fire on the cost of the next message instead — summarise when the incoming turn won't fit, not when the history is 85% full? And would you trust a recap written by the same model that reads it, or does that need an independent check before it enters history?

— Krish Verma, creator of Ankita AI

Top comments (0)