Every app we build on Kumiko comes with an AI assistant. It can do what the logged-in user can do: create a campaign, look up a contact, edit a mail template. The framework turns each handler the user's role can reach into a tool, and the assistant got the whole list on every turn.
In offlot, our marketing app, a tenant admin has 187 tools. I measured what goes into the prompt before the user types a word: about 418k characters. 143k of that was tool JSON. Another 260k was the app manifest, which tells the model what the app looks like (entities, screens, navigation, handlers). The handlers section alone was 154k characters, and it described the same handlers the tool list already described. We were sending most actions twice.
About 60% of the 187 tools came from platform and admin features rather than from the marketing domain. Someone asking how many contacts signed up this week paid for all of them.
Four tools and a directory
The model now gets four tools: navigate, ask_user, describe_action and run_action. The actions the role can use go into the system prompt as a directory, one line per action, with a summary capped at 160 characters. To use an action, the model calls describe_action for the input schema and then run_action with the name and the input.
run_action doesn't execute anything itself. Right after the provider response we rewrite it into the call it names, so it takes the same path as before: dispatch, Zod validation, the approval step for risky writes. When a conversation is replayed, the history gets rewritten back, so the model sees the calls the way it made them. An unknown name and a name the role may not use get the same reply ("No action with this name is available"), so the assistant can't be used to probe for actions outside the role.
The manifest lost its handlers section, since the directory covers it now. On the framework side, the manifest also lists only the features and entities the role can reach.
We built a third mode for comparison, lazy, where the model sees less up front and loads action details on demand.
The run
The same 11 end-to-end specs against offlot in all three modes, on Sonnet 5.5, with real API calls:
| Mode | Prompt prefix | Cost per turn | Model calls per turn |
|---|---|---|---|
| Full tool catalog | 203k tokens | $0.148 | 1.5 |
| Directory | 85k tokens | $0.062 | 2.1 |
| Lazy | 79k tokens | $0.061 | 2.6 |
All three modes passed 11 of 11. The directory costs 58% less per turn. It needs more model calls, because the model has to ask for a schema before it can use an action, but each call is cheap once the prefix is smaller and cached.
Lazy saved another 6k tokens and a tenth of a cent per turn and needed half a call more for it. Every extra round is latency the user sits through, so we dropped lazy. The directory is now the only mode, in all our apps.
What I got wrong
Before the run I estimated the directory prefix at about 8k tokens. It came out at 85k. The tools were the obvious bulk, and I assumed the rest was small. The system rules, the directory itself and the rest of the manifest (entities, screens, navigation) add up, and about half of the 85k is still manifest.
The measurement also cost more than it had to: $10.79 in total. Two runs went to the wrong model. In the first, the model setting only reached the demo tenant, and that run cost $3.90 on the old Sonnet. Then I found that our dev server ignores the environment variable that sets the model in production. By then the run script checked the model name in the first response and stopped the run after $1.10. Next time that check goes in before the first run.
Why not drop JSON?
JSON is wordy, so a compact text format looked like the next lever. I ran the same manifest through Anthropic's count_tokens endpoint as JSON and as a compact line format. The difference was 13 to 14%, not enough to justify a custom format that every consumer has to parse.
The boring content was worth more. Every label went out in German, English and Spanish, although each conversation has one locale. Feature descriptions, written for the docs, went in at full length: 46.6k characters for 77 features. The framework now sends one label in the requested locale and only the first sentence of each feature description, capped at 300 characters. For offlot that takes the descriptions from 46.6k to 12.2k characters. That change shipped in the same release, but I haven't measured it in a full run yet.
An open question
The directory is still one line per action in every prompt. At 187 actions that works. If you run an agent with a lot more actions than that: where did a flat list stop working for you, and what replaced it? Grouping by feature, a search tool, a small model that picks the candidates first?
Top comments (6)
The remaining half of the 85k prefix points to a second version of the same problem. The directory removed the handlers duplication, but entities, screens and navigation still travel in full on every turn, even though navigate is already one of the four fixed tools. A describe-style lookup for screens or entities would apply the same pattern one level up: names and one-line summaries up front, details only for the part of the app the conversation actually touches. The interesting measurement would be how often a turn touches more than one screen, because that decides whether the extra lookup stays near the 2.1 calls per turn seen here or drifts toward the lazy mode's 2.6.
The 58 percent saving is one measurement taken on the largest role you have. Savings scale with action count. The penalty does not: the jump from 1.5 to 2.1 model calls per turn is a describe_action round, and that round costs the same whether the role can reach 187 actions or twelve. On the saving side, 143k characters of tool JSON across 187 tools is roughly 765 characters per action, and the directory replaces each of those with a line capped at 160. Shrink the role and the numerator shrinks while the extra round trip stays exactly where it was, so somewhere below 187 the full catalog should win on both cost and latency. You dropped lazy because an extra round is time a user sits through, and that same objection lands on the directory for a role with twelve actions. The open question asks what happens above 187. The gap in the table is on the other side. Since about 60 percent of those 187 come from platform and admin features, a domain-only role is already sitting there, and running the same 11 specs against it once would give the table a second point. One point cannot describe a curve.
The 160-character cap deserves its own run, separately from the structural change. It shipped in the same release, it has not been through a full run, and it fails in the opposite direction. Your earlier post on undocumented handlers covered the loud case: no description, no tool, the model never learns the name exists. Truncation produces the quiet case. Two actions that used to read differently can land on summaries that no longer tell them apart, the model picks the plausible one, and 11 specs that never name the action it skipped still come back green. This part is measurable with zero API calls. Cut all 187 summaries at 160 characters and count the pairs that stop being distinguishable, then count the actions whose cut drops the operative verb or the clause saying the write cannot be undone, which is the wording delete-golden leans on entirely. Feature descriptions reduced to a first sentence under 300 characters belong in the same count.
run_action leaves two representations of one event. There is the call the model emitted and the call that reached dispatch after the rewrite, and replay restores the first of them. Which one is persisted is the part the post leaves open. If the stored transcript holds only the run_action form, then what actually reached dispatch is not stored at all; it is recomputed by running the rewrite again, and the rewrite is code that may have moved on by the time anyone audits the conversation. Does the record keep both forms, or is the rewritten dispatch call never written down?
Fair on the curve, so I ran the second point. Offlot's Editor role reaches 99 actions. Same model, same specs, minus one spec whose final check reads an action Editor doesn't have, so 10 per mode.
Full catalog: $0.103 per turn, largest prefix 139k tokens, 1.47 model calls per turn. Directory: $0.059, 74k, 2.06 calls. The saving drops from 58% at 187 actions to 43% at 99, and the extra describe round shows up at the same size in both roles, as you expected. The directory cost hardly moves between the two roles ($0.062 vs $0.059), because most of its prefix is system rules and manifest, not the action lines. So the crossover sits well below 99, and I'd guess it depends as much on what else is in the prompt as on the action count. Two points still aren't a curve; a small role would be the third.
The directory passed 10 of 10, the catalog 9 of 10. The one catalog failure was a 5-second client timeout on an approve while the machine was under load; the proposal itself was correct.
The summary count needed no API calls, so I ran it. Of the 187 TenantAdmin summaries, 110 hit the 160-character cap, and 27 more lose everything after their first sentence. No two summaries came out identical, but that's a weak test: each line also carries the action name and its kind (read, write, read that needs approval), and I didn't measure near-duplicates. 11 descriptions say a write can't be undone or is permanent, and all 11 keep that wording in the summary. The approval step for risky writes comes from the handler declaration, not from the text, so a cut there would cost the model a hint but wouldn't skip the approval. I haven't counted the feature descriptions yet.
The count also found a bug. The generated description for a fetch action starts "Fetch a single A dealer's marketing run for one vehicle…", because the entity description gets glued onto the verb. I'd have missed that without reading all 187 lines at once.
On the record: we store the dispatched form. The
run_actioncall is unwrapped right after the provider response, before the turn is persisted, so the stored transcript has the call that reached dispatch. Therun_actionform is rebuilt on replay, and that rebuild is mechanical: any tool call that isn't one of the four fixed tools becomesrun_actionwith name and input. What we don't keep verbatim is the model's rawrun_actioncall. A malformed one stays as it was and fails, so it's stored as the model sent it. One small asymmetry: a call with no input is stored with an empty input object.A search tool adds a failure mode that the flat directory doesn't have: the right action exists, but never reaches the model's candidate list. Before replacing the directory, I'd add requests that need two actions with unrelated names and record retrieval misses separately from execution failures. In particular, a missing candidate should not turn into a confident claim that the app cannot do the job. Keeping the same end-to-end specs, plus those routing cases, would make the next comparison more informative than prompt size alone.
That's the property I'd lose. Today the directory lists every action the role can use, so "not in the list" really means the role can't do it, and the model can say so. With search, "not found" only means the search didn't find it, and the model can't tell the two apart.
Separate counts for retrieval misses and execution failures make sense. Retrieval would need its own log line, since a miss never reaches dispatch and leaves nothing to count afterwards. The two-action cases with unrelated names go on the list as well. A few of our specs already chain actions, like setting up a vehicle, its campaign and its photos from one message, but none was written to check whether the model finds an action whose name doesn't match the words in the request.
Before you trim the prompt, count how many of those tokens are tool schema. They are sent with each call, used or not, so that count is the first thing to measure.