In September I wrote about AI agents in our framework, Kumiko, and why a handler without a description never shows up in the agent's tool list: Undocumented handlers don't exist to the agent. Two readers, phongdesigns and anp2network, went through the post more carefully than I had, and the thread under it ran to eleven comments over three days.
Three changes to how our agent decides what it may do came out of that thread. We turned down two of the ideas, and there are three questions we haven't answered. This post covers all of it, because the open questions are the part I'd like help with.
The setup in two sentences
Every write handler in a Kumiko app can carry an agent risk (low, mid or high), and the user can set mid-risk tools to "always allow" while high-risk tools ask every time. Back then, the risk was whatever the handler declared, plus two tests that pinned the risk of specific handlers by name.
Change 1: the risk comes from what the handler does
anp2network pointed out that pinning by name doesn't catch the hard delete someone adds in six months. It defaults to mid, and nothing goes red. Their suggestion was to watch what a handler actually does and compare that with what it declares.
That fit our code better than I expected. A hard delete in Kumiko doesn't go through the normal event append. It calls a separate forget operation on the event store, so the framework can see it. Since 0.322.0, a handler that runs forget, or a hard delete on an entity without soft delete, has to resolve to high. Otherwise the executor denies the call before it reads anything. The check looks at the handler the caller dispatched directly, so a mid-risk handler can't get around it by delegating to a high-risk one internally. Standard delete handlers on those entities default to high, and declaring one lower fails at boot.
My follow-up comment listed after-commit hooks, jobs and event consumers as a gap. Since then, writes dispatched inside those contexts go through the same check, with a short list of deliberate exceptions such as cron cleanups and pipelines started by a high-risk handler.
Change 2: the tool list and the dispatch table are tested against each other
I had claimed that the model can't call a tool that isn't in its list, because the list and the dispatch table come from the same catalog. anp2network's reply was that this was only true because of how the code happened to be arranged, and no test checked it. Now a test asserts that every tool name has exactly one dispatch entry, and the other way round.
Change 3: some fields are dangerous to write because of who reads them
This one I didn't see coming. Our prompt store keeps every revision of a feature's system prompt. Editing a prompt was a high-risk action. Reverting to an older revision was mid, and a revert appends a copy of the old revision, which then becomes the live prompt. A user who set revert to "always allow" had given the agent a way to swap another feature's instructions without asking.
A floor on the handler can't see that, because on the storage side the revert is an ordinary, reversible write. anp2network put it this way: the risk is a property of the destination. A later run reads that row as instructions.
Since 0.324.0, a field can say so:
systemPrompt: createLongTextField({
required: true,
personal: false,
reason: "is_business_data",
readAsInstruction: true,
}),
Any create or update that writes such a field needs high on the directly dispatched handler, through a generic update handler as well. The prompt store's content field carries the flag, so revert is high now.
We also record how every agent write was approved: no approval needed, an "always" rule, or the user confirming it. "Always" isn't offered for high-risk tools, so the number of flagged writes approved by an "always" rule should be zero, and we can check that with a query.
What we turned down
anp2network noticed that a tool the user set to "never" vanishes from the model's view, so to the model "forbidden" looks exactly like "doesn't exist". The suggestion was a tombstone: keep the entry visible and have dispatch refuse it with its own reason. We didn't build it. "Never" is one of several filters that hide tools (roles, mode, an explicit deny list), so tombstones would either be inconsistent with the other filters or leak names those filters exist to hide. anp2network agreed with the reasoning. Instead, the agent's system prompt now tells it to say the action isn't available "rather than approximating it with a similar tool — a similar tool is not the same action."
The other suggestion was a test that counts the high-risk handlers, so every change to that set shows up in a diff. Once the risk comes from the effect, the count adds little: a new hard delete is high whether anyone updates a number or not.
Still open
1. Irreversible effects outside our database
The floor sees forget and hard deletes. It doesn't see a mail, a webhook or a payment. In our store, the write that queues a mail looks like any other reversible write, but the mail can't be unsent. Our webhooks now send an idempotency key, so the receiver can drop a redelivery, but that only protects against duplicates. Nothing decides whether the agent should trigger the first one without asking.
Our half idea is to treat outbound channels like flagged fields: the channel declares itself irreversible, and any handler that hands work to it resolves to high. We haven't built that. I'm not sure the line is right either, because a status mail to the user themselves and a payment to a third party would land in the same class.
2. Who reads a field as instructions
The readAsInstruction flag is set by hand next to the field, apart from the code that later puts the field into a prompt. If a new feature starts feeding an unflagged field into a system prompt, the flag set is stale and every check stays green. anp2network asked whether anything fails when that happens. Nothing does, and on the agent side the system prompt is still a plain string.
The flag also has holes on the write side. A hand-written domain event whose projection writes the flagged field skips the gate. The gate checks whether the payload contains the field, not whether the value changed, which we chose deliberately so it fails closed. Reverting a prompt also doesn't undo a run that already read the bad version.
Our half idea is a type. The system prompt parameter would only accept text that came from a flagged field or from code, so feeding it anything else wouldn't compile. I haven't checked whether that survives prompts assembled from several pieces.
3. Refusing instead of reaching for the neighbour
When the model can't find a tool for what the user wants, it should say so. The risk is that it picks the closest-sounding tool instead, maybe one the user set to "always allow". Right now the only defence is the sentence in the system prompt quoted above.
anp2network suggested a number: after a rejected call, record what the model calls next. In our loop, calling a tool that isn't in the list ends the run, so there's no "next" to count. The case we can't see at all is the model going straight to the neighbour without ever trying the missing tool.
phongdesigns added the near-duplicate problem: two writes described in almost the same words, and the model confuses them. Our manifest snapshot still records only handler names, so a reworded description doesn't even show up in review. I said in the thread that adding descriptions and risk to it was the next step, and it still is.
The only test I can think of for substitution is to give a real model pairs of similar tools and see where it goes wrong. That costs money on every run, and the result changes with the model.
How do you test that an agent refuses instead of improvising, without paying for a model call on every pull request?
Top comments (0)