DEV Community

Cover image for $840 Refund vs a $500 Cap: Microsoft's agent-hooks Blocked It Before the Tool Ran
fable7
fable7

Posted on

$840 Refund vs a $500 Cap: Microsoft's agent-hooks Blocked It Before the Tool Ran

#ai #security #python #devtools

A support agent tries to refund $840. Your policy says $500 is the line. The only question that matters is where that line gets enforced — in the system prompt the model may or may not respect, or in the code path the tool call has to walk through before it executes.

I wanted to know which one Microsoft's agent-hooks actually delivers, so I ran it. Short version: the guard fires at the tool seam and the refund function is never entered. But I got there the hard way, and the way I got it wrong first is the part worth writing down.

What I installed (first trap)

The announcement post tells you to install it like this:

pip install agent-framework-core[agent-hooks]
Enter fullscreen mode Exit fullscreen mode

That extra does not exist. On agent-framework-core==1.21.0 the only extra published is all:

$ python -c "import importlib.metadata as m; print(m.distribution('agent-framework-core').metadata.get_all('Provides-Extra'))"
['all']
Enter fullscreen mode Exit fullscreen mode

pip is polite about it — warning: The package agent-framework-core==1.21.0 does not have an extra named 'agent-hooks' — and then installs the core anyway. The middleware is in there (agent_framework/_agent_hooks.py, 1,710 lines), but the enforcement SDK it calls into is a separate package you have to name yourself:

pip install agent-hooks-sdk   # 0.1.0b2 at the time of writing
Enter fullscreen mode Exit fullscreen mode

Without it, importing the module still works and the factory raises a clear ModuleNotFoundError telling you exactly what to install. That part is good behaviour. Not every project does that.

The rule I wanted to test

One tool, one guard, one cap. The interceptor is deliberately dumb:

class RefundGuard:
    def intercept(self, context):
        if context["interception_point"] != "pre_tool_call":
            return Verdict.allow()
        call = context["tool_call"]
        if call["name"] == "issue_refund" and float(call["args"]["amount"]) > CAP:
            return Verdict.escalate(
                reason="refund_over_limit",
                message=f"refund {call['args']['amount']} exceeds the {CAP} cap",
            )
        return Verdict.allow()
Enter fullscreen mode Exit fullscreen mode

Wired to the agent the way the docs say:

bundle = create_agent_hooks_middleware([RefundGuard()], record_sink=RECORDS.append)
agent = Agent(client=client, instructions="support agent",
              tools=[issue_refund], middleware=[bundle])
Enter fullscreen mode Exit fullscreen mode

No real model is involved. I wrote a fake chat client that emits exactly one issue_refund call, so the run is deterministic and costs nothing. The record_sink gives me the decision stream as data instead of as vibes.

Two runs, one cap

Run A asks for $840. Run B asks for $100. Everything else is identical.

A: refund $840 B: refund $100
pre_tool_call verdict DENY / refund_over_limit ALLOW
issue_refund() actually called no — calls: [] yes — one call, amount=100
interception records 9 10

The $840 case never reaches the function body. The framework hands the model a tool error and the loop continues, which is the behaviour you want: the agent does not crash, it just cannot buy anything with that call.

Both runs show pre_model_call / post_model_call twice — the tool hop costs one extra model round trip — and only run B has a post_tool_call, because in run A there was no tool call to post about. The record stream is the receipt:

A: agent_startup, input, pre_model_call, post_model_call,
   PRE_TOOL_CALL(DENY), pre_model_call, post_model_call, output, agent_shutdown
B: agent_startup, input, pre_model_call, post_model_call,
   PRE_TOOL_CALL(ALLOW), POST_TOOL_CALL(ALLOW), pre_model_call, post_model_call, output, agent_shutdown
Enter fullscreen mode Exit fullscreen mode

Interception record for both runs: a $840 refund is denied at pre_tool_call, a $100 refund is allowed and executed

Where I nearly fooled myself (second trap)

My first attempt looked fine and was completely broken. I built the fake client by stacking the framework's own layers, in the wrong order:

class OneShotClient(ChatMiddlewareLayer, FunctionInvocationLayer, ChatTelemetryLayer, BaseChatClient):
    ...
Enter fullscreen mode Exit fullscreen mode

With ChatMiddlewareLayer first in the MRO, the chat seam is the outermost get_response, so it receives the mixed middleware list the agent passes down — chat and function middleware together. It registers what it recognises and drops the rest:

Ignoring unrecognized middleware of type _AgentHooksFunctionMiddleware:
it is neither a ChatMiddleware nor a callable and will not be executed.
Enter fullscreen mode Exit fullscreen mode

One line in the log. No exception, no failed run. And then the $840 refund went through: tool_executed: true, calls: [{"customer": "acme", "amount": 840.0}], and the interception records contained no pre_tool_call entry at all — the guard was never asked. Swap the order so FunctionInvocationLayer comes first, and run A blocks as designed.

I'm not going to call that a framework bug: hand-rolling a chat client is on me, and the failure mode is visible if you read logs. But it is worth naming what it is. A spend cap, a destructive-command check, or a PII gate that silently does nothing because of a class ordering detail is worse than no gate, because you now believe you have one. If you assemble your own client, assert on the record stream — "did pre_tool_call fire?" is a one-line test, and it is the difference between enforcement and decoration.

The design does get one thing right here: put the bundle on the client and it refuses to run rather than half-installing:

MiddlewareException: MiddlewareBundle member _AgentHooksAgentMiddleware is agent middleware,
but this middleware seam supports only chat, function middleware. A bundle is one indivisible
feature and cannot be partially installed; pass the bundle to the agent instead.
Enter fullscreen mode Exit fullscreen mode

That is the correct failure: loud, at configuration time, with the fix in the message.

What I did not test

Being precise about the boundaries, because a lot of "I tested it" posts quietly overreach:

  • No real provider client and no real LLM. A fake client emitting one tool call is enough to prove the seam blocks, not enough to prove the whole loop holds up under a chatty model.
  • Streaming, transform verdicts, and approval resolvers were not exercised. Verdict.escalate is a deny with approval metadata; nobody approved anything in my run.
  • Hosted, provider-side tools are documented as never passing through the framework's tool seam. A deny at pre_tool_call cannot stop a tool the provider executes on its own — that is a stated limitation, not a bug I found.
  • One version pair: agent-framework-core==1.21.0 + agent-hooks-sdk==0.1.0b2, 2026-10-14. Both are experimental and the API says so out loud.

The takeaway

Dollar-denominated policy at the tool boundary is real and it works — the $840 refund died before the function body, which is exactly the claim in the announcement. The install line in that announcement does not work as written, and the failure mode when you wire it yourself can be silent. Read the record stream, not the log line.

The contract itself is framework-neutral and the same demo is described for LangGraph, the OpenAI Agents SDK, Semantic Kernel, LlamaIndex, CrewAI, and the Claude Agent SDK. If someone re-runs my two scenarios on one of those and gets a different decision stream, that is the interesting follow-up — one contract, eight hosts, and I only checked one.

Top comments (2)

Collapse
 
reidmarlow profile image
Reid Marlow •

The silent fall-through on unrecognised middleware is terrifying. In an enforcement framework, an unhandled hook type should raise a hard configuration exception at startup rather than logging a warning and letting the transaction proceed.

The other practical wall with that float check is multi-turn splitting. Once the model receives a tool error that 840 dollars exceeds the 500 dollar ceiling, it tends to split the request into two 420 dollar calls across consecutive turns. Unless the interceptor writes to an external session ledger to track cumulative spend, a stateless pre-tool gate only stops single large calls.

Collapse
 
fable7 profile image
fable7 •

Both points land. On the startup exception: the contract can't give you that. It declares the host fully trusted, and Microsoft's own writeup says the conformance kit can't detect a host that skips interception points or ignores verdicts — coverage is testable, not automatic. A hard failure on an unknown hook type has to come from the host framework's config loading, not from the spec.

On the split: agreed, and that's the more dangerous one. A per-call gate sees two $420 calls and no violation. What the contract does give you is content-bound identity — an approval covers exactly the action the human saw — so the second call can't inherit the first approval. But a cumulative ceiling is interceptor state, and state that doesn't survive a restart isn't a ceiling. External ledger is the only shape that holds.