Give an AI agent a real toolset (25, 40, more) and every call pays for it. The agent reads every tool description to use one, so input tokens grow with each tool you add. And with many similar tools in view (search, search_hotels, search_flights...), the model picks the wrong one more often.
Two costs on every turn: tokens you did not need to spend, and picks you did not want. Giving the agent fewer tools is not the fix, because different turns need different tools. You want the full toolset available but only the relevant few in front of the model each turn.
In an earlier post I solved this with semantic tool selection over a vector store: embed every tool description into an index (FAISS), embed the query, and keep the nearest few. It works, and that walkthrough covers it in depth. But it has a cost of its own: you maintain an embedding index for your tools, every query pays an embedding call, and the index has to stay in sync as tools change.
A new class of model changes this. A decision model reads the request and the list of tools and returns a calibrated choice directly, with no embeddings and no index. It generates no text, so there is nothing to parse, and the choice comes with a confidence score.
Three have arrived recently: Strands Decider, an open-source 2B model that runs locally; Jev (TypeSafe), a hosted one; and the OpenAI Decisions API (gpt-6-luna, public beta since Oct 2026), also hosted. All three answer the same bounded question, which tool fits this request?, and all three skip the vector store entirely. You get the token and accuracy win of tool selection without building or maintaining an index.
How Strands fits in
Strands lets you plug any of these in the same way, through a hook that runs before the model and swaps the agent's tools for the turn. The selector inside the hook can be the vector store or a decision model, and the same idea extends to model routing, choosing which model serves the request.
class ToolFilterHook(HookProvider):
def register_hooks(self, registry):
registry.add_callback(BeforeModelCallEvent, self.filter)
async def filter(self, event):
tools = await self.select(query) # vector, decider, Jev, or OpenAI Decisions
swap(event.agent, tools) # the model now sees only these
agent = Agent(tools=ALL_TOOLS, hooks=[ToolFilterHook(select)])
The agent is built once with every tool; the hook trims the registry per request. This is where Strands earns its place: it swaps tools on the live agent at runtime, so the conversation survives. Frameworks that force you to rebuild the agent to change its tools lose that history. Here you keep the memory, and you switch strategy by swapping select in one line.
The swap itself is a few lines against the agent's tool registry. Clear it and register the chosen tools; the running agent and its message history stay intact:
def swap(agent, tools):
reg = agent.tool_registry
reg.registry.clear() # drop the previous turn's tools
reg.dynamic_tools.clear()
for t in tools: # register only the ones picked this turn
reg.register_tool(t)
# agent.messages is untouched — the conversation is preserved
References: Strands hooks, hook events,
and model routing.
Keeping the live conversation is memory within a session. For memory that lasts across sessions, Strands has a dedicated
memory library: pluggable memory stores on a unified storage interface. You attach a store, a local file store for prototyping or Amazon Bedrock Knowledge Bases with semantic search for production (or your own backend), and the agent gets recall, automatic injection into the prompt, and extraction of new memories. So the hook keeps this turn's context, and the memory store carries durable knowledge between runs.
Model routing
The hook decides which tools. Strands model routing decides which model. Give a ModelRouter a few candidates and a strategy; the strategy picks one per request. A decision model fits this role well: picking the model is a choice among a few unordered options — exactly what these models are built for — and it votes with a calibrated pick instead of a chat call.
All three decision models plug into the same RoutingStrategy interface: Jev (hosted), Strands Decider (local 2B), and the OpenAI Decisions API (gpt-6-luna, a choice question over the candidate models). The example below uses Jev — the one wired in the demo (model_routing_with_jev.py); swapping in Strands Decider or OpenAI Decisions is a one-class change, so you pick on hosted-vs-local and which provider you are already on. Routing to a cheaper model for the easy requests is where this pays off — OpenAI reports a run that cut cost 54% by classifying each job before picking the model.
router = ModelRouter(models=[small, large], strategy=JevRoutingStrategy())
agent = Agent(model=router)
Same shape as the tool hook: a cheap decision in front, the expensive model only when it is needed.
The comparison
Same 12 queries, same ~25-tool pool, one demo run (AWS Bedrock Claude Haiku as the agent):
| Approach | Accuracy | Chat tokens | Latency (end to end) |
|---|---|---|---|
| Send all tools | 10 / 12 | ~56k | ~1.1s, fastest |
| Hook + vector | 11 / 12 | ~20k | ~1.8s (+ an embedding call) |
| Hook + decision model | 11 / 12 | ~16k | ~2.3s (+ a decision call) |
Both selectors reach the same accuracy and cut chat tokens by about two-thirds. But each adds a call before the model: vector embeds the query (and pays embedding tokens), the decision model runs one choice, so here they are slower end to end, not faster. At 25 tools, sending everything is the quickest.
That ordering flips as the toolset grows. The model's prefill (reading the prompt before it answers) scales worse than linearly with input length, so a bigger pile of tool descriptions makes every call slower, and an agent makes many calls. Past some tool count the baseline's growing prefill overtakes the fixed cost of one selection step, and selecting first becomes the faster path, not just the cheaper one.
So the real takeaway is not a fixed latency winner; it is that you spend a small, constant selection cost to keep the prompt, the token bill, and the prefill small, and that pays off more as you add tools.
Where the delay is
Follow the request through the pipeline and you can see exactly where each approach spends time:
- Send all tools: one step, the model call. No selection, so nothing extra, but the model reads all ~25 tool descriptions.
- Hook + vector: three steps before the answer: embed the query (a network call to the embedding model), search the index (local, microseconds), then the model call. The added delay is the embedding round-trip.
- Hook + decision model: two steps before the answer: the decision (local inference for Strands Decider, or a network call for Jev), then the model call. The added delay is that one decision.
So the extra latency lives in the selection step, before the model runs. The model call itself is cheaper afterwards (a short tool list), but not cheap enough at this scale to beat skipping selection entirely.
Which selector should I use?
- Already embedding things? Vector is the smallest add.
- Want a calibrated pick and no index? A decision model.
- Run a small model locally? Strands Decider 2B. Prefer a managed endpoint? Jev.
-
Already on OpenAI, or need image context? The OpenAI Decisions API — hosted, calibrated
choice+ confidence, billed on input tokens only ($0.10/1M), and it accepts images as well as text (which Jev does not yet).
They are not exclusive. The hook makes them interchangeable in one line.
A quick selection benchmark
Measuring only the pick — which tool each selector returns — over 20 labelled queries against a ~25-tool pool (warm, single run on 2026-10-07):
| Selector | Accuracy | Mean | p50 | Cost |
|---|---|---|---|---|
| Vector (FAISS/Titan) | 95% | 155 ms | 144 ms | — (embedding call) |
| Jev (hosted) | 100% | 210 ms | 200 ms | — |
OpenAI Decisions (gpt-6-luna) |
100% | 275 ms | 258 ms | $0.054 / 1k picks (538 input tok each) |
Small set, one run — treat it as an early signal, not a verdict. Both decision models matched the right tool on every query here; vector missed one. (This is the selection step in isolation; the end-to end table above measures a different 12-query run including the agent's answer.)
Try it
All of this runs in the open-source demo (Demo 02),
why-agents-fail-sample-for-amazon-agentcore. It has chat.py (vector), chat_decider.py (local decision model), model_routing_with_jev.py (model routing), and a notebook that measures accuracy and tokens for each — including the hosted OpenAI Decisions API as a third selector.
FAQ
Does trimming tools hurt accuracy? No. In the demo it raised accuracy, because fewer similar tools means less confusion. The risk is dropping the right tool; tune how many you keep.
Why a hook instead of rebuilding the agent? Rebuilding loses the conversation. The hook changes the tools on the live agent, so memory survives and you switch strategy in one line.
Does a decision model save tokens? It sends fewer tool descriptions to the agent and generates no text itself. It is a separate, cheap call, not a chat turn.
Is this tied to Strands? The idea (filter before the model) is general. Strands makes it first-class: hooks, the tool-registry swap, and model routing are built in.
What do I need to run the OpenAI Decisions selector? An OPENAI_API_KEY and openai>=3.26.0 (or just POST to /v1/decisions). It is in public beta on gpt-6-luna, returns a typed choice with confidence, and bills input tokens only — no output or cache charges.
Gracias!
Top comments (1)
The token numbers check out (56k to 20k is -64%, to 16k is -71%, so "about two-thirds" holds), but the accuracy rows can't separate the three yet: 10/12 vs 11/12 is one query, and 95% vs 100% on 20 is one query. With counts that small I'd read all three selectors as tied and the token cut as the only measured difference.
Two things I'd add from building tool selection over a larger pool (306 tools): (1) the failure that hurts is the right tool being dropped, and the hook then gives the model no way to recover, so it's worth wiring the confidence score to a fallback (low confidence or a "no tool fits" turn restores the full list for that turn). (2) Tell us how the 20 labelled queries were written. If they were written from the tool descriptions, plain-word queries will do better than real traffic; for us a plain lookup bug (the stemmer turned "rates" into "rat") only showed up on requests nobody had tuned for, and no model-size comparison would have shown it. 46 of 48 never-tuned requests came back first for us.
Do you log the tool the model actually called next to the one the selector returned? That's the number that tells you whether trimming dropped something.
I built the pool I tested on, canli-mcp (free, MIT, no key;
npx -y canli-mcp, github.com/arhancanli/canlicapital): 306 tools behind three front tools, 1,109 tokens of context against 34,508 for listing them all.