Your agent can't use a tool it never finds

An agent is asked to find GitHub issues and put a summary in a spreadsheet.

It finds three issue-search tools. It can read tickets, inspect comments, and filter bugs. The spreadsheet tool exists too. It just never makes it into the agent's context.

Now the agent has to work with an incomplete menu. It might stop, ask for help, or try to finish with the wrong tools. Better reasoning alone won't put the missing capability back on that menu.

This is the trade-off behind on-demand tool discovery: you stop loading every tool definition into the prompt, but you take on a retrieval problem before tool selection can even begin.

The useful question isn't only "Did the model choose the right tool?"

It's "Was the right tool available to choose?"

The optimization is real

Loading every schema upfront gets expensive as an agent connects to more services. Definitions compete with the conversation, task data, and intermediate results for context.

Anthropic's advanced tool-use introduction describes deferring tool definitions until a model searches for them. In one illustrated setup, it reports roughly 77,000 tokens of initial context consumption with the traditional approach versus 8,700 with tool search. It also reports better results on its internal MCP evaluations.

Those are vendor-reported results for particular setups, not a promise that adding search will improve every agent. The same post says discovery adds a search step and is less useful for small, compact tool libraries whose tools are used frequently.

This isn't confined to one API. Spring AI's implementation supports semantic, keyword, and regex search over tool definitions. Its guide describes the same basic sequence:

1. Keep a catalog of tools outside the model's initial context.
2. Let the model search for a capability.
3. Expand matching definitions into context.
4. Let the model choose and call a tool.

That third step is where a new kind of failure can become invisible if you only inspect the final tool call.

Discovery and selection are different tests

Suppose the task needs two capabilities:

• Search issues.
• Append spreadsheet rows.

A discovery result containing five issue tools and no spreadsheet tool is incomplete, even if every returned tool is relevant to part of the request.

The Dynamic ReAct paper gives a closely related example: retrieval for a Twitter-to-Google-Sheets workflow returns a Twitter-only subset. Its proposed search-and-load design uses separate, focused queries for different operations instead of treating the whole request as one search query.

MCP-Zero addresses another limit of one-shot retrieval: later steps can require tools that weren't obvious from the original request. Its design lets an agent request more capabilities as the task develops.

These papers don't establish one universally best retriever. They do make a useful architectural point: finding tools is part of the workflow, not necessarily a one-time setup step.

An original conceptual diagram, not benchmark data.

A small test you can run without a model

You don't need to call an LLM to catch the most basic discovery regressions. Start with a few tasks and the capabilities each task requires.

The script below deliberately uses a weak lexical search over a tiny, invented catalog. It is a constructed example, not a measurement of MCP, embeddings, or any model. Its purpose is to show how a retrieval test can expose a missing capability before you evaluate agent behavior.

import re

CATALOG = {
    "issues.search": "search issue tickets bugs github project",
    "issues.read": "read issue ticket details github project",
    "issues.comments": "read issue ticket comments github project",
    "chat.search": "search team chat messages slack",
    "chat.thread": "read team chat message thread slack",
    "sheets.append": "append rows spreadsheet google sheets",
    "sheets.read": "read rows spreadsheet google sheets",
}

def terms(text):
    return set(re.findall(r"[a-z]+", text.lower()))

def search(query, k=2):
    q = terms(query)
    ranked = sorted(
        CATALOG,
        key=lambda t: (-len(q & terms(CATALOG[t])), t),
    )
    return [t for t in ranked if q & terms(CATALOG[t])][:k]

task = (
    "Find github issue tickets and bugs in a project "
    "and log results to a spreadsheet"
)
required = {"issues.search", "sheets.append"}

whole_task = set(search(task))
split_task = set(search("search issue tickets")) | set(
    search("append rows spreadsheet")
)

print("Missing, whole task:", sorted(required - whole_task))
print("Missing, split task:", sorted(required - split_task))

Output:

Missing, whole task: ['sheets.append']
Missing, split task: []

The whole-task query is dominated by issue-related vocabulary. Both result slots go to issue tools. Separate queries recover the spreadsheet capability.

There's a cost: in this example the split queries return four unique candidates rather than two. This is not evidence that splitting is always better at an equal token budget. It shows why "we reduced the number of visible tools" is an incomplete success metric.

For a real evaluation, compare approaches at the same total schema-token budget, or report the extra cost alongside the coverage improvement. Use tasks your users actually ask, including their vocabulary, not only phrases copied from your tool descriptions.

Measure the part that failed

I would keep these measurements separate:

Discovery coverage. For each task, did the retrieved set include at least one valid tool for every required capability? Some capabilities have multiple valid implementations, so don't grade every task against one hard-coded tool name.

Selection correctness. Given a complete candidate set, did the agent choose an appropriate tool? A tool that is visible but ignored is a different problem from one that was never retrieved.

Argument and task correctness. Did the agent supply valid, appropriate arguments and complete the actual job? A perfect retrieval score doesn't establish either.

Cost. Track schema tokens, discovery calls, latency, and extra candidates. A high-coverage search that loads the entire catalog has given up much of the intended saving.

A useful trace records the discovery query, candidate IDs and ranks, definitions actually loaded, selected tool, and any later search. Keep sensitive arguments out of broadly shared logs.

Then an unsuccessful run has somewhere to point:

• Required capability never surfaced: inspect retrieval, catalog freshness, or filtering.
• Capability surfaced but wasn't loaded: inspect the search-to-load step.
• Definition loaded but wrong tool chosen: inspect selection.
• Correct tool chosen but task failed: inspect arguments, execution, and the result.

That is more actionable than putting every failure in a bucket called "the model got confused."

Don't mistake a partial catalog for poor search

Before tuning embeddings or rewriting descriptions, check that the catalog is complete for the caller.

The MCP tools specification supports pagination in tools/list. It also allows the returned tool set to vary with the authorization on the request. That means two failures can look identical from the model's seat:

• The search index didn't contain the second page of tools.
• The caller wasn't authorized to see those tools.

One needs a catalog fix. The other may be correct access control.

There's concrete community evidence for testing the first case: a May 2026 Claude Code issue reports tools beyond the first page disappearing after a list-change notification. The issue is closed as a duplicate. This is a historical report, not a claim that the bug remains in current releases.

It is still a useful regression test: paginate the catalog, change it, refresh it, and verify that tools from later pages remain discoverable.

A practical starting point

Keep a small set of frequently used tools directly available. Defer the rest when the catalog's size justifies the added machinery.

For compound tasks, allow more than one focused discovery query. If a later step reveals a new capability requirement, allow another search instead of forcing the agent to finish with its initial set.

Write tool descriptions around actual operations and objects. "Append rows to a spreadsheet" is easier to distinguish from "Read spreadsheet rows" than two descriptions that both say "Manage sheets." Include important constraints, but don't pack every description with unrelated keywords to win retrieval.

Give discovery a bounded failure path too: a reformulated query, a broader search, or a clear stop when the needed capability cannot be found. Don't quietly substitute a different action just because it ranked highly.

Finally, keep authorization at the execution boundary. A search result is a candidate, not permission. A tool's presence in context does not authorize a particular call, account, recipient, or side effect.

Tool search can make a large agent practical. But it changes the system you're testing.

Once capabilities are loaded on demand, the catalog and retriever become part of what the agent can do. Test whether the right capability reaches the model before asking whether the model used it well.

Sources

• Anthropic: Introducing advanced tool use, November 24, 2025. Vendor examples and internal evaluations.
• Spring AI: Dynamic tool discovery. Implementation guide; its preliminary benchmarks use a different setup from Anthropic's.
• Dynamic ReAct. Research architecture and multi-service retrieval example.
• MCP-Zero. Research on iterative capability discovery.
• MCP tools specification, July 28, 2026. Catalog pagination and authorization-dependent availability.
• Claude Code issue #59538. Historical community report, closed as a duplicate; not a current product verdict.