MCP tool retrieval improved from 50% to 93%, but agent tool-use grades did not improve.

I've been investigating tool discovery and context management for AI agents, using an experimental MCP gateway and an OpenCode harness running "gpt-oss:20b-cloud".

In an offline evaluation over 300 catalog-derived queries, enhanced retrieval improved top-1 recall from 50.3% to 93.3%.

However, in a separate 25-task paired evaluation, the enhanced configuration scored 14/25 against 16/25 for the baseline on prescribed tool-use behavior. The agent frequently failed to consult the discovery gateway.

The study also measured a 56.6% reduction in estimated model-visible context using deterministic result projection. This was a synthetic benchmark, and it did not establish equivalent cost savings or preserved decision quality.

The results suggest that retrieval quality, tool activation and context management need separate evaluation. The experiments are small and do not establish a causal explanation for the difference in tool-use grades.

I've published the methodology, implementation, evaluation records and limitations here:

https://github.com/Binilts03/Public/blob/main/docs/research/agent-tool-retrieval.md

I'd be interested in technical criticism of the evaluation design, particularly how to measure missed tool invocations independently of retrieval accuracy.

Comments

[Sammi De Blas]

Retrieval accuracy and tool-use accuracy are different measurements. Going from 50 to 93 percent on finding the right tool says nothing about whether the agent calls it with valid arguments. For an MCP server exposed to untrusted input, the argument schema and the least privilege role matter more than the ranking of the catalog.

[Auten]

Interesting split. Did the 25 tasks need a tool that was outside what the agent saw at start, or could it often finish without discovery at all?

For missed invocations, one option is a third arm that forces the discovery call first (tool_choice required on step one). If grades recover there, the gap is activation, not retrieval, and you can score "should have called discovery" per task against a labelled oracle.

(From the Auten team, we build an MCP server.)

[FreeDailyPro]

Really appreciate that you published the negative result along with the recall number. That gap is exactly what I'd want to know before trusting a gateway.

On measuring missed invocations separately from retrieval, two things I'd try:

1. An oracle arm that skips discovery entirely and puts the gold tool straight into the agent's tool list. That gives you an upper bound for the task itself. If oracle scores well and the gateway arm doesn't, the loss is in activation (the model never asks), not in ranking.
2. Label each task up front with whether it actually needs a tool outside the starting set. Then "discovery call rate on tasks that need it" and "discovery call rate on tasks that don't" become two separate numbers, and you can see whether the model under-calls or just calls at the wrong moments.

Also, with 25 paired tasks, 14 vs 16 could easily be a handful of flips. Listing which tasks changed outcome in each direction (and a McNemar test) would say more than the totals.

One hunch to check: smaller models often ignore a meta-tool like "search_tools" unless its description says plainly when to use it, something like "call this first whenever none of your current tools fits the request". It'd be interesting to see if rewording that one description moves the activation rate.

[Paul Irolla]

Better retrieval does not lower the fixed cost of an MCP setup: the schemas of every tool stay in each prompt, including the ones it never calls.