Why Cosine Similarity Fails to Catch Confusable MCP Tools

MCP (Model Context Protocol) servers expose tool definitions — names,
descriptions, JSON schemas — that an LLM agent reads at runtime to decide what
to call. When a server declares several tools with overlapping capabilities
(read_file, read_text_file, read_media_file), agents frequently pick the
wrong one or construct invalid arguments. As tool counts grow, catching
confusable definitions before deployment becomes a real quality and safety
problem.

The obvious fix is text similarity: vectorize the descriptions, compute
cosine similarity, flag anything above a threshold. I ran that against real
data and it doesn't work — here's the empirical case for why, and what
worked instead.

The naive approach

TF-IDF vectorize each tool description, compute pairwise cosine similarity,
flag pairs above ~0.85.

Why it fails

Tested against the 14 tools of the official
@modelcontextprotocol/server-filesystem server — 91 distinct pairs.

• Zero detections at the standard threshold. At 0.85 cosine similarity, TF-IDF flagged 0 of 91 pairs.
• No threshold works. Lowering it doesn't help — confusable pairs (read_file vs. read_text_file) and clearly distinct pairs (read_file vs. write_file) produce overlapping similarity distributions. There's no cut point that separates them.

Two reasons this collapses:

1. Shared domain vocabulary dominates. Tools on the same server repeat the same nouns (path, directory, file, permissions) regardless of whether they're confusable. High term overlap reflects the server's domain, not tool ambiguity.
2. Opposing verbs barely move the needle. read_file and write_file share 80%+ of their tokens. Cosine similarity treats read vs. write as one differing word among many — even though it's the whole difference that matters.

What worked: gate on schema substitutability first

Instead of scoring text first, gate on structure first: can one tool's
arguments satisfy the other tool's schema? Two schemas are substitutable if
every required property in A exists in B with a compatible type, and neither
has a required parameter that would immediately break the other. If the
schemas aren't substitutable, an agent literally can't confuse the two calls
— so those pairs are discarded before any text comparison happens.

On the filesystem server, this structural gate removes 63 of 91 pairs before
scoring starts.

For the remaining 28, score on:

• name affinity (edit distance, shared prefix/suffix)
• description similarity (non-domain token overlap)
• a hard veto if names or descriptions contain opposing verbs (read/write, create/delete, encrypt/decrypt)

Results

Four pairs flagged:

• read_file / read_text_file
• read_file / read_media_file
• read_text_file / read_media_file
• list_directory / list_directory_with_sizes

Their scores sit between 0.42–0.48, cleanly separated from every non-flagged
pair (0.00–0.32) — a real 0.33–0.50 gap between the two classes, which cosine
similarity alone never produced.

The full 91-pair dataset — raw cosine scores, substitutability decisions,
affinity scores, veto flags — is public:
docs/data/filesystem-server-ambiguity-scan.json.

Limitations

• Single server. This evaluation covers one server, 14 tools, 91 pairs. Whether this generalizes to other MCP servers is untested.
• The verb veto is a heuristic, not a proof. It's tuned to common CRUD antonyms and will miss scope-based ambiguity that isn't verb-opposed — e.g., read_file vs. a hypothetical read_all_files.
• This catches documentation gaps, not runtime bugs. A flagged pair might be implemented perfectly safely; an unflagged pair might still have bugs. This tool audits what the agent is told, not what the server actually does.

Try it

This is implemented in mcplock, an
open-source CLI (pip install mcplock) that runs this as a lint check
against any MCP server, plus a separate hash-based baseline/diff mode for
catching drift after the fact. Feedback on the substitutability approach —
especially against other MCP servers — is genuinely useful; that's the main
open question right now.

Comments

[Max Quimby]

Gating on schema substitutability before any text scoring is the insight here, and it's the kind of thing that seems obvious only after someone shows it to you. Dropping 63 of 91 pairs structurally means you never even ask the ambiguous question for cases where the agent literally can't confuse the calls — that's a much cleaner formulation than trying to find a magic cosine threshold that was never going to exist, because shared domain vocabulary guarantees the distributions overlap.

The opposing-verb veto (read/write, create/delete) is a nice touch — that's precisely the signal cosine drowns out by treating one differing token as noise, when it's actually the whole semantics. Two things I'd be curious about as you generalize past the single filesystem server: (1) does substitutability need to account for description-implied preconditions, not just JSON schema types? Two tools can have type-compatible args but one says "must be an existing path" — an agent confuses those far less often. (2) For the remaining scored pairs, have you thought about validating against actual agent mis-selection logs rather than human-judged confusability? The ground truth that matters is which pairs models actually swap at runtime, and that sometimes surprises you. Really like that you published the full 91-pair dataset — that's what makes this checkable.

[yash161004]

Really appreciate this comment; this is exactly the kind of pushback I was hoping for.

On preconditions—yeah, you're right, that's a real gap. Right now the gate is purely type-level: does the schema shape match? It doesn't parse anything like "must be an existing path" from the description, so two tools can pass the structural gate and still be pretty different in practice. The honest answer is I haven't built that part yet, but I think the fix is pulling precondition language (existence checks, value ranges, that kind of thing) into the gate itself rather than leaving it to the scoring step—otherwise you're just pushing the same problem downstream into text again.

On runtime validation—also no, not yet. The 4 flagged pairs are validated against what a human would call confusable, not against what agents actually mix up in practice. And I think you're right that those two things could diverge more than I'd like. The next real step is probably logging actual tool-call mismatches across a few agent harnesses and checking how well that lines up with what the linter flags—versus pairs that score close but never actually get confused for real.

Good questions; honestly, they're both things I'd been sort of aware of but hadn't said out loud yet. Appreciate you reading closely enough to push on it.