Why Cosine Similarity Fails to Catch Confusable MCP Tools

MCP (Model Context Protocol) servers expose tool definitions — names,
descriptions, JSON schemas — that an LLM agent reads at runtime to decide what
to call. When a server declares several tools with overlapping capabilities
(read_file, read_text_file, read_media_file), agents frequently pick the
wrong one or construct invalid arguments. As tool counts grow, catching
confusable definitions before deployment becomes a real quality and safety
problem.

The obvious fix is text similarity: vectorize the descriptions, compute
cosine similarity, flag anything above a threshold. I ran that against real
data and it doesn't work — here's the empirical case for why, and what
worked instead.

The naive approach

TF-IDF vectorize each tool description, compute pairwise cosine similarity,
flag pairs above ~0.85.

Why it fails

Tested against the 14 tools of the official
@modelcontextprotocol/server-filesystem server — 91 distinct pairs.

• Zero detections at the standard threshold. At 0.85 cosine similarity, TF-IDF flagged 0 of 91 pairs.
• No threshold works. Lowering it doesn't help — confusable pairs (read_file vs. read_text_file) and clearly distinct pairs (read_file vs. write_file) produce overlapping similarity distributions. There's no cut point that separates them.

Two reasons this collapses:

1. Shared domain vocabulary dominates. Tools on the same server repeat the same nouns (path, directory, file, permissions) regardless of whether they're confusable. High term overlap reflects the server's domain, not tool ambiguity.
2. Opposing verbs barely move the needle. read_file and write_file share 80%+ of their tokens. Cosine similarity treats read vs. write as one differing word among many — even though it's the whole difference that matters.

What worked: gate on schema substitutability first

Instead of scoring text first, gate on structure first: can one tool's
arguments satisfy the other tool's schema? Two schemas are substitutable if
every required property in A exists in B with a compatible type, and neither
has a required parameter that would immediately break the other. If the
schemas aren't substitutable, an agent literally can't confuse the two calls
— so those pairs are discarded before any text comparison happens.

On the filesystem server, this structural gate removes 63 of 91 pairs before
scoring starts.

For the remaining 28, score on:

• name affinity (edit distance, shared prefix/suffix)
• description similarity (non-domain token overlap)
• a hard veto if names or descriptions contain opposing verbs (read/write, create/delete, encrypt/decrypt)

Results

Four pairs flagged:

• read_file / read_text_file
• read_file / read_media_file
• read_text_file / read_media_file
• list_directory / list_directory_with_sizes

Their scores sit between 0.42–0.48, cleanly separated from every non-flagged
pair (0.00–0.32) — a real 0.33–0.50 gap between the two classes, which cosine
similarity alone never produced.

The full 91-pair dataset — raw cosine scores, substitutability decisions,
affinity scores, veto flags — is public:
docs/data/filesystem-server-ambiguity-scan.json.

Limitations

• Single server. This evaluation covers one server, 14 tools, 91 pairs. Whether this generalizes to other MCP servers is untested.
• The verb veto is a heuristic, not a proof. It's tuned to common CRUD antonyms and will miss scope-based ambiguity that isn't verb-opposed — e.g., read_file vs. a hypothetical read_all_files.
• This catches documentation gaps, not runtime bugs. A flagged pair might be implemented perfectly safely; an unflagged pair might still have bugs. This tool audits what the agent is told, not what the server actually does.

Try it

This is implemented in mcplock, an
open-source CLI (pip install mcplock) that runs this as a lint check
against any MCP server, plus a separate hash-based baseline/diff mode for
catching drift after the fact. Feedback on the substitutability approach —
especially against other MCP servers — is genuinely useful; that's the main
open question right now.