The Shortlist Decides First ended with a caveat I promised to resolve: the retriever in that experiment was BM25, a lexical matcher, and its catastrophic 5 percent recall on paraphrased requests was exactly the failure lexical matching should produce. Would a semantic retriever change the verdict? That is a measurable question, so here is the rematch: the same 100 tools, the same 200 tasks, the same four description conditions and k-sweep, with one change. Instead of BM25, the retriever embeds every tool document and every request with text-embedding-3-small on Azure OpenAI and ranks by cosine similarity. The harness is measure/eval_recall_embed.py in the companion repo; nothing else moved.
The results
Recall of the correct tool into the shortlist, by description condition and task kind:
| condition | vocab @3 | vocab @5 | vocab @10 | para @3 | para @5 | para @10 |
|---|---|---|---|---|---|---|
| terse | 95% | 99% | 100% | 59% | 73% | 87% |
| realistic | 91% | 98% | 100% | 59% | 76% | 94% |
| verbose | 97% | 99% | 100% | 65% | 82% | 98% |
| realistic + aliases | 98% | 99% | 100% | 99% | 100% | 100% |
Three findings, one win, one surprise, one standing verdict.
The win: embeddings rescue paraphrase
BM25 scored 5 percent on paraphrased requests at k=5. Embeddings score 76 percent on the same tasks with the same descriptions. That is the rematch headline, and it lands exactly where the theory says it should: a user who says "I want to cancel my booking" shares almost no tokens with a tool described as "deletes a reservation", but the two sentences live close together in embedding space.
The k-sweep behaves differently now too, and this is worth noticing. With BM25, widening the shortlist from 3 to 10 bought paraphrase recall from 3 to 10 percent, because the correct tool usually was not even near the top of the ranking. With embeddings, the same widening buys 59 to 94 percent. Semantic misses are near misses; lexical misses were not misses at all, they were absences. When your retriever is at least semantically in the neighborhood, paying for a longer shortlist actually purchases something.
The surprise: verbosity finally pays, and exactness costs
The shortlist post concluded that verbosity is not coverage, because doubling description tokens moved BM25's paraphrase recall not at all. The rematch forces me to refine that: it is true lexically and less true semantically. Under embeddings, verbose descriptions beat terse ones on paraphrase at every k, 82 against 73 at k=5, because every extra sentence adds semantic surface for the query vector to land on. Verbosity is still a poor deal, you pay roughly triple the tokens per tool for nine points, but it is no longer a zero.
The reverse surprise sits in the vocabulary columns. On requests that echo the tool's own vocabulary, BM25 scored 97 to 100 percent at k=3; embeddings score 91 to 97. Embeddings blur exact matches. When the user says precisely the words in the description, nothing beats matching those words literally, and a semantic retriever occasionally ranks a similar-sounding sibling above the exact hit. The two retrievers fail on opposite task populations, which is the textbook argument for hybrid retrieval, and now the series has the numbers for it.
The standing verdict: the alias sentence still wins
Look at the last row. With the 13-token alias sentence per tool, recall is 99 to 100 percent everywhere, at every k, and it was also 100 percent under BM25. The cheapest intervention in the entire series makes the retriever choice irrelevant.
That reframes what an embedding index buys you. If you control the tool descriptions, the alias sentence gets you to perfect recall for about 13 tokens per tool and zero infrastructure. The embedding pipeline, an embedding model, an index, a lookup on every request, buys you from 5 to 76 percent on paraphrase when you cannot touch the descriptions, which is precisely the situation of a gateway or catalog fronting other teams' MCP servers. Both answers are right; they answer different ownership questions.
The end-to-end multiplication from The Model Was Never the Problem updates accordingly. Recall times selection on paraphrased requests with realistic descriptions: BM25 gave 5 percent times 93 percent, roughly 5 percent of tasks ending in the right tool call. Embeddings give 76 times 93, roughly 71 percent. Aliases give 100 times 97, roughly 97, with either retriever underneath.
Honest limits
One embedding model, text-embedding-3-small; a larger model or a hybrid lexical-plus-vector ranker would likely narrow both gaps, and the harness takes any Azure OpenAI embedding deployment if you want to try. The paraphrases are template-generated, which makes them systematic but tamer than real user phrasing. The catalog is 100 tools; at 1,000 the embedding index gets more attractive because the alias discipline gets harder to enforce across teams. And this measures retrieval in isolation; the selection stage was measured separately and multiplies on top.
The takeaway
The rematch confirms the retriever was the right suspect and the description was still the right fix. Embeddings turn paraphrase recall from a catastrophe into a tax, 5 to 76 percent, and they are the honest recommendation when the descriptions belong to someone else. But when the descriptions are yours, one alias sentence per tool beats the entire vector pipeline, at one ten-thousandth of the operational surface. Fix the words first. Add the index when the words stop being yours to fix.
The harness, both retrievers, and the full results are in the companion repo: measure/eval_recall.py for BM25, measure/eval_recall_embed.py for the rematch.
Top comments (1)
The alias row is a vocabulary test in its own right. The function with_aliases in eval_recall.py builds the alias sentence from VERB_SYNONYMS and NOUN_SYNONYMS, the same two tables make_tasks uses to write the paraphrases, and eval_recall_embed.py imports both functions, so under either retriever each of the 100 paraphrase tasks finds both of its phrases verbatim in its own tool's alias sentence. The 99 to 100 percent shows what the sentence does when it names the exact words the user will use.
A held-out table shows what it does when a user picks a word nobody listed. With a second synonym per word that appears in no description and no alias sentence, fetch for get, scrap for cancel, statement for invoice and so on, BM25 with aliases is back at 3, 5 and 10 percent at k of 3, 5 and 10. That is the BM25 paraphrase baseline too, and it is list order: under the terse and realistic descriptions all 100 paraphrases score zero against every tool, so the shortlist is the first k tools in the list. Running eval_recall_embed.py on the same held-out table is the number that would say whether the alias sentence adds semantic reach or only lexical overlap.