BM25 vs Embeddings: Why My Vector Search Couldn't Find Error E4012

A user typed E4012 into my docs assistant. The top result was the page for E4021. The second was E4013. The page that actually explained E4012 sat at rank 14, well below my top-5 cutoff, so the LLM never saw it and confidently explained the wrong error.

That's the BM25 vs embeddings problem in one screenshot. Dense vector search is great at "what does this mean" and bad at "find this exact string." Error codes, SKUs, function names, ticket IDs and version numbers fall straight into that gap.

I spent an evening convinced my index was corrupted. It wasn't. The embedding model was doing exactly what it was built to do.

TL;DR

• Embedding models compress text into meaning. E4012 and E4021 mean almost the same thing to them ("some error code in this product"), so their vectors land next to each other.
• BM25 scores exact token matches weighted by rarity. A code that appears in one document gets a huge IDF weight, so BM25 ranks that one document first.
• The fix is hybrid retrieval: run BM25 and vector search in parallel and merge them with Reciprocal Rank Fusion (RRF).
• Check your BM25 tokenizer. A default analyzer that splits ERR_CONN_RESET or v2.3.1 into pieces throws away the exact-match advantage you added BM25 for.
• On my own 120-query test set, hybrid fixed every identifier query that dense-only missed without hurting the natural-language ones.

Why does vector search fail on exact identifiers like error codes?

Vector search fails on identifiers because an embedding encodes what a passage is about, not which exact characters it contains. Two pages that describe neighboring error codes are about the same thing, so they get nearly the same vector.

Walk through what happens to E4012:

1. Tokenization. The embedding model's subword tokenizer breaks E4012 into fragments, something like E, 40, 12. None of those fragments is special. E4021 becomes E, 40, 21. Most of the pieces overlap.
2. Contextual mixing. The transformer blends every token with its neighbors. On a docs page, the code is surrounded by sentences like "this error occurs when the connection times out." That context dominates.
3. Pooling. Mean pooling (or a CLS token) squashes everything into one vector of 384 to 1024 floats. A few digit tokens out of 300 barely move the average.

The result: the E4012 page and the E4021 page sit a hair apart in vector space, and which one wins for the query E4012 is basically noise.

A short query makes it worse. E4012 alone has almost no context, so its embedding is a vague "error-code-ish" vector. It's closer to every error page than to any particular one.

How does BM25 rank documents differently?

BM25 ranks documents by counting exact term matches and weighting each term by how rare it is across the corpus. A term that appears in only one document carries a very high weight, so that document wins.

The scoring formula has three moving parts:

• IDF (inverse document frequency): log((N - n + 0.5) / (n + 0.5) + 1), where N is the number of docs and n is how many contain the term. In my corpus of about 2,300 chunks, a word like "connection" appeared in hundreds of chunks. e4012 appeared in 3.
• Term frequency with saturation: the k1 parameter (default around 1.2 to 1.5) means mentioning the term 10 times isn't 10 times better than mentioning it once.
• Length normalization: the b parameter (default 0.75) stops long chunks from winning just by being long.

Plug in the numbers and e4012 gets an IDF around 6.5 while "connection" sits near 1.5. BM25 doesn't know what an error code is. It just knows this token is rare and the query contains it. For identifier lookups, that's exactly the right instinct.

Here's the minimal version with rank_bm25:

import re
from rank_bm25 import BM25Okapi

def tokenize(text: str) -> list[str]:
    # keep codes like E4012, ERR_CONN_RESET, v2.3.1 as single tokens
    return re.findall(r"[A-Za-z0-9_][A-Za-z0-9_.\-]*[A-Za-z0-9_]|[A-Za-z0-9]", text.lower())

corpus_tokens = [tokenize(c.text) for c in chunks]
bm25 = BM25Okapi(corpus_tokens, k1=1.2, b=0.75)

scores = bm25.get_scores(tokenize("E4012"))

Why didn't my first BM25 attempt work either?

My first BM25 attempt failed because the tokenizer split identifiers into pieces before indexing them. Once ERR_CONN_RESET becomes err, conn, reset, three common words, the rare-term advantage is gone.

I started with text.split(). That kept E4012, with the trailing comma as a different token from E4012, so half my matches silently vanished. Then I switched to a "proper" word tokenizer that stripped punctuation, and it shredded v2.3.1 into v2, 3, 1 and ERR_CONN_RESET into three generic words.

Things to check in whatever BM25 you use (Elasticsearch, OpenSearch, Postgres full-text, Tantivy, rank_bm25):

• Does the analyzer split on _, ., or -? Paste one of your real identifiers through it and look.
• Does stemming touch codes? A stemmer that turns errors into error is fine. One that mangles HTTP_403s is not.
• Is the query run through the same tokenizer as the documents? Mismatched analyzers are a classic zero-hit bug.
• Lowercasing is usually fine for codes, but not if your product has E4012 and e4012 as distinct things (I hope it doesn't).

The regex above is crude, and that's the point. It keeps anything that looks like an identifier in one piece. Run your nastiest IDs through it before trusting it.

How do you combine BM25 and embeddings? Reciprocal Rank Fusion

You combine them by running both searches and merging the two ranked lists with Reciprocal Rank Fusion. RRF ignores raw scores and only uses rank positions, which matters because BM25 scores and cosine similarities live on completely different scales.

The formula: for each document, sum 1 / (k + rank) across every list it appears in. k = 60 is the common default from the original RRF paper.

def rrf(result_lists: list[list[str]], k: int = 60, top_n: int = 5) -> list[str]:
    scores: dict[str, float] = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]

dense_ids = vector_search(query, limit=50)
sparse_ids = bm25_search(query, limit=50)
context_ids = rrf([dense_ids, sparse_ids])

Why this works for E4012: BM25 puts the right page at rank 1 (score 1/61). Dense search had it at rank 14 (1/74). Total about 0.030. The E4021 page was dense rank 1 but didn't appear in BM25's list at all, so it scores 1/61 or about 0.016. The right page wins by almost double.

And for a fuzzy question like "why does my upload hang forever," BM25 returns weak, scattered matches while dense search nails it. RRF lets the stronger signal win each time without you writing a router.

Pull 50 candidates from each side, not 5. If the right doc is rank 14 in one list and you only fetch 10, fusion never sees it.

What did hybrid search change on my test set?

On my own test set, hybrid search recovered every identifier query that dense-only search missed and left natural-language queries roughly where they were. These are numbers from my corpus and my embedding model, not a general benchmark.

The setup: 120 real queries pulled from support tickets, each labeled with the chunk that answers it. 38 of them contained an identifier (error code, config key, CLI flag, version string). I measured whether the correct chunk landed in the top 5.

• Dense only: 21 of 38 identifier queries hit. Natural-language queries were fine.
• BM25 only, with the naive tokenizer: worse on identifiers than I expected, because of the shredding problem above.
• BM25 only, with the identifier-preserving tokenizer: 36 of 38 identifier queries hit, but it fell apart on paraphrased questions.
• Hybrid with RRF: 38 of 38 identifier queries, and the natural-language queries lost nothing I could measure on a set this size.

120 queries is small. Treat it as "this is the direction" rather than "this is the number." Build your own eval set from real queries; it's the only one that tells you anything about your corpus.

Should I just add a reranker instead?

A cross-encoder reranker helps ordering, but it can only rerank what retrieval hands it. If dense search buried the right chunk at rank 14 and you fetched 10 candidates, no reranker will ever see it.

Rerankers also read the same subword fragments, so they're better at identifiers than bi-encoders but not immune. Put the reranker after hybrid retrieval, not instead of it.

The cheap checks I run now

• Grep my query logs for anything matching [A-Z]+\d{3,}, \w+_\w+, or v\d+\.\d+. That's my identifier traffic share. Mine was about a third.
• For each identifier query, check whether the doc containing that exact string is in the top 5. If not, the retriever is guessing.
• Pipe five real IDs through the BM25 analyzer and eyeball the tokens.
• Log both ranked lists before fusion. When an answer is wrong, I can see which side dropped the ball.

So, BM25 vs embeddings: which one should you use?

Use both. Embeddings find passages that mean the same thing as the query; BM25 finds passages that contain the same rare tokens. Vector search misses error codes like E4012 because embedding models compress the code's digits into a vector dominated by surrounding meaning, which puts neighboring codes almost on top of each other. BM25 gives that rare exact token a high IDF weight and ranks the matching document first. Run both retrievers, fetch around 50 candidates from each, merge with Reciprocal Rank Fusion at k = 60, and make sure your BM25 tokenizer keeps identifiers in one piece. That combination handles both "what does this error mean" and "find error E4012."

---

Written by the developer behind Preterview, an interview prep platform.

Comments

[kevinbai]

Your tokenizer section is where I recognised something from today. I was diffing a list of slugs with a regex that stopped at the second slash, so every nested path collapsed to the same prefix and the diff reported 2 matches instead of 7. Nothing errored, and the wrong answer looked like a clean result. Did you find any case where the query side and the document side disagreed but still returned results, rather than the usual zero-hit?