BM25 vs Embeddings: Why Your RAG Can't Find Error Code PGX-4012

A support engineer pinged me with a screenshot. They had asked our internal docs bot "what does PGX-4012 mean?" and it answered, with total confidence, by explaining PGX-4021.

Different error. Different fix. Same four digits, shuffled.

The page for PGX-4012 was in the index. It was chunked fine. It had a heading with the exact string in it. The vector search just didn't care. That was my introduction to the real BM25 vs embeddings tradeoff: dense retrieval is great at meaning and bad at exact identifiers, and a RAG pipeline built on embeddings alone will quietly fail on the queries your users care about most.

TL;DR

• Embedding models compress a whole chunk into one vector. A short identifier like PGX-4012 is a handful of subword tokens out of hundreds, so its signal gets averaged away.
• Embedding models are trained to find similar meaning. To them, PGX-4012 and PGX-4021 mean almost the same thing: "a database error code."
• BM25 scores exact term overlap weighted by rarity (IDF). A term that appears in one document out of 3,000 gets a huge weight, which is exactly what you want for error codes, SKUs, function names and ticket IDs.
• The fix is hybrid search: run BM25 and vector search in parallel, then merge with Reciprocal Rank Fusion (RRF), which combines ranks instead of raw scores.
• Your BM25 tokenizer matters. If it splits or strips PGX-4012, you lose the advantage.

Why does vector search miss exact error codes?

Vector search misses exact identifiers because an embedding represents the gist of a chunk, and an identifier is a tiny, semantically boring part of that gist.

Here's the pipeline for a typical sentence-transformer model:

1. The tokenizer splits text into subword pieces.
2. The transformer produces one vector per token.
3. A pooling step (usually mean pooling or the CLS token) squashes all of those into a single vector.

Run this and look at what your identifier becomes:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-small-en-v1.5")
print(tok.tokenize("PGX-4012 connection pool exhausted"))

You won't get one token for PGX-4012. You'll get several fragments: some letters, a hyphen, digit chunks. Now put those fragments inside a 300-token chunk about connection pooling, timeouts and retry settings. After pooling, the identifier is a few drops of dye in a bucket of water.

Then there's training. Models like this are tuned on pairs of text that mean the same thing. Nothing in that objective teaches the model that 4012 and 4021 are unrelated. Both are "four digits after an error prefix in a Postgres-ish context." Semantically, they are neighbors. So the PGX-4021 page, which happens to share more surrounding vocabulary with the user's question, wins.

The model isn't broken. It's doing its job. Its job just isn't exact match.

How does BM25 find what embeddings miss?

BM25 finds exact identifiers because it scores documents by literal term overlap, and it weights each term by how rare it is across the whole corpus.

The core of the score for one query term looks like this:

score(term, doc) = IDF(term) * tf * (k1 + 1)
                   / (tf + k1 * (1 - b + b * doc_len / avg_doc_len))

Two parts matter here:

• IDF (inverse document frequency) is large when a term appears in very few documents. connection appears in hundreds of pages. pgx-4012 appears in one. That one term dominates the score.
• tf with saturation (controlled by k1) means repeating a term helps, but with diminishing returns. b normalizes for document length so long pages don't win by volume.

Common defaults are k1=1.2, b=0.75 in Elasticsearch and Lucene, and k1=1.5, b=0.75 in the Python rank_bm25 package. You rarely need to touch them.

BM25 has no idea what "my database keeps dropping connections" means. It will whiff on paraphrases, synonyms and questions written in different words than the docs. But hand it a rare exact string and it goes straight to the right page.

What did I actually see in my own pipeline?

I pulled 50 real questions from our support bot logs that contained some kind of identifier: error codes, config keys, CLI flags, internal service names. Then I checked by hand whether the correct page appeared in the top 5.

This is a spot check on my own docs, not a benchmark. But the pattern was loud:

• Vector only: the right page was in the top 5 for 31 of 50.
• BM25 only: 44 of 50.

The misses were not random. Vector search failed on near-twin identifiers (4012 vs 4021), on config keys that shared a prefix (pool.max_idle vs pool.max_lifetime), and on flags buried in long chunks. BM25 failed on the handful of questions where the user typed the code wrong or described it instead of pasting it.

Then I ran the opposite check: 50 conceptual questions with no identifiers ("why do my workers time out after deploys?"). There, vector search clearly beat BM25. Which is the whole point. Neither retriever is the good one. They fail on different queries.

How do you combine BM25 and embeddings in a hybrid search?

Run both retrievers, then merge their result lists with Reciprocal Rank Fusion. RRF ignores raw scores and only looks at each document's rank in each list:

RRF(doc) = sum over retrievers of 1 / (k + rank)

k=60 is the constant from the original RRF paper by Cormack, Clarke and Buettcher, and it's the default rank constant in Elasticsearch's RRF too. It keeps a single first-place finish from completely dominating.

Why not just add the scores with a weight? Because they live on different planets. BM25 scores are unbounded and shift with query length and corpus stats. Cosine similarities from many embedding models bunch up in a narrow band. A 0.7 * bm25 + 0.3 * cosine formula that works on Monday breaks when someone asks a longer question on Tuesday. Ranks are always comparable.

Quick math with k=60:

Document	BM25 rank	Vector rank	RRF score
A	1	30	1/61 + 1/90 = 0.0275
B	not found	3	1/63 = 0.0159
C	5	5	2/65 = 0.0308

Consensus wins, a strong single-retriever hit still beats a document only one side found, and you never had to normalize anything.

What does a minimal hybrid retriever look like in Python?

About 30 lines with rank_bm25 and sentence-transformers:

import re
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

def tokenize(text):
    # keep identifiers like pgx-4012 or pool.max_idle as one token
    return re.findall(r"[a-z0-9]+(?:[-_.][a-z0-9]+)*", text.lower())

docs = load_chunks()  # list[str]
bm25 = BM25Okapi([tokenize(d) for d in docs])

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
doc_vecs = model.encode(docs, normalize_embeddings=True)

def hybrid_search(query, top_k=5, pool=50, rrf_k=60):
    s = bm25.get_scores(tokenize(query))
    bm25_ids = [i for i in np.argsort(-s)[:pool] if s[i] > 0]

    q = model.encode(query, normalize_embeddings=True)
    dense_ids = np.argsort(-(doc_vecs @ q))[:pool]

    fused = {}
    for ranking in (bm25_ids, dense_ids):
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1 / (rrf_k + rank)
    return sorted(fused, key=fused.get, reverse=True)[:top_k]

Two details that bit me:

1. The s[i] > 0 filter. Without it, BM25 "ranks" hundreds of documents that share zero terms with the query, in arbitrary order, and they pollute the fusion with fake votes.
2. The tokenizer. My first version used text.lower().split(), so PGX-4012. with a trailing period became its own useless token and never matched. A tokenizer that strips punctuation entirely is worse: pgx and 4012 become separate, and 4012 alone might also match a port number somewhere. Keep identifiers whole.

If you're on Elasticsearch, OpenSearch, Weaviate, Qdrant or pgvector plus Postgres full-text search, most of them have some form of hybrid or fusion built in. Check what fusion method they use and what analyzer tokenizes your text field. The analyzer is where identifier handling quietly dies.

Should you add a reranker on top?

Yes, if latency allows. A cross-encoder reranker reads the query and each candidate together, token by token, so it can actually notice that 4012 in the query matches 4012 in the passage. Take the top 20 to 50 from hybrid retrieval, rerank, keep the top 5.

But a reranker can only reorder what it's given. If pure vector search never put PGX-4012 in the candidate pool, no reranker will save you. Retrieval recall comes first. That's what BM25 buys you.

When is pure embedding search actually fine?

Pure vector search is fine when your users ask in natural language and your corpus has few exact-match anchors: support articles written as prose, meeting notes, essays, product FAQs. The moment your data contains error codes, API names, SKUs, version strings, legal clause numbers or ticket IDs, you want lexical matching in the loop.

A cheap test before you build anything: grep your query logs for anything matching [A-Z]{2,}-?\d{3,} or containing backticks, dots between words, or --flags. If that's more than a sliver of traffic, go hybrid.

So why can't your RAG find PGX-4012?

Your RAG can't find error code PGX-4012 because embedding models turn each chunk into one averaged vector of meaning, and an exact identifier is a few subword tokens that barely move that vector, while its near-twin PGX-4021 means almost the same thing to the model. BM25 solves the exact-match side because it scores literal term overlap weighted by rarity, so a code that appears in one document dominates. In the BM25 vs embeddings debate, the answer is both: run them in parallel, merge with Reciprocal Rank Fusion at k=60, keep identifiers intact in your BM25 tokenizer, and put a cross-encoder reranker on top if you can afford the latency.

---

Written by the developer behind Preterview, an interview prep platform.