The Boss Fight, Attempt Number Five
If you've read straight through from Article 03, you've met our old adversary Q8 by now.
It's query number 8 of our 12-query test set: process payment and create Stripe charge. Its ground truth has two functions, one of which, calculate_order_total, has evaded us for four full articles.
- Article 03, vector-retrieval baseline — it missed.
- Article 04, three chunking strategies — it missed again.
- Article 05, graph-augmented retrieval finally scooped it back — at the cost of a Q1 regression, leaving the total unmoved.
- Article 06, encoding
called_bystructural info into the embedding lifted the similarity from 0.46 to 0.51, the right direction, but couldn't push past the wall of literal vocabulary, and it missed once more.
Four articles, five encounters (Article 04 tried three chunking variants), and Q8 is like that boss that keeps respawning — every time you think you've found the opening, it stands back up in a new stance.
For this article, I brought the "ultimate weapon" the industry universally endorses — BM25 keyword retrieval — and braided it together with vector retrieval into hybrid search. This combination has been validated as the best-practice augmentation for vector retrieval across countless production systems. The logic is straightforward: vector retrieval relies on semantics, BM25 on literal term frequency; their failure modes differ, so in theory they should cover for each other. That's exactly what I planned at the end of Article 06 — use two orthogonal paths to cover each other's blind spots, and Q8 should finally be saveable.
Let me lay out the conclusion up front so you don't cheer prematurely halfway through: Q8 still failed. Worse, hybrid search inherited BM25's regression on another query, dropping the total from 0.958 to 0.931 — worse than pure vector retrieval.
But this failure is the most valuable one in the entire series. Because it isn't "yet another approach that didn't pan out" — it's the one that fully unmasked Q8's root cause and measured exactly where the boundary of the whole text-matching route lies. By the end of this article, you'll understand why every failure in the previous four articles was inevitable, and why they were all, in fact, doing the same thing.
Two Concepts First: BM25 and RRF
Before we get hands-on, let's spend two minutes making two terms clear.
BM25 is a classic information-retrieval algorithm. Think of it as "weighted keyword matching." It doesn't understand semantics — it counts term frequency: the more often a query term appears in a document, and the rarer that term is across the whole corpus, the higher that document scores. Search engines, and Elasticsearch's default relevance ranking, run on it underneath.
An analogy: vector retrieval is like a well-read linguist — say 'collect money' and they know you might mean 'checkout,' 'charge,' 'payment gateway'; BM25 is like a meticulous librarian — say 'collect money' and they go find which book has the exact words 'collect money' most often. The linguist understands meaning but might overthink; the librarian is literal but never mixes things up.
RRF (Reciprocal Rank Fusion) is a way to merge multiple retrieval result lists into one ranking. It ignores each channel's raw scores (vector cosine similarity and BM25 scores aren't even on the same scale, so you can't just add them) and looks only at rank. If a function ranks 2nd in the vector results and 5th in the BM25 results, RRF scores it 1/(k+2) + 1/(k+5), where k is a smoothing constant — the industry standard is 60. Whatever ranks near the top across multiple channels gets the highest fused score.
The RRF code is charmingly short:
def rrf_fuse(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
"""Reciprocal Rank Fusion. k=60 is the standard constant."""
scores: dict[str, float] = {}
for ranked in ranked_lists:
for rank, name in enumerate(ranked):
scores[name] = scores.get(name, 0.0) + 1.0 / (k + rank + 1)
return sorted(scores, key=lambda n: scores[n], reverse=True)
That's it. RRF's virtues: no weight tuning, insensitive to scale, robust — the most low-maintenance fusion method in hybrid search.
So this article's three contenders are:
- A_vector: pure vector retrieval. The old baseline since Article 03.
- B_bm25: pure BM25 keyword retrieval.
- C_hybrid: vector + BM25, fused with RRF.
[Image: hybrid search pipeline. A query box on the left splits into two arrows: top arrow to a "Vector search" box, bottom arrow to a "BM25 search" box. Each produces a ranked list. Both lists feed into an "RRF fusion" box on the right, which outputs a final ranked list. A small note under RRF reads "merge by rank, k=60".]
The Crucial Step: How BM25 Tokenizes Code
BM25's effectiveness rides almost entirely on tokenization — it only recognizes the tokens it splits out, and if it splits wrong, everything downstream collapses. Code isn't natural language, and names like calculate_order_total, processCheckout, create_payment_intent fed into an ordinary tokenizer just yield an unmatched blob of joined strings.
So we need a tokenizer that understands code naming conventions: first split on non-alphanumeric characters, then break snake_case, then break camelCase:
def tokenize(text: str) -> list[str]:
# split on non-alphanumeric characters
raw = re.split(r"[^a-zA-Z0-9_]", text)
tokens = []
for tok in raw:
if not tok:
continue
# snake_case split
parts = tok.split("_")
for part in parts:
# camelCase split: insertBefore -> [insert, Before]
sub = re.sub(r"([a-z])([A-Z])", r"\1 \2", part).split()
tokens.extend(s.lower() for s in sub if len(s) > 1)
return tokens
With this, create_payment_intent splits into ['create', 'payment', 'intent'], and processCheckout into ['process', 'checkout']. Words get broken down to the finest grain, so BM25 can match them against the query's words.
Now, before we reveal the final results, let's do one thing — run the tokenizer on Q8 and see what the repeatedly-missed calculate_order_total can actually match. This step is the whole article's deciding move.
Q8 query, tokenized:
Query: 'process payment and create Stripe charge'
Tokens: ['process', 'payment', 'and', 'create', 'stripe', 'charge']
Six tokens. Now look at three candidate functions and how much overlap each one's full token set has with the query:
calculate_order_total:
Token overlap with query: set() <- zero overlap!
Total tokens: 59
create_payment_intent:
Token overlap with query: {'stripe', 'create', 'payment'}
process_checkout:
Token overlap with query: {'stripe', 'create', 'payment', 'process'}
Look at the first line. Across all 59 tokens of calculate_order_total, its overlap with the query is — set(). The empty set. Zero. Not a single word matches.
This isn't "the match wasn't good enough" — it's "there's nothing to match at all." BM25 is that literal librarian; ask it "which book has payment, stripe, charge most often," and it flips through the entire calculate_order_total book, finds not one such word, and confidently gives it a score of 0.
Contrast with create_payment_intent and process_checkout — their code plainly contains payment, stripe, create, process, overlap by the handful, and BM25 immediately ranks them up front.
By now you can probably already sense the result: BM25, the "ultimate weapon," is completely ineffective on calculate_order_total, because it shares not a single word with the query. And vector retrieval already proved in Article 06 that it can't reach it either (semantically too far). Two paths — one by semantics, one by frequency — both fail on the same function.
Let's run the results and confirm the hunch.
Result: Not Only Not Fixed, but Regressed
Running all three approaches against the 12 queries, the totals:
Approach R@3 R@5 vs Vector
───────────────────────── ─────── ─────── ──────────
A_vector 0.889 0.958 base
B_bm25 0.889 0.931 -0.028
C_hybrid 0.889 0.931 -0.028
The first glance stings: both BM25 and Hybrid have Recall@5 of 0.931, 0.028 lower than pure vector's 0.958. We eagerly added a keyword-retrieval channel, and after fusion the total went down, not up.
Now the per-query Recall@5, to see exactly where the points were lost:
Query Vec BM25 Hyb
────────────────────────────────────────────────── ────── ────── ──────
verify user identity and check JWT token validity 1.00 1.00 1.00
encrypt and store user password securely 1.00 1.00 1.00
generate JWT access token for authenticated user 1.00 1.00 1.00
check if user has permission to perform an action 1.00 1.00 1.00
store and retrieve data from Redis cache 1.00 1.00 1.00
limit how many times a user can call an API 1.00 1.00 1.00
execute SQL query safely against the database 1.00 0.67 0.67 <- Q7, BM25 and Hybrid both regress
process payment and create Stripe charge 0.50 0.50 0.50 <- Q8, all three fail
issue refund to customer 1.00 1.00 1.00
send email notification to user 1.00 1.00 1.00
send mobile push notification 1.00 1.00 1.00
delete a record without permanently removing it fr 1.00 1.00 1.00
Two arrows tell the whole story:
-
Q8 (
process payment and create Stripe charge): Vec, BM25, Hyb are all 0.50. Identical to the previous four articles — hitting only one of the two ground-truth functions. Our "ultimate weapon" misfired. -
Q7 (
execute SQL query safely against the database): vector is a perfect 1.00, but BM25 drops to 0.67, and Hybrid drops to 0.67 along with it. That's the culprit dragging down the total — and Hybrid, because it fused BM25's results, inherited this regression wholesale.
One query we wanted to fix stayed broken; one query that was perfect got dragged down by BM25, and the damage spread to Hybrid. That's the entire source of the "total score regression."
Let's look at Q8's details first and nail down the hunch.
Q8: Semantics Can't Reach It, Frequency Can't Either
Pull out the top-5 for all three approaches on Q8:
Q8 top-5:
A_vector: ['process_checkout', 'process_refund', 'create_payment_intent', 'get_payment_history', 'verify_webhook_signature']
B_bm25: ['process_checkout', 'process_refund', 'create_payment_intent', 'verify_webhook_signature', 'get_payment_history']
C_hybrid: ['process_checkout', 'process_refund', 'create_payment_intent', 'get_payment_history', 'verify_webhook_signature']
calculate_order_total: token overlap with Q8 = set() <- completely zero overlap
All three approaches' top-5 hold the same batch of "payment-vocabulary-rich" functions: process_checkout, process_refund, create_payment_intent, get_payment_history, verify_webhook_signature. Their names and bodies are full of payment, stripe, charge, refund — whether you measure by semantics or by frequency, they rank comfortably up front.
And the one we actually want, calculate_order_total, can't reach the top-5 in any of the three. The reason is that set() we saw ahead of time:
- Vector retrieval can't reach it — Article 06 already measured this: its semantic distance from the query is too far, and even with a structural prefix it only got to 0.51, unable to climb over that wall starting at 0.51.
- BM25 reaches it even less — its token overlap with the query is the empty set, so BM25 gives it a flat 0, denying it even the right to participate in ranking.
-
Hybrid RRF fusion — RRF lifts scores based on "who ranks near the top across channels." But
calculate_order_totalranks outside the top-5 in vector and dead-last at 0 in BM25. Both channels rank it low, so what can fusion conjure? Two out-of-reach signals added together are still out of reach. RRF isn't magic; it can't fabricate a high rank that neither channel gave.
This is Q8's most essential truth: it isn't ranked at slot 6, just barely missing — it's that no text signal, of any kind, can even perceive its existence. Semantics and frequency are the two pillars of text retrieval, and Q8 punched through both.
[Image: Q8 dual failure. Two parallel channels on the left labeled "Vector (semantic)" and "BM25 (lexical)". A function node calculate_order_total in the middle. From the Vector channel, a dashed arrow labeled "too far, sim 0.51" fails to reach it; from the BM25 channel, a dashed arrow labeled "token overlap = empty set" also fails to reach it. Both arrows greyed out. On the right, an RRF box receives two "not found" signals and outputs "still not found".]
Q7: BM25's Crash Is Kin to Article 06's
Now the query BM25 dragged down and infected Hybrid with, Q7: execute SQL query safely against the database. Its ground truth is three functions: execute_query, bulk_insert, paginate_query.
Vector retrieval hit all three cleanly (1.00), but BM25 dropped to 0.67, missing one.
Why? The problem is the word execute. execute_query is a classic hub function (high-out-degree utility), called by a whole crowd of functions in the database module. And the word "execute" — it's just too common. Plenty of functions with nothing to do with SQL queries also have execute in their body (execute an action, execute a callback, execute a task...), and BM25 indiscriminately matches their execute token against the query's execute, inflating their scores, scrambling the ranking, and bumping one of bulk_insert or paginate_query right out of the top-5.
This pitfall — look familiar? It's the same root cause as Strategy B's crash in Article 06. There, Strategy B indiscriminately printed the hub function execute_query's name at the head of every caller's embedding, creating execute query literal noise, and Q7 regressed. Here, BM25 lifts a crowd of irrelevant functions via literal matching on the high-frequency generic word "execute," and again Q7 regresses.
Different mechanisms, same code smell: hub node + high-frequency generic word = literal noise polluting the ranking. Article 06 was stuffing the hub name into embeddings to pollute; this one is BM25 over-matching hub-related generic words directly. On the text route, as long as there's a word like execute that's both generic and high-frequency, pure frequency matching will crash.
And Hybrid? It fused vector and BM25 rankings via RRF. Vector is perfect on Q7 and should have corrected BM25's mistake. But RRF is a "weighted average of the two rankings" — BM25 ranked the bumped-out ground-truth function too far back, and even though vector ranked it near the top, the fused ranking was still dragged down by BM25 and couldn't squeeze back into the top-5. Hybrid failed to isolate BM25's error; instead it inherited it, diluted. That's why its total is also 0.931.
Five Experiments, All Doing the Same Thing
Reaching this point, it's time to connect Articles 03 through 07 and see what we've actually been doing.
On the surface, these five articles are five different technical approaches. But step back and you'll see they're essentially systematically investigating one question: is it even possible to find calculate_order_total via text matching?
We tried every variant of the "text route":
-
Semantic vectors? No.
calculate_order_total(compute tax) and "Stripe payment" (collect money) are two different things in the real world; the semantic distance is too far, and the vector simply can't get close. (Articles 03, 04) - Comment / docstring augmentation? No. Its docstring is also "sum prices, apply discount, compute tax" — still order, discount, tax, and no payment flavor can be mixed in. (Article 06 Strategy C)
-
Structural prefix? Right direction, not enough. Adding
# called_by: process_checkoutpushes similarity from 0.46 to 0.51, but an entire row of payment functions starting at 0.51 blocks the way. (Article 06 Strategy B) - BM25 keyword? Even less. Token overlap is the empty set — it can't even get above 0, no right to participate in ranking. (this article)
- Hybrid RRF? Still no. Fusing two out-of-reach signals is still out of reach, and it inherits BM25's Q7 regression to boot. (this article)
See it? All five approaches are the "text route" — all trying to find, in calculate_order_total's text content, a thread connecting to "Stripe payment." And every failure told us the same thing:
That thread doesn't exist in the text at all.
calculate_order_total's body has subtotal, discount, tax, price, quantity. From start to finish it's computing an order's total, with none of the words payment, stripe, charge. Measure it from any text angle — semantics, frequency, comments, structural prefix — and you won't find its link to "Stripe payment," because that link simply isn't written in its text.
So where is that link hidden? In one code structural relationship: calculate_order_total is called by process_checkout, and process_checkout is the function that actually calls the Stripe payment gateway. It's the structural fact of "who calls it" that binds calculate_order_total to the payment flow. That information can never be encoded in text content — it isn't inside any function body; it's on the edges between functions.
This is why Article 05's graph retrieval was the only approach to ever fix Q8: it didn't take the text route; it walked directly along the call edge process_checkout → calculate_order_total and dragged the function into the candidate set. It bypassed text and read structure directly.
Series Recap: The Boundary of the Text Route, Measured
Aggregate all five articles' results into one table, and the whole route's outline becomes crystal clear:
| Article | Approach | Q8 Status | Total R@5 |
|---|---|---|---|
| 03 | AST function-level + raw-code embedding | Fail (0.50) | 0.958 |
| 04 | Chunking comparison (fixed-line / file / AST) | AST fails | 0.958 |
| 05 | Graph-augmented retrieval (BFS 2-hop) | Fixed! but Q1 regresses | 0.958 |
| 06 | Structural embedding (called_by prefix) | Fail (+0.05 not enough) | 0.958 |
| 07 | BM25 + vector RRF | Fail (zero token overlap) | 0.931 ↓ |
Five approaches, and the total never exceeded the baseline 0.958. Q8 was truly fixed only once, in Article 05 — and that one time was precisely the one approach that stepped outside text and read code structure directly. Article 07's hybrid search, supposedly the strongest move on the text route, not only failed to fix Q8 but, by inheriting BM25's literal noise, pulled the total to the series-low 0.931.
This isn't a demoralizing "failed again" ending. Quite the opposite — these five experiments precisely measured the boundary of the text-matching route. They give us a conclusion that was once only a vague feeling and now has numbers behind it:
The link between
calculate_order_totaland "Stripe payment" is purely structural. It exists in the code's call graph, not in any function's text content. Therefore, no pure text-matching approach — vector, BM25, or their hybrid — can cover this class of query. This isn't an algorithm-choice problem, not a parameter-tuning problem, not a chunk-size problem — it's the physical boundary of the route.
Knowing where the boundary is, in itself, is extremely valuable engineering knowledge. It stops us from drawing more water from the dry well of text.
The Way Out: Promote Graph Retrieval to a First-Class Citizen
Since the root cause is structural, the way out is clear too: stop treating graph retrieval as an add-on patch to vector retrieval, and promote it to a first-class citizen.
Article 05's naive graph expansion (mindless 2-hop BFS) was double-edged because it inflated the candidate set and introduced noise. But that isn't graph retrieval's own fault — it's "too crude a usage." The right usage should be precise, conditional structural supplementation:
If a function (say
process_checkout) appears in the vector top-k, explicitly query itscalled_by/callsneighbors, filter by business rules, and add the relevant neighbors (likecalculate_order_total) into the results.
This approach differs fundamentally from Article 05's mindless expansion:
- Trigger expansion only for high-confidence functions in the top-k, not an indiscriminate BFS over all candidates — avoiding candidate-set explosion.
-
Walk only deterministic
CALLSedges, capped at 1 hop, within the same module — avoiding pulling in unrelated functions. - Filter neighbors by business rules — e.g., only add neighbors that "participate in the payment flow," excluding pure utility functions.
This way, calculate_order_total gets scooped back precisely via the structural fact of "being called by process_checkout," without breaking Q1 the way Article 05 did. Graph retrieval is no longer an after-the-fact patch, but a first-class recall channel alongside vector retrieval — vector handles semantics, graph handles structure, each to its own job.
This is the right engineering posture for a "fundamental limit": not hunting for a more magical embedding trick, nor tuning better BM25 parameters, but admitting where the text route can't reach, and covering it with an orthogonal structural signal.
Final Chapter: The Closing of a Technical Route
Writing to this point, the vector-retrieval technical route is now formally complete.
From Article 03's embedding baseline, to Article 04's chunking comparison, to Article 05's graph augmentation, to Article 06's structural encoding, to this article's hybrid search — across five articles, we've all but exhausted the mainstream variants of "retrieving code by text similarity." We didn't find a pure-text approach that fixes Q8 without side effects, but we got something more valuable than a "fixed": a clear boundary line, and the right direction beyond it.
Q8, the ghost that haunted us for five articles, was in the end not tamed by a cleverer text trick, but seen through for what it is — a structural problem, not a text problem. That realization itself is the entire point of these five experiments.
So what's next? With the boundary of the text route now measured, continuing to chip away at vector and BM25 details yields diminishing marginal returns. The direction truly worth investing in is bringing code structure in as a first-class signal — which requires us to rise from the vantage point of "retrieval" to the more macroscopic engineering vantage point of "how to build a complete knowledge representation for a codebase."
So in the next series, we'll shift altitude: no longer fussing over a single retrieval algorithm's recall, but starting from the whole codebase knowledge base's system architecture, to see how vector, graph, and symbol-index signals should be organized into a genuinely usable production system. Q8's story ends here, but the line it leaves behind will keep reminding us:
Some links, by their very nature, are not in the text.
Summary
-
BM25 is completely ineffective on
calculate_order_total. Its token overlap with the Q8 query is the empty set (set()) — not one of its 59 tokens matchespayment/stripe/charge. BM25 gives it a flat 0, no right to participate in ranking. - All three approaches fail Q8 (0.50). Vector can't reach it (semantically too far), BM25 can't reach it (zero frequency), and Hybrid RRF fusing two out-of-reach signals is still out of reach. RRF isn't magic; it can't produce a high rank neither channel gave.
- Hybrid actually lowered the total. BM25, via over-matching the high-frequency generic word "execute," regressed Q7 from 1.00 to 0.67; Hybrid inherited this error through RRF and dropped to 0.67 too. Overall Recall@5 fell from 0.958 to 0.931 — worse than pure vector.
- Q7's crash is kin to Article 06's. hub function + high-frequency generic word = literal noise polluting the ranking. Article 06 stuffed the hub name into the embedding; this one has BM25 over-matching hub-related generic words — same code smell.
-
Five experiments measured the boundary of the text route. Semantics, comments, structural prefix, BM25, Hybrid — every text approach tripped on Q8. Because the link between
calculate_order_totaland "Stripe payment" is purely structural (it's called byprocess_checkout), and simply isn't written in any function's text. -
The way out is promoting graph retrieval to a first-class citizen. Not adding tricks to embedding or tuning BM25, but admitting where text can't reach and covering it with an orthogonal structural signal: for high-confidence top-k functions, explicitly query their
called_by/callsneighbors, filter by business rules, and add them into the results.
The vector-retrieval technical route closes here. In the next series, we reorganize vector, graph, and symbol-index signals from a more macroscopic system-architecture vantage point.
References
- Full demo code: codebase-kb-07-hybrid
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)