Making Good on a Promise
At the end of Article 05, I left a clear piece of homework. Since naive graph expansion is double-edged (it fixed Q8 but broke Q1), let's switch approaches — don't traverse the graph at retrieval time; encode the structural information into the embedding content at index time instead.
That article included a sketch of the code: add the line # called_by: process_checkout to calculate_order_total's chunk so its own vector carries the signal "I'm called by the checkout flow." In theory, when the query "process payment" hits process_checkout-related tokens, calculate_order_total's vector itself becomes perceptible — no candidate-set inflation, and thus no Q1-style crowd-out noise.
Sounds beautiful. This article is where we throw that hypothesis into the experimental furnace.
I prepared three strategies, all embedding the same payment-module code:
- Strategy A (raw): raw code, nothing added. This is the baseline.
-
Strategy B (struct prefix): add a comment line before the function body spelling out
called_byandcalls. - Strategy C (struct doc): fold the structural info into the function's existing docstring, without a separate line.
Then I ran all three embeddings against the same 12 queries to see whether Q8 gets fixed.
Let me spoil the conclusion so you don't think I'm building suspense: structural info does work — calculate_order_total's similarity really did go up. But not enough, and Q8 still failed. And Strategy B managed to break another query that was previously perfect.
This isn't a demoralizing "all that effort for nothing" story. Quite the opposite — this failure precisely measured where the boundary of embedding lies, and that number itself is more valuable than a "fixed."
What the Three Strategies Look Like
First, an intuitive feel for how the text fed to the embedding differs across strategies. Take our repeatedly-missed protagonist calculate_order_total:
Strategy A (raw):
def calculate_order_total(items: list[dict], discount_code: str = None) -> dict:
"""Sum item prices, apply discount, compute tax. Returns breakdown dict."""
subtotal = sum(item["price"] * item["quantity"] ...
Strategy B (struct prefix - comment before the function body):
# called_by: process_checkout
def calculate_order_total(items: list[dict], discount_code: str = None) -> dict:
"""Sum item prices, apply discount, compute tax. Returns breakdown dict."""
subtotal = sum(item["price"] * item["quantity"] ...
Strategy C (struct doc - structural info added into the docstring):
def calculate_order_total(items: list[dict], discount_code: str = None) -> dict:
"""Sum item prices, apply discount, compute tax. Returns breakdown dict. Called by: process_checkout."""
subtotal = sum(item["price"] * item["quantity"] ...
That's the whole difference: B jams # called_by: process_checkout at the very top, C stitches Called by: process_checkout. onto the end of the docstring. The core intent is identical — both want the token process_checkout to appear in calculate_order_total's embedding text, so it drifts a little closer to the payment flow in vector space.
The difference is where. B puts it outside the function body, on the most prominent first line; C hides it in the docstring, mixed in among natural-language description. This "where" difference leads to an unexpected side effect later.
Both B and C are short to implement:
# Strategy B: struct prefix
def strategy_b_struct_prefix(func, call_graph, called_by):
callers = called_by.get(func["name"], [])
callees = call_graph.get(func["name"], [])
prefix = []
if callers:
prefix.append(f"# called_by: {', '.join(callers)}")
if callees:
prefix.append(f"# calls: {', '.join(callees)}")
if prefix:
return "\n".join(prefix) + "\n" + func["text"]
return func["text"]
# Strategy C: struct info fused into docstring
def strategy_c_struct_doc(func, call_graph, called_by):
callers = called_by.get(func["name"], [])
callees = call_graph.get(func["name"], [])
if not callers and not callees:
return func["text"]
struct_note = []
if callers:
struct_note.append(f"Called by: {', '.join(callers)}.")
if callees:
struct_note.append(f"Calls: {', '.join(callees)}.")
# inject to the end of the existing docstring
old_doc = f'"""{func["docstring"]}"""'
new_doc = f'"""{func["docstring"]} {" ".join(struct_note)}"""'
return func["text"].replace(old_doc, new_doc, 1)
Note that B adds a prefix to every function that has a call relationship — not just calculate_order_total. This detail is the seed of Q7's crash later on.
[Image: three strategies fed to embedding. Three side-by-side code cards labeled Strategy A (raw), Strategy B (prefix comment on top), Strategy C (note fused into docstring). An arrow from each card points into a shared box labeled "Embedding model", and a note reads "same intent: inject 'process_checkout' token".]
The Main Comparison: One Stable, One Regresses, Neither Fixes Q8
Running all three embeddings against the 12 queries, the totals look like this:
Strategy R@3 R@5 vs A
───────────────────────── ─────── ─────── ───────
A_raw_code 0.889 0.958 base
B_struct_prefix 0.889 0.931 -0.027
C_struct_doc 0.889 0.958 base
Don't rush to find Q8 yet. The first thing that jumps out is that Strategy B actually regressed: Recall@5 dropped from 0.958 to 0.931. We added structural info, and the total went down, not up. Meanwhile Strategy C held steady at the baseline, not moving a point.
Now the per-query Recall@5, where the truth comes out:
Query A B C
────────────────────────────────────────────────── ────── ────── ──────
verify user identity and check JWT token validity 1.00 1.00 1.00
encrypt and store user password securely 1.00 1.00 1.00
generate JWT access token for authenticated user 1.00 1.00 1.00
check if user has permission to perform an action 1.00 1.00 1.00
store and retrieve data from Redis cache 1.00 1.00 1.00
limit how many times a user can call an API 1.00 1.00 1.00
execute SQL query safely against the database 1.00 0.67 1.00 ←
process payment and create Stripe charge 0.50 0.50 0.50 ← (Q8, all strategies fail)
issue refund to customer 1.00 1.00 1.00
send email notification to user 1.00 1.00 1.00
send mobile push notification 1.00 1.00 1.00
delete a record without permanently removing it fr 1.00 1.00 1.00
Stare at the last two arrows — the whole story is there:
-
Q8 (
process payment and create Stripe charge): A, B, and C are all 0.50. We went to great lengths to encode structural info into the embedding, and Q8 didn't move — still hitting only one of the two relevant functions. The hypothesis is falsified. -
Q7 (
execute SQL query safely against the database): A and C both scored a perfect 1.00, but Strategy B dropped to 0.67. That's the culprit dragging B's total down.
One query we wanted to fix stayed broken; one query that was fine got broken by B. This shape sounds familiar — Article 05's graph expansion also "fixed one, broke one." But this time the reason is entirely different, and more fundamental.
Let's take them one at a time.
Q8: Structural Info Really Did Help — Just Not Enough
Q8 first — the true protagonist of this article. I pulled out the cosine similarities between the query and a few key functions:
Query: process payment and create Stripe charge
Cosine similarity to the query:
0.4617 calculate_order_total (raw)
0.5137 calculate_order_total (+called_by: process_checkout prefix)
0.5970 create_payment_intent (raw)
0.6341 process_checkout (raw)
Look at lines one and two. After calculate_order_total got the # called_by: process_checkout prefix, its similarity to the query rose from 0.4617 to 0.5137 — a full +0.052.
This is good news, and the direction is exactly right. Structural info isn't useless — it genuinely pulled calculate_order_total closer to the query. Article 05's hypothesis, "structural encoding raises a related function's vector similarity," is confirmed in the numbers. We didn't waste our time.
The bad news is in lines three and four. In the same data, create_payment_intent sits at 0.5970 and process_checkout at 0.6341 — both naturally a good chunk higher than calculate_order_total. Why? Because their code plainly spells out payment, Stripe, charge — words that literally match the query.
Now do the math: calculate_order_total rose from 0.4617 to 0.5137, up 0.052. But the crowd standing in front of it — not just create_payment_intent and process_checkout, but process_refund, get_payment_history, verify_webhook_signature, all functions carrying payment vocabulary — every one of them is above 0.51. Look at where it lands in the end:
Q8 top-5 hits:
A_raw_code: ['process_checkout', 'process_refund', 'create_payment_intent', 'get_payment_history', 'verify_webhook_signature']
B_struct_prefix: ['create_payment_intent', 'process_checkout', 'process_refund', 'get_payment_history', 'verify_webhook_signature']
C_struct_doc: ['process_checkout', 'process_refund', 'create_payment_intent', 'get_payment_history', 'verify_webhook_signature']
Across all three strategies, top-5 slots 4 and 5 are always get_payment_history and verify_webhook_signature — impostors carrying payment vocabulary that aren't ground truth. And the one we actually want, calculate_order_total, is pinned firmly beyond slot 6 by these five "payment-vocabulary-rich" functions.
A +0.052 boost sounds substantial, but it's simply not enough. It pushed calculate_order_total from 0.46 to 0.51, yet couldn't push it past that wall — a wall that starts at 0.51 and is held up by literal vocabulary. So close, yet forever uncrossable — because standing in front of it is an entire row of functions "with payment in the name."
[Image: Q8 similarity ladder. A vertical axis of cosine similarity. calculate_order_total(raw) at 0.46, an upward arrow labeled "+0.052 struct" to calculate_order_total(struct) at 0.51. Just above, a cluster of bars at 0.51-0.63 labeled create_payment_intent / process_refund / get_payment_history / verify_webhook_signature / process_checkout, drawn as a wall. calculate_order_total still sits just under the wall, tagged "still below top-5".]
Root Cause: This Is a Semantic Gap, Not a Ranking Bug
At this point, many will think: what if I add structural info more aggressively? A few more comment lines, repeat process_checkout several times, or even splice the entire call chain in?
Stop. That direction is a dead end, and it's worth explaining why.
calculate_order_total's semantics are "sum prices, apply discount, compute tax." That is the true meaning of its code, and the embedding model faithfully encodes it into an "order-computation" vector. Meanwhile "create Stripe charge" means "call a third-party payment gateway, create a charge." These are two different things in the real world — one computes money owed, the other collects it. Their semantic distance is determined by the embedding model's knowledge; it isn't something our comment tricks can bridge.
The line we added, # called_by: process_checkout, is essentially mixing a small pinch of "checkout flow" flavor into a vector for "tax-computing code." It did tilt the vector toward payment a bit (+0.052), but that pinch of flavor can't overpower the code body's own strong signal of "I am computing tax." When the model reads this code, the substance is still sum, discount, tax — one comment line can't change that fundamental base.
Put differently: structural injection is a fine-tune of the vector, not a redirect. It can push a "close but not quite" function over the line, but for a function like calculate_order_total, separated from the query by an entire business concept, +0.052 is a drop in the bucket.
This is the essence of the semantic gap: it isn't a bug in the ranking algorithm, isn't bad chunking, isn't insufficient embedding dimensions. It's the objective distance between "the true meaning of this code" and "the intent the query expresses." That distance cannot be fundamentally crossed by any trick applied to embedding content — swapping chunk strategies, adding structural prefixes, stuffing docstrings.
Look back at Article 05: graph expansion fixed Q8 precisely because it didn't go through the embedding path. It relied on the deterministic call edge process_checkout → calculate_order_total to drag the function into the candidate set directly — bypassing the verdict of semantic similarity. This article's three strategies all still operate within the framework of embedding similarity, so naturally they all crash into the same wall.
Q7: Strategy B's Crash Is Another Warning
Now the query Strategy B broke, Q7: execute SQL query safely against the database. Its ground truth is three functions: execute_query, bulk_insert, paginate_query.
A and C both hit all three cleanly (1.00), but B dropped to 0.67 — missing one.
Why only B? Recall Strategy B's implementation: it added a structural prefix to the very top of every function with a call relationship. execute_query is a classic utility function, called by a whole crowd of functions in the database module — bulk_insert calls it, paginate_query calls it, and even the payment module's get_payment_history calls it. So the embedding text of every one of these callers got a line # calls: execute_query slapped at the top by Strategy B.
There's the problem. That line # calls: execute_query carries the words execute and query — which happen to overlap heavily with Q7's query "execute SQL query." As a result, a batch of functions that weren't really that relevant to Q7 had their vector scores artificially inflated because their prefix now contained "execute query," scrambling the ranking and crowding a genuinely relevant ground-truth function out of the top-5.
This echoes Article 05's hub-node lesson: execute_query is that high-out-degree utility function, and once its name is indiscriminately printed at the head of every caller's embedding text, it pollutes the ranking of a whole swath of queries. Article 05 was "expanding along a hub node detonates the candidate set"; this one is "stuffing a hub node's name into every caller's embedding detonates literal noise" — different mechanisms, same code smell.
Why doesn't Strategy C have this problem? Because it stitches the structural info into the docstring, mixed in among a whole stretch of natural-language description, rather than sitting alone at the very top. When # calls: execute_query stands on its own line, execute and query are naked high-weight tokens; but Calls: execute_query. buried among a docstring's sentences is diluted by the surrounding semantics, perturbing the overall vector far less. So C kept Q7's perfect score, at the cost of a gentler boost to Q8 too (we'll cover that trade-off next).
[Image: Q7 regression from prefix noise. Left, execute_query as a central hub node with edges from bulk_insert, paginate_query, get_payment_history. Right, each caller's embedding text shown with a red top line "# calls: execute_query", arrows pointing to a query box "execute SQL query" with a "spurious token match" tag. A bumped-out function marked red at slot 6.]
B vs C: Where You Put It Matters More Than What You Put
Putting Q8 and Q7 together, the contrast between Strategy B and C gets interesting:
-
Same structural info (
called_by/calls) — what B and C insert is nearly identical. - Different placement: B opens a separate line at the very top of the function; C stitches it into the docstring, mixed among the description.
-
Wildly different results: B's boost to Q8 is larger (
calculate_order_totalreached the edge of the top-5, even bumpingcreate_payment_intentto slot 1), but at the cost of Q7 crashing and the total regressing; C's boost to Q8 is gentler (not enough to reach top-5), but it wins on stability — introducing no regression.
There's a counterintuitive engineering lesson here: in embedding text, where the information sits and what form it takes can affect the vector as much as the information itself.
# called_by: process_checkout on its own line is a high-weight, high-purity signal — its boost to the target function (calculate_order_total) is strong, but its side effect on innocent high-out-degree utility callers is equally strong. Called by: process_checkout. fused into a docstring is a signal diluted by natural language — a small boost, and a small side effect.
No free lunch. B trades "stronger perturbation" for "a bigger boost to the target function," but that same strong perturbation brings bigger collateral noise. C trades "gentler perturbation" for "zero side effects," but is gentle enough that it fails to fix Q8.
And whether B or C, neither actually solved Q8 — because the magnitude of their perturbation (a few percentage points of similarity) was never in the range that could cross the semantic gap in the first place. Where you put it and how only decide whether "those few percentage points" are more precise or more noisy; they can't change the ceiling of "just those few percentage points."
What Did We Actually Measure?
Reaching Article 06 in this series, it's worth pausing to take stock. We've now tried the main variants of the "vector-retrieval technical route" one by one:
- Article 03: swapping embedding input (raw code vs signature-added vs doc-added) — Q8 failed.
- Article 04: swapping chunking strategy (whole-file vs function-level vs sliding window) — Q8 failed.
- Article 05: bolting on graph traversal at retrieval time — Q8 fixed, but at the cost of a Q1 regression, total unchanged.
- Article 06 (this one): encoding structural info into embedding content — Q8 failed, and Strategy B introduced a Q7 regression.
See the pattern? As long as we stay within the framework of "retrieval via embedding similarity," no matter how we optimize input, chunking, or content, Q8 won't pass. The one time Q8 got fixed (Article 05) was precisely by a mechanism that stepped outside embedding (deterministic call-edge traversal) — and that mechanism came with its own side effects.
This isn't because we implemented it poorly, or didn't tune enough. This is the boundary of the route. The semantic gap between calculate_order_total and "Stripe charge" is determined by the embedding model's real understanding of code semantics. Any effort to bypass it inside the embedding (whether input, chunking, or content augmentation) is essentially mixing "payment" flavor into a "tax-computing" vector — you can mix in a little, but not enough to climb over that wall built from literal vocabulary.
Every failure precisely measured the height of that wall. Articles 03 and 04 told us "the wall is there"; Article 05 told us "the cost of going around the wall"; Article 06 measured "how high you can push against the wall" — 0.052, far from enough. Together, these numbers point to a clear conclusion: the problem isn't how embedding is used — it's relying on embedding alone as the route.
Admit the Limit to Find the Way Out
So what should we do about Q8? Since the conclusion is "the embedding-similarity route is a dead end," the way out shouldn't be to keep piling tricks onto embedding — it should be to add mechanisms outside of embedding: admit its limit, then use other means to cover what it can't reach.
The direction is already clear:
1. BM25 keyword retrieval. In Q8's ground truth, calculate_order_total really is semantically distant from the query, but what it shares with the query is "business context" — it lives inside the payment flow. Embedding's weakness is purely semantic. Introducing a term-frequency-based keyword retrieval like BM25, with suitable query expansion, can scoop candidates from an orthogonal dimension. Keyword and vector retrieval have different failure modes, and combining them (that is, hybrid search) often covers each other's blind spots.
2. Explicit graph traversal (with denoising). Article 05 already proved the call edge process_checkout → calculate_order_total can fix Q8 directly. Its problem isn't uselessness — it's that naive bolt-on introduces noise. Given the constraints Article 05 described — walk only CALLS edges, restrict to the same module, cap at 1 hop — graph traversal can serve as a precise supplementary channel alongside vector retrieval, rather than a wildly swung double-edged sword.
3. A code symbol index. Many "misses" don't need semantic matching at all — the user is querying an explicit symbol, call relationship, or dependency path. A deterministic symbol index (who defines what, who references what) is more accurate than any embedding on this class of queries.
These three paths share one thing: none of them tries to make embedding better — they admit embedding has places it can't reach, then cover them with other mechanisms. That's the right engineering posture for a "fundamental limit" — not hunting for a more magical embedding trick, but building a multi-channel-recall hybrid system where each channel plays to its strengths.
In the next article, we'll formally build hybrid search: braiding BM25 keyword retrieval and vector retrieval into one rope, to see whether these two complementary failure modes can finally let the ghost Q8 — which has haunted us for four articles — rest.
Summary
-
Structural info really does work. After adding the
# called_by: process_checkoutprefix,calculate_order_total's cosine similarity to the Q8 query rose from 0.4617 to 0.5137 (+0.052), exactly the right direction. Article 05's hypothesis is confirmed in the numbers. -
But the boost is far from enough. "Payment-vocabulary-rich" functions like
create_payment_intent,process_refund, andget_payment_historyall sit above 0.51, andcalculate_order_totalremains pinned outside the top-5. Q8 failed under all three strategies (0.50). -
The semantic gap is a fundamental limit.
calculate_order_total(compute tax) and "Stripe charge" (collect money) are two different things in the real world, and their semantic distance is set by the embedding model's knowledge. A comment line is a "fine-tune," not a "redirect" — +0.052 can't bridge this gap. -
Naive prefix injection is double-edged. Strategy B prefixed all functions, printing the high-out-degree utility
execute_query's name at the head of every caller's embedding, creating literal noise that regressed Q7 (1.00 → 0.67). - Where you put it matters more than what you put. With identical structural info, a separate line (B) perturbs strongly with large side effects; fused into a docstring (C) perturbs gently with zero side effects but a small boost. Placement and form affect the vector on the same order as the information itself.
- The value of failure is measuring the boundary. The series tried the main variants of the vector-retrieval route, and Q8 tripped every time — not an implementation problem, a route problem. The way out isn't inside embedding but outside it: BM25 keyword retrieval, denoised explicit graph traversal, a code symbol index.
In the next article, we build hybrid search (BM25 + vector) and let two orthogonal failure modes cover for each other.
References
- Full demo code: codebase-kb-06-struct-embed
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)