Granularity is not a tuning parameter you sweep. A file vector and a function vector answer different questions, and choosing wrongly produces a specific, recognisable failure: results that are in the right area and point at the wrong lines.
What each vector encodes
A function-level embedding encodes one unit of behaviour. Its neighbours in the vector space are other functions that do something similar. It answers “show me code that does X”.
A file-level embedding encodes a topic. Because almost every embedding model produces one vector by pooling over token representations, a file vector is approximately an average of its contents, and the average of six functions about billing is a point that means “billing” and no longer means any of the six. It answers “which part of the system deals with X”, which is a real and useful question but not the same one.
The consequence people trip over is that a file vector’s neighbours are not necessarily files containing any function similar to the query. They are files whose overall subject matter resembles the query. Those come apart badly when a file is not thematically coherent — and most files in a real codebase are not, because they accreted.
The dilution example
Take src/billing/invoice.py, 400 lines, five functions:
build_line_items(order) # 120 lines, the core of the file
apply_tax(items, jurisdiction) # 90 lines
render_invoice_pdf(invoice) # 110 lines
format_currency(amount, locale) # 20 lines
_log_invoice_event(kind, id) # 60 lines, telemetry, unrelated
Query: “how do we format a monetary amount for a German customer”.
The correct answer is format_currency, twenty lines out of four hundred. In a function-level index it is one chunk whose entire content is about formatting money by locale, and it ranks first. In a file-level index, that chunk’s contribution to the file vector is roughly 5% by token count. The file vector is dominated by line items, tax and PDF rendering, so it sits near “invoice generation” rather than near “currency formatting”.
What comes back instead is src/reporting/exports.py — a file that mentions currency, locale and formatting throughout because it exports figures for finance, and whose vector is therefore genuinely closer to the query. It is a plausible-looking result containing nothing the reader wants. And when the developer opens invoice.py from a lower-ranked hit, they land at the top of a 400-line file with no indication that line 340 is the answer, because a file-level chunk has no sub-file position to report.
That second failure is the underrated one. Even when a file-level index retrieves the right file, it cannot tell you where to look, so the reader does the last mile by hand — or, in a retrieval-augmented pipeline, the whole file is pushed into the context window and the irrelevant 380 lines compete for attention with the 20 that matter.
When file-level is correct
It is worth being precise about the arithmetic, because the intuition “the file contains it, so the file matches it” is what makes this surprising. Under mean pooling, the file vector is approximately a token-weighted average of its parts. A chunk that is 5% of the tokens contributes roughly 5% of the direction. Cosine similarity is dominated by the remaining 95%, so a file vector is near a query only when a majority of the file is about that query. The dilution is not a subtle degradation that better models fix; it is what averaging does, and it gets worse linearly as files get longer.
There are three cases where the coarse vector is the better choice and function-level actively hurts.
- Orientation queries. “Where does authentication live” wants a place, not a function. A function-level index answers it with twelve fragments from nine files and no sense of structure.
- Files that are one thing. A migration, a configuration module, a schema definition, a Terraform file. There are no meaningful sub-units to split, and splitting one produces fragments that are individually meaningless.
- Languages you cannot parse. If you have no grammar for a language, you have no reliable function boundaries. A file-level vector is honest about its granularity; a fixed-size window pretending to be a function is not.
A two-level index
Both, stored together, is usually right and costs less than it sounds. Embed every function, and additionally embed a short synthesised file summary — the path, the module docstring, and the list of defined symbols — rather than the whole file text. That summary vector behaves like a file-level vector for orientation queries without the dilution, because it contains no bodies to average over.
file summary chunk for src/billing/invoice.py
src/billing/invoice.py
Invoice construction, tax application and PDF rendering.
defines: build_line_items, apply_tax, render_invoice_pdf,
format_currency, _log_invoice_event
imports: decimal, reportlab, .tax_tables, .locale
That is perhaps 60 tokens against the file’s 4,000, so the storage cost of the second level is a rounding error against the function vectors, and it retrieves for the orientation query precisely because every token in it is a name rather than an implementation. Route by result type at query time: return summary hits as places to explore and function hits as lines to read.
Neither, when the function is huge
The granularity argument assumes functions are the natural unit, and in a legacy codebase a great many are not. A 900-line function has exactly the dilution problem the file had, one level down, and it also may not fit in the embedding model’s context window at all — in which case most clients truncate silently and you have embedded the first third of the function while believing you embedded all of it.
Handle it explicitly: measure the token length of every chunk before embedding, and for anything over your model’s limit, split at the outermost block boundaries the parser gives you and store each piece with a reference back to the enclosing symbol. Log the count of over-length chunks. It is a small number that tells you something real about the codebase, and a silent truncation is the kind of defect that shows up months later as “search never finds anything in that module”. The general treatment of splitting decisions is in the chunking guide; what differs for code is that the boundaries are given to you by a grammar, so guessing is never necessary.
Top comments (0)