File vs Vector for RAG

File-driven context management has been the rage in the last few days, especially since Claude CoWork launched. It made me curious and I tried a few things.

I ran an experiment across 5 distinct domains: from Python code to scientific papers. I ingested 50,000 documents from popular datasets and fired 5,000 queries at them.

The goal? To find out what all the noise about file-based search is. And if it is even real!

I started this expecting Vector search to crush the benchmarks across the board. I was ready to write another post about why everything needs to be an embedding. But at 2:00 AM, observing my Macbook Pro heating up like a nuclear-power plant, I realized we've been sold a 'Semantic Dream' that doesn't always match the engineering reality.

What these two systems actually do

Before the numbers, the distinction that explains all of them.

Keyword search, in this case Tantivy, builds an inverted index and matches terms exactly. It has no model of meaning. If the query says "sort a list" and the document says bubble_sort, keyword search finds nothing, because those are different strings. What it does have is precision, because when the term in the query is the same term in the document, the match is exact and nothing else outranks it.

Vector search, in this case ChromaDB, converts text into a numeric representation and finds documents whose representation sits nearby. It has an approximate model of meaning and no notion of exactness. It will bridge "sort a list" to bubble_sort, and it will equally happily return a document about cell structure when you asked about mitochondria, because those two things live close together in the same space.

Exact matching and approximate meaning are not two implementations of the same idea. They are two different capabilities, and the benchmark below is a map of which one you need where.

The results

We measured with MRR@10, which scores how high the correct document ranks in the first ten results. A score of 1.0 means the right document was ranked first every time.

Dataset	Domain	Keyword, exact match	Vector, semantic	Winner	Margin
CodeXGLUE	Natural language to code	0.2901	0.9143	Vector	+0.6242
MS MARCO	Real Bing queries	0.4035	0.5225	Vector	+0.1190
SQuAD	Wikipedia question answering	0.6048	0.6136	Tie	+0.0088
HotpotQA	Multi-hop reasoning	0.5494	0.4953	Keyword	+0.0541
SciQ	Science exam questions	0.8145	0.6142	Keyword	+0.2003
Mean, all five		0.5325	0.6320	Vector	+0.0995
Mean, excluding CodeXGLUE		0.5931	0.5614	Keyword	+0.0317

Read the final two rows together, because that is the entire result. Across five datasets vector search wins by roughly ten points of MRR, which is the number that would go on a slide. Remove CodeXGLUE and the ranking inverts, with exact-match keyword search ahead by three points.

Vector search is not generally better at retrieval. It is 3.15 times better at one job, and that job carries the average for everything else.

CodeXGLUE is the purest semantic-gap task in the set, because a natural language description of what code should do shares almost no vocabulary with the code that does it. Exact matching has nothing to match on, and scores 0.2901. This is the case vector search exists for, and it wins it decisively.

SciQ is the inverse. Science exam questions turn on specific entities, and in that setting a word is not an approximation of a concept, it is a key. Mitochondria is not a thing that is similar to a cell. Exact matching locks onto the term and scores 0.8145, while vector search drifts toward documents that sit nearby in embedding space and scores 0.6142.

SQuAD is a tie. The gap is 0.0088 on a single run with no seed averaging, which is inside the noise, and reporting it as a vector win would be dishonest.

The vector tax, with a number

The earlier version of this post said embedding generation is a tax and did not say how large. It is 76.4 times.

Dataset	Keyword indexing	Vector embedding and indexing	Ratio
CodeXGLUE	445.75 ms	43,098.93 ms	96.7x
MS MARCO	418.56 ms	26,335.61 ms	62.9x
SQuAD	410.60 ms	35,920.01 ms	87.5x
HotpotQA	394.98 ms	29,538.14 ms	74.8x
SciQ	444.30 ms	26,670.01 ms	60.0x
Total, 50,000 documents	2.11 seconds	161.6 seconds	76.4x

That is 23,650 documents per second against 309, on the same machine over the same corpus.

The consequence is about when an agent can use what it just learned. If an agent needs to read a repository or a hundred-page document and act on it within the same turn, exact-match indexing is effectively instant and embedding is a coffee break. If it reads now and acts later, the tax amortises and stops mattering.

The result that points past both systems

HotpotQA has the smallest margin in the table and the most instructive failure underneath it.

Those questions require linking two documents, for example establishing which of two magazines was founded first. Vector search would reliably retrieve the document for the first entity and miss the bridge document for the second, because the second document was not similar enough to a query that was mostly about the first. The answer was retrievable and the evidence was not.

That is not a similarity problem and no reranker fixes it. Neither exact matching nor semantic similarity has any representation of the fact that two documents are connected. Both systems rank documents independently against a query, and a bridge document is by definition the one that does not look like the query. Answering that class of question requires storing the relationship itself, which is a third mechanism and the reason graph structure exists in memory systems at all.

Finding the answer is not the same as holding enough context to prove it, and an agent that retrieves the first without the second produces confident, unsupported output rather than an error.

Method, and everything wrong with it

Anyone can rerun this, and anyone evaluating it should know where it is weak.

10,000 documents and 1,000 queries per dataset, 50,000 and 5,000 in total, executed on January 14, 2026 on an Apple M4 with 16GB of RAM. Keyword search was Tantivy 0.22.0 with the default analyzer. Vector search was ChromaDB 0.4.0 or later with all-MiniLM-L6-v2 at 384 dimensions. No chunking on either side. Scoring was MRR@10.

On scoring, which is the part most benchmarks in this category get wrong. Relevance was evaluated programmatically by exact document ID matching against each dataset's ground truth, in src/evaluation/metrics.py. There is no LLM judge anywhere in this pipeline. Nothing here can be moved by a prompt file, a judging bias, or an equivalence rule, which is a claim we cannot make about most published numbers in this category, including some of our own work on conversational benchmarks. A rerun on the same corpus produces the same table.

Six limitations, in the order they would be raised against us.

Single run rather than an average over seeds, so small margins carry no weight and SQuAD should be read as a tie.

all-MiniLM-L6-v2 is a small and relatively old embedding model. A stronger embedder would very likely raise the vector numbers and could change the sign on HotpotQA. The honest statement is that this measures a common default configuration rather than the best available vector setup.

Tantivy ran on its default analyzer with no tuning, so the keyword side is equally untuned.

No chunking, which affects both approaches and affects vector retrieval more on longer documents.

Ingestion timings come from local embedding on an M4. A hosted embedding API moves that number in both directions, worse for network round trips and better for batched inference, so treat 76.4x as the shape of the cost rather than a constant.

MRR@10 measures where the correct document ranks. It does not measure whether the agent then produced a correct answer. Retrieval rank and task success are different things, and we would rather say that ourselves than have it said back to us.

What this does not settle

This benchmark runs over a fixed corpus of documents that were written by someone else and do not change. Agent memory is neither of those things. It grows on every turn, it is written by the agent itself, and yesterday's fact can contradict today's.

The next post covers what happens to both of these systems once the corpus is conversation history rather than documents, which is where the interesting failures live. The architecture argument, including where graph structure and extraction fit, is on our build vs buy agent memory page, and the formal treatment of the token economics is in our paper on Agentic Context Management.