DEV Community: David Liu

I've Built RAG Infrastructure Several Times. Last Week Was the First Time I Actually Benchmarked It.

David Liu — Tue, 25 Aug 2026 11:08:09 +0000

I have a confession, and I suspect I'm not alone in it: I've built RAG infrastructure multiple times, and until last week I had never benchmarked any of it.

Unit tests, sure. Integration tests, sure. Everything green, every pipeline connected. But if you'd asked me "is the retrieval actually good?", the honest answer was a shrug with a deployment attached.

For context: I'm building Langhuan, an open-source knowledge base that agents query over MCP. Retrieval is the product. Which made the shrug harder to live with.

The audit that started it

Before writing any eval code, I sat down and listed the retrieval decisions I had shipped. The chunking contract had been through three revisions. Vector and keyword search get fused with RRF. A reranker sits on top. Every one of those decisions had a rationale I could defend in a code review. Not one of them had a number behind it.

I was running on architecture taste. Taste doesn't fail loudly.

What I built

The expensive part of an eval is never the harness, it's the labeled data. I labeled nothing. MIRACL-zh is a Chinese Wikipedia corpus with human-annotated passage relevance, Apache-2.0, so I deterministically sampled 200 real queries from it — same seed, same dataset fingerprint, same exam every time.

Two tracks, because retrieval loses quality at different stages:

Passage track: ~5,300 single-passage documents. Isolates embedding, FTS, and the fusion itself.
Long-document track: ~700 full articles through the real pipeline — chunking, parent-child chunks, retrieval.

Each track runs four configurations: vector only, FTS only, hybrid, hybrid + rerank. Metrics are recall@10, MRR@10, nDCG@10 against fixed qrels. No LLM judge anywhere — I wanted the uncertainty of an LLM judge to be the thing this system eliminates.

One rule I'm glad I held onto: determinism. Same fingerprint (dataset, chunker params, models, code version) must produce metrics that match bit for bit across runs, different port, fresh instance and all. If it doesn't, everything you observe is noise and "the change made it better" is folklore.

The smoke test ate first

Step one was a smoke run with a mock embedding — same text always maps to the same vector, zero semantics — to check the harness wasn't biased. Scores hugged the random baseline, so the ruler was straight.

But the smoke run also has to spin up a real standalone instance, create a knowledge base, and write to it. That path returned a 500. Every time. A forward foreign key was missing its deferred check on SQLite — meaning anyone bringing up a fresh instance following my own README would have hit it first thing. The second bug was nastier: vector search had never worked in the production binary at all. The vec extension simply wasn't linked into the build.

Why did my existing tests miss both? Unit tests mock the database. Integration tests run against Postgres. The eval harness was the first thing in the repo to behave like a brand-new user.

The row of zeros

Real model in (bge-m3), full 200 queries, and the results table came back with a row of 0.0000s. The FTS channel. Zero recall. Not "weak on interrogative queries" — zero, across the board.

My first assumption was that the eval was broken. I spent a while proving it wasn't. It was the product.

The kicker sat one column over: hybrid scored 0.9799 — identical to vector-only, digit for digit. My hybrid search had been running as a plain vector search with extra steps, for who knows how long. No errors. No alerts. Users got results, the results looked fine. One of the two channels just... wasn't there.

The bug itself was almost enjoyable

The tokenizer (gse) splits the query 埃及有哪些民族？ — "What ethnic groups does Egypt have?" — into five tokens: 埃及 / 有 / 哪些 / 民族 / ？. And FTS5 matches with AND semantics: a document hits only if it contains every token.

Body text about Egypt will contain 埃及 and 民族. It will never contain 哪些, and it definitely won't contain the question mark. One missing token and the whole query is the empty set. So every query phrased as a question returned nothing, always.

What I like about this bug is that nobody is wrong. The tokenizer segments every word correctly. FTS5 faithfully implements AND. The bug lives in the seam between two correct components — which is exactly the kind of bug component-level tests can never see.

The fix filters the query side: strip punctuation, single-character function words (有, 的, 了), question fillers (哪些, 怎么, 为什么). I kept the stopword list deliberately conservative, because over-filtering silently kills keyword queries — the exact disease being treated. New house rule: touch the list, rerun the eval.

channel	before	after
vector only	0.9799	0.9799
FTS only	0.0000	0.1314
hybrid	0.9799	0.9826

0.13 looks tiny until you remember what FTS is for: exact strings — file names, model numbers, IDs, proper nouns. Vector search covers "similar meaning"; FTS covers "exactly these characters." Questions get handled by the vector side, keywords are FTS's home turf. Both channels alive is the entire point of hybrid. And 0.9826 > 0.9799 is the first time "hybrid is worth it" was confirmed by my own data instead of by an architecture diagram.

Where the effort actually goes

Two more findings rewired my priorities. Swapping the embedding model (bge-m3 vs Qwen3-Embedding-0.6B) moved recall by 0.4 percentage points — I had spent more agony than that on model choice. Chunk-size experiments? ±0.5pp, not worth changing defaults over. Rerank, meanwhile, actually moved MRR (0.9975 — hits pinned to position one). For the first time the lever hierarchy is measured rather than vibes.

One honesty note: none of these scores are comparable to public leaderboards. My passage track retrieves over a ~5k-passage sample; MIRACL's official numbers are computed over the ~4.9M full pool, and bigger pools mean lower scores. The value is one ruler measuring every change: same fingerprint, diff two metrics.json files, and every gain or loss has a cause.

What I actually got out of it

Not the metrics. For the first time, I feel confident about my own work.

Before this, every statement I made about the retrieval was conditional. "It should handle that." "We use hybrid search" — except functionally, we didn't. Now I know what the system does, where it's strong, where it leaks (about 80% recall on long documents, and I know which failure modes eat the remaining 20%). Every future change — tokenizer tweaks, stopword edits, the traditional-Chinese normalization I have planned — will arrive with a number attached instead of an opinion.

The bugs would still be in there otherwise. That's the part I keep coming back to: two production bugs and an entire missing retrieval mode, and the system looked healthy the entire time. Quietly. If you've shipped RAG more than once and never measured it, you probably have a dead channel you don't know about either.

The harness is a separate binary in the repo (make eval), and the full report with every run and fingerprint is RETRIEVAL_BENCHMARK.md. Steal the approach — it's cheaper than the uncertainty.

Local RAG starts with retrieval, not infrastructure

David Liu — Thu, 20 Aug 2026 08:42:11 +0000

RAG projects have a way of collecting infrastructure before they collect evidence.

A database gets provisioned. A vector store appears. Then Redis, object storage, a parser service, a queue worker, and a few dashboards. By the time the first PDF is imported, there are enough moving parts that a bad search result can mean almost anything.

Was the parser wrong? Did chunking lose the useful context? Was the vector index empty? Did the query just not match the document?

For early work, I would rather make the problem smaller. Use a handful of documents people already depend on. Ask questions they have actually asked before. Look at the passages returned by search and decide whether they are useful.

That is what I mean by starting local.

An empty database is not a knowledge base

A knowledge base becomes interesting once it contains the annoying files: a PDF with a table in the middle, a spreadsheet someone edited last quarter, a product manual with the same heading on every page, a contract with a number that people need to find exactly.

Those files are much more informative than a clean benchmark dataset.

When I test retrieval, I usually want a short list of questions with different failure modes:

one with an exact name, number, or abbreviation;
one that needs context rather than a single matching sentence;
one the document set cannot answer.

The last question matters more than it sounds. A system that returns nothing useful should make that clear. Otherwise, people start blaming the model for an answer that retrieval never supported in the first place.

I also want every result to point back somewhere specific: the source document, its revision, and a page, row, or other usable anchor. “The search found something” is not enough when someone needs to check it.

Chinese queries make weak retrieval obvious

Chinese is often where a retrieval setup reveals what it has been getting away with.

Vector search can be good at questions with enough semantic context. It is less reassuring when the query is a product code, a contract number, a person’s name, or an internal abbreviation. Those queries are often better served by lexical matching.

The opposite problem is also real. Keyword search alone does poorly when the source and the question use different wording.

That is why I prefer hybrid retrieval: keep a semantic path and a lexical path, then let both participate in recall.

The local version does not need to look exactly like production. SQLite with FTS5, a vector extension, and Chinese tokenization can be enough to test the idea. A larger deployment might move to PostgreSQL, pgvector, and a Chinese full-text extension. The components change, but the question stays the same: can this query find the passage a person would expect to see?

I wrote more about that trade-off in this hybrid-search guide.

Local first does not mean local forever

I do not think SQLite is a substitute for a production database, and I would not run a shared, business-critical knowledge service from a developer machine.

Local mode has a different job. It lets a team test document processing and retrieval before the environment becomes the main thing being tested.

A single binary with a local database is useful when you want to import some files, try a few searches, reset the library, and repeat. There is no need to bring up PostgreSQL, Redis, Docker, and object storage just to learn that the documents were chunked badly.

The boundary changes when people start relying on the system:

more than one workspace needs it;
imports need retrying and queuing;
backups and monitoring are no longer optional;
a missed result can affect real work.

At that point, adding production infrastructure is not overengineering. It is responding to an operating need you can now name.

The agent should receive evidence, not a folder

Once local retrieval is working, connecting it to an agent is usually simpler than building a chat interface around it.

The knowledge service can expose search and document operations over an API or MCP. The agent asks for relevant evidence when it needs it; it does not carry a whole library through every conversation.

That distinction matters when a second agent needs the same documents. Copying the files into another application may be quick, but it creates a second versioning and retrieval problem. Sharing a retrieval boundary is often the cleaner move.

I have been building Langhuan around that boundary: document ingestion and retrieval on one side, agents and application workflows on the other. Its standalone mode uses SQLite for local validation, while the production path can use PostgreSQL and Redis when the workload earns them.

If I were starting a new RAG project tomorrow, I would begin with five messy documents and a short list of real questions. I would not add the next service until I could explain which problem it solves.