DEV Community: Lev Riabov

I measured the RAG technique menu on 46,000 chunks. Four things mattered.

Lev Riabov — Sun, 02 Aug 2026 19:16:41 +0000

Search "advanced RAG techniques" and you'll get a list of twenty things: hybrid search, reranking, HyDE, query decomposition, contextual retrieval, RAPTOR, GraphRAG, parent-child chunking, multi-vector retrieval, semantic chunking. They're presented as a menu you work through, each one making your system a bit better.

They don't compose like that. Techniques at the same pipeline stage mostly compete — they fix the same failure, so the second one you add finds nothing left to fix. And the only way to know which one your system actually needs is to measure on your own corpus.

I built a RAG system over 62 ancient-history books (~46,000 chunks) and put every retrieval technique I could through the same door: implement, measure against a fixed 161-question test set, keep or reject, write down the number. This article is the retrieval half of that ledger — what shipped, what got rejected, and what the rejections taught me. My production system has no hybrid search and no reranker, and that's a result, not a shortcut.

The map: four stages, two ledgers

Everything before the LLM writes a word lives in one of four stages:

A. INGEST  →  B. QUERY TRANSFORM  →  C. RETRIEVAL CORE  →  D. POST-RETRIEVAL

Same stage → the techniques compete. Different stages → they compose. That one rule kills most of the listicle's implied ordering.

The second rule is about money, and it's asymmetric:

Ingest-time cost is paid once, offline, on hardware you already have. One clever pass over the corpus is cheap forever.
Query-time cost is paid on every request, in latency and tokens, for the life of the system.

So query-side cleverness has to clear a far higher bar than ingest-side cleverness. A technique that buys +1.7 points for a paid API call on every query is a worse deal than the same +1.7 for one overnight batch job — even though the leaderboard row looks identical.

One more thing, and it's the part nobody labels. Evidence comes in three grades, and mixing them is how bad advice spreads:

Measured here — on this corpus, 46k chunks, 161 questions.
Measured smaller — on my predecessor project (4 books, 950 chunks, 50 questions). Weaker. Re-tested whenever the finding looked scale-sensitive.
Reasoned away — never measured. A cost/benefit judgment, and I'll say so when it is one.

Here's the whole retrieval menu with verdicts. The rest of the article explains the interesting rows.

Technique	Stage	Verdict	Evidence
Strong embedding model	C	Ship — +18 recall@5	measured here
Contextual chunk notes + heading path	A	Ship — synthesis +18.2	measured here
Structure-aware chunking, canonical locators	A	Ship — architecture	measured smaller
Dedup, lost-in-the-middle reordering	D	Ship — free, one line each	not worth measuring
Metadata-filtered search	C	Ship — enables source isolation	measured here
Cross-encoder reranking	D	Kept, then dropped — +1.7, paid per query	measured here
Multi-query expansion	B	Off by default — +2.1	measured smaller
Hybrid BM25 + RRF	C	Rejected — zero, byte-identical	measured here
HyDE	B	Rejected — −9.7	measured smaller
Parent-child / small-to-big	A	Rejected — completeness 3.22 → 2.67	measured smaller
Contextual compression	D	Rejected — prompt caching already won	reasoned away
Multi-vector (ColBERT), SPLADE	A	Rejected — 10–50× storage	reasoned away
Semantic chunking, step-back, embed-summaries	A/B	Rejected — no expected ROI	reasoned away

The baseline everything is measured against: naive dense retrieval, 500-token chunks, top-5 — recall@5 = 35.2%. Two out of three questions never reached the model with the right passage.

Stage A — ingest, the stage that pays

At four books you can eyeball every quirk. At sixty-two you can't, and running one splitter over raw Gutenberg text means page headers, tables of contents and translator footnotes all leak into your chunks and get embedded as if they were content.

The architecture that scales puts a hard interface in the middle:

raw book ──(per-format parser)──▶ normalized document tree ──(ONE uniform chunker)──▶ chunks

All per-book weirdness lives in parser adapters. The chunker is a single well-tested function that never learns which book it's processing: split on structural boundaries in priority order (section → paragraph → sentence), pack greedily to ~500 tokens, never merge across a section wall, never cut mid-sentence.

Chunking at scale is a parsing problem, not a token-count problem. The size discussions get all the attention and are the least interesting decision in the stage.

Two things I'd insist on again. Every chunk carries character offsets into the normalized text — which makes test-set gold spans chunking-invariant, so I could change chunking strategy without rewriting the test set. And every chunk carries its canonical citation locator (Caesar, Gallic War 4.25), because classical texts have stable reference schemes that survive every edition. That bought professional citations in answers and mechanical test-set authoring, for the cost of teaching the parsers to recognize book/chapter numbering.

Contextual retrieval: shipped, and instructive about how to read a number

A raw slice from the middle of chapter 12 reads "he then marched north" — the embedder has no idea who "he" is or which war this is. The fix is an ingest-time pass: a cheap local LLM writes 1–2 sentences situating each chunk, and the embedding covers context_note + heading_path + chunk_text instead of bare text. 46,159 of 46,170 chunks enriched in one local batch pass on a consumer GPU, zero query-time cost.

The headline result was +1.7 recall@5. Noise. The internals were not:

category	Δ recall@5
synthesis	+18.2
literal	+8.7
multi-hop	+2.1
synonym	+0.0
contradiction	−5.3
cross-book	−9.0

Synthesis — the worst category in the project — transformed. Cross-book got worse. Overall ranking sharpened well above noise (recall@1 +5.8, MRR +0.060). A flat headline hiding ±18-point internals is the normal case, not the exception. If I'd only looked at the aggregate I'd have called this a no-op and moved on.

The generation-side number was even more misleading. Answer completeness appeared to drop, 4.45 → 4.30. It hadn't. On the 113 questions both runs answered, completeness was flat (4.46 → 4.40). What actually happened: contextual retrieval converted 12 previously-refused questions into answered ones, and those 12 — the retrieval-starved hard ones — scored 3.42, dragging the mean down while every prior answer held. In-scope false refusals fell from 15.6% to 7.4%.

A win wearing the disguise of a regression. Any change that converts refusals into answers will do this to you, and the only defense is to always compute the metric on the set of questions both runs answered.

The rest of the stage was cheap to reject. Parent-child retrieval (embed small chunks, hand the LLM their parent section) regressed completeness 3.22 → 2.67 on my predecessor project — re-testable, but I'd want evidence of context starvation first. ColBERT-style multi-vector costs 10–50× vector storage; rejected on the storage budget without measuring, and I'll call that what it is. Semantic chunking and embedding summaries instead of chunks were skipped on expected ROI — the latter is subsumed by contextual notes, which keep the original text and add the context.

Stage B — query transforms, the cheapest stage to skip

Every technique here costs an extra LLM call before you've even searched.

HyDE — have the model write a hypothetical answer and embed that instead of the question — scored −9.7 recall@5 on my predecessor. The mechanism is worth understanding because it generalizes: HyDE replaces your query, discarding the discriminative terms the user actually gave you. It's built for a query/document vocabulary mismatch. If your embedder is strong enough not to have that mismatch, you're throwing away signal and paying a second of latency for the privilege.

Multi-query expansion (paraphrase the question n ways, union the results) measured +2.1 — real but marginal, and it's an extra call plus n searches on every request. Left in as an off-by-default flag.

Query decomposition and step-back prompting I skipped as standalone techniques for a different reason: a retrieval loop that can search more than once does both of these adaptively, driven by what it actually found. Don't hand-build a static version of a behavior a loop gives you for free.

Stage C — the embedder is the whole ballgame

This is the one that mattered most, and it's one line of configuration.

Swapping the default embedder for a strong one (qwen3-embedding-8b, hosted) took recall@5 from 35.2% to 53%. The hardest-hit category — modern-English questions against Victorian translation prose — gained +41.7 points. Nothing else in this article comes close.

How I picked it is the transferable part: shortlist by constraints, decide by ablation. The constraints were concrete — can it serve queries on a cheap CPU container or does it have to be an API, what's the license, does it need instruction prefixes, does the context window fit contextual notes. That produced four candidates. The leaderboard never got a vote in the final decision, because no leaderboard contains Victorian translation prose. Your corpus is the only leaderboard that counts.

Two footguns cost real projects real quality here:

Prefix parity. Most modern embedders want different instruction prefixes for queries versus documents, and hosted APIs serving open-weight models generally don't inject them for you. Corpus embedded with prefixes, queries without, is a silent multi-point loss that looks like nothing. Wrap embedding in one module that owns the prefix policy and never call the model from two places.
Same model ≠ same vectors. Runtime differences (sentence-transformers vs llama.cpp vs a hosted API), fp16 vs fp32, and quantization all shift vectors. The cheap defense: embed 20 fixed sentences on both stacks and assert cosine ≥ 0.999 before trusting them as one index.

Hybrid BM25 + RRF: rejected, and it's the best receipt in the project

The standard 2024-era advice is that hybrid search — keyword BM25 fused with vector search — always wins at scale, especially for rare proper nouns. My corpus is full of rare proper nouns (Vercingetorix, Pharsalus) in inconsistent Victorian spellings. I had rejected hybrid once already at 950 chunks, and I wrote down a prediction before running it: at 46k chunks this flips to a win.

It didn't flip. It returned nothing.

metric	dense	hybrid	Δ
recall@1	32.5	30.7	−1.8
recall@5	56.7	56.3	−0.4
recall@50 (pool)	82.4	82.4	0.0
MRR	0.580	0.561	−0.019

Not "roughly the same" — byte-identical pool recall, category by category. Every answer BM25 could find by exact token match, the 8B embedder already had. And fusion made the top ranks slightly worse, because RRF injects keyword-noise chunks that displace well-ranked dense hits.

The mechanism is the finding: where a strong dense retriever misses, the answer is distributed, not keyword-findable — so BM25 can't reach it either. The hybrid-always-wins advice assumes a weak lexical first stage. With a modern 8B embedder that assumption is just false on this corpus.

Notice what made that result readable at all: recall@50, treated as pool recall. Recall@5 alone would have shown −0.4 and left me guessing whether BM25 had contributed new candidates that fusion then mis-ranked. A metric designed to separate "widened the pool" from "reordered the pool" turned an ambiguous wash into a clean rejection. Design your metrics to distinguish mechanisms, not just to score outcomes.

Stage D — the reranker that failed its own rationale

A cross-encoder rescores your top-50 and returns the best 5. It's the most-recommended technique in RAG, and I measured five of them.

reranker	host	recall@5 vs no-rerank
qwen3-reranker-0.6b	local	−3.0
bge-reranker-v2-m3	local	−2.1
cohere/rerank-v3.5	API	+0.0
cohere/rerank-4-pro	API	+1.7

This is the exact inverse of the embedder gate. There, the component was so weak that anything better was a huge win. Here the embedder is so strong that a 0.6B cross-encoder is worse than the 8B embedder's own ranking — it adds noise. Only a state-of-the-art hosted reranker helps at all, which means shipping reranking means shipping a paid per-query dependency, forever.

One architectural law came out of this, and it's free to obey: rerank the same text you embedded. Scoring the bare chunk text while the index holds contextualized text made the reranker fight the retriever and undid the contextual gains outright (47.9% vs 51.6% on my predecessor). Retrieval and rerank must share a representation.

The interesting part is why I dropped it. The reranker was kept provisionally for a specific stated reason: contextual retrieval had cost me 9 points on cross-book questions, and reranking the top-50 was supposed to recover them. It didn't — cross-book landed at 26.0 against a 34.4 floor, and no reranker recovered it, though the pool demonstrably held the answers. Cross-book was a candidate-pool problem, not an ordering problem, and reranking cannot surface what isn't in the pool.

So: the reranker helped a little, everywhere except the place it was hired to help. Marginal, paid, per-query, forever, and falsified in its own rationale. When a stronger generator arrived later it came out of the pipeline entirely.

One honest caveat. The model swap and the reranker drop happened in the same run, so that drop was never cleanly isolated — I skipped the arm that would have separated them, for cost. It's a receipt gap in an otherwise complete ledger, and I'd rather name it than let the table imply more rigor than it has.

The rest of the stage: dedup and lost-in-the-middle reordering are one line each, free, and I shipped them without measuring. Contextual compression — LLM-summarize the retrieved chunks before stuffing them — I rejected by reasoning: prompt caching already makes raw chunks cheap, compression adds latency, and a summary can silently delete the exact sentence you were going to cite.

Four things mattered

After all of it, here's what actually determined retrieval quality on this corpus:

The embedding model. +18 points from one config line. Everything else in this article combined is smaller. Don't inherit the tutorial's embedder.
What goes inside the chunk. Not how big it is — what it contains. An ingest-time context pass transformed the worst category by +18.2 and costs nothing at query time.
The parsing layer. Where boundaries fall and what metadata rides along. Unglamorous, decided before any technique gets a vote, and expensive to change later.
The metrics themselves. Two metric redesigns changed my conclusions more than any technique except the embedder: pool recall, which turned the hybrid result from ambiguous into decisive, and a recall metric that I had to rewrite after discovering it penalized questions with corroborating sources across multiple books — scoring 0.2 for retrieving one of five passages that each fully answered the question. A metric that punishes the exact diligence your system is for will quietly steer you wrong for months.

And what didn't matter: hybrid search (zero), four of five rerankers (negative or nil), HyDE (−9.7), and every fashionable ingest architecture I skipped. Not because they're bad techniques — because on this corpus the binding constraint was somewhere else. That's the pattern underneath all of it: at any moment exactly one thing is the binding constraint, and every technique aimed anywhere else returns noise. The embedder was binding, so fixing it paid 18 points. Once it wasn't, contextual retrieval was marginal, reranking was marginal-and-paid, and hybrid was nothing at all.

The shipped retrieval stack is boring: contextual dense embeddings, a strong embedder, top-k, metadata filters. No hybrid. No reranker. recall@5 = 56.7%.

Which raises the obvious question, and it's the reason there's a second half to this story: the finished system answers 100% of in-scope questions and refuses 96% of the unanswerable ones, on a retriever that finds the right passage 56.7% of the time in a single shot.

It manages that because it doesn't do a single shot. Once retrieval was closed, the remaining headroom turned out to be architectural — and that's the next article.

I build RAG and LLM-evaluation systems, and I'm available for contract work. Everything above is open: the code, the 161-question golden set, and the full append-only eval log with every run record behind these numbers.

Live demo: historian.loroplanner.com
Code + case study + eval log: github.com/LevRiabov/antic-historian
Previous article: RAG for developers who aren't AI engineers

If your team is trying to make an LLM answer reliably from your own data — or trying to figure out whether the one you built already can be trusted — reach out: levriabov@zohomail.eu

RAG for developers who aren't AI engineers: what actually matters

Lev Riabov — Sun, 19 Jul 2026 13:53:54 +0000

RAG for developers who aren't AI engineers: what actually matters

Most non-AI developers have a mental model of RAG that is either wrong or dangerously incomplete. Not because they're bad engineers — because almost everything written about RAG is either a 10-minute framework tutorial that hides every real decision, or a research paper aimed at people who already do this for a living. There's very little in between.

This article is the in-between. I recently built a production RAG system over 62 ancient-history books (~46,000 chunks) and measured every design decision against a fixed test set — including the decisions that didn't work. I'm going to skip the framework marketing and tell you the small number of things that actually determine whether your RAG system is trustworthy, with the numbers to back them up.

You don't need ML knowledge to follow this. If you can build a REST API and query Postgres, you can build everything described here.

The 60-second mental model

RAG (Retrieval-Augmented Generation) is just this:

question → find relevant text in your corpus → paste it into the LLM prompt → "answer ONLY from this text"

That's the whole idea. A search engine bolted onto an LLM, with instructions to answer from the search results instead of from its training data.

The only genuinely new component for a typical backend developer is embeddings. An embedding model turns text into a vector (an array of ~1,000 floats) where similar meaning produces nearby vectors. So "find relevant text" becomes nearest-neighbor search over vectors: embed all your document chunks once at ingest time, embed the user's question at query time, fetch the k closest chunks.

If you know Elasticsearch: it's like that, but matching on meaning instead of keywords. "Why did the ship sink" will match a paragraph about "the vessel foundered in the storm" even though they share zero words.

Everything else — chunking, retrieval, prompting — is ordinary engineering. Which is exactly why regular developers can and should build these systems. What's missing from most tutorials isn't skill, it's knowing where the traps are.

The trap: naive RAG works in the demo and lies in production

Here's the part that shocks people.

The standard tutorial setup — split documents into fixed 500-token chunks, embed with whatever the tutorial used, retrieve top-5, feed to the model — appears to work immediately. You ask a question, you get a fluent, confident, well-structured answer with a citation. Demo done, ship it, right?

I measured that exact baseline on my corpus with a 135-question test set where I knew which passages contained each answer. Result: recall@5 = 35.2%.

Read that again. For roughly two out of three questions, the correct passage never reached the model at all. And here's the dangerous part: the model answered anyway. Fluently. With the same confident tone as when the retrieval worked.

This is the core thing non-AI developers get wrong about RAG. The failure mode is not "no results found" like a search engine, and not an exception like an API call. The failure mode is a confident, plausible fabrication that is indistinguishable from a correct answer unless you already know the answer. A system that's right 65% of the time and silently makes things up the other 35% is worse than useless — because you can't tell which answer you're looking at.

Nothing about the demo tells you this is happening. The only way to know is to measure. Which brings us to the single most important piece of advice in this article.

Before you improve anything: build a test set

This is the step every beginner tutorial skips, and it's worth more than any technique you'll ever add.

Before touching chunk sizes, embedders, rerankers, or agent frameworks, do this:

Write 30–50 real questions about your corpus — the kind your actual users will ask, not the kind your documents happen to answer neatly.
For each question, record where the answer lives (which document, roughly which passage).
Crucially, include 5–10 questions your corpus cannot answer. These are your hallucination traps. The correct behavior is a refusal; anything else is fabrication you'd otherwise never see.
Measure two things separately:
- Retrieval: is the right passage in the top-k results? (Pure code, no LLM judging needed — this is the number to optimize first.)
- Generation: given the right passages, is the answer grounded in them? Does the system refuse when it should?

This costs one or two days and it changes everything, because RAG advice on the internet is wildly corpus-dependent. Things that transformed one system do nothing on another. Without your own test set, every blog post — including this one — is a coin flip. With it, you can check any technique against your data in an hour.

Concrete example of why this matters: the standard 2024-era advice says "hybrid search (keyword BM25 + vector) always beats pure vector search at scale." I believed it. I expected it to win on my corpus and pre-registered that prediction. When I actually ran it, hybrid retrieval was byte-identical to plain dense retrieval — same recall, category by category, not one result changed. I only know that because I measured. (The full write-up of that negative result is its own story.)

Your test set is not test infrastructure you write once and forget. It's the steering wheel.

What actually moved the needle (ranked)

With a test set in place, here's what mattered on my corpus, in order of impact.

1. The embedding model — the biggest single lever

Swapping the default embedder for a strong one (qwen3-embedding-8b, via a hosted API) took recall@5 from 35% to 53% — an 18-point jump from changing one line of configuration. The hardest-hit category improved by +41.7 points: questions phrased in modern English against text written in formal Victorian translation. The weak embedder simply couldn't bridge the vocabulary gap; the strong one could.

The lesson: don't inherit the tutorial's embedder. It's the component doing the actual "understanding" in your retrieval, models differ enormously, and swapping it is trivial. Check the MTEB retrieval leaderboard, pick a few strong candidates, and measure them on your test set.

2. What you put inside a chunk

Chunking discussions usually obsess over token counts. The bigger win for me was what the chunk contains. A raw 500-token slice from the middle of chapter 12 loses all context — the embedder sees "he then marched north" with no idea who "he" is or which war this is.

The fix: at ingest time, prepend context to each chunk before embedding it — the book title, the chapter heading path, a one-line note about what this section covers. On my worst-performing question category (questions requiring synthesis across a section), this transformed results: +18.2 recall points. Chunking is about meaning-per-chunk, not arithmetic.

3. Things that did NOT help (measure before adding)

Two techniques that every "advanced RAG" listicle recommends:

Hybrid BM25 + vector search: as described above — zero change on my corpus. A strong embedder already found everything keyword search could find.
Cross-encoder reranking: I measured five rerankers. Only the most expensive hosted one beat the no-reranker baseline at all, by ~1.7 points — for a permanent paid dependency on every query. Dropped.

The point is not "never use these." On your corpus they might win. The point is that both add cost, latency, and complexity, and the only way to know whether they pay for themselves is your test set. "Fashionable" is not evidence.

Making it refuse instead of hallucinate

Retrieval quality gets the right text in front of the model. The second half of trustworthiness is what the model does when the text isn't there — and this you have to engineer deliberately.

Three parts:

Prompt for grounded answering. Instruct the model to answer only from the provided sources, cite which source supports each claim, and explicitly refuse when the sources don't contain the answer. This is necessary but nowhere near sufficient.

Test refusal like a feature. This is what the out-of-scope traps in your test set are for. My favorite trap type: a question about a named work the corpus doesn't contain but does mention. For example, my corpus includes a footnote about the poet Sappho but none of her actual poetry. Ask "what does Sappho's poetry express?" and a weak setup confidently answers from the footnote, sounding completely authoritative. The correct behavior is: "the corpus mentions this work but does not contain it," with the source of the mention. That distinction — between having information and having a reference to information — is exactly the kind of failure you'll never notice without deliberately testing for it.

Know that model choice matters here too. In my measurements, honest-refusal ability varied dramatically between LLMs — swapping the generator model took out-of-scope refusal from 73% to 100% with the same prompt. Prompting alone had a ceiling; I iterated the prompt through five versions and watched edits slide along a fixed trade-off curve without moving it.

End state on my system, measured on the full test set: 0% false refusals (it answers every answerable question) and 96% honest refusals on the trap questions. Those two numbers together are what "reliable" means for RAG — and both are ordinary integration-test numbers you can put in a CI run, not vibes.

A concrete starting path for a TypeScript developer

You do not need an AI framework, a vector-database vendor, or Python to start. Here's the stack I'd hand a TS developer today:

Postgres + pgvector — vector search inside the database you already run and know how to operate. At hundreds of thousands of chunks it's more than enough.
A hosted embedding API — one HTTP call per chunk/query.
Any decent LLM API for generation.
~200 lines of TypeScript before you even consider a framework. Writing the pipeline by hand first is the best way to actually understand it — frameworks make far more sense once you know what they're abstracting.

The five steps:

Ingest: split documents into chunks along natural boundaries (sections, paragraphs), prepending title/heading context to each chunk's text.
Embed and store: one API call per chunk, insert into a table with a vector column, add an index.
Retrieve: embed the incoming question, SELECT ... ORDER BY embedding <=> $1 LIMIT 10. That operator is a cosine-distance nearest-neighbor search — that's the whole "vector database."
Generate: build a prompt with the retrieved chunks (labeled with their sources), the grounding-and-refusal instructions, and the question.
Build the test set and iterate. Measure retrieval first. Fix retrieval before blaming the model. Re-run on every change.

Steps 1–4 are a weekend. Step 5 is what separates a demo from a system.

If you want a reference implementation of the full production version — the eval harness, the measured ablations including the failed ones, cost controls, prompt-injection defenses — the entire project this article is based on is open:

Live demo: historian.loroplanner.com (try asking it something the corpus can't answer — that's the interesting part)
Code + case study: github.com/LevRiabov/antic-historian

It's Python, but nothing about the architecture is Python-specific — the structure maps one-to-one onto the TypeScript stack above.

I build RAG and LLM-evaluation systems, and I'm available for contract work. If your team is trying to make an LLM answer reliably from your own data — or trying to figure out whether the one you built already can be trusted — my case study above shows how I approach it, with published numbers. Reach out: levriabov@zohomail.eu.