Split a document into chunks and you destroy the one thing retrieval depends on: context. Chunk 47 says "The margin fell to 8.2% in the same period." Same period as what? The sentence that established the quarter is 3,000 tokens away in chunk 12. Your embedding model sees "the same period," encodes a vague vector, and when the user asks "what was Q3 gross margin," chunk 47 doesn't rank. The answer was in your index. Retrieval just couldn't find it.
Contextual retrieval fixes this by rewriting each chunk to carry its own context before you embed it. It's a preprocessing step, not a new index type, and it's one of the highest-leverage changes you can make to a production RAG pipeline.
TL;DR
- Standard chunking strips context. A chunk that references "the company" or "this period" embeds into a vector that doesn't match the queries that should retrieve it.
- Contextual retrieval prepends a short, chunk-specific context blurb (generated by a cheap LLM that sees the whole document) to each chunk before embedding and BM25 indexing.
- Prompt caching makes it affordable. Cache the full document once, then generate context for every chunk against the cached prefix — you pay near-nothing per chunk after the first call.
- Anthropic reported large retrieval-failure reductions: contextual embeddings alone cut failures meaningfully, contextual embeddings + contextual BM25 cut them further, and adding a reranker on top produced the biggest drop.
- Combine dense (embeddings) + sparse (BM25), then rerank the top ~150 down to top ~20. Each layer catches what the others miss.
Why do isolated RAG chunks fail retrieval?
Because embeddings encode what the text says, and a stripped chunk says less than it means. Chunking optimizes for fitting inside a context window and token budgets, not for semantic self-sufficiency. The result is chunks full of dangling references: pronouns, relative dates, "the above table," "as noted earlier."
Two concrete failure classes:
Anaphora. "It grew 40% year over year." The embedding has no idea what "it" is. A query mentioning the actual entity — "revenue growth" — lands in a different region of vector space.
Lexical mismatch for BM25. Sparse retrieval matches tokens. If the chunk never contains the string "gross margin" because the section header two chunks up carried it, an exact-keyword query scores zero. Dense retrieval is fuzzy enough to sometimes recover; BM25 is not.
You can widen chunks to reduce this, but bigger chunks dilute embeddings (one vector averaging more topics) and waste context window at generation time. Overlap helps at the seams but not for long-range references. The reference that disambiguates chunk 47 can be pages away.
What is contextual retrieval?
Contextual retrieval is a preprocessing step: for every chunk, use an LLM that can see the entire source document to write a 1–2 sentence context string, then prepend that string to the chunk before you embed it and before you index it for BM25.
So chunk 47 goes from:
The margin fell to 8.2% in the same period.
to:
This chunk is from Acme Corp's Q3 2025 10-Q, discussing
gross margin trends in the Hardware segment. The margin
fell to 8.2% in the same period.
Now the vector encodes "Acme Q3 2025 gross margin Hardware." The query "Acme Q3 hardware gross margin" retrieves it. BM25 now matches the literal tokens too. You store the augmented text for retrieval; you can pass either the augmented or original chunk to the generator (I keep the context line — it helps the model too).
The context is situating information, not a summary. You're answering "where does this sit in the document?" not "what does this say?"
How do you generate chunk context without exploding cost?
Prompt caching. Naively, generating context for every chunk means re-sending the whole document once per chunk — a 50-page doc with 200 chunks re-processes those 50 pages 200 times. That's the entire cost objection to contextual retrieval, and caching removes it.
Put the full document in the prompt with a cache_control breakpoint, then loop over chunks reusing the cached prefix. Cache reads are far cheaper than fresh input tokens, so after the first (cache-writing) call, each chunk costs roughly one cache read of the document plus a tiny generated output.
import anthropic
client = anthropic.Anthropic()
DOC_CONTEXT_PROMPT = """<document>
{doc}
</document>"""
CHUNK_PROMPT = """Here is a chunk from the document above:
<chunk>
{chunk}
</chunk>
Give a short, specific 1-2 sentence context situating this
chunk within the overall document, to improve search retrieval.
Answer only with the context, nothing else."""
def contextualize(doc: str, chunk: str) -> str:
resp = client.messages.create(
model="claude-haiku-4-5",
max_tokens=120,
messages=[{
"role": "user",
"content": [
{
"type": "text",
"text": DOC_CONTEXT_PROMPT.format(doc=doc),
"cache_control": {"type": "ephemeral"}, # cache the doc
},
{
"type": "text",
"text": CHUNK_PROMPT.format(chunk=chunk),
},
],
}],
)
return resp.content[0].text
# Index-time loop
augmented = []
for chunk in chunks:
ctx = contextualize(full_document, chunk)
augmented.append(f"{ctx}\n\n{chunk}")
# embed(augmented) and bm25_index(augmented)
Two practical notes. First, the cache has a short TTL (a few minutes on the default ephemeral cache), so process all chunks of one document in a tight loop before moving on — don't interleave documents. Second, use a cheap, fast model. This is a mechanical situating task; Haiku-class models handle it, and you're calling it once per chunk across your whole corpus, so latency and price compound. This is index-time cost, paid once, not query-time cost.
Why combine contextual embeddings with BM25?
Because dense and sparse retrieval fail on different queries, and contextualizing helps both. Embeddings win on paraphrase and semantic similarity ("earnings" ~ "net income"). BM25 wins on exact tokens embeddings smear together: error codes, function names, product SKUs, proper nouns, TLS_ERR_4032. An embedding model may place 4031 and 4032 next to each other; BM25 knows they're different strings.
Run both, fuse the ranked lists (rank fusion or normalized-score fusion), dedupe, and take the union. Contextual retrieval improves both indexes simultaneously because the prepended context adds both semantic framing (helps the vector) and literal keywords (helps BM25). Anthropic reported that contextual BM25 stacked on top of contextual embeddings reduced retrieval failures further than contextual embeddings alone — the two layers are additive, not redundant.
Where does a reranker fit?
At the end, as a precision filter. Retrieval is tuned for recall — you pull the top ~100–150 candidates from the fused dense+sparse pool, accepting noise. A cross-encoder reranker then scores each (query, chunk) pair jointly — the query and chunk attend to each other in one forward pass, instead of being embedded into independent vectors and compared by cosine — and you keep the top ~20 by that score.
This is the biggest single-layer win in the stack, because bi-encoder embeddings compress each chunk into one fixed vector before the query ever exists, while a cross-encoder reads the pair together. The cost is latency: reranking is a per-candidate forward pass at query time, so cap the candidate count. Anthropic's reported numbers put contextual embeddings + contextual BM25 + reranking as the strongest configuration, cutting the top-20 retrieval failure rate by roughly two-thirds versus the naive baseline.
What are the failure modes and costs?
Index-time compute. You now run an LLM call per chunk during ingestion. Prompt caching makes each call cheap, but it's still a real batch job. Budget for it, and re-run it when documents change.
Cache TTL discipline. If your ingestion pipeline interleaves chunks from many documents, you'll blow the cache between calls and pay full document input every time. Group by document.
Context that hallucinates. A weak model can invent a quarter or misattribute a segment. The prompt should demand specific situating detail and forbid speculation; spot-check outputs. This is why "answer only with the context" and a low token cap matter — you're not asking for analysis.
Not a fix for bad chunk boundaries. If a chunk splits a table row from its header mid-cell, context helps but doesn't reassemble the structure. Fix chunking (structure-aware splitting) and contextualize; they're orthogonal.
Small corpora don't need this. If your whole knowledge base fits in a long context window, skip retrieval and just put it in the prompt with caching. Contextual retrieval earns its keep at scale, where the corpus is far larger than any context window.
Direct answer: what is contextual retrieval and why does it fix broken RAG chunks?
Contextual retrieval is an index-time preprocessing step that fixes the core failure of chunked RAG: chunks lose the surrounding context that makes them findable. For each chunk, you use a cheap LLM (Claude Haiku works well) that can see the entire source document to write a 1–2 sentence context blurb, then prepend it to the chunk before embedding it and before BM25 indexing. Prompt caching keeps this affordable by caching the full document once and reusing it across every chunk. Combine the contextualized dense embeddings with contextualized BM25, fuse the results, and rerank the top ~150 down to ~20 with a cross-encoder. Anthropic reported this stack cuts retrieval failures substantially — contextual embeddings help, contextual BM25 adds to that, and reranking on top produces the largest drop. If your RAG system retrieves the wrong chunks despite the right answer sitting in your index, contextualize your chunks before you touch anything else.
Top comments (0)