Quick answer: Diagnose a RAG retrieval failure by checking the pipeline in order. First, confirm the answer was ingested and parsed, that chunking kept it whole, that the embedder ranked it, that no filter dropped it, and that the reranker and the language model kept it. Log the retrieved set per query. When the failing stage is a model, swap it on Superlinked Inference Engine to get one API for embeddings, reranking, OCR, and extraction, on SIE Cloud or self-hosted.
A customer support agent asks the model assistant about the refund window for enterprise contracts. The answer cites the correct policy document, but it gets the number wrong. The policy exists. It is in the index. But the page with the refund table is a scanned PDF, and the parser read its two columns as one. The chunk that should have matched now reads like a ransom note. The dev team spends two days tuning the prompt. The language model, as usual, takes the blame for the PDF parser's crimes.
Most retrieval failures look like model failures from the outside. The answer is wrong; the language model produced it, so you replace the language model, or the embedder. But the main problem is often several stages earlier, so a bigger model cannot fix a table it never saw.
This guide is a stage-by-stage triage for any production RAG pipeline. It assumes nothing about your vector database, framework, or model host. Vertical-specific causes, such as defined terms in contracts, superseded amendments, or restated reporting periods, have their own deep dives in this series. What follows is the generic order of operations.
Here is how to track down a retrieval problem in your RAG pipeline.
Step 1: Reproduce the failure and log the retrieved set
You cannot diagnose a retrieval failure from the final answer. First, you need the intermediate state. For each failing query, capture:
- the raw query and any rewritten form the pipeline sent to search
- the metadata filters applied
- the top k chunk IDs from vector search, with scores
- the order after reranking, with scores
- the chunks actually placed in the prompt
- the final answer and its citations
Read the rewritten query before anything else. A rewriter that omits "enterprise" or expands an acronym incorrectly sends a different question to search, which means every later stage answers that wrong question. Rerun the search with the raw query. If the correct chunk appears, the rewrite is the stage to fix.
Then identify the gold chunk: the passage a human would say answers the question. Every later step asks one question about it, which is where it was lost. Without the accurate chunk, triage becomes opinion.
Step 2: Check the answer is in the index at all
Search the index for the gold passage directly, by document ID or by an exact phrase from the source. If it is not there, stop looking at models.
Ingestion failures are common and hard to see. They can occur because:
- A connector skipped a folder.
- A file type was silently ignored.
- OCR produced text from a scan, but the text is wrong: merged columns, dropped table rows, headers read into the body.
Open the stored text for the source page and compare it with the original by eye. If a human cannot find the answer in the stored text, no embedder will.
Step 3: Check whether chunking split the answer
If the passage is indexed, look at how it was cut. Fixed-size chunking often separates a heading from its content, a table from its caption, or a condition from the clause it qualifies. The refund window can end up in one chunk and the words "enterprise contracts" in the next, so neither chunk matches the query well.
Signs of a chunking problem include:
- a gold passage that spans two chunks
- a chunk that starts mid-sentence, and
- a table row stored without its column headers
The fix is usually to switch from fixed-size to structural chunking, which chunks by section or table and carries the heading into each chunk. This requires re-embedding the affected documents, so confirm it on the failing queries first.
Step 4: Check whether the embedder ranked it
With the gold chunk intact, find its rank in the raw vector search results. Recall@k on a small set of failing queries tells you whether the embedder is at fault. If the gold chunk sits at rank 3, the embedder did its job. If it sits at rank 80 or doesn't appear in the top 200, the embedding stage is the problem.
An embedder failure can be caused by:
- The query using shorthand the model never learned. The answer often depends on an exact identifier, such as a product code, that dense vectors blur.
- The query and the documents being in different languages.
Test a hybrid search with a sparse component for identifier queries, or a domain embedder for vocabulary gaps, before replacing the model outright.
Two causes look like embedder failures but aren't. The first is approximate search. HNSW settings such as a low ef, or aggressive quantisation, can drop a chunk that exact nearest-neighbour search would return. To fix this, rerun the query with exact search, which Qdrant and most vector databases expose, and if the gold chunk shows up, tune the index, not the model. The second is version skew: the query is embedded by a newer model version than the one that built the corpus, after an upgrade that re-embedded only new documents. Log the embedder version on both sides and compare them.
Step 5: Check whether a filter dropped it
Metadata filters fail silently. A date filter excludes a document whose effective date was parsed incorrectly. A tenant filter excludes a shared policy stored under a different owner. A document-type filter excludes a PDF labelled as an image. The gold chunk ranks well, then vanishes.
Rerun the failing query with filters removed. If the gold chunk appears, inspect the metadata on that document against the filter the pipeline built. Many teams find the filter logic is correct, but the metadata isn't, which points the problem back to ingestion.
Step 6: Check whether the reranker or the language model discarded it
If the gold chunk reached the candidate list, follow it through the last two stages. Compare its rank before and after reranking. A reranker that pushes it from rank 4 to rank 30 is making a mistake you can see and measure, often on long chunks or on queries unlike its training data.
If it survives reranking and reaches the prompt, the failure belongs to generation. Check first that the chunk survived prompt assembly: context truncation and deduplication run after reranking and can cut it. If it is in the prompt, the model may under-weight a chunk buried in a long context, prefer a more fluent passage that is less precise, or be told by the prompt to answer when it should decline. Only at this point is changing the language model or the prompt the right move.
A trace across the stages makes the pattern obvious. The log below is illustrative, written to show the method, and it records no real system:
| Query (illustrative) | In index | Chunk intact | Vector rank | After filter | After rerank | In prompt |
|---|---|---|---|---|---|---|
| Refund window, enterprise | yes | no, split | 41 | 41 | 22 | no |
| Data retention period | yes | yes | 3 | dropped | n/a | no |
| SSO setup steps | yes | yes | 6 | 6 | 1 | yes |
The first row is a chunking fix. The second is a metadata fix. Neither calls for a new model. The third row retrieved correctly, so if its answer was still wrong, look at generation.
Where SIE fits
Triage is an experiment where you try a different OCR model on the mangled page, a sparse embedder for identifier queries, or another reranker for the long-chunk case. When each model runs on its own server, every one of those experiments means deploying something new before you can learn whether it helps.
Superlinked's inference engine, SIE, puts the model stages of the triage behind one API. The three retrieval primitives, encode, score, and extract, cover the embedder, the reranker, and document parsing, and a fourth, generate, serves smaller language models. On SIE Cloud, the managed product, the menu includes OCR with LightOnOCR-2-1B, document-to-markdown with Docling, dense, sparse, and multi-vector search models, and Qwen3 rerankers. Testing a different reranker is a different model ID in the same call:
items = [{"id": str(i), "text": c} for i, c in enumerate(candidate_chunks)]
before = client.score("Qwen/Qwen3-Reranker-0.6B", {"text": query}, items)
after = client.score("Qwen/Qwen3-Reranker-4B", {"text": query}, items)
Self-hosted, the same engine runs in your own cluster, which matters when the traces you are debugging contain customer documents that should not leave your environment. The deployment guide covers Kubernetes, Docker, and offline installs.
Honest take: SIE is the production layer for the model experiments triage ends in: a different OCR model, a sparse embedder, another reranker, on one API for embeddings, reranking, OCR, and extraction. SIE Cloud or self-hosted, open weights, GPUs shared across those models, so the swap is a model ID. The stage-by-stage log still lives in your pipeline. When the failing stage is a model, that model belongs on SIE, in the cloud or on your own cluster.
FAQs
Is it the embedder or the chunker?
Check the chunk before the embedder. Open the stored text of the gold passage and ask whether a reader could answer the query from that chunk alone. If not, the chunker failed, and any embedder would struggle. If the chunk reads well and still ranks low in vector search, the embedder is the suspect. Confirm by embedding the query and the gold chunk directly and comparing their similarity with that of the chunks that outranked it.
What should we log per query?
Log enough to replay the retrieval without rerunning the model: the query and its rewrite, filters, the top k chunk IDs with scores before and after reranking, the chunks sent to the language model, and the model versions of the embedder (for queries and for the corpus) and the reranker. Store chunk IDs rather than full text to keep volume down, with a way to resolve them later. If the text contains personal data, apply the same retention and access rules as the source documents.
Would a bigger language model fix this?
Often, no. A larger model reasons better over the context it receives, but it cannot use a passage that was never ingested, was split in half, or was filtered out. In the illustrative log above, two of three failures happened before generation. A bigger model also raises the cost of every query, including the ones that were already correct. A specialist model at the failing stage, such as a different reranker or OCR model from the SIE model catalog, is the experiment to run. Fix the stage that lost the answer, then judge whether model generation still needs an upgrade.
Triage first, models second
Walk the stages in order, find where the gold chunk disappeared, and fix that stage. Most of the time it will be data. When it is a model, test the alternative before replacing anything. For the model experiments, SIE Cloud lets you swap OCR, embedding, and reranking models behind one API, with the same engine available to self-host.
Top comments (1)
Reading the rewritten query before anything else is the step I'd underline. A rewriter that drops one qualifier like "enterprise" looks exactly like an embedding problem from the outside, and teams can lose days tuning the wrong stage. One thing that makes step 1 cheaper over time: store the gold chunk ID next to each failing query in the eval set. Then the next regression run tells you right away which stage lost it, instead of starting the triage from scratch.