I built a RAG app on LlamaIndex in about four lines. Wire an index to a query engine, point it at my documents, ask a question, get an answer. The first hundred queries were great. I was impressed with how little code it took.
Then a user asked something my little setup had never been tested on. The answer came back confident and wrong. I checked my eval, which reported a single faithfulness score, and all it told me was that faithfulness had dropped. Not which part of the pipeline broke. Not why. Just a number going down.
That was the moment I understood the trap. LlamaIndex makes RAG incredibly easy to assemble, and that ease hides how many separate moving parts you just wired together. One quality score cannot debug a thing made of five different pieces. Here is how I evaluate it now, layer by layer.
Why one RAG score is not enough here
Plain RAG is basically two steps: retrieve some chunks, write an answer from them. Two steps means two things to check, and one score almost covers it.
LlamaIndex is rarely that flat. The interesting apps compose. You add a router that picks between several engines. You add a step that breaks a big question into smaller ones. You pick a synthesizer mode that decides how the final answer gets assembled. You maybe turn the whole thing into an agent that calls tools. Every one of those pieces is its own little component with its own way of failing.
So when a composed pipeline gives a wrong answer, the single faithfulness score is measuring the very end of a long chain and telling you nothing about where the chain broke. The fix is to stop scoring the program as one blob and start scoring each piece against its own job. I think of it as four layers.
Layer one: score the retrieval on its own
The first layer is the retrieval, scored completely independently of the answer that comes after it. This is the upstream signal. If retrieval is broken, every later score will look bad too, so this is where the debugging should start.
Three things I check on the retrieved chunks:
- Are the chunks actually relevant to the question? This catches the classic case where the search grabbed a passage that shares words with the query but means something different. You asked about Section 12, it handed you Section 9.
- Do the chunks actually support the claims in the answer? This catches invented citations, where the answer points at a chunk that does not say what the answer claims.
- Did the answer even use the chunks it fetched? This one surprised me. If you fetch ten chunks and the answer uses two, you are over-fetching, and that is both wasteful and a sign your retrieval settings are loose. The fix is usually a reranker before the answer step, or just pulling fewer chunks.
The rule that made this useful: score retrieval per retriever, not per app. Different retrieval methods fail in different shapes, and tagging the score by which retriever ran means a regression points straight at the one that moved.
Layer two: score the routing and the decomposition
This is the layer generic RAG eval completely misses, and it is where two of my nastiest bugs lived.
If you use a router that picks between engines, you have to score the routing decision on its own. Here is why it is sneaky: if the router sends a question to the wrong engine, the chunks that engine retrieves are still relevant to whatever it retrieved, and the answer is still grounded in them. Both your normal scores look fine. The answer is wrong purely because the wrong engine ran. So I score routing as its own question: given the query and the choices, did it pick the right engine? A right answer from the wrong engine is luck, and the next harder question will expose it.
If you use a step that breaks a big question into smaller ones, score the decomposition too. The failure I hit here was a dropped condition. A user asked for X under a specific condition Y. The decomposer split it into "tell me about X" and "tell me about Y" as two separate questions, and the final answer lost the "under condition Y" part entirely. Each sub-question was answered perfectly. The combined answer was still wrong, because the join between them got lost.
So for a decomposing engine I score each sub-question on its own, then score the final merged answer on completeness against all of them together. If the sub-questions score well but the final answer does not, the merge step is dropping something. If a single sub-question scores badly, that branch is your problem. Same data, two clearly different bugs.
Layer three: watch which synthesizer mode you are using
LlamaIndex gives you a few ways to assemble the final answer from the retrieved chunks, and they genuinely behave differently. One mode stuffs everything into a single call. Another refines the answer chunk by chunk. Another summarizes in a tree.
The one that bit me: the tree-style summarizer drops citations on long contexts that the simpler mode handled fine. If you switch modes in a deploy and only score the final answer, all you see is that quality dropped after Tuesday, with no hint that the synthesizer mode was the cause. So I treat the synthesizer mode as a thing worth scoring on its own, especially citation validity, and I re-check it whenever the mode or the context length changes. Long-context behavior is exactly where these modes diverge.
Layer four: run the same checks in production, not just in CI
Your offline eval catches the failures you thought to test. Production catches the ones you did not. So the same four layers of checks that run before release should also run against real traffic after release.
You do not have to score everything. The cheap checks, like whether a citation actually exists, can run on every request. The expensive judge-based ones can run on a small sample of live traffic. Then you watch for a sustained drop in any one of them per route.
The most useful signal turned out to be the gap between my offline scores and my live scores. When they agree, my test set still looks like reality. When they drift apart, my test set has gone stale and is no longer testing what users are actually asking. That gap is worth watching on its own.
A note on sensitive data
If your app retrieves over anything sensitive, like medical, legal, or financial documents, the same checks that score offline should also gate at request time. Run them before the answer goes out, and if the answer is not grounded in what was retrieved, fall back to a safe response instead of shipping the confident wrong one. It is the same rubric, just used as a gate instead of a report. For high-stakes paths I make it strict, every check has to pass. For casual ones, a majority is enough.
The mistakes that ship LlamaIndex regressions quietly
- One faithfulness score for the whole app. It tells you something broke, never which piece.
- Not scoring the router. A wrong-engine answer looks perfectly grounded, so it hides in plain sight.
- Not scoring the decomposition. The dropped-condition bug passes every per-sub-question check.
- Ignoring the synthesizer mode. Switching modes can drop citations with no other visible cause.
- Over-fetching and never checking. If the answer ignores most of what you retrieved, you are paying for chunks that do nothing and loosening your relevance.
- CI-only eval. Production drifts past a frozen test set within a quarter. Score live traffic too.
The lesson I keep coming back to is that LlamaIndex's biggest strength, how easily it composes primitives, is also what makes it hard to debug. Every primitive you add is another place a failure can hide, and a single score averages all of them into one useless number. Once I scored each layer on its own, the bug that had been invisible for a week was obvious in an afternoon.
If you want the deeper version, with the exact rubric for each layer and how to wire routing and decomposition checks specifically, this piece walks through all four layers in detail.
If you run LlamaIndex in production, I am curious which primitive broke on you first. For me it was the router, an answer that was perfectly grounded in evidence the wrong engine went and fetched.
Top comments (0)