I almost shipped a RAG pipeline that, on certain questions, cited exactly the right document, and then told the user the answer wasn't in it.
Every unit test was green. The retrieval returned the correct chunk. The API returned 200. The citation was attached to the response. By every check I had, it worked. The first run of my eval harness scored it 0.57, and that number is the only reason I found out before users did.
This is the story of those two bugs, why no unit test I could have written would have caught them, and why I now believe an eval harness belongs in a GenAI project from day one, not "once it's stable."
What the eval harness actually does
For the RAG starter I was building, "chat with your documents," I wanted a test that exercised the thing users actually do, end to end. So the harness:
- Ingests a small fixture corpus (a few public-domain documents) with real embeddings into a dedicated database, no mocks.
- Runs the real agent loop for each question: the same tool-calling, the same retrieval, the same prompts as production.
- Grades every answer with an LLM-as-judge, scoring two things: faithfulness (is the answer supported by the retrieved context?) and citation correctness (do the citations point to the right places?).
- Includes negative cases, questions whose honest answer is "that's not in these documents", where the correct behavior is a refusal, not a confident guess.
It's deterministic-ish (temperature 0, a fixed judge prompt), and the score is committed per-commit, so a regression in retrieval visibly drops the number in a PR.
Then I ran it for the first time. 0.57.
Bug #1: the model was shown a preview, not the evidence
Digging into the low-faithfulness cases, I found answers where the agent retrieved the correct chunk, attached it as a citation, and still said the information wasn't there. It was citing a source while denying its contents. That makes no sense, until you look at what the model was actually handed.
When the agent called the search tool, the tool result it received back was the 400-character UI snippet, the little preview you show in a citation chip, not the full chunk text. The snippet was built for humans skimming a sidebar. The model was being asked to answer from the preview while the actual evidence sat in a field it never saw. If the answer lived past character 400 of the chunk, the model genuinely couldn't see it, and faithfully reported it wasn't there.
The fix was to separate the two concerns: a snippet for the UI, and the full content for the model.
class Citation:
snippet: str # short preview, for the UI chip
content: str # full chunk text, for the model to reason over
Here's the part that matters for this post: every unit test still passed after I found this, and every unit test passed before. Retrieval returned the right chunk (✓). The citation was attached (✓). The endpoint returned 200 (✓). The bug wasn't in any unit, it was in the contract between units: what one component handed another, inside a loop, at runtime. The only way to see it was to look at the behavior of the whole system on a real question and ask "is this answer actually good?" That question is precisely what an eval harness asks and a unit test does not.
Bug #2: the agent answered from memory instead of searching
The negative cases surfaced the second bug. For general-knowledge questions, the agent would often just... answer. Confidently. From the model's own parametric memory, without ever calling the search tool. Sometimes it was even right, which is worse, because it means the behavior is unreliable in a way that looks fine in a demo.
For a "chat with your documents" product, that's a correctness bug: the whole value proposition is that answers are grounded in your documents, with citations. An answer that skips retrieval is off-contract even when it happens to be correct, and it's how you get confident hallucinations on the questions that matter.
The fix was prompt-level: a hardened, search-first system prompt and a sharper tool description that makes "search before you answer" the default, plus a relevance-score floor on citations so weak matches don't get dressed up as sources. The eval harness is what told me the fix worked rather than just felt better, the negative cases started passing without dragging down the answers that should be grounded.
After both fixes, the next run came back at 0.96 (faithfulness 0.99, citation correctness 0.93). Same code path, same corpus, the harness measured the difference instead of me guessing at it.
Why this changed how I build GenAI features
Unit tests verify that a function does what you wrote it to do. They're necessary, and both of my bugs slipped past a green suite because both functions did exactly what they were written to do, the system just behaved badly. LLM-powered features fail at a different layer: not "this function returned the wrong value" but "this non-deterministic pipeline produced an unfaithful answer." You can't assert your way to that with assertEqual.
An eval harness is the test for that layer. A few things I'd now treat as non-negotiable:
- Run the real pipeline. Mocks would have hidden both bugs, the snippet/content split and the skipped search only exist in the real loop.
- Grade behavior, not strings. "Is this answer faithful to the context?" is the question; an LLM judge with a fixed rubric and structured output is a practical way to ask it at scale.
- Include negative cases. "The answer is not in the corpus" catches the confident-hallucination failure mode that positive cases never will.
- Commit the score. A number in every PR turns "did retrieval regress?" from a vibe into a diff.
- Do it on day one. The harness cost me an afternoon and caught two ship-blockers on its first run. "We'll add evals later" means shipping the 0.57 version.
The whole harness, runner, fixture corpus, judge, and the per-commit results, is open source (MIT) in the starter, if you want the exact shape of it:
👉 github.com/delmalih/saas-genai-starter
The uncomfortable takeaway: my pipeline was "done" and fully green at 0.57. The only thing standing between that and production was one number from a test that actually asked whether the answers were any good. If you're building anything RAG- or agent-shaped, write that test first.
Top comments (7)
Bug #1 is the one I wish more people talked about: "the model never saw what you think you retrieved" is an entire bug class, and it's invisible to unit tests precisely because it lives in the contract between components at runtime. The snippet-vs-content split is exactly right — the UI preview leaking into the model's evidence is a trap I've watched bite a few teams.
Two things that extended this for us. First, the judge needs its own eval: an LLM-as-judge will quietly reward confident wording, so we calibrate it against a small human-labeled set and re-check it on every model bump. Otherwise a 0.96 today and a 0.96 after a model upgrade don't mean the same thing, and the number you commit per-commit drifts out from under you. Second, beyond clean negative cases, add near-miss cases — questions where the answer is almost-but-not-quite in the corpus. Those catch the Bug #2 behavior (answering from memory) far more reliably than clean negatives, because the model is most tempted to skip retrieval when it "sort of" knows already. How are you keeping the judge stable across model versions?
Both additions are exactly the kind of thing I wish I'd thought of before run #1. The near-miss cases especially, you're right that clean negatives are almost too easy to refuse correctly; the model has nothing pulling it toward an answer. A "sort of in the corpus" question is a much better stress test for Bug #2, because that's precisely the situation where skipping retrieval feels harmless.
On keeping the judge stable across model bumps: right now it's pretty manual. I keep a small, frozen set of ~20-30 human-labeled (question, answer, expected verdict) pairs, and whenever I touch the judge prompt or swap the underlying model, I re-run the judge against that set before trusting any new score. If the judge's agreement with the labels drops, I treat that as a red flag before even looking at the pipeline's score. It's not continuous monitoring, more of a gate at the moment of change, but your point stands that without re-checking it on every model bump, the 0.96 silently stops being comparable across time. I'll probably formalize that into something closer to a "judge eval" that runs alongside the main harness rather than only on demand. Appreciate the push on this, it's a real gap.
The "green at 0.57" framing is the right gut-punch, and run-the-real-pipeline is the part most people skip because mocks are easier. The question I'd put to the harness: the judge is itself a non-deterministic component you're now trusting to grade the others, so what grades the judge? Your 0.96 is the judge's opinion that the answers are faithful, not ground truth that they are. If the judge shares a blind spot with the pipeline, say both are loose about what "supported by the context" means, then a subtly unfaithful answer scores faithful and the harness reads green on a real miss, which is the exact 0.57-looks-like-passing failure one level up. The negative cases help because a refusal is harder to fake past a judge than a plausible-but-wrong answer. So I'd ask: do you ever check the judge against human labels on a slice, or treat the judge score as the oracle? Not a knock on the approach, it's the strongest version of the thing you're already doing, I just want to know where the ground truth actually bottoms out.
This is the question I was hoping someone would ask, because you're right that I glossed over it in the post. The 0.96 is the judge's opinion, full stop. I don't currently have an independent ground truth that the harness checks itself against on every run.
What I do have is a small human-labeled slice (a few dozen question/answer/verdict triples) that I use to sanity-check the judge whenever I change the judge prompt or the underlying judge model, basically the same gap another commenter raised. But you're pointing at something sharper: even with that check, if the judge and the pipeline share the same blind spot about what "supported by context" means, a confidently-wrong-but-plausible answer can sail through both. The negative cases are a partial defense, like you said, because a flat refusal is a much higher bar to fake than a plausible paraphrase that quietly drifts from the source.
So no, I don't treat the judge score as oracle, but I also don't have a rigorous answer for "where does the ground truth bottom out" beyond that human-labeled slice. I think the honest next step is to grow that human-labeled set specifically with subtly-unfaithful answers, not just correct/incorrect, things that sound supported but quietly aren't, since that's exactly the failure mode you're describing the judge being vulnerable to. Good prompt to leave with me.
the honest "no rigorous answer beyond the human-labeled slice" is the right place to land, and i think growing that slice with subtly-unfaithful answers is exactly the move. the thing i'd add is that the failure you're defending against has a specific shape: it's not correct-vs-incorrect, it's plausible-but-drifted, and those two need different labeling effort. a flat wrong answer is cheap to label and the judge catches it anyway. the expensive, valuable labels are the ones where the answer sounds supported and quietly isn't, because that's the region where judge and pipeline share a blind spot. so if you're growing the set, i'd weight it hard toward that region: paraphrases that preserve fluency while dropping or inverting one load-bearing fact from the source. the other thing that helps is authoring those hard negatives with a different generator than the one that produced the pipeline outputs, so the unfaithful examples don't inherit the same drift patterns the judge is already tuned to miss. otherwise you're sampling the failure mode from inside the distribution you're trying to audit. curious how you're planning to source the subtly-unfaithful set, hand-write them or mine them from real pipeline runs that you then relabel?
It's great to see your evaluation harness paying off! The challenges you faced with citation accuracy highlight a crucial need for robust document verification in RAG pipelines. Using DocImprint's MCP server could help you capture and prove the provenance of the documents your model cites. The evidence bundles generated include SHA-256 hashes, ensuring that you can verify the integrity of the documents at any point. This not only enhances the reliability of your citations but also provides an audit trail that can be vital for compliance. You might want to explore how integrating these features could further improve your pipeline.
Thanks for reading! To be clear, the two bugs here weren't about document integrity or provenance, the chunk content itself was correct and unaltered both times. The failures were about what got passed to the model (snippet vs. full content) and whether retrieval was even triggered. Hash-based provenance is a different problem than the one this harness was built to catch.