Every RAG demo looks great with clean inputs. The failure mode that matters in production is quieter: the system answering fluently from the wrong evidence, or inventing the missing piece, and nobody noticing because the output reads fine.
A regression suite with fixed questions (LLM-as-judge grading, retrieval-vs-generation failure tagging) is worth building — but it has a blind spot: it measures the cases you thought to ask. Here is a five-minute manual test that complements it.
The setup
Write two synthetic documents about the same event:
- Document A (older): "The policy change takes effect on March 1. Applies to the Standard plan only."
- Document B (current): "The policy change takes effect on March 15. Applies to all plans."
Upload both, then ask questions whose correct answer depends on which document is current:
- "When does the policy change take effect?" — should cite Document B.
- "Does it apply to the Basic plan?" — only answerable from Document B, and only if the tool tells you which document it used.
What to check
- Current source citation. Does the answer cite Document B, or blend the two dates into something plausible?
- Missing-fact honesty. Ask about something neither document covers ("who approved the change?"). A good system says it is not in the documents. A bad one interpolates a name.
- Access boundaries. If the tool supports per-user or per-file permissions, remove Document B and re-ask. The answer should degrade honestly, not silently fall back to Document A as if it were still authoritative.
The step most people skip
Swap the dates and re-run: now Document A says March 15 and Document B says March 1. If the tool was passing by memorizing file order or filenames, it fails now. A system that genuinely resolves currency survives the swap.
Why this matters more than accuracy scores
The failures this test catches are generation-side failures that look correct — fluent answer, wrong provenance. They are invisible in retrieval metrics and dangerous in real workflows (meeting notes, policy Q&A, client research), where a confident wrong date costs more than an honest "not found".
Where I use it
I maintain a small public collection of open-source knowledge-base and RAG candidates (RAGFlow, AnythingLLM, and others) and run this test across them as a first-pass filter before anything deeper:
https://useaistation.com/githubai/collections/enterprise-knowledge/
Free to browse, no sign-in or install. It is a discovery aid — not a benchmark and not a license audit; passing this test is necessary, never sufficient.
Disclosure: I build AI Open Source Radar, the site hosting the collection linked above. This article was prepared with AI assistance; the test method is the point, and it works with any RAG tool you already use.
Top comments (0)