A five-minute test for any RAG assistant: two documents with conflicting dates

Wolf Zhang — Sat, 12 Sep 2026 11:08:30 +0000

Every RAG demo looks great with clean inputs. The failure mode that matters in production is quieter: the system answering fluently from the wrong evidence, or inventing the missing piece, and nobody noticing because the output reads fine.

A regression suite with fixed questions (LLM-as-judge grading, retrieval-vs-generation failure tagging) is worth building — but it has a blind spot: it measures the cases you thought to ask. Here is a five-minute manual test that complements it.

The setup

Write two synthetic documents about the same event:

Document A (older): "The policy change takes effect on March 1. Applies to the Standard plan only."
Document B (current): "The policy change takes effect on March 15. Applies to all plans."

Upload both, then ask questions whose correct answer depends on which document is current:

"When does the policy change take effect?" — should cite Document B.
"Does it apply to the Basic plan?" — only answerable from Document B, and only if the tool tells you which document it used.

What to check

Current source citation. Does the answer cite Document B, or blend the two dates into something plausible?
Missing-fact honesty. Ask about something neither document covers ("who approved the change?"). A good system says it is not in the documents. A bad one interpolates a name.
Access boundaries. If the tool supports per-user or per-file permissions, remove Document B and re-ask. The answer should degrade honestly, not silently fall back to Document A as if it were still authoritative.

The step most people skip

Swap the dates and re-run: now Document A says March 15 and Document B says March 1. If the tool was passing by memorizing file order or filenames, it fails now. A system that genuinely resolves currency survives the swap.

Why this matters more than accuracy scores

The failures this test catches are generation-side failures that look correct — fluent answer, wrong provenance. They are invisible in retrieval metrics and dangerous in real workflows (meeting notes, policy Q&A, client research), where a confident wrong date costs more than an honest "not found".

Where I use it

I maintain a small public collection of open-source knowledge-base and RAG candidates (RAGFlow, AnythingLLM, and others) and run this test across them as a first-pass filter before anything deeper:

https://useaistation.com/githubai/collections/enterprise-knowledge/

Free to browse, no sign-in or install. It is a discovery aid — not a benchmark and not a license audit; passing this test is necessary, never sufficient.

Disclosure: I build AI Open Source Radar, the site hosting the collection linked above. This article was prepared with AI assistance; the test method is the point, and it works with any RAG tool you already use.

Why I separated live discovery from the AI chat box

Wolf Zhang — Sat, 29 Aug 2026 15:32:58 +0000

Most AI workspaces start with the same useful primitive: a chat box. I kept one in AI Workstation because it is still the fastest interface for many research and writing tasks.

But while using the product for day-to-day work, I found two questions that did not belong in a general chat flow:

What current topic is worth researching today?
Which open-source AI project is worth evaluating now?

Both questions depend on live evidence. They also have different failure modes from ordinary drafting. A model can produce a fluent answer while using stale memory, mixing project identities, overlooking a license, or treating popularity as proof of quality.

That led me to split AI Workstation into three layers: a general workspace, public discovery Radars, and installable Agent Skills.

Layer 1: the workspace

The main AI Workstation handles everyday knowledge work: questions, links, documents, images, drafting, proofreading, reusable templates, and exports.

The point is not to hide every operation behind one large prompt. It is to keep routine work accessible while letting tasks that need current data move into a more explicit flow.

Layer 2: public Radars for live discovery

The first Radar is Global Topic Radar. It is designed for creators and editors who need current candidates rather than generic content ideas. It keeps the topic lane, freshness, market context, evidence state, and original sources visible.

The second is Open-Source AI Radar. It is designed for developers and researchers comparing active AI projects. It presents dated rankings, categories, collections, and project cards with direct links to upstream repositories. Stars, forks, licenses, languages, and practical summaries are treated as research inputs.

The important design choice is what the Radars do not claim:

A topic score is not a prediction that a post will go viral.
Project popularity is not a security audit or a quality guarantee.
A generated summary does not replace the upstream repository or license text.

The Radars are intentionally usable without signing in. A visitor can inspect the current data before deciding whether the workflow is useful.

Layer 3: Agent Skills as research contracts

Discovery and research are different jobs. A Radar can show a promising lead, but an Agent still needs rules for what to do next.

Topic Intelligence turns one current Radar item into a structured content brief. The output includes research questions, must_verify, avoid_claims, and visual requirements. Those fields matter more to me than a long generated script because they expose the work that remains uncertain.

AI Open Source Intelligence handles the open-source side. It resolves project identity, examines license evidence, builds comparison matrices, and plans candidate stacks under explicit constraints. It also exposes nine read-only MCP tools so an Agent can retrieve public Radar data without executing code from third-party repositories.

Both Skills are open source:

They can be inspected as normal repositories with SKILL.md, scripts, references, Agent metadata, and release assets. The goal is to make the operating contract reviewable before installation.

Why not let the model decide everything?

A model is very good at organizing a bounded set of material. It is much less reliable as an invisible authority for freshness, identity, provenance, or legal interpretation.

Separating the layers makes failures easier to see:

If discovery is stale, inspect the Radar data and source timestamps.
If a project was confused with a similarly named repository, inspect identity resolution.
If a claim is not ready to publish, it should remain in must_verify instead of being smoothed into confident prose.
If an integration needs to read public data, keep it read-only rather than giving it execution privileges by default.

This is also why I do not describe the system as a one-click content factory. Topic selection and early research happen here. Script review, asset production, publishing, and platform performance belong to later workflows, and performance is not something the Radar can guarantee.

The trade-off

The product is less dramatic to explain when each component has a narrow job. There is no single button that claims to discover a trend, verify every fact, generate production assets, publish the result, and predict its reach.

The benefit is that the boundaries are inspectable:

the workspace handles general work;
the Radars surface current leads and sources;
the Skills structure repeatable Agent-side research;
the user keeps final judgment.

That separation is the central idea behind AI Workstation today. I would especially value feedback from people building Agent workflows: which parts of live discovery should stay outside the model, and which research checks should be enforced as part of the Skill contract?

This article was drafted with AI assistance and reviewed, revised, and published by the builder. Product claims and links were checked against the current public pages before publication.

DEV Community: Wolf Zhang