Your RAG Problem Is a Content Operations Problem

A support assistant gives a customer the wrong refund window. Fourteen days, when the policy is thirty.

The engineering response is predictable. Check the chunk size. Try semantic chunking. Add a reranker. Tune the top-k. Maybe the embedding model is wrong for this domain. Maybe try a hybrid BM25 plus dense approach.

All reasonable moves. And in most cases I have looked at, none of them are the problem.

The problem is that there are two articles in the knowledge base. One says fourteen days, one says thirty. Both are indexed. Both are plausible. The retriever did its job perfectly and returned a document that says fourteen days, because that document exists and nobody deleted it when the policy changed in 2024.

No retrieval architecture rescues a corpus that contradicts itself.

What is actually in your knowledge base

Run an audit before you tune anything. The findings are consistent across organisations.

Contradictions. The same question answered differently in two places, because one was updated and the other was not. Nobody knew the second one existed.

Stale content. Policies that changed, products that were discontinued, processes that were replaced. The articles remain, indexed, competing for retrieval with the correct ones — and often winning, because older content has had longer to accumulate the phrasing that matches how customers ask.

Missing discriminators. An answer correct for one market and wrong for another, with nothing in the document to distinguish them. The retriever cannot filter on information that is not there.

Audience mismatch. Content written for internal staff who already have context, being served to customers who do not. Technically correct, practically useless, and occasionally alarming.

Orphaned fragments. Sections of longer documents that make no sense retrieved in isolation, because they depend on context three headings up.

Model choice will not fix any of these. Neither will chunking strategy.

The metadata that makes retrieval work

Most retrieval failures in customer service are filter failures wearing a relevance costume. The document you want is in the index; the document you do not want ranked above it.

Minimum viable metadata per document:

market:          [UK, IE]           # or ALL
product:         [premium, plus]    # or ALL
effective_from:  2024-06-01
effective_to:    null               # null = current
audience:        customer           # customer | internal
owner:           billing-team       # who keeps it correct
last_verified:   2026-07-14

With this, retrieval becomes a filter followed by a search rather than a search and a hope:

candidates = index.search(
    query,
    filter={
        "market":   user.market,
        "product":  user.product,
        "audience": "customer",
        "effective_from": {"$lte": today},
        "$or": [{"effective_to": None},
                {"effective_to": {"$gte": today}}],
    },
    top_k=8,
)

The effective_to field alone eliminates a large share of wrong answers, because it lets superseded content stay in the archive without competing for retrieval.

And last_verified gives you something you probably do not currently have: a measurable indicator of corpus decay. Documents nobody has confirmed in a year are where your wrong answers concentrate.

Detecting contradictions at scale

You cannot read ten thousand articles. You can find the ones that disagree.

A workable approach:

1. For each document, generate a set of question–answer
   pairs it supports.
2. Cluster the questions semantically.
3. Within each cluster, compare the answers.
4. Where answers diverge, flag for human review.

This surfaces the fourteen-versus-thirty-days problem without anyone auditing the corpus by hand, and it is exactly the sort of high-volume, low-judgement work that is now cheap to run.

The output is a review queue, not a fix. A human decides which answer is right — and that decision frequently requires someone from the business rather than from the content team, which is itself a finding worth surfacing.

Ownership is the durable fix

Audits decay. A corpus cleaned in March is dirty again by September unless something maintains it.

The mechanism that works is ownership at the document level, with a review cadence. Every document has a team, and that team confirms or updates it on a schedule. Documents past their review date get flagged, and eventually deprioritised in retrieval rather than silently serving stale answers.

This is a content operations investment wearing an AI project's clothing, and it is the part most often absent from budgets. Teams fund the engineering and not the curation, then wonder why quality degrades after the launch quarter.

The measurement that tells you where you are

Before tuning anything, establish whether retrieval or generation is failing. They need different fixes and get conflated constantly.

Take a sample of wrong answers and check manually: was the correct document in the retrieved set?

correct doc retrieved, answer still wrong  → generation problem
                                             (prompt, model, context length)

correct doc NOT retrieved, but exists      → retrieval problem
                                             (metadata, chunking, ranking)

correct doc does not exist / contradicted  → CONTENT problem
                                             (most common)

In the deployments I have reviewed, the third row dominates — and it is the only one that no amount of engineering effort addresses.

Full guide — containment versus resolution, escalation design, agent assist, the cost curve and sequencing: AI Customer Experience. We also audit knowledge corpora before deployment.

Frequently Asked Questions

Why does my RAG system give confidently wrong answers?

Usually because the corpus contains contradictions or stale content that retrieval faithfully returns. Generated answers read fluently regardless of source quality, which makes bad content harder to spot than it was with keyword search.

How do I tell whether the problem is retrieval or content?

Sample wrong answers and check whether the correct document was in the retrieved set. Retrieved but wrong is a generation issue; not retrieved but present is a retrieval issue; absent or contradicted is a content issue — and that is the most common.

What metadata does a support knowledge base need?

Market, product, effective dates, audience, owner and last-verified date at minimum. Effective dating alone removes a large share of wrong answers by letting superseded content stay archived without competing for retrieval.

How can I find contradictions without reading everything?

Generate question–answer pairs per document, cluster the questions semantically, and compare answers within each cluster. Divergence goes to a human review queue — often needing someone from the business rather than the content team.

How do I stop the corpus decaying again?

Document-level ownership with a review cadence. Flag documents past their review date and deprioritise them in retrieval rather than letting them silently serve stale answers.

Should I fix content before tuning retrieval?

Yes. Chunking strategy and rerankers cannot resolve a corpus that contradicts itself, and tuning against bad content optimises for retrieving the wrong document more reliably.