Relevance Is Not Enough: How to Evaluate Enterprise RAG for Authority, Permissions, and Time

A retrieval system can return highly relevant context and still produce the wrong enterprise answer.

The retrieved policy may be obsolete. The document may belong to the wrong region. A newer version may exist in another system. The user may not have permission to see the source. Two equally relevant records may conflict, with no survivorship rule to decide which one wins.

None of these failures is captured by a relevance score alone.

For enterprise RAG and knowledge-graph RAG, evaluation needs to test a harder question:

Did the system use the right knowledge for this user, at this time, for this purpose?

That requires evaluating the knowledge supply chain beneath retrieval.

Why relevance-only evaluation creates false confidence

Most retrieval evaluations begin with a reasonable pattern:

1. Prepare a question.
2. Define one or more relevant passages.
3. Run retrieval.
4. Measure whether those passages appear near the top.
5. Score the generated answer against an expected response.

This is useful, but incomplete.

Imagine an organization has two versions of a travel-expense policy:

• Expense Policy — 2025, approved by Finance and currently effective.
• Expense Policy — 2023, copied into a team wiki and never removed.

Both documents can be semantically relevant to “What is the hotel limit?” A relevance benchmark may reward the system for retrieving either one—or even both.

The enterprise requirement is different. The system should choose the currently authoritative policy, confirm that it applies to the user’s region and role, cite the exact version, and avoid answering if the conflict cannot be resolved.

Relevance is necessary. It is not sufficient.

Evaluate five independent controls

1. Relevance

Does the retrieved evidence actually address the question?

Keep the familiar retrieval metrics, but do not combine them with governance controls too early. A system can be relevant and still fail every control below.

Useful tests include:

• Does the evidence cover the requested topic?
• Is the evidence specific enough to support the answer?
• Does the answer stay within the retrieved evidence?

2. Authority and survivorship

When multiple sources disagree, does the system select the record that should survive?

Authority is not simply “newest timestamp wins.” A survivorship policy may consider:

• system of record;
• content owner;
• approval status;
• applicable region or business unit;
• effective and expiration dates;
• supersession links;
• human verification;
• exception policies.

The evaluation dataset should deliberately include conflicting versions. If every test case contains only one clean answer, the benchmark never exercises the system’s authority logic.

3. Permission validity

Does the system preserve access boundaries from ingestion through retrieval and generation?

Test the same question using identities with different permissions. A correct answer shown to the wrong audience is still a failure.

The evaluation should verify that:

• source permissions survive ingestion;
• retrieval applies permissions at query time;
• citations do not reveal restricted titles or metadata;
• cached answers do not cross identity boundaries;
• generated summaries do not reconstruct restricted facts from mixed evidence.

4. Time validity

Would the answer have been correct at the time of the query?

Freshness is not enough. A document can be recently edited but not yet effective. An older policy may still be the correct answer for a historical question.

Create time-aware test cases with:

• multiple snapshots of the same knowledge object;
• effective-from and effective-to dates;
• scheduled policy changes;
• retroactive corrections;
• questions anchored to a specific date.

Then score whether the system selects the version that was authoritative at the requested time.

5. Abstention and escalation

What happens when the system cannot safely resolve a conflict?

Abstention should be a product behavior, not a generic error.

A useful abstention response should explain:

• which sources conflict;
• which versions were found;
• who owns them;
• which permission scopes apply;
• when each source was last updated;
• why no survivorship rule produced a safe answer;
• where the case has been routed for review.

The evaluation should reward appropriate abstention and penalize confident guesses.

Build adversarial knowledge fixtures

Clean evaluation corpora hide governance failures. A stronger benchmark intentionally creates the conditions found in real enterprise knowledge:

Fixture	Expected behavior
Exact duplicate in two systems	Collapse duplicate evidence without inflating confidence
Near-duplicate with one changed number	Identify the conflict and apply authority rules
Newer draft versus older approved policy	Prefer the approved effective version
Restricted source with a public summary	Return only evidence permitted for the querying identity
Two authoritative owners disagree	Abstain and route to review
Historical question after a policy update	Select the version valid at the requested time
Superseded source still highly ranked	Exclude or clearly demote the superseded version

These fixtures test whether the system understands knowledge state, not just text similarity.

Keep the scorecard separable

Avoid compressing everything into one opaque “RAG quality” score.

Report the dimensions separately:

• relevance;
• authority selection;
• permission compliance;
• temporal validity;
• citation precision;
• abstention precision;
• escalation completeness.

This makes failures actionable. A team can see whether it needs better embeddings, stronger source metadata, permission enforcement, survivorship rules, or a human-review workflow.

It also prevents strong retrieval performance from masking a severe governance failure.

A minimum evaluation workflow

For each high-value enterprise scenario:

1. Identify the systems that contain relevant knowledge.
2. Map source authority, owners, permissions, and version rules.
3. Create clean and conflicting knowledge fixtures.
4. Test multiple user identities and query times.
5. Score relevance and governance controls independently.
6. Verify that every answer cites an exact source version.
7. Confirm that unresolved conflicts trigger abstention and review.
8. Re-run the suite whenever sources, policies, or retrieval logic change.

This turns evaluation from a one-time model test into an operating control for the knowledge supply chain.

The larger lesson

Enterprise AI reliability does not begin at the prompt or the model.

It begins with knowing which source should be trusted, which version applies, who may access it, how it changed, and what the system should do when the evidence remains unresolved.

If your benchmark asks only “Was the retrieved text relevant?”, it may reward a system that finds both the right policy and its outdated copy.

The better question is:

Can the system prove that this was the right knowledge to use?

Assay is exploring this problem as a governed knowledge supply chain for enterprise AI applications and Agents. You can run the free knowledge-readiness check here:

https://assay.20190628.xyz/en/readiness-checklist?leadPrompt=1&utm_source=dev&utm_medium=article&utm_campaign=knowledge-readiness-pilot&utm_content=20260822-kg-rag-authority-evaluation