DEV Community: Garfield Assay

Relevance Is Not Enough: How to Evaluate Enterprise RAG for Authority, Permissions, and Time

Garfield Assay — Sat, 22 Aug 2026 16:12:11 +0000

A retrieval system can return highly relevant context and still produce the wrong enterprise answer.

The retrieved policy may be obsolete. The document may belong to the wrong region. A newer version may exist in another system. The user may not have permission to see the source. Two equally relevant records may conflict, with no survivorship rule to decide which one wins.

None of these failures is captured by a relevance score alone.

For enterprise RAG and knowledge-graph RAG, evaluation needs to test a harder question:

Did the system use the right knowledge for this user, at this time, for this purpose?

That requires evaluating the knowledge supply chain beneath retrieval.

Why relevance-only evaluation creates false confidence

Most retrieval evaluations begin with a reasonable pattern:

Prepare a question.
Define one or more relevant passages.
Run retrieval.
Measure whether those passages appear near the top.
Score the generated answer against an expected response.

This is useful, but incomplete.

Imagine an organization has two versions of a travel-expense policy:

Expense Policy — 2025, approved by Finance and currently effective.
Expense Policy — 2023, copied into a team wiki and never removed.

Both documents can be semantically relevant to “What is the hotel limit?” A relevance benchmark may reward the system for retrieving either one—or even both.

The enterprise requirement is different. The system should choose the currently authoritative policy, confirm that it applies to the user’s region and role, cite the exact version, and avoid answering if the conflict cannot be resolved.

Relevance is necessary. It is not sufficient.

Evaluate five independent controls

1. Relevance

Does the retrieved evidence actually address the question?

Keep the familiar retrieval metrics, but do not combine them with governance controls too early. A system can be relevant and still fail every control below.

Useful tests include:

Does the evidence cover the requested topic?
Is the evidence specific enough to support the answer?
Does the answer stay within the retrieved evidence?

2. Authority and survivorship

When multiple sources disagree, does the system select the record that should survive?

Authority is not simply “newest timestamp wins.” A survivorship policy may consider:

system of record;
content owner;
approval status;
applicable region or business unit;
effective and expiration dates;
supersession links;
human verification;
exception policies.

The evaluation dataset should deliberately include conflicting versions. If every test case contains only one clean answer, the benchmark never exercises the system’s authority logic.

3. Permission validity

Does the system preserve access boundaries from ingestion through retrieval and generation?

Test the same question using identities with different permissions. A correct answer shown to the wrong audience is still a failure.

The evaluation should verify that:

source permissions survive ingestion;
retrieval applies permissions at query time;
citations do not reveal restricted titles or metadata;
cached answers do not cross identity boundaries;
generated summaries do not reconstruct restricted facts from mixed evidence.

4. Time validity

Would the answer have been correct at the time of the query?

Freshness is not enough. A document can be recently edited but not yet effective. An older policy may still be the correct answer for a historical question.

Create time-aware test cases with:

multiple snapshots of the same knowledge object;
effective-from and effective-to dates;
scheduled policy changes;
retroactive corrections;
questions anchored to a specific date.

Then score whether the system selects the version that was authoritative at the requested time.

5. Abstention and escalation

What happens when the system cannot safely resolve a conflict?

Abstention should be a product behavior, not a generic error.

A useful abstention response should explain:

which sources conflict;
which versions were found;
who owns them;
which permission scopes apply;
when each source was last updated;
why no survivorship rule produced a safe answer;
where the case has been routed for review.

The evaluation should reward appropriate abstention and penalize confident guesses.

Build adversarial knowledge fixtures

Clean evaluation corpora hide governance failures. A stronger benchmark intentionally creates the conditions found in real enterprise knowledge:

Fixture	Expected behavior
Exact duplicate in two systems	Collapse duplicate evidence without inflating confidence
Near-duplicate with one changed number	Identify the conflict and apply authority rules
Newer draft versus older approved policy	Prefer the approved effective version
Restricted source with a public summary	Return only evidence permitted for the querying identity
Two authoritative owners disagree	Abstain and route to review
Historical question after a policy update	Select the version valid at the requested time
Superseded source still highly ranked	Exclude or clearly demote the superseded version

These fixtures test whether the system understands knowledge state, not just text similarity.

Keep the scorecard separable

Avoid compressing everything into one opaque “RAG quality” score.

Report the dimensions separately:

relevance;
authority selection;
permission compliance;
temporal validity;
citation precision;
abstention precision;
escalation completeness.

This makes failures actionable. A team can see whether it needs better embeddings, stronger source metadata, permission enforcement, survivorship rules, or a human-review workflow.

It also prevents strong retrieval performance from masking a severe governance failure.

A minimum evaluation workflow

For each high-value enterprise scenario:

Identify the systems that contain relevant knowledge.
Map source authority, owners, permissions, and version rules.
Create clean and conflicting knowledge fixtures.
Test multiple user identities and query times.
Score relevance and governance controls independently.
Verify that every answer cites an exact source version.
Confirm that unresolved conflicts trigger abstention and review.
Re-run the suite whenever sources, policies, or retrieval logic change.

This turns evaluation from a one-time model test into an operating control for the knowledge supply chain.

The larger lesson

Enterprise AI reliability does not begin at the prompt or the model.

It begins with knowing which source should be trusted, which version applies, who may access it, how it changed, and what the system should do when the evidence remains unresolved.

If your benchmark asks only “Was the retrieved text relevant?”, it may reward a system that finds both the right policy and its outdated copy.

The better question is:

Can the system prove that this was the right knowledge to use?

Assay is exploring this problem as a governed knowledge supply chain for enterprise AI applications and Agents. You can run the free knowledge-readiness check here:

https://assay.20190628.xyz/en/readiness-checklist?leadPrompt=1&utm_source=dev&utm_medium=article&utm_campaign=knowledge-readiness-pilot&utm_content=20260822-kg-rag-authority-evaluation

Your Enterprise AI Has a Knowledge Problem, Not Just a Model Problem

Garfield Assay — Fri, 21 Aug 2026 02:37:39 +0000

Teams often diagnose an unreliable enterprise AI system as a model problem.

They change the prompt. They replace the embedding model. They add a reranker. They increase the context window. These changes can help, but they do not repair the knowledge supply chain underneath the system.

If the source material is duplicated, stale, contradictory, permission-blind, or ownerless, a stronger model simply retrieves and explains unreliable knowledge more fluently.

Before building or scaling RAG and enterprise agents, I find it more useful to ask a different question:

Is the organization's knowledge ready to be trusted by AI?

Here is a practical five-control framework for answering it.

1. Source connection

Can the organization identify where relevant knowledge lives and connect those systems without creating another uncontrolled copy?

A readiness review should include documents, databases, wikis, support systems, images, audio, and video. The first milestone is not migrating everything. It is establishing read-only visibility into the sources, their owners, permissions, and update patterns.

Warning signs include exports shared through personal drives, important answers that exist only in chat, and teams that cannot name the system of record.

2. Quality and authority

Can the system distinguish an authoritative policy from a convenient but outdated copy?

Basic retrieval scores are not enough. Enterprise knowledge needs signals such as source authority, freshness, completeness, review status, and intended audience. Those signals should be visible to the people who govern the content and available to downstream retrieval systems.

Without an authority model, relevance can select the wrong version with high confidence.

3. Duplicate and conflict control

Does the organization know when two items are true duplicates, near-duplicates, or conflicting versions?

The hard part is not only finding similar text. A useful control also defines survivorship rules: which version remains authoritative, which evidence supports that decision, and what happens when a source changes later.

For multimodal content, this extends beyond text similarity to stable IDs, media fingerprints, transcripts, timecodes, and rights metadata.

4. Permissions and lineage

Can every answer be traced back to its source, transformation history, and access policy?

An AI system should not flatten enterprise permissions during ingestion. It should preserve the chain from source system to extracted knowledge, deduplication decision, authoritative object, and downstream answer or agent action.

This is both a security control and a debugging tool. When an answer is wrong, teams need to see whether the failure came from the source, preprocessing, ranking, governance decision, or application layer.

5. Human governance

Is there a clear queue for decisions that automation should not make alone?

AI can identify suspicious duplicates, stale content, missing owners, or conflicting claims. It should not silently declare every ambiguous item authoritative. Mature systems route high-impact exceptions to the right human owner, record the decision, and use that feedback to improve future recommendations.

The goal is not to review everything manually. It is to make the important uncertainty visible and assignable.

A simple readiness test

For each control, choose one of four states:

Controlled — the capability is defined, consistently applied, and measurable.
Partial — it exists for some sources or teams but is not dependable across the organization.
Missing — the capability is not yet implemented.
Unknown — the organization cannot currently verify the answer.

"Unknown" should not be treated as neutral. Not knowing where knowledge lives, who owns it, or which version is authoritative is itself a visibility risk.

The output should not be a single vanity score. It should show the weakest control, the evidence gap, and the next governance action that reduces risk before the AI system expands.

What to do before tuning the model again

Pick one high-value workflow and trace its knowledge path end to end:

Which systems supply the answer?
Which item is authoritative when versions disagree?
Which permissions must survive retrieval?
Can the answer be traced to its source and transformations?
Who decides when the evidence is ambiguous?

If these questions are difficult to answer, model tuning is probably not the highest-leverage next step. Start by making the knowledge supply chain visible.

Assay is exploring this problem through a 60-second Enterprise AI Knowledge Readiness Check. It produces a directional view of the five controls and the first governance priority—without claiming that a short assessment replaces a full technical review.

Run the 60-second readiness check

Discussion

When a RAG system gives a confident but wrong answer, which layer do you investigate first: source quality, version conflict, permissions, retrieval, or model behavior?

Your AI has a knowledge problem, not just a model problem

Garfield Assay — Thu, 20 Aug 2026 06:44:05 +0000

Enterprise AI often looks impressive in a controlled demo and becomes unreliable when it meets real company knowledge.

The model may be capable. The retrieval pipeline may work. Yet the source material still contains:

Several versions of the same policy
Documents whose owners have left
Fresh content mixed with outdated instructions
Duplicate files spread across multiple systems
Answers without traceable sources
Restricted knowledge that should not be exposed to every user

When these materials enter a retrieval system unchanged, the AI does not simply inherit the company’s knowledge. It inherits its contradictions.

RAG is necessary, but it is not the whole control system

Chunking, embeddings, hybrid search and reranking can improve retrieval. They do not determine which of two conflicting documents should be trusted.

That requires a governed knowledge supply chain.

Before an enterprise AI application or Agent uses a piece of knowledge, five controls should be clear:

1. Source connection

Where did the knowledge come from?

The connection should preserve source identity, permissions, timestamps and content type—not reduce everything to anonymous text.

2. Quality and authority

Is the material current, complete and owned?

Human verification, source authority, freshness and usage signals can help distinguish an approved policy from an abandoned draft.

3. Duplicate resolution

Are multiple documents repeating or contradicting the same knowledge?

Duplicate detection should go beyond filenames. It should identify semantically similar materials and apply explicit survivorship rules to determine the authoritative version.

4. Permissions and lineage

Who is allowed to use this knowledge, and how did it reach the answer?

Every output should remain traceable from its source through processing, classification and retrieval.

5. Human governance

What happens when the system is uncertain?

Low-confidence matches, conflicting sources and high-risk knowledge need a review queue with clear ownership. Human judgment should be part of the operating model, not an emergency fallback.

The practical question

Before asking, “Which model should we use?”, enterprises should ask:

Is our knowledge ready to be used by AI?

If the answer is unclear, improving the model alone will not create trustworthy outcomes.

We are building Assay to help enterprises diagnose and govern this layer before unreliable knowledge reaches their AI applications and Agents.

Take the 60-second AI Knowledge Readiness Check:

https://assay.20190628.xyz/en?leadPrompt=1&utm_source=devto&utm_medium=organic_community&utm_campaign=knowledge-readiness-pilot&utm_content=knowledge-supply-chain-post