Build a Tiny Citation Gate Before Trusting RAG Answers

#ai #rag #python #beginners

A RAG answer can cite a real document and still make an unsupported claim. Retrieval answers “which text was nearby?” Citation verification asks a different question: “does the cited text contain enough evidence for this sentence?”

Start with a deterministic gate before adding another model.

from dataclasses import dataclass

@dataclass
class Claim:
    text: str
    citation_ids: list[str]
    required_terms: set[str]

def verify(claim: Claim, sources: dict[str, str]) -> dict:
    missing = [cid for cid in claim.citation_ids if cid not in sources]
    evidence = " ".join(sources.get(cid, "") for cid in claim.citation_ids).lower()
    absent = sorted(term for term in claim.required_terms if term.lower() not in evidence)
    return {
        "claim": claim.text,
        "valid_source_ids": not missing,
        "missing_sources": missing,
        "missing_terms": absent,
        "supported": not missing and not absent,
    }

sources = {
    "doc-1": "The service retains audit logs for 30 days.",
    "doc-2": "Enterprise plans can export logs as JSON."
}

claim = Claim(
    "All plans retain exportable audit logs for 90 days.",
    ["doc-1", "doc-2"],
    {"all plans", "90 days"},
)

print(verify(claim, sources))

Expected result:

... 'missing_terms': ['90 days', 'all plans'], 'supported': False}

This is intentionally not semantic reasoning. It catches missing references and obvious term mismatches while remaining easy to inspect. That makes it a useful first test and a poor final verifier.

Extend it carefully

Split an answer into atomic claims. One citation at the end of a paragraph is ambiguous.
Store source ID, URL, revision, and retrieved span together.
Reject citations to spans the retriever did not actually return.
Add numeric normalization so 30 and thirty can be compared.
Only then test an entailment model, using a labeled dataset with supported, contradicted, and insufficient-evidence examples.

Measure precision and recall separately. A gate that blocks every answer has perfect recall for bad answers and no product value. A gate that approves everything is fast but meaningless.

The public MonkeyCode repository describes AI task and project-requirement workflows. Citation gates can be useful anywhere an assistant summarizes repository requirements or task evidence, but this exercise does not test MonkeyCode or describe its implementation.