Why can an AI answer cite a source it never saw?
A team's AI assistant answers a question with a citation that looks legitimate: eval-note-7. The retrieval step for that request supplied eval-note-1 and eval-note-2. If the application simply renders the citation, a reader can click what appears to be evidence that was never part of the answer's context.
The first fix is mechanical: compare the structured citation IDs in the AI response with the source IDs in the retrieval snapshot for the same request. Reject unknown IDs before returning the answer. This catches one failure; it does not tell you whether an allowed document actually supports the words beside its citation.
AI disclosure: This post was drafted with AI. The author must review the code and claims, then select the DEV disclosure tier that matches the final published article.
What exactly is the contract?
The retriever supplies a list of sources with stable IDs for one request. The generator returns an answer with status, text, and a list of citations using those IDs. In this example, every answered response needs at least one citation. An unknown response may have no citations because it is explicitly saying the supplied material cannot answer the question.
This is a contract for a particular application, not a universal format for model output. Your production adapter must capture the retrieval snapshot and parse the generator's structured output. Do not scrape source-like strings out of free-form prose and call them verified citations.
Save a small retrieval fixture as retrieved.json:
{
"sources": [
{"id": "eval-note-1", "text": "The evaluation covered routine prompts."},
{"id": "eval-note-2", "text": "Unusual prompts were not tested."}
]
}
Save an intentionally bad output as answer.json:
{
"status": "answered",
"text": "The evaluation covered routine prompts only.",
"citations": ["eval-note-1", "eval-note-7"]
}
The sentence could be reasonable. The second citation is still impossible to justify from this retrieval snapshot because eval-note-7 is absent.
Implement the source-ID check
Save this as check_citations.py. It reads trusted local files, rejects oversized inputs, checks schema shape, and exits with a failure code on invalid citations. It never sends documents or prompts to a network service.
import json
import sys
from pathlib import Path
MAX_BYTES = 1_000_000
def load_json(path):
if path.stat().st_size > MAX_BYTES:
raise ValueError(f"{path}: file exceeds {MAX_BYTES} bytes")
return json.loads(path.read_text(encoding="utf-8"))
def validate(retrieval, response):
errors = []
if not isinstance(retrieval, dict) or not isinstance(response, dict):
return ["both inputs must be JSON objects"]
sources = retrieval.get("sources")
if not isinstance(sources, list) or not sources:
return ["sources must be a nonempty list"]
ids = []
for source in sources:
if not isinstance(source, dict):
return ["each source must be an object"]
source_id = source.get("id")
if not isinstance(source_id, str) or not source_id:
return ["each source needs a nonempty string id"]
ids.append(source_id)
if len(ids) != len(set(ids)):
errors.append("retrieved source IDs must be unique")
status = response.get("status")
if status not in ("answered", "unknown"):
errors.append("status must be answered or unknown")
if not isinstance(response.get("text"), str) or not response["text"].strip():
errors.append("text must be a nonempty string")
citations = response.get("citations")
if not isinstance(citations, list):
return errors + ["citations must be a list"]
if any(not isinstance(c, str) or not c for c in citations):
return errors + ["citation IDs must be nonempty strings"]
if len(citations) != len(set(citations)):
errors.append("citation IDs must be unique")
unknown = sorted(set(citations) - set(ids))
if unknown:
errors.append(f"IDs not retrieved: {unknown}")
if status == "answered" and not citations:
errors.append("answered output needs a citation")
return errors
def main():
if len(sys.argv) != 3:
raise SystemExit("Usage: python3 check_citations.py retrieved.json answer.json")
try:
errors = validate(load_json(Path(sys.argv[1])),
load_json(Path(sys.argv[2])))
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
print(f"INPUT ERROR: {exc}")
return 2
if errors:
for error in errors:
print(f"FAIL: {error}")
return 1
print("PASS: citation IDs are present in this retrieval snapshot")
return 0
if name == "main":
raise SystemExit(main())
Run python3 check_citations.py retrieved.json answer.json. The bad fixture prints FAIL: IDs not retrieved: ['eval-note-7'] and exits 1. Replace the citations array with ["eval-note-1"] and it prints PASS: citation IDs are present in this retrieval snapshot with exit 0. Remove all citations while keeping status as answered and it fails because the contract requires a citation.
What does a passing result not prove?
A retrieved document can be cited for the wrong sentence. A document can be outdated, adversarial, or misread. A source ID can exist while its text says the opposite of the generated claim. This program cannot compare natural-language claims with passages, evaluate source authority, detect prompt injection, or determine whether the answer should have abstained.
In this Ranknod editorial example, PASS means only that the citation ID appears in the retrieved set; claim-level support still needs review.
That boundary is the point of the tool. A known structural failure can be caught cheaply and consistently. Factual support needs a separate review that compares each consequential claim with the exact source passage, plus tests of behavior on relevant and irrelevant retrieval sets. NIST's AI RMF calls for defined evaluation and human oversight in context; a source-ID check is only one small layer.
Keep the retrieval snapshot and model output together for a permissioned test run. If the output fails, do not render the apparently supported answer and hope the reader notices. Return a controlled error or an approved fallback, then inspect why the generator produced an unknown identifier.
Top comments (0)