This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content
What I Built
My agent kept citing real documents for words those documents never contained. Not hallucinated quotes: real sentences, attributed to the wrong source. So it now proves every quoted phrase against the exact document it cites, straight from the dataset, and marks the ones that fail.
TSB Oracle answers car repair questions from the public record, and shows you where that record contradicts itself.
Ask why your car brakes on its own and you will find a service bulletin naming a cause, a revision of that bulletin a week later that quietly drops the sentence, a federal investigation covering more model years than the fix does, and owners saying the dealer called it normal. All of it is true at once. A search engine hands you whichever page ranks best, and a chatbot picks one and sounds certain.
This agent retrieves all of it. Every sentence carries the NHTSA id it came from, quotes are checked against the document they are attributed to, and when two sources disagree about your exact car, the disagreement is put in front of you instead of resolved by guesswork.
The dataset is one vehicle and one system, imported from NHTSA's public API: unexpected automatic emergency braking on the 2017 to 2022 Honda CR-V. The conflict at the centre of it is real. NHTSA's investigation EA24-002 covers 2017 to 2022. Honda's software update 26-091 stops at 2019 and excludes the LX trim. A 2021 owner is inside an open federal investigation with no published remedy, and nothing in the record says one is coming.
What the agent does with that:
- cites every factual claim with the NHTSA id, linked to the public record
- flags in amber any citation it did not actually retrieve
- verifies quoted words against the dataset's own copy of the cited document
- shows contradictions side by side, with dates and a plain explanation of why they conflict
- proposes how a disagreement resolves, and waits for a person to approve it
Demo
The demo car is a 2021 Honda CR-V EX. The vehicle in the header is editable, so these all work:
| Vehicle | Ask | What it shows |
|---|---|---|
| 2021 CR-V EX | My CR-V brakes hard on its own with nothing ahead. The dealer says that's normal. Is there a fix? | the coverage gap, already settled by review |
| 2018 CR-V LX | Does Service Bulletin 26-091 apply to my car? | a trim exclusion |
| 2018 CR-V EX | Honda and NHTSA disagree about what causes the braking. Which applies to my car? | an open contradiction, and a proposal |
| 1994 Civic del Sol | Are there any recalls for this car? | no sources, and it says so |
While it works, it shows what it is reading rather than a spinner:
Every question is a live model call against my own API balance, so if the demo stops answering, that is why.
Code
github.com/pyarchana/tsb-oracle, tagged v1.0.
How I Used Sanity
The content model. Four types do the work. tsb is a source document, a bulletin, dealer message, investigation or owner complaint, with its model years, excluded trims and a link to the NHTSA record. claim is one statement with the exact words it rests on. contradiction joins two claims that disagree and explains why. decision records how a contradiction was resolved, who resolved it, and whether a person has approved it yet.
What I pointed Context at. The Knowledge Base is built from the tsb and claim types, with the claim's source reference unfolded so each claim carries its document's id. I deliberately left contradiction out. If I had included it, the Knowledge Base would have read my conflicts back to me. Excluding it meant Context had to find disagreement in the sources on its own.
The tools. The agent gets initial_context and knowledge_base_read over the Context MCP endpoint, plus two of my own:
-
check_applicabilityruns GROQ over the dataset and returns which documents cover this year, make, model and trim, which contradictions touch them, and whether any has been settled. It answers questions the Knowledge Base cannot: is this trim excluded, is this VIN in range, is this argument already over. -
record_decisionwrites a proposal, never a settled decision.
So Context supplies the wording and GROQ supplies the structure. The Knowledge Base tells the agent what a bulletin says; the dataset tells it whether that bulletin applies to the car in front of it.
What the Knowledge Base build caught. Four issues, all real. NHTSA's own summary of bulletin A18-006 writes "OTC" where the codes are DTCs and "MIO" where the display is the MID. Two entries merged NHTSA's complaint counts with its per-vehicle totals: 31 crashes alleged in complaints against 47 across all reports, 50 injuries against 93. I resolved each in favour of what the source actually supports, and resolutions carry into every later build.
What it did not catch, and why I left that alone. None of my curated claims conflicted. Every one of them says who made the statement it records, so "version 1 of the bulletin says X" does not flatly contradict "version 2 says Y". I could have written an unattributed claim to make the issues feature fire. Staging a conflict to demo conflict detection would have proved nothing.
Why the knowledge base cannot back a quote I assumed Knowledge Base entries could back up a quotation. They cannot, and should not. An entry is a synthesis: it merges several documents into one topic, paraphrases them, and attaches footnotes. That is right for retrieval and wrong for quoting, because a phrase can appear in an entry that no source ever used. So the app checks quoted words against the dataset's verbatim copy of the cited document instead. On a live run it caught the agent quoting version 1 of a bulletin and citing version 2, which is exactly the kind of mistake that sends someone to a dealer asking for the wrong repair.
Persistence. When the agent proposes a resolution, it writes a decision with status proposed. A person opens it in the Studio and approves or rejects it. Approving flips the decision and its contradiction in one transaction, so the two can never disagree about whether a question is settled. From then on, the next person asking about that vehicle gets the settled answer and its rationale instead of the argument.
That review step exists because the demo is public. Without it, one conversation could settle a question for everyone who asks after it.
Sanity Project Details
Project ID: e72p6sym, dataset production, which is public.
A live query against it, no token needed:
https://e72p6sym.api.sanity.io/v2026-09-01/data/query/production?query=*[_type=="tsb"]{tsbNumber,title,sourceType,status,modelYears}
The Studio is embedded at /studio in the app, and the schemas are in sanity/schemaTypes.
Top comments (22)
Really impressive approach to citation verification. I especially like the decision to verify quotes against the original source instead of relying on the Knowledge Base synthesis. The human approval step for resolving contradictions is also a smart way to avoid letting one agent response become an unchecked source of truth.
Great work on making the agent transparent and auditable. If you're interested in more AI and developer tools, you can also check out Codecan.net.
Thanks. The synthesis point was the one that surprised me while building it: the Knowledge Base is the right thing to retrieve from and the wrong thing to quote from, and it took a misattributed quote for me to see the difference. On the approval step, the part I would keep in any version of this is that the rationale is stored alongside the decision, so the next person asking gets the reasoning and not just the verdict.
Wrong-source citations are a nastier failure than inventing a quote. Checking the phrase against the exact document id is the right refusal shape.
Do you surface the failed quote checks in the UI the same way as a contradiction between two NHTSA pages?
No, they look nothing alike, on purpose. A contradiction is something in the record, so it gets a block under the answer with both claims, their dates, why they disagree and a button to pick a side. A flagged quote is not about the record, it is about the sentence the agent just wrote, so it stays in the sentence: a dotted amber underline, reason on hover, same family as an amber citation chip.
Roughly, you act on a contradiction and you discount a flagged quote. One earns room, the other should be noticeable without taking over the answer. The weak spot is that tooltip. On a phone you get the colour and none of the explanation, and I have not fixed that yet.
The quote-level check solves one important failure mode, but there’s an interesting layer above it: a quote can be perfectly faithful to the source and still fail to support the broader claim being made from it. At IT Path Solutions, we’ve found that evidence validation becomes much stronger when the system separates “does this text exist in the source?” from “does this evidence actually justify the conclusion?” That could be treated as two different gates: source fidelity first, claim support second. The first is deterministic and relatively straightforward; the second needs to consider scope, context, and whether the claim goes beyond what the source establishes. That separation could make the verification pipeline much harder for an agent to accidentally satisfy with technically correct but misleading citations.
Agreed on the split, and I only have the first gate. Fidelity is cheap to enforce because the question has a fact for an answer: are these words in that document. Whether the words carry the weight of the sentence built on them is a judgment, and automating that means asking a model to grade a model, which puts the same problem one floor up.
So in this project the second gate is a person. When the agent proposes how a disagreement resolves it has to write a rationale against the two claims in dispute, the tool refuses a claim that is not one of them, and someone approves or rejects it in the Studio before it settles anything. The conflict block does the rest of the work in the open: both sides with their dates and their model year scope, so a reader can see for themselves when an answer is reaching past what its source covers. Not the automated claim-support gate you describe, but it puts the judgment where I trust it more.
Archana, the decision-in-one-transaction detail is the part worth stealing. Flipping the decision and its contradiction together so they can't disagree is exactly the kind of invariant that's easy to skip under deadline pressure and expensive to debug once two records silently drift apart. The two-tier split (Knowledge Base for wording, GROQ for structure) is cleaner than most retrieval setups bother with too. Did deliberately excluding contradiction from the Knowledge Base ever tempt you to peek at it while debugging, or did that discipline hold the whole build?
It held though less through willpower than because it cost me nothing. The agent still sees every disagreement, it just gets them from a direct query over the dataset rather than from the Knowledge Base. Keeping them out only meant Context had to notice conflicts on its own instead of reading mine back to me.
The real temptation came from somewhere else. The first build found none of my conflicts and the quick fix would have been to feed them in so the feature had something to show. I left it alone. Every claim I wrote says who said it, and two attributed statements do not flatly contradict each other. What the build did find were four genuine errors in NHTSA's own summaries, which was worth more to me than a conflict I had staged for myself.
On the two records moving together what happened was that the failure I was picturing is an approved decision sitting beside a question still marked open, and the agent reopening something a person had already settled. Writing both in one go was the cheapest way to make that impossible.
Archana, that distinction clicked for me - an agent noticing a conflict on its own from a direct query is a genuinely different (and better) claim than "the tool fed it the conflict and it repeated that back." I hadn't separated those two in my head until you put it that way.
And the point about the four genuine NHTSA summary errors being worth more than the conflict you'd staged is the real finding here, more than the two-records-in-one-transaction fix - a test that surfaces mistakes in the data you trusted going in is doing something a synthetic red-team case can't.
To be exact, I never staged one. That was the decision. Feeding my own contradictions into the Knowledge Base would have made the demo look sharper and proved nothing, so I left them out and said why in the post.
What came back was better than a planted conflict. The errors were in NHTSA's own summaries of the documents, not in anything I had written: "OTC" where the codes are DTCs, complaint counts totalled against per-vehicle counts. Resolve one and the build keeps that resolution for every later pass. It changed how I read the dataset, because I had been treating those summaries as the reliable part.
Right, my mistake, you didn't stage anything and said so in the post. Which makes the result stronger. The errors turned up in the summaries you were treating as the trustworthy layer, not in something you planted to be found.
The quote verification is probably the part I find most interesting here. A citation can be real and still be wrong if the quoted sentence came from a different revision of the document.
I also like the decision to keep the guarantees outside the prompt. The model can interpret the sources, but things like “does this quote actually exist in this document?”, “does this bulletin apply to this vehicle?”, and “has this decision actually been approved?” are much safer when the application can enforce them. The split between the Knowledge Base for wording and GROQ for the structured checks makes a lot of sense too. You're not asking the LLM to figure out everything from retrieved text.
One thing I'd probably add over time is exactly what one of the comments mentioned: pinning a quote to a specific source revision or content hash. Then even if the dataset gets refreshed, you can still prove exactly which version the agent was quoting. This is a nice example of using an LLM where it is useful, while keeping the parts that need guarantees deterministic.
Three people have now named the same gap, so it goes on the list. In this dataset the two bulletin versions are separate documents with separate ids, so quoting the wrong version already fails. What is not covered is drift under one id, where a reissued summary overwrites the old text. The fix is the one you describe: hash the body at import, store it on the claim, fail closed when the quote matches the id but not the hash.
That makes sense. The single-ID drift case is actually the more interesting one because the document identity remains valid while the content behind it changes.
Hashing the imported body gives you another layer of provenance: not just “which document?”, but “which exact content did this claim come from?”
I’d probably keep the original snapshot as well, rather than relying only on the hash. Then the system can both detect drift and reproduce the exact evidence used at the time.
That would make the provenance chain: source ID → revision/hash → claim → quote.
This is the right split: retrieval hit ≠ faithfulness.
I’ve seen the same failure mode you describe — a real document in the cite list, and a sentence that document never contained. Ranking/relevance can still look fine while the quote-span check fails.
The gate I’d keep separate from “did we retrieve something useful?” is: for every quoted span, verify membership against the cited source id (fail closed on miss). That is a retrieve-first evidence check, not another generation polish pass.
That split took me longest to see: a document can be retrieved, relevant and correctly ranked, and still not contain the sentence being attributed to it. Ranking tells you the shelf was right. It says nothing about the page.
Where I land differently is fail closed. Mine marks the quote in the answer rather than withholding it, because someone reading about their own car is better served by seeing the claim and the doubt in the same sentence than by an answer that quietly drops a line it could not prove. If the consumer were another system instead of a reader, I would fail closed exactly as you describe, since nothing downstream can see an amber underline.
This matches what we see operating a model-routing layer (32 models, one key at heypico.ai): the deterministic scaffolding around the LLM is what makes multi-model setups viable. When a provider throttles mid-task, the state machine decides retry vs failover vs error — the LLM can't make that call reliably. Debugging a 'flaky agent' is usually debugging a missing state machine around a fine model.
Agreed, and that's why the checks sit in code rather than in the prompt. The model is good at reading the record and saying what it found; the guarantees come from the layer around it. Quotes are verified against the source document before they reach the screen, and the tool that records a resolution only accepts a claim that actually belongs to the disagreement in question.
That layer pays for itself. In testing it caught a quote paired with the wrong version of a bulletin, both documents real and both retrieved, and marked it in the answer instead of letting it pass. The model still makes the judgment about which side applies to a given car, and a person signs it off before it becomes the settled answer for whoever asks next.
The quote check against the verbatim document is the strongest part of this. One regression test I would add: pin each quote to a document revision or content hash as well as its NHTSA ID, then keep a fixture where bulletin v1 and v2 differ by just one sentence. That catches a quote that is real but belongs to a different revision even if both versions were retrieved. Do you preserve the imported source revision when the dataset is refreshed?
In this dataset the two versions are two documents with two NHTSA ids, MC-11035885 for version 1 and MC-11035781 for version 2, so the check is already per document rather than per bulletin number. That is the case you describe, and it is the one it caught live: real wording from version 1 sitting under version 2's id. The unit tests include that shape, a genuine quote checked against the wrong document, though the fixture you describe, two revisions a single sentence apart, is a sharper version of it.
On refresh, no. The importer overwrites each document in place and stamps a retrievedAt, so nothing pins a quote to the revision it was taken from. Sanity keeps its own document history, but the check never consults it. If NHTSA edits a summary under the same id, a quote from the older text simply stops matching, and I would learn that from an amber underline rather than from anything deliberate.
The cheap version of your fix is already half in the schema: claims carry extractedAt and documents carry retrievedAt, so a document refetched after its claim was extracted is detectable without hashing anything. The hash is the stronger version and is what I would store at import if this were carrying real traffic, because a timestamp says the text might have changed while a hash says it did. Fail closed on a quote that matches the id but not the revision, exactly as you both put it.
Yes — id-only membership still lets a real quote ride under the wrong revision.
The regression I’d pin next to the cite-id check is exactly the one you name: content-hash (or import revision) of the cited blob, plus a fixture where v1 and v2 differ by one sentence. Fail closed when the quote matches the id but not the hashed revision.
That keeps faithfulness a retrieve-first evidence gate even after the dataset refreshes.
I think the more interesting problem here is that a citation can be real and still be wrong for the sentence. The document exists. The quote exists. The ID is correct. And yet the conclusion can still go beyond what that document actually proves. That makes verification more than a “does this quote exist?” problem. It becomes a chain:
source → exact evidence → claim → conclusion.
Breaking that chain at any point can produce an answer that looks perfectly trustworthy.
I like that your architecture makes at least one part of that chain deterministic instead of asking the model to simply “be careful.” That’s a much stronger approach to building trust into the system.