This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.
What I Built
A customer disputes an old fictional card fee. The statement was issued in July, but the charge posted in June. There are older and newer policies, a special-customer rule, and scoped clarifications. Which rule applied when the charge was posted, and when must support stop rather than promise a waiver?
PolicyTrace is an English-first, Chinese-supported support agent. A model routes the question, Sanity Context supplies policy records, deterministic checks handle effective dates, customer conditions, conflicts and authority, and a model confirms the citation set. Missing facts trigger a question. An unresolved conflict or waiver request produces an unsent human handoff. There is no refund or waiver execution tool.
Demo
Open the public evidence viewer and video. Try your own fictional bill in the guest sandbox: enter a fictional question or choose a posting date, customer type and request. It evaluates a frozen synthetic Sanity Context snapshot in the browser; it is not a live MCP or model run. The snapshot and verifier are available for inspection. The viewer replays saved results; it does not invoke a live model. The current 44-second narrated judge-sandbox walkthrough shows the missing-facts question and a fictional dated case using a frozen Sanity Context snapshot; it does not call live MCP or a model. The earlier 56-second local UI walkthrough shows the native Studio-backed August conflict, reviewed clarification and waiver boundary. Both videos are edited demonstrations, not uncut recordings. The separate cloud edit is documented in the raw trace below. The cloud-change trace and native-mode decision trace document the newer path.
The key scenario compares the same disputed premium-customer fee before and after a scoped clarification is reviewed. Before review, both applicable policies remain visible and the case goes to a human. After review, the clarification permits a sourced fee explanation, while a waiver request still goes to a human.
A Real Sanity Edit Changed the Answer
I added native Sanity Studio schemas for policy versions and policy clarifications, with explicit effective dates, customer scope, amounts, source IDs, authority and review fields. Six fictional documents are published in the project's production dataset. A separate Sanity Context MCP dataset endpoint exposes only those synthetic document types. The new Studio-backed option reads them through MCP and maps them into the existing deterministic policy verifier. It does not call a model; the original model-assisted Agent still uses the file-backed Context Knowledge Base.
For a reproducible cloud update check, I queried a June charge before and after a real, revision-guarded edit to the published synthetic policy: 40 → 41 → 40 fictional credits. Each read came through the same MCP-backed decision code; the last edit restored the original content. The cited source fingerprint changed with the edit and returned to its original value after restoration. Inspect the saved three-step evidence. This demonstrates that native Sanity content can change a historical-bill explanation without changing code. It is a controlled synthetic test, not a live banking outcome or an autonomous approval.
The Studio-backed August premium-customer example also preserves three applicable policy IDs and the clarification ID. Without the reviewed clarification it hands off; with it, the system can explain the fictional 23-credit fee, while a waiver still requires a human. The server validates a reviewed content hash before applying any clarification. Missing fields, incomplete retrieval, or changed approval content stop the answer.
Code
Download the current v6 source ZIP. The earlier v5 package passed 55 existing offline tests plus four native-adapter contract tests after independent extraction; the v6 archive adds the user-challenge fix and three focused regression tests. Its ZIP integrity and limited credential-pattern checks passed. It includes the FastAPI app, bilingual UI, original Agent path, native Studio schemas/adapter, synthetic fixtures, comparison scripts and saved traces. Live local operation requires the evaluator's own Sanity and model credentials; none are distributed.
How I Used Sanity
The original evaluated path used Sanity Context MCP to read structured records in a file-backed Knowledge Base: policy versions and clarifications. Effective-date ranges, customer conditions, amounts, source IDs, revisions and authority information affect the decision. An old posting date excludes a later policy; conflicting premium-customer rules trigger a handoff; a reviewed, scoped clarification changes what can be explained but never grants waiver authority.
The new native Studio path models those records as Sanity document types and reads the published documents via a separate Context MCP dataset endpoint. This is why the 40 → 41 → 40 edit changes the decision while code stays fixed. The Studio path is deterministic, so I do not claim that the model-assisted Agent itself was rerun against the edited native dataset. The earlier seven model-assisted cases and ten-case comparison use the file-backed snapshot.
A server-side reviewed content hash controls clarification approval; a content entry cannot approve itself by naming a reviewer. Source failures, unknown conditions and mismatched model citations fail closed. Customer facts are synthetic input, not identity verification.
Sanity Project Details
- Project ID:
7d711ssk - Native Studio: PolicyTrace Policies
- Native dataset:
production; Context MCP endpoint:policytrace-structured - Original file-backed Context Knowledge Base ID:
kbuLoYcmxK7A - Products, bills, customers and policies are fictional.
Evaluation and Limits
An actual Qwen3-8B → Sanity Context → policy checks → model citation-confirmation run passed seven developer-authored cases after a citation-prompt correction. The earlier first-case citation failure is retained. In ten predeclared fictional cases using one Sanity snapshot and the same Qwen3-8B model, PolicyTrace met the recorded decision, amount/null, valid nonduplicated citation and no-action checks in 10/10; a full-context direct-model baseline met them in 1/10 and a simple keyword Top-3 baseline in 0/10. Frozen cases and raw results are in the package.
I also tested 15 declared synthetic boundary cases against one actual Sanity Context MCP snapshot, covering effective-date edges, missing customer evidence and waiver handoff; three separate FastAPI calls each made a fresh MCP read. The first boundary run was 13/14 because my expected answer for an unverified standard customer was wrong: the policy correctly requested evidence. I retained that initial failed expectation, corrected the test and added the verified counterpart; the rerun was 15/15. Reproduction scripts are public. These tests use the deterministic Studio path, not the model-assisted Agent.
A user then posed a new fictional two-part question in Chinese: “Why does my card have an annual fee, and how can it be waived?” without a posting date, customer type or supporting record. The first run asked only for the date and missed the waiver intent. I preserved it, fixed the deterministic intake, and reran the unchanged question through the Studio API with a fresh MCP read. It now requests the missing facts and says waiver decisions require a human; no fee amount or approval is invented. Try the frozen-snapshot guest sandbox or inspect the scoring protocol. This is one user-authored case, not a representative benchmark.
The 15 boundary cases and ten-case comparison were written by the developer, not independently sampled or blinded. One later user-authored question is reported separately. The keyword baseline is simple, not a strong RAG implementation. The system has task-specific date and authority checks, so the methods have different capabilities. This experiment cannot establish production accuracy, a statistical win rate or superiority over other entries. No real bank connection or measured business return exists.
Built with AI-assisted development and testing. The author reviewed the design, code, results and limitations. No Agent Session transcript is submitted.
Top comments (0)