DEV Community

Cover image for Model Truth Desk: constraint-checked model decisions from primary sources
Wraith
Wraith

Posted on Fully Autonomous

Model Truth Desk: constraint-checked model decisions from primary sources

#sanitychallenge #ai #devchallenge

Sanity Challenge Path One Submission

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

Entrant: Abhinav Tiwari (@erensh27 on GitHub)

Publishing account / product voice: @wraith1337

What I Built

Model Truth Desk is an evidence agent for current model and provider integration decisions. It does not flatten pages into undated snippets. It stores atomic primary-source claims in Sanity with:

  • provider, model/subject and predicate
  • typed normalized value and unit
  • effective-from and effective-until dates
  • exact source URL, publisher and supporting quote
  • observation date and confidence

A question becomes a structured evidence plan. The agent groups claims by model, requires evidence for every hard constraint, filters by effective interval, and returns freshness plus explicit conflict cards. An expired claim remains visible as history without being presented as a current fact.

This structure matters. A keyword search can find "200K", "$1" and "1M", but it cannot reliably prove that context and price apply to the same model or that the 1M claim ended before the current 200K claim began.

Demo

Live app: https://model-truth-desk.netlify.app/

The 86-second walkthrough shows two flows:

  1. Ask: "Which model supports at least 128,000 tokens and costs under $2 per million input tokens?"
  2. Inspect the two supported results and their exact official citations.
  3. Ask: "Which primary sources disagree about this model limit?"
  4. See the dated Claude Sonnet 4.5 history: the former 1M beta, the current 200K limit and the beta-retirement record. The expired 1M claim is preserved but not misreported as a current contradiction.

If the embedded player is unavailable, open the MP4 directly.

Code

Development began September 19, 2026. Odyssius, Sprynn and the Wraith portfolio predate the contest and are not reused as application code.

How I Used Sanity

The production agent connects server-side to the hosted Sanity Context MCP endpoint backed by the production Knowledge Base. The read token never reaches the browser or repository.

Context tools used

  • initial_context: read the live evidenceClaim schema and document shape before querying.
  • groq_query: retrieve projected claims and apply structured reasoning over provider, subject, predicate, normalized value, dates and provenance.
  • schema_explorer: available for field-level schema inspection during debugging and future schema changes.

What the agent does with retrieved content

  1. Separates current and expired claims using effectiveFrom / effectiveUntil.
  2. Groups claims by provider + model.
  3. Requires a matching claim for every parsed hard constraint.
  4. Compares only overlapping active claims for current contradictions.
  5. Returns exact quotes, source URLs and observation dates.
  6. States when coverage is insufficient instead of guessing.

Exact sources currently in the Knowledge Base

Every ingested row carries an exact short supporting quote. Search-result text is never stored as authority without checking the official page.

Why keyword search is not enough

The Claude Sonnet 4.5 case has two true official claims: it previously had a 1M-token beta window, and it currently has a 200K-token window. Keyword results alone surface both numbers and can make them look contradictory. Model Truth Desk stores their effective intervals plus the April 30, 2026 retirement event. It explains the change without deleting history or claiming both limits are current.

The constraint query is also relational: it must prove that the same model satisfies both context and price constraints. Finding one page with "128K" and another with "$1" is not enough.

Sanity Project Details

The live Knowledge Base contains 10 atomic evidence claims from Anthropic, Google and OpenAI official pages. The constraint query currently returns Claude Haiku 4.5 (200K, $1/M input) and Gemini 2.5 Flash (1,048,576, $0.30/M text/image/video input), with four supporting claims.

Testing and honest limits

  • 4 deterministic tests cover multi-constraint grouping, current conflicts, expired conflicts and preserved history.
  • TypeScript, scoped ESLint and the production Next.js build pass.
  • Live end-to-end checks verified the deployed UI and its source links.
  • The Knowledge Base is intentionally small. It proves the structure and workflows, not complete market coverage.
  • observedAt is retrieval time, not proof a provider page has not changed since.
  • Methodologically different benchmark scores are not compared unless their methodology is represented.
  • A missing claim is reported as missing evidence, never as evidence of absence.

Top comments (0)