How Do You Stop an AI Agent from Publishing Claims Your Repo Cannot Support?

Vivek Yadav — Sat, 07 Nov 2026 16:46:50 +0000

Inside Switch Studio: how we combined multi-agent drafting in Switch with a deterministic Python verifier and a human chat gate so every release-note sentence cites a real git fact. #HowISwitch

Watch the 92-second walkthrough: https://youtu.be/4DPnYG44RQw

THE PROBLEM

Our baseline example shows the failure mode in two sentences. demo/baseline_prompt.md is a plain one-shot prompt over our 3-commit repository snapshot, and the baseline output we compare against (demo/baseline.md) reads:

"The Switch Studio verifier landed in 3 commits from 3 contributors, including Saurabh's change to add the claim verifier CLI. That change improved correctness by 35% and runs in under 2 seconds."

The first sentence matches the git history. The second is completely invented. Nothing in the repository mentions 35% or 2 seconds.

When an LLM turns git log into release notes, changelogs, or build-in-public updates, it can blend real commit subjects with plausible-sounding metrics, fabricated short hashes, or inflated file counts.

For the Flip the Switch hackathon, our four-person student team (Ritweek Dubey, Saurabh, Uday Jain, and I) built Switch Studio (https://github.com/Ritweek17/switch-studio) around one rule: no sentence survives into the final post unless it cites a repository fact verified by deterministic code and passes human review in #studio.

WHY WE SPLIT AGENTS, DETERMINISTIC CODE, AND HUMAN REVIEW

Instead of asking a single agent to read git history and publish Markdown, we separated evidence gathering, drafting, deterministic verification, editorial judgment, and shipping across a shared Switch room (#studio, bridged to Slack) and five offline Python 3.11+ CLI tools (tools/):

Local Git Clone / Fixture Snapshot
-> tools.snapshot (studio/snapshot@1)
-> Scout Agent (studio/facts@1: numbered F1..Fn with 40-char commit hashes)
-> Writer Agent (studio/draft@1: numbered S1..Sn with citations in facts[])
-> tools.verify (studio/verdict@1: exact fact + hash + digit token checks)
-> Editor Agent (studio/review@1: copies verifier errors + adds OVERCLAIM)
|-- loop 1 if rejected -> Writer revises rejected sentences only (r1)
+-- loop 2 or all ok -> Handoff to Humans in #studio
-> Human Commands (approve / approve all / reject / edit / ship -> decisions.jsonl)
-> tools.apply + tools.verify + tools.render -> runs//final.md

Two architectural decisions shaped this pipeline:

Structured sentences and out-of-band citations (ADR-0004): Splitting free-form prose into sentences with regex is brittle. In Switch Studio, Writer emits an array of sentence objects (S1..Sn), and citations live only in facts: ["F1", "F2"], never inside text. The renderer (tools.render) appends [F1][F2] at the very end. That means the verifier never guesses sentence boundaries, and [F2] can never be misread as the number 2.
Exact token verification vs. editorial judgment (ADR-0005): Deterministic code cannot prove arbitrary English prose true, and we do not pretend it can. Instead, tools.verify enforces a strict, mechanical boundary:

It recomputes every count fact from a closed list of five metrics (commits, files_changed, lines_added, lines_deleted, contributors) and checks commit and file facts for exact string equality against snapshot.json (FACT_INVALID on mismatch).
For each sentence, after stripping citation tags, S IDs, and cited commit dates, every 7-to-40 character hex token containing a letter a-f must prefix a cited fact's 40-character commit hash (HASH_UNSUPPORTED).
Every remaining standalone number token must equal either a cited count fact's value or a number token in a cited commit subject or path (NUMBER_UNSUPPORTED). Derived percentages (%) always fail.

Because a sentence can pass digit checks and still exaggerate in words, the Editor agent runs tools.verify first, copies its errors verbatim, and adds one judgment code: OVERCLAIM.

HOW WE SET UP SWITCH FOR THE TEAM

Our setup is documented in docs/SETUP.md:

Console and server. We started the Switch Console and worked through its first-run checklist. The Console's server address became the base URL our agent bridge connects to.
Agents. We registered our three agents (Scout, Writer and Editor) on the Codex provider, each with its own instructions, and invited them into the room with !invite-agent @name. New agents default to answering only their owner, so opening them up to the room is an explicit step.
The #studio room. We created a shared #studio room, bridged to our team's Slack workspace through the Switch Agent Bridge, with room instructions describing the workflow. Messages to an agent start with @ followed by the agent's exact name.
The infrastructure bot. Our always-on bot (infra/bot.py) registers with Switch through a POST /agents call using a one-time registration key, then speaks Switch's WebSocket agent protocol for document validation and workflow handoffs. Its connection values live in a .env file.
Runs. Each live run starts from the terminal: demo/run_demo.py --live-only creates a fresh run ID, and the room message then points the agents at that run.

Honest setup notes: two steps cost us real time. We had to create a run ID every time, and the base URL and base API key were tough to find. Once those were in place, everything else (drafting, verification, human approval) happened inside the one shared room.

REAL IMPLEMENTATION CHALLENGES AND TRADEOFFS

Parallel Development Without Third-Party Dependencies

With four students building five modules (M0 through M4) under strict .github/CODEOWNERS boundaries, we chose the Python 3.11+ standard library only (ADR-0002, with pytest as the sole dev dependency) so nobody lost demo time to environment drift. For M0 and M2, I wrote CONTRACT.md, the eight JSON schemas, a 194-line stdlib JSON Schema validator (schemas/validate.py), and the Scout, Writer, and Editor agent modules (agents/). Ritweek built an RFC 6455 WebSocket client from scratch in infra/bot.py using socket, ssl, and struct, while Saurabh built the five CLI tools in tools/ and Uday built the offline fixture sandbox and pytest suites in fixtures/, sandbox/, and tests/.

Because I built agents/editor/agent.py (commit a7cb499) before tools/verify.py merged to main, I initially wrote a local fallback verifier in Editor and golden parity tests (agents/tests/test_verify_parity.py, commit 40988a5) against the contract examples. Once tools/verify.py landed (commit 9bb0bfc), I deleted 198 lines of duplicate verification logic (commit 681f80d) and wired Editor directly to tools.verify, and the tests stayed green.

Fixing Decision Replay Order (ed4c80b)

Human commands in #studio are appended to decisions.jsonl and replayed by tools.apply when ship runs. During integration testing, I found a bug in tools/apply.py: replay_decisions skipped any decision targeting a sentence whose status was no longer "pending". If a human ran approve all and then edited or rejected a sentence before shipping, tools.apply silently ignored the later edit or rejection. In commit ed4c80b, I fixed replay_decisions so accepted decisions apply in append order with last-decision-wins semantics, and added 112 lines of regression tests in tests/test_decisions.py.

WHAT WORKS NOW

All 70 pytest tests and all 8 schema example checks pass. Runtime varies by machine.
On the offline fixture run (demo/run_demo.py --offline-only), tools.grade scores the shipped output at 8/8 supported claims and 5/5 supported sentences (both 100.0%), compared to 3/4 supported claims (75.0%) and 1/2 supported sentences (50.0%), with 1 unsupported number, for demo/baseline.md.
On the recorded live run (demo-live-20261107-111427-4794), tools.grade scores 6/6 supported claims (100.0%) and 3/4 sentences (75.0%), because tools.grade strips [F#] tags to grade plain Markdown equally and only counts sentences containing digit or hash tokens as checkable claims.

LIMITATIONS

Documented in docs/NOT-BUILT-YET.md:

tools.verify checks digit tokens and hex prefixes, not spelled-out number words ("three", "a dozen"), which rely on Editor's OVERCLAIM check.
True derived ratios or percentages ("40% fewer lines") are rejected as NUMBER_UNSUPPORTED because count facts only store raw totals.
The optional --llm OpenAI HTTP helper in agents/_common.py is not exercised in automated CI (offline runs use the deterministic agent logic in agents/, while live #studio runs execute via Codex CLI agents registered in Switch).
tools.snapshot reads local git history (--no-merges) and does not ingest GitHub PR comments or issues.

WHAT'S NEXT

Adding a derived fact kind to studio/facts@1 that references two existing count fact IDs and an explicit operator (ratio or pct_change). That would let tools.verify deterministically recompute comparisons like percentage line reductions without weakening exact-match verification.

LINKS

Repository: https://github.com/Ritweek17/switch-studio
Demo Video (92s walkthrough): https://youtu.be/4DPnYG44RQw

Built for the Flip the Switch hackathon. #HowISwitch

DEV Community: Vivek Yadav

How Do You Stop an AI Agent from Publishing Claims Your Repo Cannot Support?