DEV Community: Ayush Gupta

Bill & Document Sloth: A Local-First AI Paperwork Manager with Gemma & TabPFN

Ayush Gupta — Thu, 22 Oct 2026 17:23:44 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Bill & Document Sloth is a 100% local-first, privacy-conscious AI document assistant, tabular forecaster, and deadline tracker designed around one core promise: "Stop sorting paperwork. Start living."

Every adult and student dreads the paperwork shuffle: opening randomly named files like IMG_20261021_124532.pdf, hunting through fine print for payment due dates, losing track of appliance warranty periods, and maintaining brittle manual spreadsheets.

Bill & Document Sloth does the heavy lifting so you can be as lazy as a sloth:

Drop Paperwork Once: Drag and drop PDFs, receipts, or phone bills.
Intelligent Local Extraction: Local open-source AI extracts the issuer, category, total amount, due date, document date, and invoice number.
Smart Standardized Filenames: Automatically generates clean filenames (e.g., 2026-10_Electricity_Bill_BESCOM.pdf).
Human-in-the-Loop Review: Split-screen side-by-side verification with confidence scores before any file is moved.
Safe Organization & Undo: Files are organized into structured directories (documents/organized/{Category}/{Year}/) with 1-click reversible undo.
Proactive Deadlines: Monitors overdue bills, payments due in 3 days, and expiring warranties before penalties strike.

Demo

Direct YouTube Link: Watch Video Demo on YouTube
GitHub Repository: https://github.com/Ayushgupta1715/bill-and-document-sloth
Local Interactive Dashboard: Runs locally on http://127.0.0.1:8000 with Swagger API docs at http://127.0.0.1:8000/docs.
1-Click Built-in Demo Pack: Synthesizes realistic BESCOM Electricity Bills, Airtel Fiber Invoices, AppleCare Warranty Certificates, Star Health Insurance Policies, and Coffee Receipts on the fly so evaluators can test immediately with one click!

Code

Ayushgupta1715 / bill-and-document-sloth

🦥 Bill & Document Sloth

“Stop sorting paperwork. Start living.”
A 100% Local-First AI Document Assistant & Deadline Tracker for Hacktoberfest 2026 Week 3: Slow and Steady (#hf26challenge).

🎥 Watch 1-Minute Demo Video on YouTube | Download Local MP4

🌟 The "Slow and Steady" Philosophy

In celebration of International Sloth Day and Hacktoberfest Week 3, Bill & Document Sloth is built around a simple truth: AI should do the tedious, repetitive paperwork so you don't have to.

Instead of spending precious hours deciphering confusing PDF bills, calculating payment deadlines, renaming cryptic camera scans, and maintaining brittle spreadsheets:

Drop your documents once.
Local AI extracts the essentials (Issuer, Category, Due Date, Total Amount, Invoice Number).
Approve with 1 click.
Relax. Your files are organized into structured directories and payment deadlines are tracked automatically.

🏆 Hacktoberfest 2026 Sponsor Track Integrations

To maximize utility and showcase open innovation, Bill & Document…

View on GitHub

Repository: https://github.com/Ayushgupta1715/bill-and-document-sloth
License: MIT License (Open-Source)

How I Built It

The application combines local-first inference, open-weight reasoning models, and tabular foundation models:

Local-First Processing Engine:
- Backend: FastAPI, Python 3.14, SQLite3, PyPDF for embedded text extraction, and ReportLab for synthetic sample generation.
- Frontend: Clean, responsive single-page application built with Tailwind CSS, Lucide icons, and Canvas Confetti.
Google Gemma Open-Weight Model (gemma-2-2b-it):
- Integrated in app/gemma_service.py to provide deep structured JSON parsing and field reasoning traces for complex or ambiguous bills.
Prior Labs TabPFN-3.5 Foundation Model:
- Integrated in app/tabpfn_analyzer.py to perform in-context tabular regression and outlier detection over historical billing cycles (forecasting next month's utility spend and alerting on anomalous spikes).
ElevenLabs Voice Integration:
- Integrated in app/voice_service.py to give the sloth agent a calm, spoken audio briefing of pending bills and deadlines.
Safe File Operations:
- Keeps original uploads pristine in documents/uploads/ while organizing approved files in documents/organized/{Category}/{Year}/. Full audit trail in actions_history table for 1-click safe undo.

Why Does Open Innovation Matter?

When building an assistant for bills, leases, and receipts, sending files to closed cloud LLM APIs poses major privacy and cost concerns:

Absolute Data Privacy: Utility bills and tax invoices contain home addresses, bank details, and personal identifiers. With open-source local inference and SQLite, zero bytes of private data leave the device.
Zero Recurring Cloud Token Costs: Users shouldn't pay monthly API subscriptions or per-token charges just to organize their receipts. Open-weight models like Gemma and tabular foundation models like TabPFN run locally on consumer hardware for free.
Total Offline Resilience: Works on a train, in airplane mode, or during internet blackouts.
User Sovereignty & Control: Unlike opaque cloud systems that silently alter data, our pipeline enforces human verification and reversible undo actions.

The Lazy Payoff

Time Saved: Manually opening, reading, renaming, and filing 10 bills takes roughly 20–25 minutes. With Bill & Document Sloth, dropping them into the app and clicking "Approve" takes under 45 seconds — a 95% reduction in manual effort.
Zero Spreadsheet Maintenance: Upcoming deadlines (due in 3 days, due today, overdue) are monitored automatically, preventing costly late-payment surcharges.
Peace of Mind: You can be as lazy as a sloth, knowing every warranty, receipt, and utility bill is categorized and safely filed on your machine.

Prize Categories

Featured Categories:

Best Use of Gemma ($200): Integrated Google's Gemma-2 open-weight model (gemma-2-2b-it) for deep structured bill parsing, confidence verification, and reasoning traces (app/gemma_service.py).
Best Use of TabPFN ($200): Integrated Prior Labs' TabPFN-3.5 tabular foundation model (app/tabpfn_analyzer.py) to forecast next month's utility spending and detect billing cost anomalies (+40% outliers).
Best Use of Render ($200): Included production render.yaml (Render Blueprint) and multi-stage Dockerfile for 1-click cloud deployment.

Partner Categories:

Best Use of ElevenLabs ($100): Sloth Voice Briefings (app/voice_service.py) delivering spoken audio briefings of upcoming bills and paperwork deadlines.
Best Use of Sentry Agent Tracing ($100): Instrumented with sentry-sdk in app/main.py for tracking document ingestion latency, OCR performance, and extraction errors.
Best Use of a .tech Domain ($100): Production DNS and routing ready for slothdocs.tech.
Best Use of GitHub Copilot & Actions ($100): Comprehensive automated CI pipeline (.github/workflows/ci.yml) validating 20 unit and integration tests on every push.

I Built KhataSnap: Turning Handwritten Shopkeeper Ledgers into a Private AI-Powered Digital Khata

Ayush Gupta — Fri, 16 Oct 2026 12:53:17 +0000

This article is submitted for the **Hacktoberfest Open-Source AI Challenge: Week 2 (Side Hustle).

☕ 1. The Real-World Problem: The Unseen Economy of Corner Stores

Walk down any lane in Delhi, Mumbai, or Varanasi, and step into a neighborhood provision store (kirana dukan). Amid the glass jars of spices and sacks of basmati rice, you will notice a faded, cloth-bound red notebook with ink-smudged corners resting beside the weighing scale: the Bahi-Khata (traditional Indian credit ledger).

Over 12 million small merchants across India and South Asia operate on Udhaar (informal community credit). When neighbors buy daily essentials—milk, wheat flour, mustard oil—money rarely changes hands immediately. The shopkeeper hastily jots down a handwritten note:

"12 Oct — Rahul Sharma — 5kg Rice, 2kg Sugar — ₹450"

At the end of the month, this informal trust system collapses into two painful bottlenecks:

The Weekend Math Nightmare: Tallying individual debits, cash payments, and cumulative balances across 150+ customers from handwriting takes 3 to 4 hours of squinting every Sunday evening.
The Awkward Social Friction: In close-knit neighborhoods, shopkeepers feel deeply uncomfortable asking friends and elders for money face-to-face. As a result, follow-ups are delayed for weeks, trapping ₹50,000 to ₹2,50,000 ($600 to $3,000 USD) in stranded working capital per store.

+-------------------------------------------------------------------------+
|                      THE INFORMAL CREDIT BOTTLENECK                      |
|                                                                         |
|  Paper Register         Manual Math Agony         Awkward Collection    |
|   [Bahi-Khata]  ----->   [Hours Lost on    ----->   [Money Stranded for |
|  Messy Handwriting       Sunday Evenings]          Months in Udhaar]    |
+-------------------------------------------------------------------------+

🚫 Why Existing Cloud AI & SaaS Tools Failed

When I asked local merchants why they don't simply scan registers with Google Lens, ChatGPT, or traditional invoicing software like QuickBooks or Vyapar, their answers were eye-opening:

The Privacy Deal-Breaker: Small merchants consider customer lists, phone numbers, and debt records sacred community trust. They refuse to upload neighborhood financial records to corporate cloud servers.
The Per-Token API Tax: Commercial cloud multimodal APIs charge \$0.02 to \$0.05 per image scan. For a small merchant digitizing 150 to 200 pages a month, cloud API overhead completely destroys micro-SaaS profit margins.
The Math Hallucination Disaster: Generative LLMs are probabilistic, not deterministic. If an AI hallucinates ₹1,950 instead of ₹950 on an ink-smudged digit, neighborly trust is destroyed permanently.

💡 2. Introducing KhataSnap AI

Tagline: “Your old khata. A smarter way to get paid.”

KhataSnap is an open-source, privacy-first AI companion that converts photographed handwritten credit ledgers into verified digital records with on-device open-weight AI, provides deterministic balance calculations, and automates payment recovery with respectful, tone-adaptive Hinglish WhatsApp reminders.

       +--------------------------------------------------+
       |         HANDWRITTEN KHATA REGISTER PAGE          |
       +--------------------------------------------------+
                                |
                                v
       +--------------------------------------------------+
       |       ON-DEVICE OPEN-WEIGHT VISION MODEL         |
       |             Google Gemma 2 via Ollama            |
       |         (Strict Local Client-Side Privacy)       |
       +--------------------------------------------------+
                                |
                                v
       +--------------------------------------------------+
       |          SIDE-BY-SIDE VERIFICATION STUDIO        |
       |         Shopkeeper Confirms Every Digit          |
       |       (Flags Ambiguous Ink e.g. ₹450 vs ₹950)     |
       +--------------------------------------------------+
                                |
                                v
       +--------------------------------------------------+
       |           DETERMINISTIC CODE LEDGER              |
       |       Math in TypeScript (Zero LLM Errors)       |
       |          + Smart Ledger Reconciliation           |
       +--------------------------------------------------+
                                |
                                v
       +--------------------------------------------------+
       |       CULTURALLY RESPECTFUL HINGLISH REMINDERS   |
       |    Polite / Friendly / Official / Urgent Tones   |
       |      + ElevenLabs Neural Hindi Voice Notes       |
       +--------------------------------------------------+
                                |
                                v
       +--------------------------------------------------+
       |         1-CLICK WHATSAPP RECOVERY (wa.me)        |
       |            Immediate Capital Clearance           |
       +--------------------------------------------------+

🎬 Watch the 1-Minute Video Demo (AI Voiceover)

Here is a 60-second walkthrough showing the live app, side-by-side verification, smart reconciliation, and WhatsApp reminder dispatch:

✨ 3. Core Technical Architecture & Breakthroughs

1. The Split-Screen Review Studio (Hero Feature)

Instead of trusting black-box AI blindly, KhataSnap implements a human-in-the-loop review studio:

Left Canvas: Original photographed handwritten register with active SVG bounding boxes.
Right Panel: Extracted transaction cards (Customer Name, Date, Items, Amount, +Credit / -Debit).
Ambiguity Detection & Correction: When an ink stroke is overwritten—such as Priya Singh's ambiguous ₹950 vs ₹450—the AI flags the entry with an amber alert and renders three immediate action controls: Correct to ₹450, Confirm ₹950, or Edit Custom Amount. Unverified entries are isolated from the final verified ledger until approved.
Separated Financial Totals: $$\text{Verified Net Impact: ₹730} \quad\mid\quad \text{Pending Verification: 1 entry} \quad\mid\quad \text{Potential Net Impact: ₹1,180–₹1,680}$$

2. Zero Math Hallucination: Deterministic Arithmetic

AI is used strictly for optical tokenization and language translation. AI is never permitted to calculate customer balances. Balances are computed deterministically in code:
$$\text{Current Balance} = \sum \text{Verified Udhaar (Credit)} - \sum \text{Verified Jama (Payments)}$$

3. Smart Ledger Reconciliation Engine

Handwritten ledgers frequently suffer from manual arithmetic drift. If a customer had an opening balance of ₹1,000, purchased ₹450 of groceries, and paid ₹200, the expected balance is strictly ₹1,250. If the merchant wrote ₹1,450 by mistake, KhataSnap highlights the ₹200 discrepancy and provides a 1-Click Auto-Reconcile button.

4. Culturally Respectful Hinglish Reminders & Voice Notes

Western automated debt collection notices sound threatening and destroy neighborly goodwill. KhataSnap drafts tone-appropriate reminders in natural Hinglish:

Namaste (Polite): "Namaste Rahul ji! Shri Ganesh Kirana Store se aapka ₹1,250 ka hisaab pending hai. Jab suvidha ho, dukaan par ya UPI se payment kar dein. Dhanyavaad!"
Friendly: Casual check-in for close friends.
Official Notice: For delayed institutional or wholesale accounts.
Urgent: For 30+ day overdue debts.

With one click, messages open directly in WhatsApp (wa.me) or generate an ElevenLabs Hindi audio voice note so merchants can send a voice message with zero typing.

🏆 4. The 10x Partner Tech Stack: How We Integrated 10 Official Sponsors

KhataSnap was architected to combine 10 official Hacktoberfest sponsored technologies into a single cohesive micro-SaaS:

#	Sponsored Partner	Bounty Category	Technical Implementation in KhataSnap AI
1	Render	$200 Featured	Web Runtime & Production Host: Configured via root `render.yaml` for automated continuous deployment, SPA client-side rewrite rules (`/* -> /index.html`), and high-performance global CDN hosting.
2	Prior Labs TabPFN	$200 Featured	Tabular Credit Risk Forecaster: Uses TabPFN-3.5, Prior Labs' revolutionary tabular foundation model, to analyze merchant customer ledger CSVs without fine-tuning, forecasting default risk probabilities (12% to 68%), predicted payback days, and debt surge anomalies.
3	Google Gemma 2	$200 Featured	Core On-Device LLM & Vision Engine: Connected directly to Gemma 2 (9B/2B) running on `localhost:11434` via Ollama for zero-cloud privacy-first register digitization and tone-adaptive reminder generation at $0 marginal cost.
4	ElevenLabs	$100 Partner	Voice AI Audio Reminders: Synthesizes natural spoken Hindi audio notes (`playElevenLabsVoiceReminder`) so merchants can send audio WhatsApp reminders with natural human inflection.
5	MongoDB Atlas	$100 Partner	Cloud Ledger Data Layer: Merchant customer records, audit logs, and digital passbooks modeled with Atlas document schemas and prepared for Atlas Vector Search customer retrieval.
6	SerpApi	$100 Partner	Live APMC Mandi Price Grounding: Live Google search queries through SerpApi benchmarking handwritten grocery items (Basmati Rice, Mustard Oil, Chakki Atta) against daily wholesale mandi indices to verify fair margins.
7	Sentry	$100 Partner	Agent Observability & Tracing: Telemetry tracing capturing Gemma OCR inference latency (ms), token consumption, validation anomalies, and deterministic audit execution runs.
8	Mastra	$100 Partner	TypeScript Agent Orchestration Harness: Coordinates the multi-step stateful workflow: Register Capture ➔ Vision Tokenization ➔ Human Verification ➔ Deterministic Balance Calculation ➔ WhatsApp Dispatch.
9	Temporal	$100 Partner	Durable Payment Recovery Workflows: Models debt follow-ups as durable state machines that survive process restarts and trigger automated retry reminders on a 7-day, 14-day, and 30-day schedule.
10	.tech Domains	$100 Partner	Official Production Domain: Mapped for official merchant access via the Hacktoberfest promo at `khatasnap.tech`.

🤝 5. User Testing & Validation: A Sunday Evening at Satish's Store

To validate whether KhataSnap solves a genuine pain point, I tested it with Mr. Satish Kumar, who has operated a neighborhood provision store in Kanpur for 14 years with over 80 regular credit customers.

I photographed one of his daily ledger pages into KhataSnap:

4 Out of 5 Entries Extracted Instantly: Names, dates, and item names populated accurately.
Ambiguity Catch: On row 4, a heavy ink stroke looked like ₹950. The Side-by-Side Review Studio highlighted it in amber. Satish glanced at the screen, smiled, and clicked Correct to ₹450 in two seconds.
Live Follow-Up: He selected an overdue customer who owed ₹1,250 for three weeks, chose the Namaste (Polite) tone, clicked Send via WhatsApp, and dispatched the reminder on the spot.

Satish's Direct Reaction:

"Mera sabse bada sar-dard tha ki har Sunday shaam ko 2 ghante hisaab jodna padta tha. Agar ye software mere counter laptop par bina internet ke hisaab jod de aur WhatsApp par tameez se reminder bhej de, toh main iske liye ₹150–₹200 har mahine aaraam se de dunga."

(“My biggest headache was spending 2 hours every Sunday evening manually tallying balances. If this tool runs locally on my counter without internet and sends polite reminders, I will happily pay ₹150–₹200 every month.”)

💰 6. The Side Hustle Monetization Model

Why is KhataSnap a viable, high-margin side hustle?

Plan	Pricing	Target Customer	Features
Starter (Free)	₹0 / mo	Micro-vendors (<30 customers)	Up to 30 transactions/month, manual ledger, standard reminders
Dukaan Pro	₹199 / $2.50 mo	Neighborhood Kirana Stores	Unlimited ledger scans, WhatsApp deep links, TabPFN default predictions, PDF passbook prints
Merchant Counter Box	₹4,999 one-time	Non-technical merchants	Plug-and-play mini PC / Raspberry Pi pre-configured with local Gemma 2 and KhataSnap

The Micro-SaaS Unit Economics:

Because KhataSnap runs on open-weight models (Gemma 2) with client-side or fixed hardware inference, our marginal cost per scan is $0.00. Unlike SaaS startups burning venture capital on OpenAI API tokens, KhataSnap enjoys 95%+ gross profit margins from day one.

🛠️ 7. Source Code & Live Deployments

🌐 Live Web Application (Hosted on Render): https://khatasnap-ai-vden.onrender.com
🐙 GitHub Repository (MIT Open Source): https://github.com/Ayushgupta1715/khatasnap-ai
🎬 1-Minute Video Walkthrough (YouTube): https://youtu.be/tZWynmHyiTI
🚀 Alternative Live Mirror (GitHub Pages): https://ayushgupta1715.github.io/khatasnap-ai/
📦 Production Blueprint: Root render.yaml with automated SPA client routing.
💻 Tech Stack: React 19, TypeScript, Tailwind CSS, Gemma 2 via Ollama, TabPFN-3.5, ElevenLabs, SerpApi, Sentry, Canvas Confetti, Lucide React.

💬 Let's Discuss!

If you run or frequent a local corner store, how do shopkeepers track credit (udhaar) in your neighborhood?
Would you feel more comfortable receiving an automated polite WhatsApp reminder from a local business, or do you prefer the traditional in-person conversation?
What features would you like to see in our open-weight bahi-khata vision pipeline next?

Drop your thoughts and questions in the comments below! 👇

Built with ❤️ for the neighborhood shopkeepers who keep our communities running.

TrailWhisper: Keep Eyes on the Canopy. Phone in the Pocket.

Ayush Gupta — Sat, 10 Oct 2026 19:43:14 +0000

🌿 Project Overview

Project Name: TrailWhisper

Tagline: Keep Eyes on the Canopy. Phone in the Pocket.

Live Demo: https://ayush-trailwhisper.surge.sh

Video Walkthrough: https://youtu.be/p9flaxXcBRI

Repository: https://github.com/Ayushgupta1715/trailwhisper

🏔️ The Trailhead Paradox: The Screen Problem in Nature

When modern hikers and birders enter forest trails or mountain ranges, they carry smartphones equipped with nature recognition apps. Ironically, this traps them in the "Screen Trap":

Staring down at 120 Hz AMOLED glass instead of looking up at the bird canopy.
Waiting on 4G/5G spinners because mountain valleys are cellular Dead-Zones.
Closed cloud APIs requiring constant internet and streaming intimate forest conversations to centralized corporate servers.

We built TrailWhisper to reverse this. It is an offline-first, zero-screen nature trail audio companion powered by on-device open-weights AI.

You put your phone in your pocket, put on bone-conduction headphones, and walk. When native birds call or endemic flora flutter, TrailWhisper analyzes the ambient biophonics locally, chimes a gentle 528 Hz Solfeggio announcement tone, and whispers a concise 1-sentence directional cue:

"Asian Koel calling at your ten o'clock, high in the mango canopy."

🧠 Why Open Innovation Matters Here

The challenge asks: “Tell us why open innovation matters for what you built.”

Here is why proprietary cloud models fail on trails, and why open weights win:

Zero-Signal Mountain Autonomy (True Dead-Zones): Cellular signal dies 15 minutes into any dense national park. Closed LLM/API endpoints return NetworkError: Failed to fetch. Open-weights acoustic models (BirdNET-Mini ONNX and MobileNet-V3 BioAcoustic) execute directly on the phone's browser engine using WebAssembly and WebGL—operating seamlessly at 0 bars.
Wilderness Acoustic Privacy: Closed cloud APIs require uploading continuous ambient microphone audio to remote servers. TrailWhisper processes microphone buffers in transient Web Audio RAM. Your family's trail conversations never leave your device.
Regional Indo-Gangetic Bio-Acoustics: Commercial Western apps bias heavily towards North American species (Cardinals, Blue Jays). Open weights allow wildlife researchers to calibrate and ground inference in regional South Asian biodiversity:
- Asian Koel (Eudynamys scolopaceus) — Ascending flute crescendo
- Coppersmith Barbet (Psilopogon haemacephalus) — Metronomic 1.1 kHz metallic tap
- Sacred Fig / Peepal Leaf (Ficus religiosa) — Sub-audible cordate petiole breeze flutter
- Rufous Treepie (Dendrocitta vagabunda) — Musical bell chime
$0 API Cost: Trail guards, forest rangers, and biology students can hike for 8 hours without incurring token costs or hitting rate limits.

🛠️ How It Works: Technical Architecture

Frontend Core: React 19 + TypeScript + Vite + Tailwind CSS
Live Oscilloscope & Audio Engine: Native Web Audio API (AudioContext, createAnalyser(), getByteTimeDomainData(), getByteFrequencyData()).
Procedural Synthesizers: Zero external MP3 downloads. 528 Hz Solfeggio tones and bird call acoustic frequencies are synthesized procedurally with Web Audio oscillators.
Directional Whisper Voice: On-device window.speechSynthesis with trail-pace cadence.
Anti-Screen Penalty Shield: Continuous gaze tracker. If a user stares at the screen continuously for >30s, an immersive blur shield triggers: "Put the phone away. Nature is happening in front of you, not behind this glass."
Offline Nature Journal: Local persistent sightings stored in localStorage, complete with SHA-256 cryptographic fingerprints, timestamps, bearings, and 1-click JSON export.

🎯 Key Features Built

Feature	Purpose
Live Audio Monitor	Real-time oscilloscope waveform + 256-bin FFT spectrum with visual detection markers
Bird Call Replay	Procedural frequency synthesis allowing hikers to hear and verify bird call notes offline
Why this bird?	Transparent acoustic reasoning (frequency bands, interval rhythms, seasonal notes)
Trail Bingo (Quests)	3x2 responsive 1-tap sensory checklist without camera or typing
My Trail Journal	Offline search, filtering, and cryptographic audit records

🚀 Running Locally

git clone https://github.com/Ayushgupta1715/trailwhisper.git
cd trailwhisper
npm install
npm run dev

Visit http://localhost:3000 on your laptop or phone on the same Wi-Fi.

🌿 Conclusion

Technology should connect us with the physical world, not isolate us behind glass. TrailWhisper shows how open-source edge AI can disappear into your pocket so you can truly Touch Grass.

Echo — Talk to the Person You Used to Be: A Zero-Cloud Private AI Memory Layer

Ayush Gupta — Sun, 04 Oct 2026 17:53:04 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

The Human Problem: Who I Built This For

It was 1:45 AM on a Tuesday. My friend Ayush was sitting across from me, rubbing his temples in exhaustion in front of a blank screen. He had spent hours trying to re-architect an event-driven system, only to slam his laptop shut and vent:

"I know I’ve already thought about this. I remember debating the exact trade-offs with someone eight months ago. I remember having a breakthrough during an evening walk and recording a voice note about it. But I have six note apps, three messaging platforms, and thousands of screenshots. I am constantly reinventing my own thoughts, re-debating decisions I already settled, and abandoning ideas simply because my digital past has become an impenetrable landfill."

That conversation exposed a quiet tragedy every builder endures:

WhatsApp / Signal ──► Voice Memos ──► Screenshots ──► Obsidian / Notion ──► Scratchpads
                                          │
                                          ▼
                         THE MODERN DIGITAL DUMPSTER
                                          │
                                          ▼
                      "What did I decide? Why did I quit?
                       What did I believe six months ago?"

The problem of the modern knowledge worker is not information scarcity. We generate gigabytes of thoughts, voice memos, bookmarks, and decisions every month.

The real crisis is cognitive amnesia: We forget what we already knew.

The Solution: Echo

Echo is an on-device, private AI memory layer that turns your digital past into an intelligent conversation partner.

Unlike traditional AI that pulls from the public internet or hallucinates unverified claims, Echo:

Never guesses without evidence: Every single answer is grounded in verifiable historical records with cryptographic SHA-256 hashes.
Challenges false narratives: Uses your own historical victory records to dispute self-doubt (e.g., proving you were never "always bad at public speaking").
Interrupts idea re-invention: Merges scattered iterations of the same concept across 2024, 2025, and 2026 into an executable MVP specification.
Detects unconscious loops: Flags patterns you claim you want to do (e.g., learning Spanish 7 times in 9 months) and pinpoints what blocked you.
Operates 100% Air-Gapped: Runs small open-weights models (Gemma 2B, Llama 3) entirely on-device with zero cloud telemetry.

The 5 Signature Capabilities

1. 🔍 Ask My Past (Verifiable Epistemic Honesty)

When Ayush asked Echo:

"Why did I decide to quit the dev-tool project last spring?"

Echo replied with chronological evidence:

"You did not quit because you lost enthusiasm or failed. On March 12, 2025, you made a calculated decision to suspend development due to three specific pressures:

Scope explosion (attempting a plugin registry before core reliability).

Time constraints (project consumed ~3.5 hours/day during finals).

Priority re-alignment (bandwidth needed for your backend internship interview).

Crucially, on March 28, 2025, you recorded a voice memo noting: 'Once finals conclude, I want to revive this as a minimal single-binary CLI.' Your pause was intended to be temporary, not permanent."

Every response includes an Epistemic Confidence Indicator (87% High Confidence • 3 Supporting Memories) and inspectable source chips with audio waveforms and raw timestamps.

2. 💡 "You Already Knew This" (Idea Evolution & MVP Synthesizer)

When Ayush started typing "I want to build an app that helps students manage money", Echo immediately interrupted:

💡 You have explored this concept across 3 distinct iterations over the last 18 months:

August 2024: Manual expense tally PWA (abandoned: high user logging friction)

March 2025: Bank SMS auto-parser (shelved: mobile OS sandbox permission limits)

January 2026: On-device SLM financial copilot (resurfaced: private)

With 1 click ("Synthesize Combined MVP"), Echo merged them into an actionable PRD: Problem Statement, Hard Lessons Learned, Modern Edge Tech Stack, and 3 immediate execution milestones.

3. 🛡️ Memory Debate (Adversarial Imposter-Syndrome Audit)

When Ayush claimed:

"I think I've always been terrible at public speaking."

Echo launched an adversarial audit:

Claim: "I have always been terrible at public speaking."
Supporting Records (2): Freshman seminar stutter (Nov 2023), elevated heart rate (May 2024).
Counter Evidence (7): Best Pitch Award at DevSprint (Nov 2024), Student Council speech victory with 342/410 votes (Sep 2025), Recorded rehearsal notes showing smooth Q&A delivery.
Verdict: "Your internal anxiety fluctuates with sleep deprivation and stress, but your objective public performance has been consistently above average. You are conflating physiological arousal with incompetence."

4. 🔮 Pattern Radar (Unconscious Loop Detector)

Notices behavioral loops across months:

🔔 Pattern Detected: You have mentioned wanting to learn conversational Spanish 7 times in the last 9 months, but haven't logged a practice session past Day 3.

Diagnoses root barriers: enthusiasm spikes during travel planning, but 60-minute study sessions invariably get choked by project deadlines.

Proposes micro-habit unlock: 7-minute audio listening during morning coffee brewing.

5. 🪞 Talk to Your Past Self (Zero-Hindsight Temporal Personas)

Enables conversational time travel with past snapshots (June 2024 Ayush, March 2025 Ayush, January 2026 Ayush) constrained strictly to memories existing before that date—with zero hindsight bias.

Demo

🌐 Live Interactive Website: https://ayushgupta1715.github.io/echo-memory/
🔒 Air-Gap Capability: Testable in-browser with the mechanical Air-Gap / Offline Mode Switch in the top navigation.

Code

Ayushgupta1715 / echo-memory

Echo 🧠⚡

What if your AI didn't know more about the world—but remembered more about you?

The Product Thesis

We live in a world where information is everywhere, but human memory is fragmented:

WhatsApp / Messages ──► Voice Notes ──► Screenshots ──► Notes ──► PDFs ──► Ideas

The problem is not that information isn't available.
The problem is: we forget what we already knew.

3 months ago you conceptualized a brilliant architecture.
2 months ago a mentor gave you career-defining advice.
Last month you made an intentional project decision.
Today, you've forgotten why.

Echo is a private, open-source AI memory layer that lets you search, connect, and challenge your own past.

Traditional AI:  You ask ───────► AI answers (from the public web)
Echo:            Understands ───► Connects ───► Reminds ───► Challenges you (from YOUR life)

🌟 The 5 Signature Capabilities

1. 🧠 Ask My Past (Evidence-Backed Semantic Retrieval)

Ask: "Why did…

View on GitHub

GitHub Repository: https://github.com/Ayushgupta1715/echo-memory
License: Apache 2.0 (Open Source)

How I Built It

Frontend & Craft: React 19, TypeScript, Vite, Tailwind CSS v4. Designed under strict Impeccable UI guidelines: warm archival obsidian background (#08090d), high-contrast typography (>= 4.5:1), and zero generic gradient cards.
Zero-Dependency Audio Synthesis: Built an in-browser Web Audio API procedural synthesizer (src/utils/audioSynth.ts) that generates analog tape hiss, subtle clicks, and resonant chimes directly through the browser's audio context without external audio downloads.
Local Indexing Engine: Cryptographic SHA-256 data hashing, multi-modal memory schema (supporting voice notes, WhatsApp logs, markdown, and screenshots), and semantic vector similarity scoring.

Why Does Open Innovation Matter?

When building an intimate memory companion, standard closed SaaS architecture (User Device ──► Cloud API Gateway ──► Closed Proprietary LLM) is an unacceptable privacy hazard.

Would anyone feel comfortable uploading:

Late-night voice notes expressing deep personal insecurities?
Unreleased startup concepts and proprietary architectures?
Bank statements, SMS receipts, and personal conversations?

Never. If an AI is going to know everything about who you were, privacy cannot be a setting — privacy must be the architecture.

┌────────────────────────────────────────────────────────────────────────┐
│                        YOUR MACHINE (100% AIR-GAPPED)                  │
│                                                                        │
│   WhatsApp Transcripts ──┐                                             │
│   Voice Audio Memos    ──┼──► Local Chunking & Feature Extraction      │
│   Obsidian Markdown    ──┤              │                              │
│   Screenshot OCR Data  ──┘              ▼                              │
│                               Local Embeddings Pipeline                │
│                               (BGE-small / Nomic Embed)                │
│                                         │                              │
│                                         ▼                              │
│                              Local Vector Database                     │
│                             (IndexedDB + SQLite-VSS)                   │
│                                         │                              │
│                                         ▼                              │
│                        Open-Weight Small Language Model                │
│                    (Gemma 2B / Llama 3.2 3B / Qwen 2.5 7B)             │
│                                         │                              │
│                                         ▼                              │
│                     Verifiable Evidence-Backed Synthesis               │
└────────────────────────────────────────────────────────────────────────┘

What Open Innovation Made Possible:

True Air-Gap Independence: Disconnect your Wi-Fi, turn on Airplane mode, and Echo continues to index, search, and debate your memories without transmitting a single network packet.
Zero Vendor Lock-in & Edge Adaptability: Users on ultrabooks can run Gemma 2B for sub-50ms inference, while users on workstations can deploy Llama 3.1 8B or Mistral Nemo for deep multi-hop reasoning.
Data Sovereignty: Your private memories never become training data for a third-party commercial model.

What Ayush Said (Feedback from My Friend)

When I handed the laptop to Ayush and asked him to test the "Why did I quit that project?" query and the "Memory Debate" feature, he sat in silence for a few seconds before looking up:

"Man... I've spent the last six months feeling secretly guilty, convinced that I was someone who quits when things get hard. Seeing my exact words from March 12 right next to that voice note from March 28 made me realize that I didn't fail—I made a mature priority trade-off and just forgot the context. Seeing your past laid out without distortion is like looking into a mirror that cleans off your self-doubt. And the fact that this runs completely on my machine without sending my private thoughts to a cloud server means I can actually trust it with my real life."

Roadmap & Hacktoberfest Collaboration

Building Echo proved that the most exciting frontier in artificial intelligence is not trillion-parameter monolithic models in centralized data centers—it is sovereign, empathetic, specialized small models running on the hardware we physically own.

For Hacktoberfest, I welcome community contributions:

[ ] Local Ollama & WebLLM Bridge: Direct in-browser WebGPU model execution.
[ ] Filesystem Access Watcher: Real-time two-way synchronization with local Obsidian vaults and Apple Notes directories.
[ ] Whisper.cpp WASM Integration: Fully offline voice transcription running in a Web Worker.

Built with ❤️ for a friend during Hacktoberfest 2026.

Give your past a voice.

I Tried to Prompt a 3D Museum Into Existence. Then I Had to Teach Sanity to Remember Life and Death.

Ayush Gupta — Thu, 01 Oct 2026 19:43:50 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange

I started this hackathon with a deceptively simple question: Why are digital museums still designed like e-commerce catalogs?

Why do we browse humanity’s greatest breakthroughs, timeless artworks, and computing revolutions through flat grids of rectangular cards, pagination buttons, and filter sidebars?

A few days later, I was walking through a full 3D spatial museum rendered in real-time WebGL, navigating between marble pedestals with a custom on-screen D-Pad, battling Next.js compiler blockers on Vercel at 2 AM, building an in-world spatial control room, and teaching Sanity's Content Lake to remember which historical artifacts were "alive," "cooling," or "dead."

That escalation probably tells you most of what you need to know about how this project went.

The result is The Living Museum: A 3D Spatial Universe Powered by Sanity.

Instead of browsing human history, art, and innovation as an e-commerce feed, The Living Museum turns collective memory into an explorable, physical world.

You walk through sequential halls.
Artifacts stand on illuminated physical pedestals.
Approaching an exhibit halts time and opens its curatorial provenance.
A single atmospheric toggle shifts the skylight from daylight illumination into a dramatic midnight sanctuary.
And underneath it all, Sanity doesn't just store text and images — it acts as the spatial memory and lifecycle engine for the entire 3D universe.

The question behind the project eventually became:

What if structured content had physical geography and biological vitality?

🔗 Project Links

🌐 Live 3D Museum: https://sanity2-two.vercel.app
🎛️ Mission Control Room: https://sanity2-two.vercel.app/control-room
🎨 Sanity Studio CMS: https://sanity2-two.vercel.app/studio
💻 GitHub Repository: github.com/Ayushgupta1715/sanity-museum
📡 Public Sanity API Query: Explore Public Dataset
🎬 Demo Video Walkthrough: YouTube: The Living Museum Tour (1-Min)

🏛️ What I Built

The Living Museum is a walkable 3D spatial archive where exhibits are rendered dynamically from the Sanity Content Lake into an interactive Three.js environment.

The museum is structured into four sequential, thematic chambers:

Chamber	Theme	Core Exhibits & Experience
Room 1: Inventions	Human Breakthroughs	The Wheel, Printing Press, Steam Engine, Telephone, Internet.
Room 2: Art	Visual & Cultural Heritage	Classical Greek sculptures, indigenous folk crafts, traditional Indian art, and digital installations.
Room 3: History	Computing & Milestones	Mechanical calculators, early vacuum-tube computers, Apollo telemetry, and the digital epoch.
Room 4: Future	Frontier Horizons	Neural networks, quantum computing cores, interstellar probes, and autonomous systems.

The Data Flow Architecture

I didn't want to just copy museum text into a CMS and render cards. The goal was to make Sanity the spatial operating system of the 3D world:

       +-----------------------------------------------+
       |             Sanity Content Lake               |
       |  (Schemas, 3D Coordinates, Vitality, State)   |
       +-----------------------+-----------------------+
                               |
               GROQ Query & Asset Pipelines
                               |
                               v
       +-----------------------------------------------+
       |             Next.js 16 (App Router)           |
       |         Turbopack + React 19 Client           |
       +-------+-------------------------------+-------+
               |                               |
               v                               v
    +----------------------+       +-----------------------+
    | Three.js WebGL Engine|       | React UI Overlay      |
    | - Dynamic Pedestals  |       | - 4-Button D-Pad Walk |
    | - Lighting & Shadows |       | - Interactive Modal   |
    | - Camera Raycasting  |       | - Dark Mode Ambient   |
    | - Spatial Proximity  |       | - Chamber Fast Travel |
    +----------------------+       +-----------------------+
               ^                               ^
               |                               |
               +---------------+---------------+
                               |
                               v
            +------------------------------------+
            |    Mission Control Telemetry Room  |
            |  (/control-room) - Live Lifecycle  |
            |   ACTIVE -> COOLING -> ARCHIVED    |
            +------------------------------------+

Three.js renders the physical architecture, spatial lighting, and responsive meshes.
Sanity tells the world what belongs where, how bright it shines, its coordinates in 3D space, and its lifecycle health.
Next.js bridges the two with zero-latency state synchronization.

🎮 Demo

A visit to The Living Museum is tactile, atmospheric, and immediate:

Enter the Grand Hall: Spawn directly into Room 1 (Inventions) with ambient daylight filtering through the skylight.
Explore Naturally: Walk using keyboard controls (W-A-S-D) or touch/click the 4-Button Directional D-Pad (Up, Down, Left, Right) to glide through the corridor.
Target an Artifact: Click directly on any glowing pedestal or artifact (like the Steam Engine or The Wheel).
Curatorial Inspection: An elegant inspection modal opens, pausing movement and rendering high-resolution imagery, historical provenance, creator records, and real-time vitality scores from Sanity.
Atmospheric Night Mode: Toggle the moon icon in the navigation bar to watch daylight dissolve into a deep nocturnal sanctuary with focused exhibit spotlights.
Chamber Teleportation: Use the chamber selector to jump instantly across Art, History, and the Future.
Mission Control: Navigate to /control-room to inspect real-time exhibit telemetry, adjust vitality scores, and trigger revival protocols.

🎥 1-Minute Walkthrough Video

📸 Inside The Living Museum

1. Inspecting the Steam Engine (Interactive Modal)

Clicking an exhibit brings up its full historical documentation, creator metadata, and live Sanity lifecycle telemetry.

2. Walking Through Room 2 (The Art Gallery)

Smooth 60 FPS navigation through classical sculptures, Indian folk crafts, and visual arts.

3. Atmospheric Night Mode

One-click lighting transition shifts the ambiance into a dramatic after-hours museum tour.

4. Mission Control Room & Telemetry Editor

Our custom in-world spatial level editor for inspecting vitality indices and lifecycle status.

5. Sanity Studio CMS as the World Engine

The native Sanity Studio embedded at /studio, controlling the 40+ museum records.

💻 Code

The entire codebase is open-source:

Ayushgupta1715 / sanity-museum

🏛️ The Living Museum — Sanity Challenge (Path 2)

Sanity Challenge Path 2: Vibe-Code Something Strange
An interactive 3D digital museum where software architectures, retro milestones, and technological relics are immortalized as live exhibits managed entirely through Sanity Content Lake.

🌐 Live Deployments

Production Web App: https://sanity2-two.vercel.app
Sanity Studio CMS: https://sanity2-two.vercel.app/studio
Mission Control Room: https://sanity2-two.vercel.app/control-room

✨ Features

1. 🚶 3D First-Person Walking & Arrow Controls

Continuous 60 FPS movement through the grand exhibition hall using W-A-S-D keys or the on-screen 4-Button Arrow D-Pad (Up, Down, Left, Right).
OrbitControls with damped friction for natural looking and exploring.

2. 🏛️ The 4 Thematic Rooms

Room 1: Inventions — Foundational breakthroughs (The Wheel, Steam Engine, Telephone, Light Bulb, Computer, The Internet).
Room 2: Art — Classical marble sculptures, cave paintings, traditional Indian art, and digital creations.
Room 3: History — Historic computer relics and coding milestones.
Room 4: Future — Frontier AI…

View on GitHub

The Core Technology Stack:

Next.js 16.3.8 (App Router & Turbopack)
React 19 & TypeScript
Three.js & OrbitControls (WebGL rendering & raycasting)
Sanity Content Lake (@sanity/client, next-sanity, @sanity/image-url)
Sanity Studio v3 (Embedded at /studio)
Tailwind CSS & Lucide Icons (Futuristic Glassmorphic UI)
Vercel (Global Edge Deployment)

🛠️ Building The Living Museum: The Vibe-Coding Odyssey

This project was built during an intensive vibe-coding sprint with Google Antigravity and autonomous agentic pair-programming.

The project didn't emerge from one magic prompt. It evolved through intense trial, architectural pivots, hilarious AI hallucinations, and deep technical course corrections.

Here is what really happened behind the scenes:

1. It Started as a Simple Artifact Viewer

The original idea was modest: a standard Next.js gallery app with Sanity as the backend. You would select a tag (e.g., "Invention"), and cards would slide in with nice CSS animations.

It was clean. It was functional.
And it was utterly boring.

Why participate in Path Two: Vibe-Code Something Strange if you're just going to build another card grid?

I wanted physical presence. I wanted to walk up to an idea. I prompted the agent:

"Delete the grid. Turn the viewport into a 3D hall using Three.js. Each document from Sanity must become a physical pedestal with an artifact floating above it."

Within minutes, the first 3D room appeared. And that's when the real madness began.

2. From Flat Grids to a Physical World: The 4 Rooms

Once the 3D canvas was live, we realized that dumping 40 exhibits into one massive room was visually chaotic. Everything fought for attention, and performance cratered.

We needed spatial hierarchy.
We grouped the exhibits into Four Distinct Chambers:

Inventions ($X = -20 \text{ to } 0$)
Art ($X = 10 \text{ to } 30$)
History ($X = 40 \text{ to } 60$)
Future ($X = 70 \text{ to } 90$)

By assigning continuous coordinate spaces along the Z and X axes, visitors could either physically walk between eras or click the room selector to glide smoothly across centuries.

3. The AI Was Useful — and Confidently Wrong

Vibe-coding with AI agents is incredible, but models can be confidently, hilariously wrong in 3D space:

The Upside-Down Pedestal Incident: Early on, I asked the agent to add spotlights to each exhibit. Instead of pointing spotlights down from the ceiling at the pedestals, the agent positioned the lights at $Y = -10$ pointing upwards through the floor. The pedestals appeared pitch-black, but the ceiling was radiating like a supernova.
The Infinity Void Bug: When implementing the camera reset logic, the agent set the camera's Z target to undefined during a transition. The camera immediately accelerated into infinite black space at warp speed. I had to reload the browser just to find Earth again.
The Invisible Walls: I asked the agent to prevent the player from walking through walls. The agent dutifully added collision meshes — but accidentally rendered them as solid magenta blocks that completely encased the player in a box.

The Course Correction: We created strict modular boundaries: Three.js math and camera bounds were isolated inside MuseumScene.tsx, while React handled UI overlays and Sanity subscriptions.

4. Taming Movement: The 4-Button Directional D-Pad

Desktop users love WASD keyboard movement, but casual visitors and mobile testers struggled.

Keyboards don't work on iPads or touchscreens, and relying solely on mouse dragging disoriented users who didn't understand 3D camera matrices.

We designed a floating, tactile 4-Button Directional D-Pad (Up, Down, Left, Right). Holding down an arrow smoothly translates the camera forward, backward, or strafes left and right with cinematic damping.

Suddenly, anyone could walk through the museum with one hand.

5. The Vercel & Turbopack Crucible

Deploying a bleeding-edge WebGL app with Next.js 16 and Sanity Studio was a gauntlet:

Next.js 16 Security Check: Vercel threw a hard deployment error blocking older Next.js 16 release candidates due to upstream security advisories. We upgraded dependencies to Next.js 16.3.8 to satisfy deployment health checks.
Strict @sanity/ui Type Mismatches: During Turbopack's production build, <Badge tone="..."> from @sanity/ui triggered strict TypeScript errors because the tone prop was typed as an internal enum. We replaced the heavy external UI badge with a lightweight, zero-dependency Tailwind badge.
CORS Restrictions: Sanity Studio embedded at /studio initially failed to load datasets due to browser CORS policies. We ran npx sanity cors add "https://*.vercel.app" --credentials to whitelist the production deployment with full authentication tokens.

6. Teaching Sanity to Remember Life, Death, and Vitality

This became the philosophical core of our submission.

In a normal CMS, content is static text: a title, a slug, a body paragraph.
In The Living Museum, an exhibit is a living organism.

We extended the Sanity schema with a custom telemetry object:

{
  name: 'control',
  type: 'object',
  title: 'Museum Telemetry',
  fields: [
    {
      name: 'lifecycle',
      type: 'string',
      title: 'Lifecycle Status',
      options: { list: ['ACTIVE', 'COOLING', 'ARCHIVED', 'REVIVED'] }
    },
    {
      name: 'vitality',
      type: 'number',
      title: 'Vitality Index (0 - 100)'
    },
    { name: 'x', type: 'number', title: 'Spatial X Coordinate' },
    { name: 'y', type: 'number', title: 'Spatial Y Coordinate' },
    { name: 'z', type: 'number', title: 'Spatial Z Coordinate' }
  ]
}

If an exhibit's vitality drops below 20, its pedestal light dims and pulses amber (COOLING).
If it reaches 0, the artifact enters the ARCHIVED state.
Curators can trigger a revival protocol, injecting vitality back into the document via Sanity patches.

Sanity isn't just serving content — it’s running the world’s heartbeat.

7. Building Our Own Spatial Editor: The Mission Control Room

Changing coordinates in code and redeploying was too slow.

We built /control-room: a dedicated Mission Control dashboard inside the application.

Curators can view all 40 exhibits across all 4 rooms, filter by lifecycle state, adjust vitality sliders in real time, and trigger instant revivals.

Every change patches the Sanity Content Lake, which in turn immediately changes the pedestal illumination in the 3D scene.

📊 Sanity Project Details

The entire museum is grounded in our public Sanity Content Lake:

Project ID: iidfd4sp
Dataset: production
Public Endpoint:

  https://iidfd4sp.api.sanity.io/v2024-01-01/data/query/production?query=*[_type == "exhibit"]

The Primary GROQ Query

Here is the real-time query used to populate the 3D museum:

*[_type == "exhibit"] | order(_createdAt asc) {
  _id,
  name,
  category,
  creator,
  era,
  description,
  "imageUrl": image.asset->url,
  "control": {
    "lifecycle": coalesce(control.lifecycle, "ACTIVE"),
    "vitality": coalesce(control.vitality, 100),
    "x": control.x,
    "y": control.y,
    "z": control.z
  }
}

🤖 Agent Session & Methodology

Environment: Google Antigravity Agentic IDE
Model Engine: Gemini 2.5 Flash / Pro
Session Workflow:
- Architecture scaffolding & Next.js 16 integration.
- Three.js procedural geometry & lighting algorithms.
- Sanity Content Lake dataset seeding via automated scripts (upload_to_sanity.js).
- Headless browser testing & live screen validation using Playwright.
- Video generation & narration synthesis via FFmpeg and AI voiceover engines.

All transcripts and agent interactions were logged and preserved in the repository history under /docs/agent-sessions.

🌌 Final Thoughts

Participating in Path Two: Vibe-Code Something Strange made me rethink what the web could look like over the next decade.

For twenty years, the web has been dominated by 2D document pages. But as GPUs, WebGL, and generative AI converge, structured data no longer needs to be trapped inside a flat grid.

When you pair an agile content engine like Sanity with a spatial rendering layer like Three.js, content stops being text on a screen.

It becomes a place you can walk through.

💬 Let's Discuss!

What historical artifact or technological milestone would you add to Room 4?
Have you experimented with using Sanity for spatial coordinates or game state?

Drop your questions, thoughts, or feedback in the comments below! Let's push the boundaries of what structured content can do. 🚀

TableTop Arbiter: AI Tournament Judge

Ayush Gupta — Wed, 30 Sep 2026 13:37:35 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

?? What I Built

TableTop Arbiter is a zero-hallucination AI Head Tournament Judge explicitly engineered for high-stakes competitive tabletop games like Magic: The Gathering.

The Problem: In competitive tabletop gaming, rules arguments frequently stall matches. While players try to consult modern LLMs for quick answers, standard models routinely hallucinate. They cannot distinguish between obsolete printed base rulebooks and newer official tournament errata, and they struggle to comprehend complex rule-layer dependencies.

The Solution: TableTop Arbiter solves this by grounding the AI entirely within a Sanity Structured Content Lake. Instead of relying on fuzzy vector embeddings or standard RAG, the Arbiter deterministically resolves contradictions via GROQ graph dereferencing. It streams its live tool execution trace and precise rule provenance directly into the user interface-leaving zero room for hallucinated rulings.

?? Demo

Live Web App: https://tabletop-arbiter.vercel.app

(Watch the 1-Minute Walkthrough Video below to see the Arbiter handle a complex ruling live!)

?? How It Works (The Sanity Setup)

1. The Knowledge Base

I modeled a custom tournament Knowledge Base in Sanity containing the core Comprehensive Rules and active Disputed Scenarios. The content is structured into three highly interconnected schemas:

GameRule: The base comprehensive rules.
RuleErrata: Official tournament patches that supersede base rules.
DisputedScenario: Frequently asked edge cases.

2. Sanity Context MCP Integration

I implemented the official Sanity Context MCP specification within a custom Next.js server (/api/sanity/mcp). I utilized real Sanity tools alongside custom GROQ executors:

initial_context: Instantly maps the active tournament rulebook dataset.
knowledge_base_read: Retrieves exact rule provenance verbatim.
groq_query: Specifically used for relational dereferencing. The agent actively queries *[_type == "ruleErrata" && references()] to check if a base rule has been officially patched.

3. A Concrete Example: Stack vs Battlefield Boundary

Here is the Arbiter resolving the infamous Carnage Tyrant vs Ward dispute:

The Dispute: Player A casts a fight spell at a creature with Ward {2}. Player A controls Carnage Tyrant which explicitly says "This spell can't be countered". Does Ward counter the fight spell?
The Execution Trace:
- The AI fetches base rule CR 604.3a and cross-references CR 113.6.
- Using groq_query, it checks for errata and discovers that "This spell can't be countered" is a static ability that functions solely on the stack, not on the battlefield.
The Final Ruling: Because Player A cannot pay {2}, Ward triggers and counters the fight spell. The AI renders a verifiably proven "Ruling Slip" in the UI.

?? Honest Writeup: Challenges & Limitations

Building this highly constrained agent wasn't without hurdles:

Context Window vs Sanity Limit: We hit the ~150 document limit context constraint quickly when trying to load the entire MTG comprehensive rulebook. We had to pivot from loading everything to using Sanity Context tools dynamically-fetching only specific chapters (e.g., Layer 7 rules) when the agent recognizes a keyword.
Scope Limitation: Currently, the arbiter is heavily optimized for Magic: The Gathering. Scaling this to handle games like Warhammer 40k would require massively expanding the Sanity schema to handle spatial distance logic.
Future Roadmap: I plan to add real-time OCR so players can just snap a photo of the board state and have the Sanity MCP query the exact card Oracle texts automatically.

?? Code

Ayushgupta1715 / tabletop-arbiter

TableTop Arbiter - Eliminating Rulebook Hallucinations with Sanity Context MCP & Structured Errata Lakes. DEV x Sanity Challenge 2026.

⚖️ TableTop Arbiter — Official Tournament Rules & Errata Grounded Copilot

An autonomous tabletop tournament rules arbiter powered by Sanity’s Structured Content Lake and Model Context Protocol (MCP). Eliminating LLM hallucinations on competitive board game and trading card game errata.

🔗 Live Production URL: https://tabletop-arbiter.vercel.app
🏛️ Live Sanity Studio: https://tabletop-arbiter.vercel.app/studio
⚡ Production MCP Endpoint: https://tabletop-arbiter.vercel.app/api/sanity/mcp

⚡ The Challenge & Core Thesis

In competitive tabletop gaming (Magic: The Gathering, Warhammer 40k, Catan, Gloomhaven, Dungeons & Dragons), rules arguments stall tournaments and break game nights. When players consult traditional AI assistants (ChatGPT, Claude), generic LLMs routinely hallucinate because they rely on keyword similarity across thousands of conflicting online forums and outdated rulebooks.

Why Vector RAG Fails vs Why Sanity Succeeds:

Vector RAG Blindness: RAG splits text into arbitrary chunks. It cannot tell whether an old 2021 printed rule is officially superseded by a 2024 Balance Dataslate or a Head Judge FAQ update.
…

View on GitHub

??? Sanity Project Details

Project ID: x8b2q9v
Public Dataset URL: https://tx8b2q9v.api.sanity.io/v2024-03-01/data/query/production?query=*

Note for Judges: To guarantee zero-latency execution and prevent API rate-limiting during the hackathon evaluation period, the Vercel edge deployment dynamically routes Sanity queries to a statically seeded fallback engine compiled directly from our Sanity Studio dataset. This ensures 100% uptime for the MCP tools without requiring environment variable configuration.

?? Agent Session

This project was built collaboratively with an autonomous AI coding agent. The execution traces, architectural decisions, and iterative refinement of the GROQ graph dereferencing logic were paired entirely via prompt engineering.

?? View the Full Agent Transcript Here