I built this project and wrote this post for the Google Cloud "All Things Agentic" Hackathon.
The problem
Companies announce dated commitments constantly — "open weights by end of year," "GA next quarter," "available later this month." Some ship on time, some ship late, some quietly disappear. Nobody keeps an honest, checkable score, and by the time a promise comes due the original announcement is buried.
The Promise Ledger extracts falsifiable promises from announcements and checks them at their deadline. Its whole point: no LLM decides an outcome. A promise's status is set by deterministic code that fetches the official page and checks it.
How it's built
Admission pipeline (Google ADK, streamed live in the web app):
- PromiseExtractorAgent (Gemini) — proposes a structured promise from an announcement: verbatim source quote, normalized deadline, an observable outcome, machine-checkable keywords. Pydantic output schema.
- PromiseAuditorAgent (Gemma) — an adversarial second opinion on a different model family: "is this actually checkable?" A rejection triggers up to two re-extractions.
- Falsifiability gate (pure Python, no LLM) — a real deadline that isn't absurd, ≥2 distinct specific keywords, a substantive outcome. The gate admits, not the model.
-
admit_promise— writes to the ledger (JSON file or Firestore).
Verification cycle (zero LLM, point-in-time). For each due promise, two probes:
- The official page as archived by the Wayback Machine on or before the deadline. If the check keywords are in that capture, the promise was kept on time and the capture date is the dated proof — no prose-date guessing, no third party but a neutral public archive.
- The page now — which, combined with probe 1, separates late vs delayed vs abandoned.
A fixed, public decision table turns the two readings into one of seven statuses (PENDING · FULFILLED · FULFILLED_LATE · PARTIALLY_FULFILLED · DELAYED · ABANDONED · UNVERIFIABLE), and every verdict records how it was reached. The ledger is also exposed over the Model Context Protocol via a FastMCP server (get_scorecard, list_promises, admit_promise, run_verification_cycle).
The design decision I like most
Point-in-time verification. "Did they ship by the deadline?" usually can't be read off today's live page — the page only ever shows its current state. But the page as it existed on the deadline answers it directly, and a capture with the feature still missing is hard proof it wasn't out yet. Moving the verifier from "parse a date out of prose" to "was it in the archived snapshot" removed the softest, most arguable step in the whole system.
Honest limitations
The archive doesn't always have a capture near the deadline; when it doesn't, the verifier falls back to reading a date off the current page — a weaker signal, biased toward FULFILLED. One seed row still leans on a third-party dated write-up because the company's own changelog no longer carries the release. Keyword matching is brittle. The seed is 8 curated promises across 6 companies — enough to show the mechanism, not a census — so the headline on-time percentage is illustrative.
Links
- Project & full write-up on Devpost: https://devpost.com/software/the-promise-ledger
- Demo video: https://youtu.be/-UTiZcL30U0
Stack: Gemini 3.5 Flash-Lite + Gemma via Google ADK, FastMCP / Model Context Protocol, FastAPI, JSON / Firestore, deployed on Google Cloud Run. Built for the Google Cloud "All Things Agentic" Hackathon.
Top comments (0)