The $5,000 hallucination
In Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), a federal judge sanctioned lawyers $5,000 after they filed a brief citing six court decisions that ChatGPT had invented. That was 2023 — the failure mode hasn't gone away, it's just everywhere now: student essays, grant proposals, court filings, and yes, dev blog posts.
CiteWarden is the checking layer. It reads text the way a paranoid reference librarian would. Built for LexHack 2026 (AI × Law × Civic Tech).
Try it: rudra496.github.io/citewarden
What it checks — against live registries
| Citation family | Verification path |
|---|---|
| DOIs | Live Crossref lookup → OpenAlex fallback. 404 on both = flagged as non-existent |
| arXiv IDs | Via their registered DOI mirror (10.48550/arxiv.*) in OpenAlex |
| PubMed IDs | Live NLM E-utilities summary |
| CFR sections | Official eCFR structure API |
| U.S. reporter cites (U.S., F., F.2d, F.3d, S. Ct., F. Supp.) | 464-case verified landmark database + forensic bounds checks |
| Case names | Landmark DB match + documented-fabrication watchlist |
| UK acts / EU regulations | Verified statute list + live Wikipedia/ELI record checks |
The offline forensic layer
Beyond "does this exist," CiteWarden runs fully offline forensics: reporter volume-range bounds, reporters cited before they existed (any F.3d cite dated before 1992 is impossible), decision years contradicting the volume era, malformed titles, placeholder artifacts, and same-cite/conflicting-year collisions.
The integrity score is transparent
- Green = matched a live registry or the verified database
- Amber = well-formed but unverifiable — honest uncertainty, never guessed
- Red = impossible values, watchlist hit, or the registry confirms non-existence
Score = 100 − 100·(red/total) − 30·(amber/total), floored at 0. You can compute it yourself from the report.
Zero-invention data policy
Every shipped row was machine-checked at build time: the 464 landmark cases were parsed from Wikipedia's curated index with volume-bounds, volume-era consistency, and year sanity gates (1791–2026); build logs live in research/. The Mata v. Avianca watchlist contains the six fabricated citations transcribed verbatim from the sanction order.
Links
- Live: rudra496.github.io/citewarden
- Source: github.com/rudra496/citewarden
If you review manuscripts, grade papers, or file briefs — paste your next document in before someone else finds the problem.
Top comments (0)