DEV Community

TecMinds GmbH
TecMinds GmbH

Posted on

I built a CLI that checks whether the references in your bibliography actually exist

#academia #opensource #node #cli

Last year I started building Acurio, a tool that verifies citations in academic theses against their sources. One pattern kept showing up in real student theses: bibliographies full of references that simply don't exist. Plausible-looking authors, plausible journal, plausible year — and no matching record anywhere. Most of them were LLM hallucinations the student never caught.

The existence check turned out to be the one piece that doesn't need any AI at all. So I extracted it into a small open-source CLI: citecheck.

What it does

Point it at a .bib, .ris, or CSL-JSON file — the export you get from Zotero, Mendeley, or EndNote — and it checks every reference against three public scholarly APIs:

  • Crossref and OpenAlex — does a matching record exist at all? A wrong DOI, a typo'd title, or a hallucinated reference shows up as not found.
  • Crossref retraction metadata — has the work been retracted? (You'd be surprised how often Wakefield 1998 still gets cited straight.)
  • DOAJ — is the journal a listed open-access journal? Shown as a positive signal. 
$ npx citecheck references.bib

  ✓  verified     watson1953      Molecular Structure of Nucleic Acids…
  ~  partial      wakefield1998   Ileal-lymphoid-nodular hyperplasia…  ⚠ RETRACTED
  ✗  not found    fabricated2099  Quantum Entanglement of Bibliographic Phantoms…

Summary: 1 verified · 1 partial · 1 not found · 1 retracted
Enter fullscreen mode Exit fullscreen mode

No API key, no signup, nothing uploaded. It only sends each reference's DOI/title to the public APIs and prints the result.

Design decisions worth stealing

Auto-detect the input format. Users don't know (or care) whether their export is BibTeX, RIS, or CSL-JSON. The CLI sniffs the format, including from stdin: cat refs.bib | citecheck -.

DOI first, title second. If a reference has a DOI, that's a precise lookup. No DOI (books, older works, grey literature) falls back to a title search with fuzzy matching — which is also where the honest uncertainty lives, hence the partial verdict instead of pretending binary confidence.

Be conservative about "not found". A missing record can mean a hallucinated reference — or a book that Crossref just doesn't cover. The output distinguishes "no match anywhere" from "partial match", because a false accusation of fabrication is worse than a missed one.

No build step at install. The npm package ships prebuilt, so npx citecheck cold-starts fast.

Why this matters right now

Every study on LLM-generated citations finds fabrication rates that should terrify anyone grading theses — GPT-4o still fabricates roughly one in five references in some benchmarks. The scary part isn't that students use LLMs; it's that a hallucinated reference looks exactly like a real one until someone actually resolves it.

Checking existence is the cheap, deterministic 80%. (Checking whether the source actually supports the claim citing it is the hard part — that's what the hosted product does.)

Try it 

npx citecheck references.bib
Enter fullscreen mode Exit fullscreen mode

Repo: https://github.com/tobiasosDev/citecheck — MIT licensed. Feedback and PRs welcome, especially around false positives on non-DOI sources: books, grey literature, and non-English titles are the current weak spots.

Top comments (0)