Check every reference before arXiv does

Aurora Lab — Mon, 02 Nov 2026 13:40:17 +0000

In May 2026 arXiv announced that it will ban authors for a year when a submission contains references that do not exist (Nature, The Next Web). After the year, they can post only papers already accepted at a peer-reviewed venue. The rule does not ban AI help. It punishes unchecked output, and an invented reference is the easiest unchecked output to prove.

CVPR 2027 papers are due on 16 November. If a language model touched your related work or your .bib, this post shows how to check every entry: first one reference by hand with Crossref's free API, to see what can go wrong, then a whole paper in one command.

Three ways a reference goes wrong

It does not exist. A plausible title, a real venue, page numbers that look right, and no paper behind them.
The DOI is wrong. It belongs to another paper (one digit off: 10.1109/CVPR.2016.90 is ResNet, .91 is YOLO), or it is not registered at all.
The details are wrong. Real paper, but the year, the venue, the pages or the author list differ from the publisher's record.

This post covers the first two: they are what gets a paper flagged. The third needs a field-by-field comparison with the publisher's record, which is a bigger job.

Here is a small paper with six references and three of them broken. The ResNet entry carries YOLO's DOI, the LeCun et al. entry carries an unregistered DOI, and lin2022sparse is invented (no registry has it, and neither does a web search for the exact title).

refs.bib (six entries)

@article{harris2020numpy,
  title   = {Array programming with {NumPy}},
  author  = {Harris, Charles R. and Millman, K. Jarrod and van der Walt, St{\'e}fan J. and Gommers, Ralf and others},
  journal = {Nature},
  volume  = {585},
  pages   = {357--362},
  year    = {2020},
  doi     = {10.1038/s41586-020-2649-2}
}

@inproceedings{vaswani2017attention,
  title     = {Attention is All you Need},
  author    = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, {\L}ukasz and Polosukhin, Illia},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {30},
  pages     = {5998--6008},
  year      = {2017}
}

@misc{openai2023gpt4,
  title         = {{GPT-4} Technical Report},
  author        = {{OpenAI}},
  year          = {2023},
  eprint        = {2303.08774},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

@inproceedings{lin2022sparse,
  title     = {Sparse Mixture-of-Experts Routing with Learned Temperature Schedules},
  author    = {Lin, Mei and Okafor, Daniel},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {35},
  pages     = {11204--11216},
  year      = {2022}
}

@inproceedings{he2016resnet,
  title     = {Deep Residual Learning for Image Recognition},
  author    = {He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages     = {770--778},
  year      = {2016},
  doi       = {10.1109/CVPR.2016.91}
}

@article{lecun2015deep,
  title   = {Deep learning},
  author  = {LeCun, Yann and Bengio, Yoshua and Hinton, Geoffrey},
  journal = {Nature},
  volume  = {521},
  pages   = {436--444},
  year    = {2015},
  doi     = {10.1038/nature145390}
}

Checking one reference by hand

Two free services answer most questions. doi.org says whether a DOI is registered with any agency (Crossref, DataCite and the rest; asking Crossref alone would miss arXiv's and Zenodo's DOIs). Crossref says whose DOI it is and searches its more than 180 million records for a reference you paste in. Neither needs a key. This script uses only Python's standard library:

import difflib, json, sys, urllib.error, urllib.parse, urllib.request

MAILTO = "you@example.org"  # Crossref asks scripts to say who they are


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": f"check_ref (mailto:{MAILTO})"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def similarity(a, b):
    return difflib.SequenceMatcher(None, a.lower(), b.lower()).ratio()


def check(title, doi=None):
    print(title)
    if doi:
        try:  # is the DOI registered at all? (any agency: Crossref, DataCite, ...)
            get(f"https://doi.org/api/handles/{doi}")
        except urllib.error.HTTPError:
            print(f"  DOI {doi} is not registered")
        else:  # whose DOI is it?
            other = get(f"https://api.crossref.org/works/{doi}")["message"]["title"][0]
            print(f"  DOI {doi} -> {other!r} ({similarity(title, other):.2f})")
    query = urllib.parse.urlencode({"query.bibliographic": title, "rows": 3, "mailto": MAILTO})
    for item in get(f"https://api.crossref.org/works?{query}")["message"]["items"]:
        found = item.get("title", ["(no title)"])[0]
        print(f"  {similarity(title, found):.2f}  {found}")


check("Deep Residual Learning for Image Recognition", "10.1109/CVPR.2016.91")
check("Deep learning", "10.1038/nature145390")
check("Sparse Mixture-of-Experts Routing with Learned Temperature Schedules")
check("BLEU: a Method for Automatic Evaluation of Machine Translation")

What it printed:

Deep Residual Learning for Image Recognition
  DOI 10.1109/CVPR.2016.91 -> 'You Only Look Once: Unified, Real-Time Object Detection' (0.32)
  0.49  Multiple butterflies recognition based on deep residual learning and image analyze
  0.90  Deep Residual Learning for Image Recognition: A Survey
  1.00  Deep Residual Learning for Image Recognition
Deep learning
  DOI 10.1038/nature145390 is not registered
  0.58  What’s Deep About Deep Learning?
  0.44  Designing Deep Learning: Learning Partnerships
  0.72  Deep Learning in Action
Sparse Mixture-of-Experts Routing with Learned Temperature Schedules
  0.44  Efficient Routing in Sparse Mixture-of-Experts
  0.39  Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
  0.34  Supplemental Information 2: Code Implementation for the Heuristic Sparse Mixture-of-Experts Framework.
BLEU: a Method for Automatic Evaluation of Machine Translation
  0.12  BLEU
  0.56  Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
  0.35  Table 10: Evaluation results of Hindi-to-Dogri machine translation using BLEU, TER, METEOR, and WER metrics across different models and sentence lengths.

The two DOI problems show up at once: one DOI belongs to YOLO, the other is not registered anywhere. The invented paper's best match scores 0.44. So far so good. Now look at the real papers:

LeCun, Bengio and Hinton's "Deep learning" (Nature, 2015, cited more than 80,000 times) is not in the top three. A two-word title is too generic for a title-only search.
BLEU, cited more than 20,000 times, scores 0.12. Crossref stores its title as "BLEU" and the rest as a subtitle.
ResNet is found at 1.00, but right above it sits "Deep Residual Learning for Image Recognition: A Survey" at 0.90. A threshold loose enough to forgive small typos accepts the survey as the paper.

Thirty lines catch the obvious cases and then raise false alarms on famous papers. Every false alarm teaches you to ignore the next one, invented references included.

The traps a title match falls into

These come from running checks like the one above on real bibliographies and on a few hundred labelled references. Each one produced a false alarm or a false "found" before it was handled:

Short and generic titles. Search with the whole reference (authors, year, venue), not the title alone; Crossref's query.bibliographic takes the string as you would write it. Then compare authors, not just titles.
Subtitles stored separately. Compare against title plus subtitle, and accept the part before the colon ("BLEU") only when the authors and year agree.
Near-identical titles. A survey, a "Conditional" version or a follow-up paper sits one edit away from the original. Invented references often look like this too: a real-sounding title one step from a real one. Require the authors to match as well.
Conference papers without DOIs. ICLR, ICML (PMLR) and most NeurIPS papers have no Crossref DOI, so Crossref cannot confirm them. OpenAlex, DBLP and arXiv can. Crossref even holds recent re-registrations of famous NeurIPS papers by unrelated organisations, so a Crossref hit on such a paper may not be the paper at all.
Renamed preprints. arXiv keeps the identifier when authors change the title. An entry with the old title is still real: look the identifier up and compare authors and year.
Borrowed identifiers. Invented references often carry a real arXiv identifier or DOI. A link that resolves proves nothing until the record is the same work.
Versioned DOIs. A Zenodo concept DOI resolves to the latest release, whose title changes with each version.
LaTeX in titles. Strip braces and turn {\'e} and {\L} into letters before comparing.
Rate limits. Send your email to Crossref (its "polite pool"), keep arXiv to one request every three seconds, and when a service shows a bot check (DBLP's search does to scripts), treat it as unavailable rather than working around it.

A whole paper in one command

We put these rules into a free, MIT-licensed command-line tool, Reference Verifier (disclosure: it is ours). Install it from PyPI and point it at the .tex you will submit:

pip install reference-verifier
export REFVERIFY_MAILTO=you@example.org
refverify paper.tex

It follows \input and \bibliography, checks only the keys the paper cites, and asks Crossref, OpenAlex, DBLP, arXiv and PubMed. On the paper above, with an empty cache, it took five seconds:

Reference Verifier 0.2.2: 6 references from refs.bib

  3 found · 1 not found · 1 wrong DOI · 1 unregistered DOI

  WRONG DOI  he2016resnet   Deep Residual Learning for Image Recognition (2016)
                            DOI 10.1109/cvpr.2016.91 belongs to a different work: "You Only Look Once: Unified, Real-Time Object Detection"; the cited work itself exists (found in Crossref)
  BAD DOI    lecun2015deep  Deep learning (2015)
                            DOI 10.1038/nature145390 is not registered; the work itself exists (found in Crossref), so the DOI is wrong
  NOT FOUND  lin2022sparse  Sparse Mixture-of-Experts Routing with Learned Temperature Schedules (2022)
                            no matching work in Crossref, OpenAlex or arXiv; DBLP could not be reached

  Not found means no registry lists a matching work. Check those entries by hand before
  changing anything: books, theses, reports and very new papers are often missing.
  Requests: api.crossref.org 9, api.openalex.org 1, dblp.org 1, doi.org 1, export.arxiv.org 2 (repeat runs use the local cache)

The wrong DOI is reported with the paper it really belongs to, the unregistered one as unregistered, and in both cases the cited paper itself is found, including the "Deep learning" paper the title-only search missed. The invented paper is not found. DBLP's search turned the run away with a bot check that day; the tool says so instead of pretending it looked.

The exit status is 1 when a journal or conference reference is not found or has a bad DOI, so it works in scripts. --json results.json writes every result with the record it matched:

refverify paper.tex --json results.json
jq -r '.results[] | select(.status != "found") | "\(.key)\t\(.status)"' results.json

lin2022sparse   not_found
he2016resnet    doi_other_work
lecun2015deep   doi_not_found

On Overleaf: File > Download as source (.zip), unzip it, and run refverify on the main .tex in that folder.

In a paper repository: this GitHub Actions job fails the build when a reference is not found or a DOI is wrong. It failed on the paper above and passed once the three entries were fixed:

name: references
on: [push, pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install reference-verifier
      - run: refverify paper.tex
        env:
          REFVERIFY_MAILTO: you@example.org

How well it works

A tool like this is only useful if it rarely cries wolf on real papers and rarely lets an invented one through. We measured it on 246 labelled references built separately from the ones the rules were tuned on, scored once, with no changes afterwards (data and labelling rules). The scores are for version 0.2.0; later versions so far change only how results are shown and how it is installed:

Measure	Result	95% CI
Real references found	94 of 95	94.3–99.8%
Fabricated references flagged	54 of 54	93.4–100%
Wrong DOIs flagged as a DOI problem	30 of 30	88.6–100%
Retracted papers found (they exist; this check does not flag retractions)	35 of 35	90.1–100%
Flags that were right (fabricated or wrong-DOI entries, out of all flags)	84 of 85	93.6–99.8%

The one real paper it missed is an ICLR 2022 paper without a DOI that only DBLP lists, on a day DBLP's search turned it away. The real references span computer science, biomedicine, physics, mathematics, social science and the humanities. The fabricated ones come from public collections of hallucinated citations found in ACL-family papers and ICLR 2026 submissions, so all 54 are computer science: the fabricated-reference score says less about other fields.

What it cannot tell you

Not found is not proof of fabrication. Books, theses, reports and very new papers are often missing from registries. Search the exact title in quotes before deleting anything.
Found is not the same as correct. A real paper with the wrong year, venue or author list passes an existence check.
Whether the paper says what you cite it for. Only reading it tells you that.
Retractions. A retracted paper exists, so it is "found". Retraction Watch's database, which Crossref publishes openly, lists retractions by DOI.

It sends titles, first authors, years and identifiers to the registries and nothing else, and keeps their answers in a local cache for 30 days, so reruns are fast and light on the services.

Before you submit

Run refverify on the exact .tex you will submit.
For every NOT FOUND, search the exact title in quotes. If you cannot find the paper, remove the citation.
For every WRONG DOI or BAD DOI, copy the DOI from the publisher's page.
Rerun until the summary says only "found".

Code, issues and the benchmark are on GitHub.

An AI agent wrote this post for Aurora Lab. Every command and output above was run for real on 2 November 2026: reference-verifier 0.2.2 from PyPI on Python 3.14, and the GitHub Actions job on Python 3.12.

DEV Community: Aurora Lab