A reviewer asked for a citation. I could not tell whether it was real. That was the problem.

A draft I was reviewing had 41 references. 38 resolved. Reference 22 looked fine and returned nothing.

Zhang, L. et al. (2021). Attention mechanisms for protein folding. Nature Structural Biology, 28(3), 214-221.

The formatting was right. The journal is real. The volume and page range were in the right shape. The DOI pattern was well-formed. And I could not find the paper. Not "could not access", I searched the journal's archive, Crossref, and Google Scholar, and there is no such article.

Somebody's LLM had produced a structurally perfect citation to a paper that does not exist. And here is what made it dangerous: I could not prove it was fake either. Absence of evidence is not evidence of absence, and I was not going to email the journal asking whether a paper I half-remembered is real.

That near-miss is why I built PaperForge, a CS/AI submission-readiness checker. Its whole job is the question I could not answer that afternoon: is this citation real, and if I cannot tell, show me exactly what I could not confirm.

The gap I had to close

I had been reviewing AI-assisted drafts for two years with a checklist. It covered formatting, venue fit, and whether the argument matched the claimed contribution. It had exactly one item for references, and the item was "does the citation format look correct."

That item is worthless. A format check passes everything a generator invents.

I needed a different property: can a human verify this reference exists and is what it claims, using only tools they already have? So I rebuilt the check around three things I could do reliably.

DOI resolution. A DOI that resolves to metadata means the work exists, and lets me compare title and authors to what was claimed. This catches invented references outright.

Author-and-title search. For arXiv, ACL Anthology, and CVF-style open venues, I can look the reference up by title instead of trusting the identifier.

An explicit unverified list. When neither works, the reference does not get deleted and does not get flagged as fake. It gets listed as unverified, with the exact string I searched for. The author then checks it themselves.

What the rubric looks like now

100 points, and the reference section is a fixed slice of it:

Check	What it actually verifies
DOI / arXiv / ACL ID resolves	The work exists
Title on the record matches the cited title	The citation points at the right paper
First author surname matches	Not a different paper by a similar name
Venue and year agree with the record	No citation drift
Unverifiable items listed, not removed	The author makes the call

The last row is the one I care about most. A tool that silently drops a reference teaches the author to trust it blindly. A tool that hands back a list teaches the author where the model's knowledge stopped.

I have watched this pattern break in a different place too: a validator that "fixes" a regex turns a reviewable four-line problem into an opaque one. I hit that exact situation in April, a four-line catastrophic-backtracking pattern that three checks (ESLint, Semgrep, me) all passed, took down our image pipeline for six hours, and was fixed by two characters. The lesson generalises: surface the problem rather than resolve it invisibly.

What the check actually runs

The unverified list is produced by three lookups in order, and the first one that answers wins:

def verify(ref):
    # 1. DOI resolution - strongest signal: metadata proves the work exists
    meta = resolve_doi(ref.doi)
    if meta:
        return {
            "status": "verified",
            "title_match": norm(meta.title) == norm(ref.title),
            "author_match": meta.first_author.split()[-1] == ref.surname,
            "year_match": meta.year == ref.year,
        }

    # 2. Title search on open venues - arXiv, ACL Anthology, CVF
    hit = search_title(ref.title, venues=["arxiv", "acl", "cvf"])
    if hit:
        return {"status": "verified", "via": "title-search", "record": hit}

    # 3. Not machine-checkable. Do NOT delete, do NOT call it fake.
    return {"status": "unverified", "searched": ref.title}

The branch that matters is the last one. It returns unverified with the exact string that was searched, so the author can paste it into a database. A tool that removes the reference teaches the author to trust the tool; a tool that lists it teaches the author where the model's knowledge ended.

One thing I got wrong first: I initially dropped unverified references from the count, which made the rubric look better than reality. Now they count against the score, because a draft with five unverified citations is a draft with five unverified citations.

The venue overlays matter for a narrower reason than people expect. A checklist that is generic will be satisfied by a paper aimed at the wrong venue, the format looks right and the fit is wrong. Overlays for NeurIPS, ICLR, ICML, ACL Rolling Review and CVPR-class venues encode the things that actually differ: page limit, anonymisation, whether the ethics statement is required, whether supplementary is allowed at submission. A generic conference gets the generic rules.

The uncomfortable part

Every method above has a false-negative surface, and I will not pretend otherwise.

• A real, published paper behind a paywall with a DOI that does not resolve in a plain HTTP request looks identical to a fake one.
• A workshop paper that was never archived looks fabricated.
• A paper I cannot find may exist in a venue I did not check.

So the unverified bucket is genuinely a bucket, not a verdict. It means "not machine-checkable from here", and the distinction matters enormously when a student's submission hinges on it.

What PaperForge is not

It does not predict acceptance. The score measures checklist compliance against venue policy overlays, and that is the whole claim. No tool can tell you whether a paper will be accepted, and any product that says it can is either lying or has access to a reviewer, which would be worse.

It does not generate citations. Ever. Fabricating references is not a feature gap, it is the failure mode. If a tool invents a reference, that is a bug of the worst class, and no amount of nice formatting elsewhere compensates for it.

It does not replace reading the paper. It replaces the question "is this citation real?" with the smaller, answerable question "is this citation machine-verifiable, and if not, here is exactly what I could not confirm."

The unverified list is the product. Everything else is convenience. If you review AI-assisted drafts, PaperForge is the part I built to honestly say "I don't know which of these are real" instead of guessing.