Do published reference lists still cite retracted papers? A dated OpenAlex audit

#showdev #api #python #research

Dated: 2026-10-01 (gate 05:13Z, old-lists 07:10Z, seeded control 09:50Z — same day, same pipeline version).
Pennyforge research artifact. $0 data (PubMed eutils + OpenAlex polite pool + Crossref production API). All data and the per-DOI results are available on request; the whole pipeline is four small scripts.

The question

Every paper's reference list is a small, checkable corpus of claims that other works exist and carry authority. A retracted paper still cited in a published list = silent authority leakage. We tested whether OpenAlex's is_retracted flag is good enough to audit real published reference lists — first on a gold set (gate), then on live 2019–2025 lists, and finally with a seeded control that proves the pipeline catches a real case by construction.

1. Gate: OpenAlex vs a Retraction Watch gold set

Gold set: Retraction Watch dataset via Crossref (72,790 rows, generated 2026-09-30, daily-updated). Sample: 120 retractions, 10 per publication year 2015–2026, deterministic stride.
OpenAlex: 120/120 found, 115/120 flagged retracted → 95.8% recall on the sample.
Crossref (authoritative source, updated-by retraction events): 117/120 found, 111/120 carry a retraction event → 7.5% of even the gold-set retractions are missing from the authoritative source itself (inherited gaps the audit cannot fix).
Agreement OpenAlex vs Crossref (117 both found): 116/117 = 99.1%.
Practical note: OpenAlex exposes only a boolean is_retracted — no retraction date. Dated notices come from Crossref updated-by (retraction-watch records, production schema since 2025-01).

2. Live lists: 0 hits, honestly framed

2025 lists (run 138): 0 retracted references in 322 DOIs.
2019–2020 lists (run 139): 8 published systematic-review reference lists (PubMed), 812 references, 606 unique DOIs, 599 resolved (98.8%) → 0 flagged retracted.
Honest frame: a retraction base rate near ~1% makes 0/928 an upper bound (~0.3% at 95% CI), not evidence of absence. The live-list pass is a negative within its confidence interval — exactly what a dated audit should report.

3. Seeded control: the pipeline finds a real case by construction

To prove the pipeline is not blind, we seeded the hardest case: a retracted, well-cited review that must still appear in a recent list.

Seed: 10.1038/s41598-023-28418-1 — "RETRACTED ARTICLE: Exogenous melatonin alleviates neuropathic pain-induced affective disorders…" (Sci Rep 2023, cited_by 50).
Citer: 10.1007/s11064-026-04725-7 — Mol Neurobiol 2026 (PMID 41824110); its published reference list contains the seed's DOI.
Result: 29 refs / 28 DOIs, 28/28 resolved; seed flagged is_retracted: true and Crossref dated retraction notice 2026-08-10; 0 false positives in the same list.
Cost: confirmation on the 21st (seed, citer) attempt — the pipeline's confirmation rate is ~1-in-20 because it only sees a retraction when the citer's reference text carries the seed's DOI string.

What this is worth

A reproducible, $0, API-only audit of any published reference list: gate 95.8% recall / 99.1% agreement, dated notices via Crossref, and a constructive proof the pipeline fires. It converts "is this citation still trustworthy?" from a manual hunt into a batch job.

Honest limitations (the ones that matter)

DOI-in-text only: the audit flags a retraction only if the cited DOI appears in the reference list text. ~1-in-20 (seed, citer) confirmation rate (21 attempts to one hit).
No dates at OpenAlex: dates require the Crossref updated-by cross-check (adds a request per flagged DOI).
Inherited gaps: the authoritative source (Crossref updated-by) itself misses 7.5% of the gold-set retractions.
Zero-hit passes are upper bounds, not absences (0/928 across both live passes).
Pipeline, not product: per-list latency and PubMed-coverage bias (non-indexed papers absent) apply.

Run the audit on a fixed weekly batch (new review lists per PubMed date) → dated series, first point = this artifact.
Extend seed sourcing to OpenAlex's own retracted, high-citation pool (done — it's what produced the passing seed).
Optional: a public "retracted-citation finder" form (paste a DOI, get its list audited).

This work was done by Pennyforge, a one-person studio: the analysis was AI-assisted with human editorial review, as we practice it across all our research posts. All APIs public; per-DOI results available on request.

DEV Community