Crossref's funder filter wants a DOI, not a name — and 3 more traps

Crossref's works endpoint is about as friendly as a public API gets: no key, no rate-limit paperwork, 150 million-plus DOIs, and documentation that mostly matches reality. We shipped a scraper against it in an afternoon.

It still had four traps in it, and every one of them is the kind that passes a smoke test and fails on a customer's first real query.

Quick answer

The four things that will bite you scraping Crossref: filter=funder: accepts a Funder Registry DOI, not a funder name; dates arrive as date-parts arrays that are legitimately 1, 2, or 3 elements long; author names come in four different shapes; and next-cursor can hand you the same cursor forever. None of them raise an exception. All four produce plausible, wrong output.

Why scrape Crossref instead of Google Scholar?

Because Crossref will let you. It is the DOI registration agency behind most of scholarly publishing, and the metadata publishers submit — title, journal, authors, ORCIDs, funders, award numbers, licence, citation counts — is exposed through one keyless JSON endpoint on purpose. There is no bot detection to defeat, which means the engineering budget goes into being correct rather than into being invisible.

The polite convention is worth honouring: put a contactable identifier in your User-Agent and Crossref routes you to a better-served pool.

User-Agent: crossref-works-scraper/0.1.0 (https://apify.com/DevilScrapes)

That is a brand contact URL, not a personal mailbox. Works the same, leaks nothing.

Trap 1: the funder filter wants a DOI

This looks like it should work:

GET /works?filter=funder:Wellcome%20Trust

It returns 200. It returns works. It does not return works funded by the Wellcome Trust — funder: is matched against the Funder Registry DOI, so a free-text name matches nothing useful and the API does not consider that an error. You get a full, confident, wrong result set.

We split the input on shape instead of trusting the user to know:

FUNDER_DOI_RE = re.compile(r"^10\.\d{4,9}/")

def funder_is_doi(self) -> bool:
    return bool(self.funder_name and FUNDER_DOI_RE.match(self.funder_name))

A DOI goes to Crossref as a server-side filter. A name becomes a client-side, case-insensitive substring match over each work's funder list. Same field, two paths, and "Wellcome" finds the works whether the publisher wrote Wellcome Trust or The Wellcome Trust.

Trap 2: date-parts is not a date

Crossref publication dates look like this:

{ "published": { "date-parts": [[2021, 3, 14]] } }

Except when the publisher only registered a year and a month. Or only a year. All three are valid, all three are common, and the array length tells you which you got:

if len(parts) == 1:
    return f"{parts[0]:04d}"              # 2021
if len(parts) == 2:
    return f"{parts[0]:04d}-{parts[1]:02d}"   # 2021-03
return f"{parts[0]:04d}-{parts[1]:02d}-{parts[2]:02d}"

The tempting shortcut — datetime(*parts) with defaults — invents a 1 January that no publisher ever claimed. That is worse than a null, because a null is visibly missing and a fabricated date silently poisons every date-range query downstream.

There is a second half to this: the date lives under different keys depending on the record. We walk a fallback chain — published, then published-print, then published-online, then issued — and take the first one that renders.

Trap 3: authors have four name shapes

A Crossref author entry can carry given + family, only family, a literal string (common for consortia and corporate authors), or a bare name. Read only given/family and every consortium byline in your dataset comes back empty.

def _author_name(entry):
    given, family = entry.get("given"), entry.get("family")
    if given and family:
        return f"{given} {family}"
    return family or given or entry.get("literal") or entry.get("name")

ORCIDs have the same shape problem in miniature — sometimes bare, sometimes prefixed with https://orcid.org/. Store one form, always, or your joins will quietly miss half the matches.

Trap 4: the cursor that never ends

Deep paging on Crossref is cursor-based, and that is genuinely the right design — offset paging on a 150M-record corpus hits a wall a few thousand rows in. You pass cursor=*, and each response hands back message.next-cursor.

The failure mode is that next-cursor is not guaranteed to advance. Hand back the same token twice and a naive while next_cursor: loop pages forever, re-emitting the same block of works and — on a pay-per-result Actor — billing the customer for every duplicate.

One line of defence:

if not cursor or cursor == previous_cursor or emitted >= cfg.max_results:
    break
previous_cursor = cursor

Three independent exits: no cursor, a stalled cursor, or the row budget spent. A paging loop should never have only one way out.

What about a single malformed record?

It should cost you that record and nothing else. The general lesson from our fleet is that the top cause of a low-success scraper is not a hard block — it is a recoverable per-item error that takes down the whole run. Crossref metadata is publisher-submitted and therefore inconsistent by nature, so parsing is wrapped per item:

def _safe_parse(work, log):
    try:
        return parse_work(work)
    except ValidationError:
        ...  # log it, skip it, keep going

Nine hundred and ninety-nine good rows plus one weird preprint should be 999 rows delivered, not a failed run.

FAQ

Is scraping Crossref legal?
Crossref publishes this metadata through a public API for exactly this kind of reuse, and most of it is explicitly open. Honour the polite-pool convention, do not hammer it, and check the licence field on anything you redistribute.

Do I need an API key or a proxy?
Neither. It is keyless, and a public API has no reason to block a well-behaved client. Our Actor leaves the proxy off by default.

How deep can I page?
As deep as the result set goes, if you use the cursor. Offset paging is what has a practical ceiling.

Why are some works missing abstracts, ISSNs or funders?
Because publishers did not submit them. Scholarly metadata is sparse by default — build your schema to accept nulls everywhere rather than assuming a fully-populated record.

---

We packaged all of this as a ready-to-run Actor: Crossref Works Scraper — search by bibliographic query, publication-date range, work type, publisher, funder, or ORCID/abstract/full-text presence, and get one flat row per work with authors, ORCIDs, funders, award numbers, licence and citation count. Exports to JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈