ORCID's search API has a ceiling it never tells you about

Quick answer

ORCID is the identity registry behind most of scholarly publishing — 20M+ researchers, each with a public profile carrying names, affiliations and employment history. The public API is keyless:

GET https://pub.orcid.org/v3.0/expanded-search/?q=<lucene>&rows=<n>&start=<offset>
Accept: application/json

The interesting part is not fetching it. It is that the query language is full Lucene, the pagination limits are enforced but undocumented in the response, and you can walk straight past the ceiling without anything telling you.

That is the ORCID Researcher Profiles Scraper. Search by name, institution, email or raw Lucene; get flat rows, with optional employment-history enrichment.

We found the limits by making the API angry 🔬

The honest way to learn an undocumented ceiling is to hit it deliberately and read what comes back. So:

• rows accepts 0–1000. Above that, ORCID's own 400.
• start accepts 0–10000 unauthenticated. Above that, 400.
• Six Lucene field names confirmed by sending a deliberately wrong field name and reading the 500 it produced, which names what it expected.

That last one is the technique worth stealing. An API's error messages are documentation it did not mean to write, and they are current in a way the published docs frequently are not.

The practical consequence: an unauthenticated caller can reach 10,000 results per query, maximum. A search for a common surname has more matches than that, so a client that just increments start until the results run out will hit a 400 it did not expect — or worse, stop early and report a complete-looking list. We re-check num-found on every page rather than trusting a stale first read, and we page honestly against the real ceilings.

Never retry a 400 🚦

Three response classes, three different behaviours:

• 429 and 5xx — transient. Retry with exponential backoff.
• 400 — ORCID telling you your query is wrong. Retrying is pure waste, and it buries the one message that would let you fix it. We surface ORCID's own error text instead.
• A malformed Lucene clause is a 400, which is why we build the query from plain fields rather than letting a hand-written clause reach the wire.

The distinction sounds obvious written down. It is routinely got wrong, because "retry on error" is a single tempting branch.

Enrichment is where runs go to die 💀

Employment history means a second request per researcher. On a 500-row list that is 500 more calls, and the failure mode is well known to us: one researcher's record 404s or times out, an unguarded exception propagates, and the entire run dies — after you have already paid for the 499 that worked.

So enrichment is per-item fault-isolated and there is a test that pins exactly that: one failing employment fetch must not take down the run. It is the single most common defect shape in our own fleet, and it is worth writing the test before the feature.

About the fingerprint check 🔐

We recently found an institutional API returning 503 to every Chrome TLS fingerprint while serving Firefox and Safari normally — our anti-blocking measure was the block signal. So every new target now gets probed across profiles before we commit.

ORCID is clean: chrome131, firefox133 and safari17_0 all return 200. No fingerprint sensitivity. Worth measuring rather than assuming, in either direction.

This is data about real people 🧭

Worth stating plainly. Every row describes a named human being, so the Actor emits only what ORCID itself publishes publicly — the fields a researcher chose to make visible on their own profile. Nothing ORCID marks private is fetched, and there is no inference, enrichment-from-elsewhere, or cross-source joining. If a researcher has restricted their employment history, you get a row without it.

What a row looks like

ORCID iD, given and family names, credit name, other names, current affiliation, institution list, and — when enrichment is on — an employment_history array of organization, role, department and start/end dates.

Verified live: affiliationOrgName: "MIT" returns num-found: 7245, and an end-to-end run produced real rows with populated employment history.

Who this is for 🎯

• Academic recruiting — a longlist of researchers at a target institution or department.
• Research ops — institutional and funder reporting.
• Publishers — reviewer and author identity resolution.
• Scientific sales — verified ORCID iDs and public affiliation history on an outreach list.

The honest limitations 🚧

• 10,000 results per query unauthenticated. Narrow the query; we do not pretend the ceiling is not there.
• Public fields only. A restricted profile stays restricted.
• Employment enrichment costs one extra request per researcher — slower, and worth turning off when you only need identities.

Pricing

$0.20 per run plus $0.003 per profile — $3.20 per 1,000. Pay for rows that land.

→ ORCID Researcher Profiles Scraper on Apify

---

Built by Devil Scrapes. We handle the Lucene syntax, the ceiling nobody documents, the 400 you must not retry, and the one bad record that should never kill a run.