DEV Community

Devil Scrapes
Devil Scrapes

Posted on

The Zenodo API Caps You At 25 Results — Above That It's Just 400

#webscraping #python #api #apify

Zenodo's API caps you at 25 results per page — until you're logged in, and the error message doesn't say that until you hit it.

Quick answer

Zenodo's /api/records endpoint accepts size up to 25 for an unauthenticated request and answers with a plain 400 above it — verified live: size=25 returns 200, size=26 returns 400. There's a second, separate ceiling: page * size can't exceed 10,000 records deep, confirmed the same way (page=400 at size=25 is fine, page=401 isn't). Neither limit is mentioned until you cross it, and a scraper built against a small QA sample (3 rows) will pass every test and then 400 on page 1 the first time a real customer asks for more than 25.

Why did this ship wrong the first time?

It shipped with PAGE_SIZE = 100, and nothing caught it locally, because the request builder always takes the smaller of the page size and what's actually needed:

def build_params(cfg, page):
    params = {
        "size": min(PAGE_SIZE, cfg.max_results),
        "page": page,
        "sort": "mostrecent",
    }
    ...
Enter fullscreen mode Exit fullscreen mode

A 3-row smoke test asks for size=3 and gets 200. The bug only reproduces once a real customer asks for 50+ records — exactly the run that should have worked. We caught it live on 2026-08-20 and dropped PAGE_SIZE to 25 with a regression test pinned to the actual boundary:

def test_build_params_never_exceeds_zenodo_unauthenticated_page_cap():
    assert PAGE_SIZE <= 25
    for max_results in (1, 25, 26, 100, 5000):
        cfg = ActorInput(searchQuery="climate", maxResults=max_results)
        assert build_params(cfg, 1)["size"] <= 25
Enter fullscreen mode Exit fullscreen mode

Why does the same client work from a laptop and 403 from the cloud?

Zenodo answers 403 to a client with no identifying User-Agent, and a bare curl-cffi session with no headers set often looks unidentified enough to trip that — a failure mode that only ever showed up running from an Apify cloud IP range, not locally, which is exactly the gap cloud QA exists to catch (run eMlNRb2litUQB5fhj):

API_HEADERS = {
    "user-agent": "DevilScrapes/1.0 (+https://apify.com/DevilScrapes)",
    "accept": "application/json",
}
Enter fullscreen mode Exit fullscreen mode

We also rotate TLS impersonation across requests — Chrome, Firefox, Safari profiles — and retry 408/429/500/502/503/504 with exponential backoff, capped at 30 seconds over 5 attempts, on top of the identifying header.

Why does the same field show up twice in one record?

Zenodo duplicates several fields — title, doi — at both the top level of a record and again nested inside metadata, and the two don't always agree on which one is populated. Rather than pick one and hope, we take the first non-empty value from either:

def _first(*values):
    """First non-empty value — Zenodo duplicates fields across two levels."""
    for value in values:
        if value not in (None, "", [], {}):
            return value
    return None

title=_first(record.get("title"), metadata.get("title")),
doi=_first(record.get("doi"), metadata.get("doi")),
Enter fullscreen mode Exit fullscreen mode

resource_type gets its own unwrap: it's a three-key object (title, type, subtype), not a string, so we split it into two flat columns — resource_type and resource_subtype — rather than shipping a nested object a spreadsheet can't sort on.

What actually stops a run — and what doesn't?

An empty page of results just ends the run cleanly. Hitting the 10,000-record paging ceiling logs a warning and stops rather than raising:

if page * PAGE_SIZE > RECORD_CEILING:
    log.warning(
        "reached Zenodo's %d-record paging ceiling — narrow the query to go deeper",
        RECORD_CEILING,
    )
    break
Enter fullscreen mode Exit fullscreen mode

A search that runs and genuinely matches nothing finishes as a success with zero rows — the only thing that fails loud is Zenodo answering with a retry-exhausted or unexpected HTTP status on the search itself.

FAQ

Do I need a Zenodo API key or account?
No — it's a public, keyless JSON API, and this Actor never asks you to authenticate.

Can I search for a specific resource type only?
Yes — resourceType restricts to one of Zenodo's own types (dataset, publication, software, poster, presentation, image, video), or leave it any.

How deep can I page?
Up to 10,000 records for a given query, in pages of 25 — Zenodo's own unauthenticated ceiling, not a limit we impose.

Does it return closed-access records too?
By default yes; set openAccessOnly to restrict to records whose files are openly downloadable.

Packaged and ready to run: Zenodo Records Scraper — search Zenodo's open-science repository by keyword, resource type, and access rights, and get DOI, authors, license, file count, and download/view stats as one flat row per record — JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

Top comments (0)