The day our scraper fleet shipped nothing — and the afternoon it shipped three

We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of release automation worked, and the day ended with zero listings live. That same afternoon, three different scrapers went from empty directory to published in about an hour each, first attempt, no retries.

The difference was not effort, tooling, or engineering skill. It was what we pointed at.

Quick answer

If your scraping backlog is a list of consumer marketplaces, your throughput is capped by someone else's bot-detection budget, not by your own velocity. Keyless public JSON APIs — OpenAlex, Crossref, GBIF, openFDA, World Bank, SEC — ship on the first attempt because there is nothing to block. We now sort the backlog by target class before we sort it by revenue, and a walled target only enters a build wave with a specific tested plan for its defences.

The day that shipped nothing

Nine Actors sat in the publish queue. All nine were consumer marketplaces and job boards with commercial anti-bot in front of them. Every one was cloud-QA'd on datacenter proxy and on residential proxy. All nine failed.

That is a real result and worth writing down, because "we were blocked" is a useless note. The useful version names the tiers:

9/9 failed cloud QA — datacenter AND residential, same day, same fixtures

What made it expensive was that the failure was invisible from inside the process. Every stage worked. The specs were fine, the tests were green, the pricing wired up correctly, the publisher refused correctly. The lane was healthy and the output was zero, which is the worst combination to debug because nothing is flashing red.

The afternoon that shipped three

We changed one variable: target class. Instead of marketplaces, we took three public research and regulatory APIs that require no key. Scaffold, implement, cloud QA, publish. Roughly an hour each, all three passed cloud QA on the first run.

There is no clever trick here. A public JSON API has no incentive to fingerprint you, no CAPTCHA, no TLS fingerprint check, no HTML that gets rewritten on Tuesday. The whole class of failure that eats marketplace scrapers simply does not exist. What is left is ordinary engineering: paging, schema, and error handling.

Why is OpenAlex a good scraping target?

OpenAlex is the open replacement for Microsoft Academic Graph — roughly 250 million scholarly works, with authors, institutions, journals, citation counts and open-access status. It is free, it needs no API key, and it is genuinely well-behaved if you are.

Two things matter in practice.

Use cursor paging, not offset. Offset paging on a corpus this size falls apart past the first few thousand rows — the API caps how deep you can page and the cost of a deep offset grows. The cursor is the supported path:

GET https://api.openalex.org/works?per-page=200&cursor=*

Each response hands you meta.next_cursor; you pass it back until it comes back null. Stateless, stable under concurrent updates, and it does not degrade at depth.

Identify yourself and get the fast lane. OpenAlex runs a "polite pool" — include a contact in your User-Agent or a mailto parameter and your requests go to a separate, faster, more reliably-served pool. This is the rare case where telling the server who you are makes your scraper better, not more blockable. We send our public contact URL, never a personal address.

The trap that survived the easy target

Keyless does not mean trivial. The bug that cost us the most time on this batch had nothing to do with blocking.

Apify Actors can declare a dataset schema, and ours declared fields like this:

{ "doi": { "type": "string" } }

That is wrong, and it is wrong in the worst possible way: it only fails on a sparse record. Scholarly metadata is extremely sparse — plenty of works have no DOI, no abstract, no ISSN, no funder. The first record missing one of those fields produces a null, the null fails validation, and push_data dies mid-run. A fully-populated QA sample never triggers it, so the smoke test passes and the customer's real query is what breaks.

The fix is one character of JSON per field:

{ "doi": { "type": ["string", "null"] } }

We fixed it at the scaffolder so no future Actor can be born with it. The general lesson: a QA fixture that is too clean is worse than no fixture, because it converts a loud failure into a delayed one.

What this means for your backlog

Sort by class first:

1. Keyless public APIs — ship first-attempt. Government registries, research indexes, open data portals, standards bodies. Low glamour, real buyers, and they do not fight back.
2. Plain HTML sites with no commercial anti-bot — ship with ordinary care.
3. Commercially defended marketplaces — only with a specific, tested plan for their defences, and a named tier you have actually probed. Never on optimism.

We ran category 3 on optimism for a long time. The scoreboard was honest about it exactly once, on the day it produced a zero.

What the scraper actually does

OpenAlex Works Scraper searches the 250M-work catalogue and returns one flat row per work — title, DOI, authors with ORCIDs and affiliations, journal, publication date, citation count and open-access status — as JSON, CSV or Excel. It is pay-per-result at $2.05 per 1,000 rows, so a query that returns nothing costs you nothing beyond the start fee.

We are not going to pretend that any source stays static forever. APIs deprecate fields, tighten rate limits and reshape their pagination, and when OpenAlex does, absorbing that is our job, not yours — the cursor handling, the retries and the schema drift are on our side of the line.

If you work the same corpus from other angles, the neighbours are Crossref-adjacent research tooling, arXiv and EU CORDIS grants.

The rule we wrote down

A backlog sorted only by revenue potential will happily spend a month on targets that cannot ship. Sort by target class first, then by revenue inside each class.

That is the whole finding. It cost us a zero-release day to learn, and it is the cheapest thing in this post to copy.