DEV Community: Devil Scrapes

A fake 404 and a 403 that isn't a ban — the UK food ratings API

Devil Scrapes — Thu, 20 Aug 2026 04:12:06 +0000

We probed the UK Food Standards Agency's ratings API and got a 404 that said the API doesn't exist. It exists. Then we fixed that and got a 403 that looked exactly like an IP ban. It wasn't. Both responses are the API working as designed — and both designs are invisible until you've lost an afternoon to them.

Quick answer

Building on api.ratings.food.gov.uk? Two headers-and-habits rules:

Send x-api-version: 2 on every request. Without it the API answers 404 "The API 'Establishments' doesn't exist" — a message engineered to make you re-check your URL forever. The URL is fine; the header is missing.
Never query /Establishments bare. An unfiltered query returns 403 "This is a CPU intensive query: please use one of the documented filters". That's not a ban and not rate limiting — it's a query-cost guard, and it means a page-through-everything scraper design is dead on arrival. Filter by local authority, name or address on every request.

The 404 that means "missing header"

Here's the trap in its natural habitat:

GET https://api.ratings.food.gov.uk/Establishments?name=pizza
→ 404 {"Message": "The API 'Establishments' doesn't exist"}

GET https://api.ratings.food.gov.uk/Establishments?name=pizza
    x-api-version: 2
→ 200, establishments with ratings

Same URL, same method, opposite outcome. A 404 with a body claiming the API doesn't exist reads like a retired endpoint — we initially logged it as one. Every dead-endpoint conclusion about this API that doesn't mention the version header is wrong.

Why does the FSA API return 403 on a valid query?

Because the query is valid and expensive. The bare Establishments listing would walk every food business in the UK, so the API refuses with:

403 {"Message": "This is a CPU intensive query: please use one of
     the documented filters in your query (e.g. filter by LocalAuthority)."}

The failure mode this creates for scrapers is nasty: a 403 pattern-matches to "we've been blocked", which sends you down the proxy-rotation rabbit hole. No proxy fixes this, because nothing is blocked — the request shape is the problem. Narrow the query (local authority ID, business name, address, rating value) and the same client on the same IP gets a 200.

What data is actually in there?

Every food business establishment in England, Wales and Northern Ireland (plus Scotland's parallel scheme): business name and type, full address, the hygiene rating (0–5, or Scotland's pass/fail), the three inspection sub-scores (hygiene, structural, confidence in management), the rating date and the local authority. It's the dataset behind the green sticker in every UK café window — and it's genuinely useful for hospitality lead-gen, franchise diligence and food-safety monitoring.

What the scraper actually does

UK Food Hygiene Ratings Scraper queries the FSA ratings register with the version header, the mandatory filters and paging handled for you, and returns one flat row per establishment — name, type, address, rating, sub-scores, inspection date, authority — as JSON, CSV or Excel. Pay-per-result at $2.05 per 1,000 establishments; an empty match costs only the start fee.

We're not going to call this API easy: it hides behind a lying 404, refuses honest queries with a scary 403, and paginates region by region. Absorbing that is the product. 😈

For neighbouring lead-gen sources we also run UK company data via GLEIF LEI records, Norway's company register and BBB business leads.

The one-line version

On the FSA ratings API: x-api-version: 2 or every response is a fake 404, and always send a filter or the 403 you get isn't the ban it impersonates.

The API where page 21 is a 400 and 'no results' is a 404

Devil Scrapes — Thu, 20 Aug 2026 04:11:35 +0000

The Federal Register API has a results window that ends at document 2,000 — and the error you get when you cross it is an HTTP 400 that looks like your bug. It also answers a perfectly healthy "no matches" query with a 404, which most HTTP clients treat as "the endpoint is gone."

Both behaviours are real, both are undocumented where you'll actually look, and both will pass every test you write against a happy-path fixture.

Quick answer

Building on www.federalregister.gov/api/v1/documents.json? Three rules:

You cannot page past 2,000 results. The API returns HTTP 400 once page × per_page crosses 2,000. Partition your query by date range and page inside each slice.
A 404 means "no documents matched", not "dead endpoint". Treat it as an empty result set, not an error.
per_page maxes at 100. Anything higher is clamped or rejected, so a 10,000-document sweep is at minimum 100 requests before you even hit the window ceiling.

The 2,000-document window

A search for a common regulatory term matches tens of thousands of documents — we measured 10,000+ hits on a single term during our probe. The API happily reports that total, then refuses to show you most of it:

page=20, per_page=100  → HTTP 200   (document 1,901–2,000)
page=21, per_page=100  → HTTP 400   (document 2,001+ — the window is shut)

The fix is the same one every deep-paging API forces eventually: partition by publication_date, run each slice under the 2,000-document ceiling, and stitch. A scraper that silently stops at 2,000 and exits green is lying to its user — ours logs the ceiling and narrows the date slices automatically.

Why does the Federal Register API return 404 on a valid query?

Because that's its way of saying "zero documents matched". A conditions query with no hits doesn't return {"count": 0, "results": []} — it returns HTTP 404. If your client raises on any 4xx (the default in most libraries), your pipeline dies on the first quiet news day, and the stack trace points at your URL builder, not at the API's convention.

The correct behaviour is exactly what our client does: catch the 404, emit zero rows, exit clean. An empty result set is an answer, not a failure.

Is the Federal Register API free?

Yes — fully keyless, no registration, no OAuth. Which is why the whole difficulty of the target is the two conventions above plus schema discipline: documents omit fields freely (agencies, abstracts, page counts come and go per document type), so every optional field in your output schema needs to be nullable or your dataset writer dies mid-run on the first sparse record.

What the scraper actually does

Federal Register Scraper searches the Federal Register — proposed rules, final rules, notices, presidential documents — by term, agency, document type and date range, and returns one flat row per document: document number, title, type, abstract, agencies, publication date, comment deadline and the canonical HTML/PDF URLs, as JSON, CSV or Excel. Pay-per-result at $2.05 per 1,000 documents; a query that matches nothing costs you only the start fee.

The paging window, the 404-means-empty convention, the sparse-field schema — absorbing those is our job, not yours. 😈

If you track the regulatory pipeline end-to-end, we also run FDA recalls, FDA 510(k) clearances and SEC XBRL financials.

The one-line version

On the Federal Register API: 404 means "no matches", 400 past document 2,000 means "partition by date", and every optional field is optional. Three rules, whole integration.

The API where 200 OK means your request was invalid

Devil Scrapes — Tue, 18 Aug 2026 17:35:02 +0000

The World Bank's v2 API is free, keyless, and serves 20,000+ development indicators for every country on earth. It is also the only API I have met this year where a successful response is not a JSON object, and where asking for an indicator that does not exist returns HTTP 200.

Quick answer

World Bank v2 returns a two-element array, not an object: [{pagination meta}, [...rows]]. When your indicator code is wrong it returns HTTP 200 with a one-element array carrying a message block instead of rows. So the only safe way to consume it is to branch on array length before you look at anything else — status code and try: rows = data["data"] will both lie to you.

The wire format

Here is a healthy response, trimmed:

[
  { "page": 1, "pages": 66, "per_page": 1, "total": 66, "lastupdated": "2026-07-13" },
  [
    { "indicator": {"id": "NY.GDP.MKTP.CD", "value": "GDP (current US$)"},
      "country": {"id": "BR", "value": "Brazil"}, "countryiso3code": "BRA",
      "date": "2025", "value": 2279920092492.13 }
  ]
]

Element 0 is pagination. Element 1 is the rows. There is no envelope key, no data, no results. Anyone who has written a client against a modern REST API will reach for response.json()["data"] and get a TypeError on a working request.

That part is merely annoying. Here is the part that costs you a customer.

Why does an invalid indicator return 200?

Ask for an indicator code that does not exist and the API answers:

[ { "message": [ { "id": "120", "key": "Invalid value",
                   "value": "The provided parameter value is not valid" } ] } ]

Status: 200 OK. One element, not two. No exception, no error field your HTTP layer will notice.

Chain that with the naive parse and the outcome is a run that completes successfully and emits zero rows. On a pay-per-result Actor that is the worst possible failure: the customer is charged a start fee, sees a green run, gets an empty dataset, and has no idea whether their query was wrong or our scraper was.

So the shape check happens first, in one place, before pagination or retries get a say:

INVALID_INDICATOR_KEY = "message"

def _parse_envelope(payload):
    if not isinstance(payload, list) or not payload:
        raise RuntimeError("unexpected World Bank response shape")
    head = payload[0] or {}
    if INVALID_INDICATOR_KEY in head:
        raise RuntimeError(f"World Bank rejected the request: {head[INVALID_INDICATOR_KEY]}")
    ...

Isolating it means the paging loop never has to guess. Two shapes exist; exactly one function knows about both.

Sparse by design: `value` is legitimately null

Development data has holes — a country did not report, a series starts in 1990, a war interrupted collection. value: null is the normal case, not corruption.

This matters for Apify Actors specifically, because a dataset schema that declares

{ "value": { "type": "number" } }

will pass a QA run on Germany's GDP and then die on the first missing year in a real query. The fix is one character:

{ "value": { "type": ["number", "null"] } }

We now treat "does this field survive a sparse record?" as a schema review question on every Actor, because the failure only ever reproduces on customer data.

One row per country, indicator and year

The other design choice worth stating: the API nests indicator and country as objects, and a request can span many countries and years at once. Consumers almost always want a flat table.

So the output is fully flattened — country_name, country_id, indicator_id, indicator_name, year, value — one row per (country, indicator, year). That drops straight into a spreadsheet, a pandas frame, or a BI tool with no unnesting step, and it makes the billing unit legible: one row is one data point.

Is scraping the World Bank API legal?

The World Bank publishes this data under an open licence and documents the API for public reuse — it is about as unambiguous as open data gets. Standard care still applies: request only what you need, and cite the source when you redistribute.

FAQ

Do I need an API key?
No. It is keyless and unauthenticated, and there is no bot detection to work around.

What does per_page max out at?
We page at 1000, which the API accepts comfortably. Watch pages in element 0 of the envelope rather than guessing when to stop.

How do I find an indicator code?
GET /v2/indicator/{code} is the catalogue endpoint — it also confirms an indicator exists before you page its data, which is the cheapest way to fail fast on a typo.

Why is my dataset smaller than countries × years?
Because the missing combinations were never reported. Absence of a row is real information about the series, not a scraper bug.

Packaged and ready to run: World Bank Indicators Scraper — pick any indicators (GDP, population, CO2, poverty, and 20,000+ more), any countries, any year range, get one flat row per country/indicator/year as JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

A 404 from SEC's XBRL API is usually the correct answer

Devil Scrapes — Tue, 18 Aug 2026 17:34:24 +0000

SEC's XBRL API is the best-kept secret in financial data. No key, no scraping of HTML filings, no vendor contract: structured, comparable, machine-readable financial facts for every company that files with the SEC, straight from the source.

Then you ask two companies for the same concept and one of them 404s. Nothing is broken. That 404 is the most important thing to understand about the whole API.

Quick answer

A 404 from SEC's XBRL API usually means "this filer did not report this concept", not "your request was wrong." Most (company, tag) and (tag, period) combinations legitimately do not exist, because companies report under different US-GAAP concepts. So the architecture question is not how to avoid 404s — it is fault isolation: one missing combination must skip one row, never sink the run. Plus three operational musts: a descriptive User-Agent (SEC's Fair Access policy blocks you without one), 10 requests/second, and CIKs zero-padded to 10 digits.

Two endpoints, two questions

The API answers two different shapes of question and it is worth picking deliberately.

frames — one concept, one period, every filer:

GET /api/xbrl/frames/us-gaap/Assets/USD/CY2023Q1I.json

That is a cross-section: total assets for every company as of Q1 2023. Perfect for screening and peer comparison.

companyconcept — one filer, one concept, full history:

GET /api/xbrl/companyconcept/CIK0000320193/us-gaap/Revenues.json

That is a time series. Perfect for trend work on a specific company.

Same data, transposed. Frames mode gives you a wide market snapshot; company mode gives you depth on names you already care about.

The frame id format is stricter than it looks

Frame ids look casual and are not:

CY2023      annual
CY2023Q1    quarterly (duration)
CY2023Q1I   instant

That trailing I is the difference between a flow and a stock. Revenue is a flow — it happens over a quarter, so it lives in CY2023Q1. Assets are a stock — they exist at a moment, so they live in CY2023Q1I. Ask for assets without the I and you get a 404 that has nothing to do with the company and everything to do with accounting.

We validate the id shape at the input boundary rather than letting SEC's 404 be the error message:

FRAME_PERIOD_RE = re.compile(r"^CY\d{4}(Q[1-4])?I?$")

A malformed period is a mistake we can name precisely. A 404 is not.

Why most 404s are correct answers

US-GAAP is a large vocabulary and a filer only reports the concepts that describe its business. Here is the same tag against two filers, checked live:

CIK0000320193 (Apple)      InterestAndDividendIncomeOperating -> 404
CIK0000019617 (JPMorgan)   InterestAndDividendIncomeOperating -> 200

Apple is not a bank, so it has no interest-and-dividend income line to report. Nothing is wrong with the request, the tag, or the API. The concept simply does not apply.

The same thing happens across revenue tags — one filer reports Revenues, another RevenueFromContractWithCustomerExcludingAssessedTax, many report both — so no single tag covers the market, and a screen built on one tag silently omits everyone who chose another.

So a run over 50 companies × 3 tags will produce a lot of 404s, and that is the API working. The design consequence is that the fetch layer must never raise on one:

if response.status_code == HTTP_NOT_FOUND:
    logger.warning("%s: HTTP 404 — not reported, skipping", context)
    return None

None means "skip this combination", not "fail". Non-retryable 4xx behaves the same way. Only 408/429/5xx get retried, with capped exponential backoff.

This is the single highest-leverage pattern in our whole fleet. When we audited our lowest-success-rate Actors, the top cause was never a hard block — it was a recoverable per-item error crashing the entire run, so a customer lost 400 good rows because row 401 was unusual. Every Actor now isolates per item.

SEC's Fair Access rules are not optional

Two hard requirements, both easy to get wrong:

A descriptive User-Agent. SEC's Fair Access policy expects a real identifier. Send a default library UA and you will eventually get blocked — and unlike a fingerprinting wall, this one is documented policy, so complaining is not a strategy. Ours is configurable and defaults to our public brand contact URL, never a personal mailbox.

10 requests per second, maximum. With one in-flight request at a time, a 100 ms sleep before each attempt enforces the whole limit with no token bucket, no shared state, and nothing to get wrong under concurrency:

MIN_REQUEST_INTERVAL_S = 0.1

We also honour Retry-After when SEC sends it, in preference to our own backoff. If the server tells you when to come back, argue with it at your peril.

Tickers in, CIKs out

Nobody wants to look up a CIK. So the Actor accepts tickers or CIKs, and resolves tickers through SEC's own public map:

GET https://www.sec.gov/files/company_tickers.json

The one gotcha: CIKs are zero-padded to 10 digits in API paths. Apple is 0000320193, not 320193. The raw map gives you the integer, so the pad is on you:

str(int(cik_raw)).zfill(10)

Miss it and every request 404s — which, given everything above, you will initially misread as "this company did not report that."

FAQ

Is scraping SEC's XBRL API legal?
It is a public API the SEC publishes for exactly this purpose. Follow the Fair Access policy — identify yourself, stay under 10 req/s — and you are inside their stated rules.

Do I need an API key?
No. It is keyless. The User-Agent is the closest thing to identification and it is required.

Why is start_date null on some rows?
Because instant facts (balance-sheet items) have no duration — only an end_date. That is expected, not a gap.

Which tags should I use?
Start with Revenues, Assets, NetIncomeLoss — widely reported across sectors — then widen. Any US-GAAP, IFRS-full or DEI concept works, and coverage varies by filer and by industry, which is what makes the 404-tolerant design necessary rather than defensive.

Ready to run: SEC XBRL Financials Scraper — frame mode for one concept across every filer, company mode for one filer's full history; tickers or CIKs, any US-GAAP/IFRS/DEI tag, out to JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

GBIF has a hard wall at exactly 100,001 records

Devil Scrapes — Tue, 18 Aug 2026 17:27:39 +0000

GBIF — the Global Biodiversity Information Facility — indexes over three billion species occurrence records from museums, herbaria, national surveys and citizen-science platforms. It is keyless, well documented, and generous. It also has a hard wall at exactly 100,001 records that its docs mention in passing, and a name-resolution step that most first drafts skip entirely.

Quick answer

Two things decide whether a GBIF scraper works. First: resolve the species name to a usageKey via species/match before you search — free-text scientificName matching is fuzzy and silently drops synonyms. Second: offset + limit must stay under 100,001 or you get HTTP 400 mid-run. We verified the wall live: offset=200000 answers 400 "Max offset of 100001 exceeded: 200000 + 1".

Why resolve the name first?

You want Panthera leo. So does everyone else — but a species can be recorded under a synonym, a misspelling, a subspecies, or an outdated genus, and taxonomic backbones exist precisely to reconcile that.

GBIF exposes the reconciliation as its own endpoint:

GET https://api.gbif.org/v1/species/match?name=Panthera+leo

That hands back a usageKey — a stable integer for the accepted taxon — plus a matchType and a confidence score. Search by taxonKey=<usageKey> and you get the occurrences GBIF's backbone considers that species, synonyms included. Search by free-text name and you get whatever string-matched.

The response also carries a matchType and a confidence score, and they are worth reading. EXACT is what you want; FUZZY means GBIF guessed, and a fuzzy match on a typo can hand you a completely different organism with a perfectly successful HTTP 200 attached.

Our Actor logs the resolution it actually used — the key plus the canonical scientificName GBIF landed on — so the run log tells you Panthera leo resolved to Panthera leo and not to something adjacent:

resolved 'Panthera leo' -> taxonKey 5219404 (Panthera leo (Linnaeus, 1758))

If species/match finds nothing at all, it says so and falls back to an unfiltered-by-taxon search rather than silently returning zero rows.

The 100,001-record wall

occurrence/search pages with offset and limit. limit maxes at 300. offset is where it gets interesting:

offset=100000  -> 200 OK
offset=200000  -> 400 "Max offset of 100001 exceeded: 200000 + 1"

Verified live, not read off a doc page. So the real constraint is on the sum: offset + limit <= 100001. A loop that pages happily to record 99,900 and then requests offset=99900&limit=300 is asking for 100,200 and gets a 400 — a crash at the very end of a long, expensive run, which is the most infuriating place to put one.

We cap the walk at 100,000 so every request stays inside GBIF's boundary, and we say so in the Actor's own docs rather than letting a customer discover it at row 100,001. If you need more than that, the answer is not a cleverer offset — it is GBIF's asynchronous download API, which is a different tool with a different contract.

Enumerations: read them from the API, do not type them

basisOfRecord (is this a preserved specimen, a human observation, a machine observation, a fossil?) is a closed enumeration. It is tempting to hardcode the four or five values you have seen.

GBIF publishes the authoritative list:

GET https://api.gbif.org/v1/enumeration/basic/BasisOfRecord

We pulled it live and generated the input's allowed values from that, rather than from memory. It costs one request at build time and removes a whole class of "why does this filter return nothing?" support question — the same class of bug as an invented enum value in a platform manifest, which we have shipped before and would rather not again.

What about one malformed record in three billion?

It should cost you one record. GBIF aggregates from thousands of independent publishers with wildly varying data quality — a coordinate that is a string, a date that is a year, an occurrence with no taxonomy at all. All of that is normal.

So parsing is per-record and failures are skipped, logged, and counted:

try:
    row = parse_occurrence(item)
except ValidationError:
    log.warning(...)   # skip this one, keep the run
    continue

The single most common cause of a low-success-rate scraper in our fleet has never been a hard block. It is a recoverable per-item error taking down the entire run — one weird row costing the customer all the good ones.

Is scraping GBIF legal?

GBIF exists to publish this data for reuse, and most records carry an explicit open licence (CC0 or CC-BY) which the API returns per record. Honour the per-record licence when you redistribute, cite the datasets, and identify your client in the User-Agent. That is the whole etiquette.

FAQ

Do I need a key or a proxy?
Neither. It is a public keyless API with no anti-bot surface.

How many records can one run return?
Up to 100,000 via the search API, which is the platform's own ceiling. Larger extractions belong to GBIF's asynchronous download endpoint.

Why did my species name return nothing?
Check the match type. A NONE or low-confidence FUZZY match means the backbone did not recognise the name — usually a spelling or an authorship string that needs trimming.

Can I filter by country and year together?
Yes, and you should — it is the cheapest way to keep a query under the offset ceiling while still getting the slice you actually want.

Ready to run: GBIF Species Occurrence Scraper — search by species name, country, year range, or dataset; get full taxonomy, event date, coordinates, basis of record and dataset provenance as JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

Our smoke test was green and returned zero rows

Devil Scrapes — Tue, 18 Aug 2026 15:12:50 +0000

Our first cloud smoke test of the FDA 510(k) scraper came back green. Status: SUCCEEDED. Rows: zero.

The scraper was fine. The example input was the bug — and the verifier we built specifically to catch bad example inputs passed it, correctly.

Quick answer

A prefill (Apify's example input) that validates is not a prefill that returns data. Ours carried a value for every filter, so the smoke run ANDed applicant AND device name AND product code AND advisory committee AND device class AND a date range into a query openFDA legitimately matches nothing for. Validation checks shape; only a live query checks that the shape matches reality. Three more openFDA traps below: skip caps at 25,000, a + in a search term returns HTTP 500, and a no-match search answers 404 rather than an empty list.

The green run with no rows

We have a pre-publish gate called verify_input_prefill.py. Its job is to assemble the input schema's prefill and default values into a payload and confirm the Actor's Pydantic input model accepts it. It exists because a prefill that fails validation means a customer's first click errors out.

It passed. It was right to pass. Every field was well-typed, every date well-formed, the payload validated cleanly.

And the run returned nothing, because openFDA's search parameter ANDs its clauses, and the intersection of six independent filters over a database of device clearances is empty far more often than it is not. Every individual filter was plausible. The conjunction was a query about a device that does not exist.

The fix was to the prefill, not the code:

before: applicant + device name + product code + committee + class + date range
after:  applicant + date range          # confirmed live to match 24 records

Re-pushed, re-QA'd, rows arrived.

The general lesson we wrote into our build queue: a smoke test must assert on rows, not on exit status. "SUCCEEDED" is a statement about our process. Rows are a statement about the customer's experience. We had been treating the first as evidence of the second, which is the same conflation as treating "the tests pass" as evidence that the feature works.

Trap 1: `skip` caps at 25,000 — and not where you think

openFDA pages with limit and skip. limit maxes at 1,000. skip refuses to go past 25,000:

skip=25000 -> 200 OK
skip=25001 -> 400 "Skip value must 25000 or less."

There is a specific trap here for anyone who has already written an openFDA scraper. We ship a sibling Actor against the recalls endpoint, and that one has a different ceiling. Copying the constant across endpoints produces a scraper that dies 1,000 rows early or 1,000 rows late, depending on which direction you copied. Each openFDA endpoint gets its own verified constant, checked against the live API, with the verification date in the comment.

Trap 2: a `+` in a search term returns HTTP 500

This one is pure encoding. openFDA's query language treats + as meaningful, so a company name containing a literal + — url-encoded to %2B — makes the server answer 500. Encode the same character as a space (%20) and it parses fine.

It is worth naming because a 500 reads as "their server is having a bad day", which invites a retry loop rather than a fix. It is deterministic: same input, same 500, forever. Our sibling recalls Actor hit exactly this and the fix transfers unchanged.

Trap 3: no matches is a 404, not an empty list

Search for something real that simply has no records and openFDA answers 404, not {"results": []}.

So a client that treats 404 as a hard error reports "the API is broken" when the honest answer is "your filter matched nothing". And a client that retries 404s burns its retry budget on a settled question. We treat this one status as a terminal, non-error, zero-results outcome, and say so in the log.

There is a fourth, smaller one worth knowing: a cold query can 500 once and then 200 on an immediate retry. Genuinely transient, unlike trap 2. Bounded retries with backoff on 408/429/5xx cover it — but only because trap 2's deterministic 500 was fixed at the source rather than retried into the ground.

What is in a 510(k) record?

The premarket notification pathway is how most medical devices reach the US market — a claim of substantial equivalence to an already-cleared device. Each record carries the applicant, the device name, the decision date and decision code, the product code, the advisory committee, the device class and the regulation number.

That is a regulatory-intelligence dataset: who is clearing what, in which class, how fast, and through which committee. Filter by applicant to watch a competitor, by product code to watch a device category, or by decision-date range to build a clearance timeline.

FAQ

Is scraping openFDA legal?
openFDA is the FDA's own public API, published for reuse, no key required for normal use. Read their disclaimer about the limits of the underlying data before you make decisions with it.

Do I need an API key?
No. Keys exist only to raise rate limits; the unauthenticated tier is enough for ordinary extraction.

How many records can one run pull?
Up to openFDA's own skip ceiling of 25,000 per query — its own 400 response points at search_after for going deeper. Until we wire that up, narrow by date range or product code and slice a bigger set into runs that each stay inside the ceiling.

Why did my query return nothing?
Almost always because the filters ANDed into an empty intersection — the exact bug that produced our zero-row green run. Drop filters one at a time until rows appear.

Ready to run: FDA 510(k) Clearances Scraper — filter openFDA 510(k) clearances by applicant, device name, product code, advisory committee, device class, or decision-date range; export as JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

Crossref's funder filter wants a DOI, not a name — and 3 more traps

Devil Scrapes — Tue, 18 Aug 2026 09:37:37 +0000

Crossref's works endpoint is about as friendly as a public API gets: no key, no rate-limit paperwork, 150 million-plus DOIs, and documentation that mostly matches reality. We shipped a scraper against it in an afternoon.

It still had four traps in it, and every one of them is the kind that passes a smoke test and fails on a customer's first real query.

Quick answer

The four things that will bite you scraping Crossref: filter=funder: accepts a Funder Registry DOI, not a funder name; dates arrive as date-parts arrays that are legitimately 1, 2, or 3 elements long; author names come in four different shapes; and next-cursor can hand you the same cursor forever. None of them raise an exception. All four produce plausible, wrong output.

Why scrape Crossref instead of Google Scholar?

Because Crossref will let you. It is the DOI registration agency behind most of scholarly publishing, and the metadata publishers submit — title, journal, authors, ORCIDs, funders, award numbers, licence, citation counts — is exposed through one keyless JSON endpoint on purpose. There is no bot detection to defeat, which means the engineering budget goes into being correct rather than into being invisible.

The polite convention is worth honouring: put a contactable identifier in your User-Agent and Crossref routes you to a better-served pool.

User-Agent: crossref-works-scraper/0.1.0 (https://apify.com/DevilScrapes)

That is a brand contact URL, not a personal mailbox. Works the same, leaks nothing.

Trap 1: the funder filter wants a DOI

This looks like it should work:

GET /works?filter=funder:Wellcome%20Trust

It returns 200. It returns works. It does not return works funded by the Wellcome Trust — funder: is matched against the Funder Registry DOI, so a free-text name matches nothing useful and the API does not consider that an error. You get a full, confident, wrong result set.

We split the input on shape instead of trusting the user to know:

FUNDER_DOI_RE = re.compile(r"^10\.\d{4,9}/")

def funder_is_doi(self) -> bool:
    return bool(self.funder_name and FUNDER_DOI_RE.match(self.funder_name))

A DOI goes to Crossref as a server-side filter. A name becomes a client-side, case-insensitive substring match over each work's funder list. Same field, two paths, and "Wellcome" finds the works whether the publisher wrote Wellcome Trust or The Wellcome Trust.

Trap 2: `date-parts` is not a date

Crossref publication dates look like this:

{ "published": { "date-parts": [[2021, 3, 14]] } }

Except when the publisher only registered a year and a month. Or only a year. All three are valid, all three are common, and the array length tells you which you got:

if len(parts) == 1:
    return f"{parts[0]:04d}"              # 2021
if len(parts) == 2:
    return f"{parts[0]:04d}-{parts[1]:02d}"   # 2021-03
return f"{parts[0]:04d}-{parts[1]:02d}-{parts[2]:02d}"

The tempting shortcut — datetime(*parts) with defaults — invents a 1 January that no publisher ever claimed. That is worse than a null, because a null is visibly missing and a fabricated date silently poisons every date-range query downstream.

There is a second half to this: the date lives under different keys depending on the record. We walk a fallback chain — published, then published-print, then published-online, then issued — and take the first one that renders.

Trap 3: authors have four name shapes

A Crossref author entry can carry given + family, only family, a literal string (common for consortia and corporate authors), or a bare name. Read only given/family and every consortium byline in your dataset comes back empty.

def _author_name(entry):
    given, family = entry.get("given"), entry.get("family")
    if given and family:
        return f"{given} {family}"
    return family or given or entry.get("literal") or entry.get("name")

ORCIDs have the same shape problem in miniature — sometimes bare, sometimes prefixed with https://orcid.org/. Store one form, always, or your joins will quietly miss half the matches.

Trap 4: the cursor that never ends

Deep paging on Crossref is cursor-based, and that is genuinely the right design — offset paging on a 150M-record corpus hits a wall a few thousand rows in. You pass cursor=*, and each response hands back message.next-cursor.

The failure mode is that next-cursor is not guaranteed to advance. Hand back the same token twice and a naive while next_cursor: loop pages forever, re-emitting the same block of works and — on a pay-per-result Actor — billing the customer for every duplicate.

One line of defence:

if not cursor or cursor == previous_cursor or emitted >= cfg.max_results:
    break
previous_cursor = cursor

Three independent exits: no cursor, a stalled cursor, or the row budget spent. A paging loop should never have only one way out.

What about a single malformed record?

It should cost you that record and nothing else. The general lesson from our fleet is that the top cause of a low-success scraper is not a hard block — it is a recoverable per-item error that takes down the whole run. Crossref metadata is publisher-submitted and therefore inconsistent by nature, so parsing is wrapped per item:

def _safe_parse(work, log):
    try:
        return parse_work(work)
    except ValidationError:
        ...  # log it, skip it, keep going

Nine hundred and ninety-nine good rows plus one weird preprint should be 999 rows delivered, not a failed run.

FAQ

Is scraping Crossref legal?
Crossref publishes this metadata through a public API for exactly this kind of reuse, and most of it is explicitly open. Honour the polite-pool convention, do not hammer it, and check the licence field on anything you redistribute.

Do I need an API key or a proxy?
Neither. It is keyless, and a public API has no reason to block a well-behaved client. Our Actor leaves the proxy off by default.

How deep can I page?
As deep as the result set goes, if you use the cursor. Offset paging is what has a practical ceiling.

Why are some works missing abstracts, ISSNs or funders?
Because publishers did not submit them. Scholarly metadata is sparse by default — build your schema to accept nulls everywhere rather than assuming a fully-populated record.

We packaged all of this as a ready-to-run Actor: Crossref Works Scraper — search by bibliographic query, publication-date range, work type, publisher, funder, or ORCID/abstract/full-text presence, and get one flat row per work with authors, ORCIDs, funders, award numbers, licence and citation count. Exports to JSON, CSV, or Excel.

We do the dirty work so your dataset stays clean. 😈

Your proxy returned 200 OK and the wrong country's data

Devil Scrapes — Tue, 18 Aug 2026 04:32:20 +0000

The worst scraping bug we have shipped did not throw an exception. It returned HTTP 200, a full page of well-formed listings, every field populated, every assertion green — and the data was from the wrong country.

Quick answer

A rotating residential proxy pool will happily give you an exit IP in a country you did not ask for, and a geo-aware site will then serve you plausible, correct-looking data for that country instead. There is no error to catch. The defence is two-part: pin the exit country explicitly in your proxy configuration, and then verify the country in the response body before you emit a single row. If you only do the first, you are trusting a config flag with no feedback loop.

How a 200 OK lies to you

Geo-targeted sites do not usually reject a foreign visitor. They localise for them. Ask a big job board for "developer jobs in Sydney" from a German exit IP and you may well get a working page — correct HTML, real listings, sensible salaries — for an entirely different market. Or a market splash page. Or the right search, silently region-filtered.

Every failure signal your scraper watches for is absent:

Status code: 200
Content length: normal
Parse: succeeds
Required fields: all present
Row count: as expected

The only thing wrong is the answer. And because rotation is random, it is wrong intermittently — maybe one page in eight, depending on your pool's composition. That is the signature that makes this expensive: intermittent, silent, and invisible to every test that checks structure rather than meaning.

We now treat this as its own bug class. A wrong-country row is worse than a failed request, because a failed request is loud and a wrong row gets into someone's dataset.

Step 1 — pin the country, do not hope for it

Most proxy providers let you constrain the exit country. On Apify Proxy, that is countryCode on the residential group:

proxy = await Actor.create_proxy_configuration(
    groups=["RESIDENTIAL"],
    country_code="AU",
)

Leave that off and you get "some residential IP, somewhere". For a national job board, that is not a proxy configuration, it is a lottery ticket.

Worth knowing: requesting a tier is not the same as using one. We have measured runs that passed a residential configuration and billed zero residential bytes — one preferred an environment variable over the input we handed it, another's proxy helper failed and silently degraded to a direct connection. The only honest check is the run's billing record. If the residential transfer line is zero, residential was not used, whatever your config said.

Step 2 — verify geography in the response

Pinning is necessary and not sufficient. The response itself has to be interrogated before it is trusted. For an Australian job board that means checking the site's own market signal and each posting's country code, and treating a mismatch as a retryable failure, not as data:

if not looks_like_target_market(response):
    logger.warning("geo mismatch — rotating session and retrying")
    rotate_session()
    raise RetryableError("wrong market")

That is the whole guard. Rotate the session ID so you get a fresh exit IP, retry, and only emit rows once the market check passes. It costs one extra comparison per response and it converts an invisible data-quality defect into an ordinary, visible retry.

The general form, for any geo-sensitive target: find the field in the response that names the market, and assert on it. Currency symbol, country code, locale string, market identifier, a phone format — something the page states about itself. If a target does not expose one, that absence is itself a finding, and you should be far more conservative about what you claim the data represents.

Why not just use one static IP?

Because you will get blocked, and then you will have neither correctness nor coverage. Rotation is not the enemy here; unconstrained rotation is. You want a pool that rotates within a country, plus a check that confirms it did. Rotation gives you resilience, pinning gives you correctness, and verification gives you proof — you need all three, and most implementations stop after the first.

What else Seek actually needs

The geo guard is the interesting part, but it is not the only part:

Real browser TLS. We use curl-cffi impersonation so the handshake and HTTP/2 fingerprint look like Chrome, Firefox or Safari rather than Python's. Fingerprint mismatch is one of the quieter reasons a scraper gets nothing back.
Backoff that honours Retry-After. On 408 / 429 / 503, exponential backoff up to five attempts, and when the server tells you how long to wait, wait that long. Ignoring Retry-After is how a temporary throttle becomes a permanent block.
Taxonomy preservation. Seek classifies roles with its own classification and subclassification scheme. Flattening that to a free-text string throws away the most useful filtering dimension in the dataset, so we keep both levels intact alongside the suburb → state → country location hierarchy.

What the scraper actually does

Seek Australia Jobs Scraper searches au.seek.com by keyword and Australian location and returns one normalized row per posting — title, company, classification and subclassification, flattened location hierarchy, work type and arrangement, salary label, listing date and teaser. No login, no API key. Pay-per-result at $1.83 per 1,000 rows, so a search that returns nothing costs nothing beyond the start fee.

We are not going to tell you Seek is easy to scrape. Job boards actively defend their listings, they change those defences without notice, and the honest framing is that absorbing the blocks, the rotation, the retries and the geo verification is our job, not yours. When it does start pushing back, that is our problem to fix, not a support ticket for you to write.

If you need the same treatment on other markets, the neighbours are SmartRecruiters, Workable, Greenhouse and the multi-ATS aggregator.

The rule we wrote down

Geo-random residential exits return plausible wrong data with a 200 status, not errors. Pin the country, then verify the country in the body. A proxy config you never validate against the response is a hypothesis, not a control.

The JSON Schema bug that only fires on a customer's first real query

Devil Scrapes — Tue, 18 Aug 2026 04:31:49 +0000

There is a category of bug that passes every test you write, ships to production, and then fails on a customer's very first real query. We hit a clean example of it building against ClinicalTrials.gov, and the root cause was four characters of JSON.

Quick answer

If your output schema declares "type": "string" for a field that is sometimes absent, your pipeline will die on the first sparse record — and no realistic test fixture will catch it. Registry data is sparse by nature: a Phase 1 trial has no results, an observational study has no intervention arm, a terminated trial has no completion date. Declare every optional field as "type": ["string", "null"]. A QA sample chosen because it looks good is the worst fixture you can pick, because it hides exactly this class of failure.

The failure mode

Apify Actors can publish a dataset schema so downstream consumers get a typed, validated table instead of loose JSON. Ours declared fields the obvious way:

{
  "fields": {
    "primary_completion_date": { "type": "string" },
    "enrollment_count":        { "type": "integer" }
  }
}

That reads as correct. It is not, and here is why it is dangerous rather than merely wrong: validation happens at write time, per record. So the run starts, streams a few hundred perfectly good rows, and then hits a trial with no primary completion date. null is not a string. The write is rejected, the exception propagates, and the whole run dies — after the customer has been charged for the rows that already landed.

The correct declaration:

{
  "fields": {
    "primary_completion_date": { "type": ["string", "null"] },
    "enrollment_count":        { "type": ["integer", "null"] }
  }
}

Four characters. The interesting part is not the fix, it is why it survived testing.

Why a good test fixture hid a real bug

When you build a fixture, you naturally reach for a representative record — one that exercises the parser, shows off the fields, and reads well in a code review. For registry data, that means a large, well-funded, fully-reported interventional trial. Every field populated. Every assertion meaningful.

That fixture cannot fail this bug. Not "is unlikely to" — cannot. The failure requires a null, and you deliberately picked a record that has none.

The same applies to smoke tests. A three-row smoke test against the default query hits the most complete, most-cited, best-maintained records in the registry. It passes. Then a customer filters for terminated trials in a rare disease, gets a result set that is 60% sparse, and the run dies on row 4.

We changed two things:

Fixtures must include the ugliest record we can find, not the prettiest. Withdrawn studies, missing sponsors, empty location arrays.
The scaffolder now emits nullable unions by default, so no new Actor can be born with the bug. The fix belongs at the template, not in a checklist item someone has to remember.

Why is the ClinicalTrials.gov API hard to use?

Not because it blocks you — it is a public, keyless, well-run government API and it does not fight back at all. It is hard because of shape.

The v2 API returns deeply nested protocol sections that mirror the regulatory document structure, not anything you would want in a spreadsheet:

protocolSection
├── identificationModule      → nctId, briefTitle, officialTitle
├── statusModule              → overallStatus, startDateStruct, completionDateStruct
├── sponsorCollaboratorsModule→ leadSponsor.name, collaborators[]
├── designModule              → phases[], enrollmentInfo.count, studyType
├── conditionsModule          → conditions[], keywords[]
├── armsInterventionsModule   → interventions[].type, .name
└── contactsLocationsModule   → locations[].facility, .city, .country

Almost every leaf is optional, several are arrays that are sometimes absent and sometimes empty, and the date fields are structs with their own optional precision qualifier rather than plain strings. Flattening that into one predictable row per trial is the actual work, and it is the part that produces nulls everywhere — which loops straight back to the schema bug above.

The API also uses cursor paging (nextPageToken), not offset. That is the right choice on their side and a small adjustment on yours: page until the token comes back absent, and do not try to compute a page count up front.

Retry the registry's 5xx

One more that costs a run if you skip it: ClinicalTrials.gov returns transient 5xx under load, and a single one will kill an unguarded long query. Retry with exponential backoff and treat 5xx as retryable rather than fatal. The same rule we apply to every government API we touch — these are public services under real load, not enterprise SLAs, and intermittent failure is normal operating behaviour rather than an exception.

What the scraper actually does

ClinicalTrials.gov Scraper searches the registry and returns one flat row per trial — NCT ID, title, phase, status, sponsor, conditions, interventions, enrollment, locations, and start and completion dates — as JSON, CSV or Excel. It pages the cursor API, stops exactly on your result cap, and retries transient registry 5xx instead of failing the run. Pay-per-result at $2.05 per 1,000 rows, so a query that returns nothing costs nothing beyond the start fee.

We are not going to claim registry data is tidy. It is a regulatory filing system with twenty years of schema evolution in it, and absorbing the nesting, the nulls, the cursor paging and the retries is our job, not yours.

If you work adjacent regulatory sources, the neighbours are FDA recalls, NPI healthcare providers and PubMed papers.

The rule worth stealing

Your test fixture should be the worst record in the dataset, not the best one. A fixture chosen for how well it demonstrates the happy path is a fixture guaranteed to miss the sparse-record bug.

openFDA returns HTTP 500 when you follow its own docs

Devil Scrapes — Tue, 18 Aug 2026 04:31:18 +0000

openFDA's documentation tells you to join search terms with +AND+. Follow that literally in most HTTP clients and the API answers HTTP 500. The docs are not wrong; your client is helpfully breaking them.

Here is the trap, plus a second one we reproduced live this morning that will flake your CI if you do not handle it.

Quick answer

Two things will bite you on openFDA and neither is documented where you will look:

+ is not a literal plus. openFDA's docs show search=field:value+AND+other:value, but a URL-encoding HTTP client percent-encodes + to %2B, which openFDA parses as part of the term rather than as a separator — and returns a 500. Join with spaces and let your client encode them.
openFDA returns intermittent 500s on cold queries. We measured it again this morning: the first request timed out server-side after 31 seconds, and two immediate identical retries returned 200 in ~1.2 seconds each. If your client does not retry 500s, your smoke test is a coin flip.

The `+AND+` trap in full

The openFDA query syntax is Lucene-ish. To combine terms, the docs show:

https://api.fda.gov/drug/enforcement.json?search=classification:"Class+I"+AND+status:"Ongoing"

That URL works when you paste it into a browser address bar, which is why it survives in documentation. It stops working the moment a real HTTP client touches it, because + is a reserved character in a query string. Most clients — and every sane URL-building library — will percent-encode a literal + in a parameter value to %2B. openFDA then sees %2BAND%2B decoded as +AND+ inside the term, not as a boolean operator, and the query blows up with a 500 rather than a helpful 400.

The fix is boring and total: use spaces.

params = {"search": 'classification:"Class I" AND status:"Ongoing"', "limit": 100}

Your client encodes the spaces as %20 (or as +, correctly, at the parameter level), openFDA parses the boolean, and you get results. We lost a build to this once. It is now the first line of our openFDA notes.

Why does openFDA return HTTP 500 randomly?

Because it sometimes just does, and you have to plan for it. This is not a theory — here is a measurement from 2026-08-18, three identical requests to the device clearance endpoint, run back to back:

try 1: HTTP 500   146 B   31.09 s
try 2: HTTP 200   21108 B  1.30 s
try 3: HTTP 200   21108 B  1.21 s

The 500 body is explicit about what happened:

{
  "error": {
    "code": "SERVER_ERROR",
    "message": "Check your request and try again",
    "details": "Request Timeout after 30000ms"
  }
}

That is a server-side 30-second timeout on a cold query, returned to you as a 500. The identical query is then served from warm state in about a second. Note the message: "Check your request and try again" is misleading — there is nothing wrong with the request.

The consequence for anyone building on this API: a 500 from openFDA is a retryable condition, not a permanent failure. Retry with backoff, at least three attempts. If you treat 5xx as fatal, your pipeline will fail intermittently in a way that is nearly impossible to reproduce on demand, because the second time you check, it works.

The paging ceilings nobody mentions until you hit them

Two hard limits, both of which return errors rather than empty pages:

limit maxes out at 1000 per request.
skip maxes out at 25000. Past that, openFDA errors instead of returning nothing.

So the maximum reachable window for a single query is 26,000 records. If your result set is bigger, you cannot page your way through it — you have to partition the query, usually by date range, and page inside each partition. Any scraper that silently stops at 25k and reports success is lying to its user; ours logs the ceiling explicitly and tells you to narrow the range.

The nullable-schema failure that only shows up in production

One more, because it is the same shape as the retry bug — it hides from your tests.

openFDA enforcement records routinely omit whole blocks. The openfda sub-object is absent on a large fraction of records. If you declare an output schema like this:

{ "product_type": { "type": "string" } }

then the first record without that field produces a null, the null fails validation, and the write dies mid-run — after you have already emitted rows and charged for them. A curated QA sample with fully-populated records never triggers it.

{ "product_type": { "type": ["string", "null"] } }

Every optional field in an openFDA-derived schema needs the union type. We fixed this at the scaffolder level so it cannot be reintroduced.

Is the openFDA API free?

Yes, and keyless for low volume — which is exactly why it is a good foundation. There is no API key to leak, no OAuth dance, no bot detection, and no HTML to re-parse when someone redesigns a page. The entire difficulty of this target is the four items above: encoding, retries, paging ceilings, and sparse records. All four are solvable once and then permanently solved.

What the scraper actually does

FDA Recalls Scraper queries openFDA's enforcement database and returns one flat row per recall event — recall number, classification, status, recalling firm, product description, reason for recall, distribution pattern, quantity, and initiation and termination dates — as JSON, CSV or Excel. It is pay-per-result at $2.05 per 1,000 rows, so a query that returns nothing costs nothing beyond the start fee.

We are not going to tell you this API is easy. It returns 500s on healthy queries, it has undocumented ceilings, and its own docs contain a query string that breaks in real HTTP clients. Absorbing all of that — the retries, the partitioning, the schema drift — is our job, not yours.

If you need the neighbouring regulatory sources, we also run ClinicalTrials.gov, NPI healthcare providers and professional license lookups.

The one-line version

On openFDA, join with spaces not +AND+, retry every 500, partition past 25k, and make every optional field nullable. Those four rules are the whole integration.

The day our scraper fleet shipped nothing — and the afternoon it shipped three

Devil Scrapes — Tue, 18 Aug 2026 04:30:46 +0000

We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of release automation worked, and the day ended with zero listings live. That same afternoon, three different scrapers went from empty directory to published in about an hour each, first attempt, no retries.

The difference was not effort, tooling, or engineering skill. It was what we pointed at.

Quick answer

If your scraping backlog is a list of consumer marketplaces, your throughput is capped by someone else's bot-detection budget, not by your own velocity. Keyless public JSON APIs — OpenAlex, Crossref, GBIF, openFDA, World Bank, SEC — ship on the first attempt because there is nothing to block. We now sort the backlog by target class before we sort it by revenue, and a walled target only enters a build wave with a specific tested plan for its defences.

The day that shipped nothing

Nine Actors sat in the publish queue. All nine were consumer marketplaces and job boards with commercial anti-bot in front of them. Every one was cloud-QA'd on datacenter proxy and on residential proxy. All nine failed.

That is a real result and worth writing down, because "we were blocked" is a useless note. The useful version names the tiers:

9/9 failed cloud QA — datacenter AND residential, same day, same fixtures

What made it expensive was that the failure was invisible from inside the process. Every stage worked. The specs were fine, the tests were green, the pricing wired up correctly, the publisher refused correctly. The lane was healthy and the output was zero, which is the worst combination to debug because nothing is flashing red.

The afternoon that shipped three

We changed one variable: target class. Instead of marketplaces, we took three public research and regulatory APIs that require no key. Scaffold, implement, cloud QA, publish. Roughly an hour each, all three passed cloud QA on the first run.

There is no clever trick here. A public JSON API has no incentive to fingerprint you, no CAPTCHA, no TLS fingerprint check, no HTML that gets rewritten on Tuesday. The whole class of failure that eats marketplace scrapers simply does not exist. What is left is ordinary engineering: paging, schema, and error handling.

Why is OpenAlex a good scraping target?

OpenAlex is the open replacement for Microsoft Academic Graph — roughly 250 million scholarly works, with authors, institutions, journals, citation counts and open-access status. It is free, it needs no API key, and it is genuinely well-behaved if you are.

Two things matter in practice.

Use cursor paging, not offset. Offset paging on a corpus this size falls apart past the first few thousand rows — the API caps how deep you can page and the cost of a deep offset grows. The cursor is the supported path:

GET https://api.openalex.org/works?per-page=200&cursor=*

Each response hands you meta.next_cursor; you pass it back until it comes back null. Stateless, stable under concurrent updates, and it does not degrade at depth.

Identify yourself and get the fast lane. OpenAlex runs a "polite pool" — include a contact in your User-Agent or a mailto parameter and your requests go to a separate, faster, more reliably-served pool. This is the rare case where telling the server who you are makes your scraper better, not more blockable. We send our public contact URL, never a personal address.

The trap that survived the easy target

Keyless does not mean trivial. The bug that cost us the most time on this batch had nothing to do with blocking.

Apify Actors can declare a dataset schema, and ours declared fields like this:

{ "doi": { "type": "string" } }

That is wrong, and it is wrong in the worst possible way: it only fails on a sparse record. Scholarly metadata is extremely sparse — plenty of works have no DOI, no abstract, no ISSN, no funder. The first record missing one of those fields produces a null, the null fails validation, and push_data dies mid-run. A fully-populated QA sample never triggers it, so the smoke test passes and the customer's real query is what breaks.

The fix is one character of JSON per field:

{ "doi": { "type": ["string", "null"] } }

We fixed it at the scaffolder so no future Actor can be born with it. The general lesson: a QA fixture that is too clean is worse than no fixture, because it converts a loud failure into a delayed one.

What this means for your backlog

Sort by class first:

Keyless public APIs — ship first-attempt. Government registries, research indexes, open data portals, standards bodies. Low glamour, real buyers, and they do not fight back.
Plain HTML sites with no commercial anti-bot — ship with ordinary care.
Commercially defended marketplaces — only with a specific, tested plan for their defences, and a named tier you have actually probed. Never on optimism.

We ran category 3 on optimism for a long time. The scoreboard was honest about it exactly once, on the day it produced a zero.

What the scraper actually does

OpenAlex Works Scraper searches the 250M-work catalogue and returns one flat row per work — title, DOI, authors with ORCIDs and affiliations, journal, publication date, citation count and open-access status — as JSON, CSV or Excel. It is pay-per-result at $2.05 per 1,000 rows, so a query that returns nothing costs you nothing beyond the start fee.

We are not going to pretend that any source stays static forever. APIs deprecate fields, tighten rate limits and reshape their pagination, and when OpenAlex does, absorbing that is our job, not yours — the cursor handling, the retries and the schema drift are on our side of the line.

If you work the same corpus from other angles, the neighbours are Crossref-adjacent research tooling, arXiv and EU CORDIS grants.

The rule we wrote down

A backlog sorted only by revenue potential will happily spend a month on targets that cannot ship. Sort by target class first, then by revenue inside each class.

That is the whole finding. It cost us a zero-release day to learn, and it is the cheapest thing in this post to copy.

Redfin caps at 350, Apple caps at ~50, Glassdoor caps at 3 — none of them are rate limits

Devil Scrapes — Mon, 17 Aug 2026 11:21:21 +0000

Short version: three review/listing platforms, three different reasons you stop getting more data. Redfin's 350-listing cap is a countable server limit, and we bisect around it with price bands. Apple's ~50-review ceiling on its customer-reviews feed is architectural and permanent — page 2 is dead, not blocked, and no amount of retrying brings it back. Glassdoor's ~3-full-reviews-per-page-load ceiling for anonymous visitors isn't a count limit at all; it's how much of the page actually renders before you're not logged in. Treat all three as "a rate limit" and you'll either burn a week chasing a wall that was never there, or under-price a scraper that's doing far more work than the row count suggests.

Here's each ceiling's real shape, how we found it, and why the fix is different every time.

🧱 Why "just paginate harder" is the wrong first question

The instinct when a scraper stops returning data past some point is to assume a rate limit and add a delay. Sometimes that's right. It wasn't right for any of the three targets below. Each one hit a hard stop that had nothing to do with request pacing — a server-side count cap, a feed that was never built to page past one screen, and a rendering budget for logged-out visitors. Backing off and retrying does nothing for any of them. The fix has to match the shape of the ceiling, and the three shapes below cover most of what you'll run into scraping review and listing data anywhere.

🏠 Redfin: a countable ceiling, so we count around it

Every call to Redfin's gis search endpoint returns at most 350 homes, full stop, no matter what you ask it for. Ask for a dense metro's for-sale inventory and the 351st home just doesn't come back — no error, no truncation flag, nothing that tells you to keep going.

Because the cap is a count against a filterable result set, it's defeatable: split the price range into buckets, request each bucket separately, and recurse into any bucket that still comes back at 350 (mpt=99 plus a min_price/max_price pair does the splitting). We stop bisecting once a bucket's width drops below a configurable floor ($25,000 by default) so a genuinely thin price band doesn't get sliced forever. Every home gets deduped by property_id across buckets, because the same listing straddles bucket boundaries more often than you'd expect. Each bucket also fails in isolation — a slow or blocked bucket gets logged and skipped rather than taking the whole search down.

That's the Redfin Property Listings Scraper: normalized for-sale/sold/pending listings with MLS id, days-on-market, HOA, exact lat/long, and open-house windows, at $2.52 per 1,000 results. The cap here is annoying but honest — it always tells you the same number, and a countable ceiling is the one shape you can engineer straight through.

🍎 App Store reviews: a ceiling that was never a wall to climb

Apple's customer-reviews RSS/JSON feed looks like it should paginate — the URL has room for a page number, and every review-scraping tutorial on the internet assumes page=2 works. During recon on 2026-08-15 we tried every path-segment ordering we could construct for page 2 and higher against that feed. All of them came back dead. Page 1 — roughly 50 reviews — is the entire dataset Apple will hand you per app, forever, and we scoped the code to never even build a page parameter, because a permanent dead end doesn't deserve a retry loop pretending otherwise.

That's a different kind of ceiling than Redfin's. There's no bucket to bisect and no query shape that unlocks more. The only honest move is to say so — in the input field description, the README, and the pricing — instead of quietly under-delivering against an implied "as many as you ask for."

Where this Actor earns its keep is the half of the ceiling that isn't fixed: Google Play's review RPC does paginate toward whatever cap you set. The App Store Reviews & ASO Scraper fetches both stores for the same app in one run — one unified schema with a store discriminator — and computes a per-app ASO rollup on top: rating distribution, month-by-month rating trend, review velocity, and top praise/complaint keywords via a deterministic stoplist-frequency count (no sentiment ML, no black box). Every incumbent we found on the Store covers exactly one platform; nobody else stitches Play and Apple into one comparable ASO view. Priced at $1.20–1.50 per 1,000 rows blended across a $0.05 warm-up, $0.0012 per review, and $0.005 per rollup row.

💬 Glassdoor: a ceiling that isn't measured in rows at all

The Glassdoor case is the one that breaks the "it's a rate limit, add a delay" instinct hardest, because the ceiling isn't about how many pages you can request — it's about how much of one page renders for a visitor who isn't logged in. Anonymous access surfaces roughly three full review entries per page load; the rest of that page's reviews exist in the DOM in truncated or teaser form. You can absolutely request page 20. You'll get a 200 and a page. You will not get twenty pages' worth of full review text out of it.

Getting onto that page at all is its own problem — Glassdoor's review pages sit behind bot defenses that a plain HTTP request doesn't survive, so the Glassdoor Reviews Scraper runs a full browser session: a proven navigation sequence through the same browser context and page object, not a fresh request per URL, with the proxy session pinned for the life of that context. One company failing to resolve or parse never sinks the run — it's logged, skipped, and every other company still completes.

Because the real throughput is ~3 full reviews per page load, not the page count you asked for, maxReviewsPerCompany and pricing are built around that reality instead of promising cheap bulk collection we can't actually deliver without a login we refuse to use. That's also why this one sits at the top of our usual range: $10.00 per 1,000 review rows, because each row costs real browser-navigation compute, not a cheap HTTP round-trip.

🧪 The three shapes, side by side

Target	What the ceiling actually is	Can you defeat it?	The right response
Redfin	Server-side count cap (350/call) on a filterable set	Yes — partition the filter (price bisection) and dedup	Bisect, recurse, dedup by ID
Apple App Store feed	Architectural — page 2+ doesn't exist, verified dead	No	Scope it out permanently, state it in docs and pricing
Glassdoor (anonymous)	Rendering depth per page load, not a count	No, without a login we won't use	Price and cap around the real per-page yield

Mistaking any row of that table for either of its neighbors is where scrapers go wrong: bisecting against an architectural ceiling wastes requests on a wall that logs nothing new; retrying against a rendering-depth ceiling just re-fetches the same three reviews with extra latency; and quietly capping a countable ceiling "for safety" leaves real, gettable data on the table.

❓ FAQ

Can I get more than ~50 Apple App Store reviews per app?
Not from this feed. Page 2 and beyond returned nothing across every URL shape we tried during recon — it's a permanent, one-page ceiling on Apple's side, not a setting we could raise.

Does Redfin's price-bucket bisection risk getting rate-limited?
We retry with exponential backoff on 408/429/5xx and rotate the session on a block, same as every Actor in the fleet — bisection adds more requests per search, not more requests per second.

Why is Glassdoor priced at $10/1,000 when the others sit closer to $1–2.50?
Because the real unit of work is a full browser navigation per company, not a cheap API call, and anonymous access only renders a handful of full reviews per page load — the price reflects the compute that actually happens, not a theoretical row count.

Do any of these need a login or API key?
No, for all three. Apple and Google Play are public feeds; Glassdoor and Redfin are scraped from their own public pages, no account, no credentials.

Will one bad app or company stop the whole run?
No — all three Actors isolate faults per item. A resolution failure or empty result for one entry gets logged and skipped; the rest of the run completes and the run's status message reports exactly what happened.

📦 The takeaway

"That's all the data you get" means three different things depending on which wall you hit. A countable ceiling gets engineered through. An architectural one gets documented and priced around, not chased. A rendering-depth ceiling gets measured honestly so the price matches the real work, not the row count on the label. We built all three read on the ceiling into the Actor before it ever shipped, so the number on the pricing page is the number you'll actually get — not the number the target's marketing implies.

We do the dirty work so your dataset stays clean. 😈

DEV Community: Devil Scrapes

A fake 404 and a 403 that isn't a ban — the UK food ratings API

Quick answer

The 404 that means "missing header"

Why does the FSA API return 403 on a valid query?

What data is actually in there?

What the scraper actually does

The one-line version

The API where page 21 is a 400 and 'no results' is a 404

Quick answer

The 2,000-document window

Why does the Federal Register API return 404 on a valid query?

Is the Federal Register API free?

What the scraper actually does

The one-line version

The API where 200 OK means your request was invalid

Quick answer

The wire format

Why does an invalid indicator return 200?

Sparse by design: value is legitimately null

One row per country, indicator and year

Is scraping the World Bank API legal?

FAQ

A 404 from SEC's XBRL API is usually the correct answer

Quick answer

Two endpoints, two questions

The frame id format is stricter than it looks

Why most 404s are correct answers

SEC's Fair Access rules are not optional

Tickers in, CIKs out

FAQ

GBIF has a hard wall at exactly 100,001 records

Quick answer

Why resolve the name first?

The 100,001-record wall

Enumerations: read them from the API, do not type them

What about one malformed record in three billion?

Is scraping GBIF legal?

FAQ

Our smoke test was green and returned zero rows

Quick answer

The green run with no rows

Trap 1: skip caps at 25,000 — and not where you think

Trap 2: a + in a search term returns HTTP 500

Trap 3: no matches is a 404, not an empty list

What is in a 510(k) record?

FAQ

Crossref's funder filter wants a DOI, not a name — and 3 more traps

Quick answer

Why scrape Crossref instead of Google Scholar?

Trap 1: the funder filter wants a DOI

Trap 2: date-parts is not a date

Trap 3: authors have four name shapes

Trap 4: the cursor that never ends

What about a single malformed record?

FAQ

Your proxy returned 200 OK and the wrong country's data

Quick answer

How a 200 OK lies to you

Step 1 — pin the country, do not hope for it

Step 2 — verify geography in the response

Why not just use one static IP?

What else Seek actually needs

What the scraper actually does

The rule we wrote down

The JSON Schema bug that only fires on a customer's first real query

Quick answer

The failure mode

Why a good test fixture hid a real bug

Why is the ClinicalTrials.gov API hard to use?

Retry the registry's 5xx

What the scraper actually does

The rule worth stealing

openFDA returns HTTP 500 when you follow its own docs

Quick answer

The +AND+ trap in full

Why does openFDA return HTTP 500 randomly?

The paging ceilings nobody mentions until you hit them

The nullable-schema failure that only shows up in production

Is the openFDA API free?

Sparse by design: `value` is legitimately null

Trap 1: `skip` caps at 25,000 — and not where you think

Trap 2: a `+` in a search term returns HTTP 500

Trap 2: `date-parts` is not a date

The `+AND+` trap in full