An API With No Pagination Is a Billing Problem

Quick answer

The CPSC's recall API has no pagination. None — no page, no offset, no page size to negotiate. It answers your date filter by returning the entire matching result set in one JSON array, and a single one-week filter came back at over a megabyte during our probe. The only cost control that exists is the width of the window you ask for, so the CPSC Product Recalls Scraper enforces one: a 31-day maximum, rejected before a single byte moves, at $4.20 per 1,000 recalls.

Why "no pagination" is a billing problem, not a convenience problem 💸

With a paginated API, an over-broad query is recoverable: you fetch page one, see the total, and stop. With this one, the first response is the whole thing. By the time you know the query was too wide, you have already paid to transfer it.

That matters more than it sounds. On a scraping platform, external data transfer is the single largest cost line — on our own bill it is roughly three-quarters of every dollar, far ahead of compute. An Actor that defaults to "all recalls" is not merely slow; it is expensive in the one dimension that actually bills.

So this Actor treats the date window as the pagination it does not have:

recallDateStart = 2026-01-01
recallDateEnd   = 2026-06-01
  -> ValidationError: date window is 152 inclusive day(s); max allowed is 31

That rejection happens in input validation — before any HTTP request and before the run-start fee is charged. A too-wide window costs you nothing at all, not even the $0.20. Want a year? Run twelve months as twelve slices, each bounded and each one you can see the size of.

The flattening rule, and why blank is not the same as empty 🧩

A CPSC recall is a nested document. One record carries arrays of Products, Hazards, Remedies, Retailers and Images, each an array of objects. Dropped into a spreadsheet raw, it is unusable.

Each nested array is joined into a single comma-separated string, which makes the row flat and filterable. The part worth stating is the null convention, because inconsistency here is what makes a dataset painful to query:

• a nested array that is absent or empty flattens to "" — an empty string
• a plain scalar that is absent or blank becomes null

So "" always means "this record listed no retailers", and null always means "this field had no value". They never swap places depending on which field you are looking at. That sounds pedantic right up to the first WHERE retailers IS NULL that silently misses half the rows it should have matched.

Underneath, the usual defences: 429s and 5xxs retried with capped exponential backoff honouring Retry-After, a malformed recall record validated, logged and skipped rather than sinking the whole batch, and a date window that legitimately contains no recalls finishing as a clean successful run rather than a false failure.

Is the CPSC API hard to scrape? 🛡️

No, and we will not dress it up — saferproducts.gov is a public US federal endpoint with no challenge page and no anti-bot wall. We probed it before writing any client code.

What it has instead is an unbounded response and no built-in way to ask for less. The failure mode is not being blocked; it is a run that succeeds, transfers far more than anyone intended, and bills for it. That is a design problem, and the guard above is the design.

FAQ

What does CPSC cover?
US Consumer Product Safety Commission recalls — consumer goods, not food or drugs. For FDA-regulated recalls, that is a different regulator and a different dataset.

Why cap the window at 31 days?
Because the API returns everything matching in one unpaginated response and egress is the dominant cost. Run consecutive months to cover a longer period.

Do I get the hazard and remedy details?
Yes — Products, Hazards, Remedies, Retailers and Images are each flattened into their own column.

How is it billed?
$0.20 per run start plus $0.004 per recall row, so 1,000 recalls cost $4.20. An invalid window is rejected before the start fee is charged.

---

Built by Devil Scrapes. We publish the traps we hit, because the ones that return 200 OK are the expensive ones.