Our first cloud smoke test of the FDA 510(k) scraper came back green. Status: SUCCEEDED. Rows: zero.
The scraper was fine. The example input was the bug — and the verifier we built specifically to catch bad example inputs passed it, correctly.
Quick answer
A prefill (Apify's example input) that validates is not a prefill that returns data. Ours carried a value for every filter, so the smoke run ANDed applicant AND device name AND product code AND advisory committee AND device class AND a date range into a query openFDA legitimately matches nothing for. Validation checks shape; only a live query checks that the shape matches reality. Three more openFDA traps below: skip caps at 25,000, a + in a search term returns HTTP 500, and a no-match search answers 404 rather than an empty list.
The green run with no rows
We have a pre-publish gate called verify_input_prefill.py. Its job is to assemble the input schema's prefill and default values into a payload and confirm the Actor's Pydantic input model accepts it. It exists because a prefill that fails validation means a customer's first click errors out.
It passed. It was right to pass. Every field was well-typed, every date well-formed, the payload validated cleanly.
And the run returned nothing, because openFDA's search parameter ANDs its clauses, and the intersection of six independent filters over a database of device clearances is empty far more often than it is not. Every individual filter was plausible. The conjunction was a query about a device that does not exist.
The fix was to the prefill, not the code:
before: applicant + device name + product code + committee + class + date range
after: applicant + date range # confirmed live to match 24 records
Re-pushed, re-QA'd, rows arrived.
The general lesson we wrote into our build queue: a smoke test must assert on rows, not on exit status. "SUCCEEDED" is a statement about our process. Rows are a statement about the customer's experience. We had been treating the first as evidence of the second, which is the same conflation as treating "the tests pass" as evidence that the feature works.
Trap 1: skip caps at 25,000 — and not where you think
openFDA pages with limit and skip. limit maxes at 1,000. skip refuses to go past 25,000:
skip=25000 -> 200 OK
skip=25001 -> 400 "Skip value must 25000 or less."
There is a specific trap here for anyone who has already written an openFDA scraper. We ship a sibling Actor against the recalls endpoint, and that one has a different ceiling. Copying the constant across endpoints produces a scraper that dies 1,000 rows early or 1,000 rows late, depending on which direction you copied. Each openFDA endpoint gets its own verified constant, checked against the live API, with the verification date in the comment.
Trap 2: a + in a search term returns HTTP 500
This one is pure encoding. openFDA's query language treats + as meaningful, so a company name containing a literal + — url-encoded to %2B — makes the server answer 500. Encode the same character as a space (%20) and it parses fine.
It is worth naming because a 500 reads as "their server is having a bad day", which invites a retry loop rather than a fix. It is deterministic: same input, same 500, forever. Our sibling recalls Actor hit exactly this and the fix transfers unchanged.
Trap 3: no matches is a 404, not an empty list
Search for something real that simply has no records and openFDA answers 404, not {"results": []}.
So a client that treats 404 as a hard error reports "the API is broken" when the honest answer is "your filter matched nothing". And a client that retries 404s burns its retry budget on a settled question. We treat this one status as a terminal, non-error, zero-results outcome, and say so in the log.
There is a fourth, smaller one worth knowing: a cold query can 500 once and then 200 on an immediate retry. Genuinely transient, unlike trap 2. Bounded retries with backoff on 408/429/5xx cover it — but only because trap 2's deterministic 500 was fixed at the source rather than retried into the ground.
What is in a 510(k) record?
The premarket notification pathway is how most medical devices reach the US market — a claim of substantial equivalence to an already-cleared device. Each record carries the applicant, the device name, the decision date and decision code, the product code, the advisory committee, the device class and the regulation number.
That is a regulatory-intelligence dataset: who is clearing what, in which class, how fast, and through which committee. Filter by applicant to watch a competitor, by product code to watch a device category, or by decision-date range to build a clearance timeline.
FAQ
Is scraping openFDA legal?
openFDA is the FDA's own public API, published for reuse, no key required for normal use. Read their disclaimer about the limits of the underlying data before you make decisions with it.
Do I need an API key?
No. Keys exist only to raise rate limits; the unauthenticated tier is enough for ordinary extraction.
How many records can one run pull?
Up to openFDA's own skip ceiling of 25,000 per query — its own 400 response points at search_after for going deeper. Until we wire that up, narrow by date range or product code and slice a bigger set into runs that each stay inside the ceiling.
Why did my query return nothing?
Almost always because the filters ANDed into an empty intersection — the exact bug that produced our zero-row green run. Drop filters one at a time until rows appear.
Ready to run: FDA 510(k) Clearances Scraper — filter openFDA 510(k) clearances by applicant, device name, product code, advisory committee, device class, or decision-date range; export as JSON, CSV, or Excel.
We do the dirty work so your dataset stays clean. 😈
Top comments (0)