DEV Community

Devil Scrapes
Devil Scrapes

Posted on

openFDA Tells You the Outcome Was 5

Quick answer

openFDA's FAERS endpoint is free and keyless, and it will hand you a spreadsheet in which the patient's sex is 2 and the outcome of the reaction is 5. Those are not IDs to look up later — they are the values, and FDA's data dictionary is the only thing that turns 5 into fatal. The openFDA Adverse Events Scraper decodes every coded field and explodes each nested report into one flat row per reaction, at $4.20 per 1,000 rows.

The trap: a column of integers that looks like data 🔢

FAERS encodes several of its most important fields as bare numeric strings. Two you will hit immediately:

patientsex       1 = male    2 = female
reactionoutcome  1 = recovered/resolved
                 2 = recovering/resolving
                 3 = not recovered/not resolved
                 4 = recovered with sequelae
                 5 = fatal
                 6 = unknown
Enter fullscreen mode Exit fullscreen mode

Nothing in the payload hints that these are enumerations. A 1 and a 2 in a column called patientsex read like a flag, and a reactionoutcome of 5 reads like a severity score where higher is worse — which is nearly right and therefore dangerous, because 4 (recovered with sequelae) and 6 (unknown) are not more severe than 5 (fatal). Sort by that column and you have put "unknown" above "the patient died."

This Actor maps both, plus the report's dates from FDA's YYYYMMDD into ISO YYYY-MM-DD, so what lands in your dataset is female and fatal and 2026-03-14. An unparseable date is passed through unchanged rather than nulled — losing a value you cannot parse is worse than carrying it.

The shape problem: one report is not one row 🧬

A FAERS report is a nested document: patient.drug[] holds every drug the patient was taking, patient.reaction[] holds every adverse reaction observed. One report routinely carries several of each. There is no single obvious row for it.

Flattening to one row per report forces you to cram a list of reactions into one cell, and then nobody can group by reaction — which is the main question people ask this dataset. So this Actor emits one row per reaction, each carrying the report's full drug list joined into a single field. Group by reaction and the counts are right; filter by drug and the report still appears under each of its reactions.

That choice is why the billing unit is a row rather than a report: a report with four reactions is four rows and costs four rows. The README says so up front, because pricing that surprises someone after the run is a worse outcome than pricing they disliked before it.

Paging is skip/limit against the meta.results.total the API reports for your specific query, so the run stops on the real total instead of probing for an empty page. A malformed report is validated, logged with its ID and skipped — one bad record in a page of a hundred does not sink the other ninety-nine.

Is openFDA hard to scrape? 🛡️

No, and we are not going to pretend it is. It is a public federal API with no challenge page and no anti-bot wall — we probed it before writing any client code. What it does have is an unauthenticated rate limit, so the run throttles politely and backs off on 429 with Retry-After honoured rather than hammering a government endpoint until it stops answering.

The work here is in the decoding and the shape, not in getting a response. That is a less dramatic kind of difficulty and a more expensive one to get wrong, because every failure mode above produces a file that opens cleanly and is quietly incorrect.

FAQ

What is FAERS?
The FDA Adverse Event Reporting System — voluntary reports of adverse events and medication errors submitted to the FDA.

Can I use this to make medical decisions?
No. FDA says so and so do we: the reports are self-reported and unvalidated, a report is not proof a drug caused the event, and duplicate reports exist. This is research and signal-detection data.

How do I search it?
With openFDA's own query syntax, e.g. patient.drug.medicinalproduct:"ozempic". The demo input ships with exactly that so the first run returns rows.

How is it billed?
$0.20 per run start plus $0.004 per row, and a row is one reaction. 1,000 rows cost $4.20.


Built by Devil Scrapes. We publish the traps we hit, because the ones that return 200 OK are the expensive ones.

Top comments (0)