Seven government recall feeds, and what it takes to make them agree

Nate B — Thu, 30 Jul 2026 01:35:57 +0000

Every developed market publishes its product recalls. The EU has Safety Gate, France runs
RappelConso on top of it, the US splits the job across the CPSC, NHTSA and three separate FDA
enforcement databases. All of it is free, all of it is public, and most of it has an API.

So checking whether a product has been recalled should be easy.

It is not, and the reasons are more interesting than "the schemas differ." Here is what came out
of normalising 176,000 recalls from seven of these sources into a single queryable corpus.

The schemas differ, and that is the easy part

Safety Gate publishes XML per weekly report. RappelConso is an OpenDataSoft instance with a JSON
API. NHTSA ships a ZIP containing a pipe-delimited flat file. openFDA gives you clean JSON with
proper pagination. The CPSC has a REST API that mostly behaves.

Mapping five shapes onto one schema is a week of tedious work with a clear finish line. You write
a normaliser per source, you write tests, you are done. Nobody's project fails here.

Where it actually gets hard is that the sources disagree about what a recall is.

An openFDA enforcement report is one row per product in a recall event: a single event with
forty affected lot numbers is forty rows sharing an event_id. A Safety Gate notification is one
row per notification, which may cover several products from one manufacturer. NHTSA is one row
per campaign, which may cover multiple vehicle models across several years.

Deduplicate naively across sources and you undercount, because the same physical hazard
legitimately appears as three different record types. Do not deduplicate at all and a customer
searching a barcode gets forty near-identical hits for one event. Neither is what anybody wants.

We keep records at source granularity and reconcile at the identifier level instead. If a GTIN
appears in an openFDA row and a Safety Gate notification, the lookup returns both, and it is
obvious from the payload that they are two agencies describing one hazard rather than two
hazards. That decision is unglamorous and it is the single most consequential one in the project.

Identifiers live in prose, not in fields

The thing you actually want to search by — a barcode, a VIN, a model number — is frequently not a
field.

Here is a real openFDA code_info value, lightly trimmed:

Lot #1234567, Exp 03/2027; UPC 0 36000 29145 2; also lots 1234568,
1234569 (UPC 036000291452) distributed to CA, NV, AZ

That contains one GTIN written two ways and three lot codes, inside a free-text field with no
consistent delimiter. The FDA is not being careless. This is a field designed for a human
reading a recall notice, and it does that job well.

So identifiers have to be mined out of prose, which means a regex pass, and a regex pass over
free text means false positives. 0 36000 29145 2 is a GTIN. So is a 12-digit order reference
that happens to sit next to the word "UPC." The distinguishing feature is the check digit: a
real GTIN's last digit is a mod-10 checksum over the others, and a random 12-digit number passes
that test one time in ten.

Validating the check digit takes about fifteen lines and throws out roughly a fifth of the
candidates a naive regex finds. Skip it and your API confidently reports recalls against
identifiers that were never barcodes.

The corollary matters more: an invalid identifier must be rejected, never answered. If
someone queries a malformed GTIN and you return "no recalls found," you have told them their
product is clear when in fact you did not look it up. For a compliance tool that is the worst
possible failure: it is wrong in the direction the customer will not check. We return a 400.

The pagination caps nobody documents prominently

openFDA's enforcement endpoints paginate with limit and skip. The limit maxes at 1,000, which
is generous.

skip refuses anything above 25,000.

skip=25000  ->  HTTP 200
skip=25001  ->  HTTP 400  "Skip value must 25000 or less."

The missing word is theirs. I have grown attached to it.

The device enforcement dataset holds 39,538 records. Sort by report date, page through, and you
hit a wall 14,538 records short of the end, with a 200 OK on the last successful page and
nothing anywhere telling you the history is truncated. Your corpus is quietly incomplete and
every downstream number is confidently wrong.

I assumed an API key would raise the cap. It does not. A key raises the daily request quota
from 1,000 to 120,000 and leaves skip exactly where it was. Registering one would not have
recovered a single record.

The route past it is date windows. Constrain the search to one year and no window comes close to
the cap. The densest year measured is 3,757 records, so skip never exceeds 4,000 and the
ceiling stops binding. Sum the per-year totals and you get exactly 39,538. Full coverage, no key.

RappelConso has the same shape of limit with a different number, and fails more honestly: it
returns an error rather than a truncated success.

The 252 MiB surprise

This one cost a production outage, so it is worth being specific.

The pipeline archives every raw payload before parsing it, so a parser bug never means re-hitting
seven government endpoints to recover. Good rule. It assumes the payload fits in memory.

Profiling the cold path per source:

                    wall        cpu     payload
openfda-device    46,889ms   20,672ms   251.9 MiB
openfda-food      16,027ms    4,406ms    27.4 MiB
openfda-drug      13,010ms    2,891ms    21.6 MiB
cpsc              16,729ms    2,734ms    26.1 MiB
nhtsa              5,206ms    4,515ms    14.1 MiB
rappelconso       29,466ms    2,078ms    22.2 MiB
eu-safety-gate    36,083ms    1,313ms     6.4 MiB
                            ─────────
                             38,609ms

Device enforcement records carry long product descriptions listing every affected catalogue
number, so 25,000 records come to 252 MiB. This ran on Cloudflare Workers, where an isolate gets
128 MB and 30 seconds of CPU by default. The payload cannot exist. Gzipping it alone — a step
that produces no data — costs 13.6 seconds, 35% of the entire budget.

The failure mode is what makes it worth writing down. A Worker killed for exceeding its limits
does not unwind. No exception, no catch, no finally. The row we had written to say "this
source is running" stayed saying that forever, the failure counter stayed at zero, and the alert
that fires after three consecutive failures never fired, because as far as the database was
concerned nothing had failed.

Six of seven sources stopped ingesting and every dashboard reported healthy. It was caught by
building a status page and noticing it said "all sources current" beside a tile reading "1 / 7."

Two lessons, both cheap:

Reap your own abandoned work. When a run starts, close out anything still marked running from
last time and count it as a failure. That converts an invisible kill into a countable event, and
it is about eight lines.

Do not let independent jobs share one budget. Seven sources ran in one invocation, so the
most expensive decided whether the cheapest ran at all. One trigger per source costs nothing and
removes the entire failure class.

Streaming is not a memory fix

Look again at the profiling table. NHTSA sits near the bottom at 14.1 MiB, one of the cheapest
sources there. That number is a lie, and it is a lie of my own making: it measures the payload
as served, and NHTSA serves a ZIP.

A profiler that records what arrives on the wire reports this source as harmless. The number that
matters is not visible until something tries to decompress it:

$ ls -l FLAT_RCL_POST_2010.zip
   14713512  FLAT_RCL_POST_2010.zip     # 14 MB. Cheapest source in the table.

$ unzip -l FLAT_RCL_POST_2010.zip
  308496873  FLAT_RCL_POST_2010.txt     # 294 MiB. Hm.

$ # isolate memory limit: 128 MB
$ #
$ # ah.

The isolate found out before I did:

RangeError: Invalid typed array length: 308496873

That is not an approximation of the problem, it is the uncompressed size. The error message and
the file listing agree to the byte, which was the most helpful thing that happened that morning.

unzipSync allocates the decompressed member as a single buffer, in a 128 MB isolate.
Deterministic, every run, at the same instruction. It never looks intermittent once you have the
number in front of you.

The obvious fix is to stream: inflate through DecompressionStream, and collapse rows as they
arrive rather than materialising the file. That reduction is real: the flat file carries one
row per campaign per model, so 243,199 rows collapse to 15,133 campaigns, and a campaign
covering eight models arrives as eight rows carrying identical defect prose with only the model
name differing. Fold rows in as they arrive and each one becomes garbage immediately.

I wrote that, measured the retained text at 15.4 MiB, and it still ran out of memory at a
192 MB heap.

The reason is worth knowing, because it is invisible in the code. In V8,
String.prototype.split and slice return sliced strings: a length, an offset, and a
pointer to the string they were cut from. The substring does not own its characters. So
retaining one short field per campaign kept alive every 1 MB decoded chunk that any field had
been cut from, and all 294 MiB stayed reachable. Streaming had moved the allocation without
removing it. The heap profile showed 15 MiB of strings I wanted and 200 MiB of parents I did
not know I was holding.

The fix is to decode each field from the line's bytes with TextDecoder, which returns a fresh
string that owns its own characters and points at nothing:

unzipSync + split          killed at 192 MB heap
stream + split             killed at 192 MB heap
stream + decode per field  67.1 MiB peak, 1.6s parse

Two things I would take to the next project. "Stream it" is not a memory fix by itself.
Retention is decided by what the surviving objects point at, and in V8 a substring points at its
parent. And profile the size you process, not the size you fetch. Every compressed source is
a payload figure hiding a multiplier, and this one was 21×.

The amendments nobody announces

The part I did not expect to matter most.

Agencies revise recalls after publication. A recall issued in March covering three lot codes
covers eleven by June. The URL does not change. There is no changelog, no revised_at field, no
notification. The page just says something different than it did.

If you check a product against a recall in April, clear it, and the recall widens in June to
include it, nothing tells you. You checked, you have a record of checking, and your record is now
wrong.

The only way to see this is to diff. Store a content hash per record, compare on every ingest,
and write a field-level delta when it moves. Across this corpus that produces a steady stream of
amendments: models added, date ranges extended, hazard descriptions sharpened. None of it was
announced anywhere.

For anyone doing this for compliance rather than curiosity, that stream is the actual product.
Under the EU's General Product Safety Regulation the obligation is continuous diligence, and
"we subscribed to a feed" is not evidence of it. What you need is a dated record of what you
checked, against a corpus of known freshness, with the result frozen as it stood, so that a
later amendment does not silently rewrite your own audit trail.

If you are building this yourself

Worth doing. It is a genuinely interesting normalisation problem and the data is free. Budget for
these, which is where the time actually goes:

Reconciling record granularity across sources — days, and the decision is irreversible
Mining and check-digit validating identifiers out of prose — a week, and skimping shows
Finding every undocumented pagination cap — you find these by auditing totals, not by reading docs
Amendment detection — cheap to build, and worth more than the rest combined
Not letting one oversized source take down the run — one afternoon, after it bites you once
Measuring decompressed sizes, not download sizes — five minutes, and it would have saved me two

The schemas are the part that looks hard and isn't.

The corpus described here is available as an API and as a dated, checksummed export at
recallproven.com. Every source is under an open licence permitting
commercial reuse; the attribution obligations are published at
/v1/sources and travel with every export.

DEV Community: Nate B