openFDA returns HTTP 500 when you follow its own docs

openFDA's documentation tells you to join search terms with +AND+. Follow that literally in most HTTP clients and the API answers HTTP 500. The docs are not wrong; your client is helpfully breaking them.

Here is the trap, plus a second one we reproduced live this morning that will flake your CI if you do not handle it.

Quick answer

Two things will bite you on openFDA and neither is documented where you will look:

1. + is not a literal plus. openFDA's docs show search=field:value+AND+other:value, but a URL-encoding HTTP client percent-encodes + to %2B, which openFDA parses as part of the term rather than as a separator — and returns a 500. Join with spaces and let your client encode them.
2. openFDA returns intermittent 500s on cold queries. We measured it again this morning: the first request timed out server-side after 31 seconds, and two immediate identical retries returned 200 in ~1.2 seconds each. If your client does not retry 500s, your smoke test is a coin flip.

The +AND+ trap in full

The openFDA query syntax is Lucene-ish. To combine terms, the docs show:

https://api.fda.gov/drug/enforcement.json?search=classification:"Class+I"+AND+status:"Ongoing"

That URL works when you paste it into a browser address bar, which is why it survives in documentation. It stops working the moment a real HTTP client touches it, because + is a reserved character in a query string. Most clients — and every sane URL-building library — will percent-encode a literal + in a parameter value to %2B. openFDA then sees %2BAND%2B decoded as +AND+ inside the term, not as a boolean operator, and the query blows up with a 500 rather than a helpful 400.

The fix is boring and total: use spaces.

params = {"search": 'classification:"Class I" AND status:"Ongoing"', "limit": 100}

Your client encodes the spaces as %20 (or as +, correctly, at the parameter level), openFDA parses the boolean, and you get results. We lost a build to this once. It is now the first line of our openFDA notes.

Why does openFDA return HTTP 500 randomly?

Because it sometimes just does, and you have to plan for it. This is not a theory — here is a measurement from 2026-08-18, three identical requests to the device clearance endpoint, run back to back:

try 1: HTTP 500   146 B   31.09 s
try 2: HTTP 200   21108 B  1.30 s
try 3: HTTP 200   21108 B  1.21 s

The 500 body is explicit about what happened:

{
  "error": {
    "code": "SERVER_ERROR",
    "message": "Check your request and try again",
    "details": "Request Timeout after 30000ms"
  }
}

That is a server-side 30-second timeout on a cold query, returned to you as a 500. The identical query is then served from warm state in about a second. Note the message: "Check your request and try again" is misleading — there is nothing wrong with the request.

The consequence for anyone building on this API: a 500 from openFDA is a retryable condition, not a permanent failure. Retry with backoff, at least three attempts. If you treat 5xx as fatal, your pipeline will fail intermittently in a way that is nearly impossible to reproduce on demand, because the second time you check, it works.

The paging ceilings nobody mentions until you hit them

Two hard limits, both of which return errors rather than empty pages:

• limit maxes out at 1000 per request.
• skip maxes out at 25000. Past that, openFDA errors instead of returning nothing.

So the maximum reachable window for a single query is 26,000 records. If your result set is bigger, you cannot page your way through it — you have to partition the query, usually by date range, and page inside each partition. Any scraper that silently stops at 25k and reports success is lying to its user; ours logs the ceiling explicitly and tells you to narrow the range.

The nullable-schema failure that only shows up in production

One more, because it is the same shape as the retry bug — it hides from your tests.

openFDA enforcement records routinely omit whole blocks. The openfda sub-object is absent on a large fraction of records. If you declare an output schema like this:

{ "product_type": { "type": "string" } }

then the first record without that field produces a null, the null fails validation, and the write dies mid-run — after you have already emitted rows and charged for them. A curated QA sample with fully-populated records never triggers it.

{ "product_type": { "type": ["string", "null"] } }

Every optional field in an openFDA-derived schema needs the union type. We fixed this at the scaffolder level so it cannot be reintroduced.

Is the openFDA API free?

Yes, and keyless for low volume — which is exactly why it is a good foundation. There is no API key to leak, no OAuth dance, no bot detection, and no HTML to re-parse when someone redesigns a page. The entire difficulty of this target is the four items above: encoding, retries, paging ceilings, and sparse records. All four are solvable once and then permanently solved.

What the scraper actually does

FDA Recalls Scraper queries openFDA's enforcement database and returns one flat row per recall event — recall number, classification, status, recalling firm, product description, reason for recall, distribution pattern, quantity, and initiation and termination dates — as JSON, CSV or Excel. It is pay-per-result at $2.05 per 1,000 rows, so a query that returns nothing costs nothing beyond the start fee.

We are not going to tell you this API is easy. It returns 500s on healthy queries, it has undocumented ceilings, and its own docs contain a query string that breaks in real HTTP clients. Absorbing all of that — the retries, the partitioning, the schema drift — is our job, not yours.

If you need the neighbouring regulatory sources, we also run ClinicalTrials.gov, NPI healthcare providers and professional license lookups.

The one-line version

On openFDA, join with spaces not +AND+, retry every 500, partition past 25k, and make every optional field nullable. Those four rules are the whole integration.