The API where page 21 is a 400 and 'no results' is a 404

The Federal Register API has a results window that ends at document 2,000 — and the error you get when you cross it is an HTTP 400 that looks like your bug. It also answers a perfectly healthy "no matches" query with a 404, which most HTTP clients treat as "the endpoint is gone."

Both behaviours are real, both are undocumented where you'll actually look, and both will pass every test you write against a happy-path fixture.

Quick answer

Building on www.federalregister.gov/api/v1/documents.json? Three rules:

1. You cannot page past 2,000 results. The API returns HTTP 400 once page × per_page crosses 2,000. Partition your query by date range and page inside each slice.
2. A 404 means "no documents matched", not "dead endpoint". Treat it as an empty result set, not an error.
3. per_page maxes at 100. Anything higher is clamped or rejected, so a 10,000-document sweep is at minimum 100 requests before you even hit the window ceiling.

The 2,000-document window

A search for a common regulatory term matches tens of thousands of documents — we measured 10,000+ hits on a single term during our probe. The API happily reports that total, then refuses to show you most of it:

page=20, per_page=100  → HTTP 200   (document 1,901–2,000)
page=21, per_page=100  → HTTP 400   (document 2,001+ — the window is shut)

The fix is the same one every deep-paging API forces eventually: partition by publication_date, run each slice under the 2,000-document ceiling, and stitch. A scraper that silently stops at 2,000 and exits green is lying to its user — ours logs the ceiling and narrows the date slices automatically.

Why does the Federal Register API return 404 on a valid query?

Because that's its way of saying "zero documents matched". A conditions query with no hits doesn't return {"count": 0, "results": []} — it returns HTTP 404. If your client raises on any 4xx (the default in most libraries), your pipeline dies on the first quiet news day, and the stack trace points at your URL builder, not at the API's convention.

The correct behaviour is exactly what our client does: catch the 404, emit zero rows, exit clean. An empty result set is an answer, not a failure.

Is the Federal Register API free?

Yes — fully keyless, no registration, no OAuth. Which is why the whole difficulty of the target is the two conventions above plus schema discipline: documents omit fields freely (agencies, abstracts, page counts come and go per document type), so every optional field in your output schema needs to be nullable or your dataset writer dies mid-run on the first sparse record.

What the scraper actually does

Federal Register Scraper searches the Federal Register — proposed rules, final rules, notices, presidential documents — by term, agency, document type and date range, and returns one flat row per document: document number, title, type, abstract, agencies, publication date, comment deadline and the canonical HTML/PDF URLs, as JSON, CSV or Excel. Pay-per-result at $2.05 per 1,000 documents; a query that matches nothing costs you only the start fee.

The paging window, the 404-means-empty convention, the sparse-field schema — absorbing those is our job, not yours. 😈

If you track the regulatory pipeline end-to-end, we also run FDA recalls, FDA 510(k) clearances and SEC XBRL financials.

The one-line version

On the Federal Register API: 404 means "no matches", 400 past document 2,000 means "partition by date", and every optional field is optional. Three rules, whole integration.