SEC's XBRL API is the best-kept secret in financial data. No key, no scraping of HTML filings, no vendor contract: structured, comparable, machine-readable financial facts for every company that files with the SEC, straight from the source.
Then you ask two companies for the same concept and one of them 404s. Nothing is broken. That 404 is the most important thing to understand about the whole API.
Quick answer
A 404 from SEC's XBRL API usually means "this filer did not report this concept", not "your request was wrong." Most (company, tag) and (tag, period) combinations legitimately do not exist, because companies report under different US-GAAP concepts. So the architecture question is not how to avoid 404s — it is fault isolation: one missing combination must skip one row, never sink the run. Plus three operational musts: a descriptive User-Agent (SEC's Fair Access policy blocks you without one), 10 requests/second, and CIKs zero-padded to 10 digits.
Two endpoints, two questions
The API answers two different shapes of question and it is worth picking deliberately.
frames — one concept, one period, every filer:
GET /api/xbrl/frames/us-gaap/Assets/USD/CY2023Q1I.json
That is a cross-section: total assets for every company as of Q1 2023. Perfect for screening and peer comparison.
companyconcept — one filer, one concept, full history:
GET /api/xbrl/companyconcept/CIK0000320193/us-gaap/Revenues.json
That is a time series. Perfect for trend work on a specific company.
Same data, transposed. Frames mode gives you a wide market snapshot; company mode gives you depth on names you already care about.
The frame id format is stricter than it looks
Frame ids look casual and are not:
CY2023 annual
CY2023Q1 quarterly (duration)
CY2023Q1I instant
That trailing I is the difference between a flow and a stock. Revenue is a flow — it happens over a quarter, so it lives in CY2023Q1. Assets are a stock — they exist at a moment, so they live in CY2023Q1I. Ask for assets without the I and you get a 404 that has nothing to do with the company and everything to do with accounting.
We validate the id shape at the input boundary rather than letting SEC's 404 be the error message:
FRAME_PERIOD_RE = re.compile(r"^CY\d{4}(Q[1-4])?I?$")
A malformed period is a mistake we can name precisely. A 404 is not.
Why most 404s are correct answers
US-GAAP is a large vocabulary and a filer only reports the concepts that describe its business. Here is the same tag against two filers, checked live:
CIK0000320193 (Apple) InterestAndDividendIncomeOperating -> 404
CIK0000019617 (JPMorgan) InterestAndDividendIncomeOperating -> 200
Apple is not a bank, so it has no interest-and-dividend income line to report. Nothing is wrong with the request, the tag, or the API. The concept simply does not apply.
The same thing happens across revenue tags — one filer reports Revenues, another RevenueFromContractWithCustomerExcludingAssessedTax, many report both — so no single tag covers the market, and a screen built on one tag silently omits everyone who chose another.
So a run over 50 companies × 3 tags will produce a lot of 404s, and that is the API working. The design consequence is that the fetch layer must never raise on one:
if response.status_code == HTTP_NOT_FOUND:
logger.warning("%s: HTTP 404 — not reported, skipping", context)
return None
None means "skip this combination", not "fail". Non-retryable 4xx behaves the same way. Only 408/429/5xx get retried, with capped exponential backoff.
This is the single highest-leverage pattern in our whole fleet. When we audited our lowest-success-rate Actors, the top cause was never a hard block — it was a recoverable per-item error crashing the entire run, so a customer lost 400 good rows because row 401 was unusual. Every Actor now isolates per item.
SEC's Fair Access rules are not optional
Two hard requirements, both easy to get wrong:
A descriptive User-Agent. SEC's Fair Access policy expects a real identifier. Send a default library UA and you will eventually get blocked — and unlike a fingerprinting wall, this one is documented policy, so complaining is not a strategy. Ours is configurable and defaults to our public brand contact URL, never a personal mailbox.
10 requests per second, maximum. With one in-flight request at a time, a 100 ms sleep before each attempt enforces the whole limit with no token bucket, no shared state, and nothing to get wrong under concurrency:
MIN_REQUEST_INTERVAL_S = 0.1
We also honour Retry-After when SEC sends it, in preference to our own backoff. If the server tells you when to come back, argue with it at your peril.
Tickers in, CIKs out
Nobody wants to look up a CIK. So the Actor accepts tickers or CIKs, and resolves tickers through SEC's own public map:
GET https://www.sec.gov/files/company_tickers.json
The one gotcha: CIKs are zero-padded to 10 digits in API paths. Apple is 0000320193, not 320193. The raw map gives you the integer, so the pad is on you:
str(int(cik_raw)).zfill(10)
Miss it and every request 404s — which, given everything above, you will initially misread as "this company did not report that."
FAQ
Is scraping SEC's XBRL API legal?
It is a public API the SEC publishes for exactly this purpose. Follow the Fair Access policy — identify yourself, stay under 10 req/s — and you are inside their stated rules.
Do I need an API key?
No. It is keyless. The User-Agent is the closest thing to identification and it is required.
Why is start_date null on some rows?
Because instant facts (balance-sheet items) have no duration — only an end_date. That is expected, not a gap.
Which tags should I use?
Start with Revenues, Assets, NetIncomeLoss — widely reported across sectors — then widen. Any US-GAAP, IFRS-full or DEI concept works, and coverage varies by filer and by industry, which is what makes the 404-tolerant design necessary rather than defensive.
Ready to run: SEC XBRL Financials Scraper — frame mode for one concept across every filer, company mode for one filer's full history; tickers or CIKs, any US-GAAP/IFRS/DEI tag, out to JSON, CSV, or Excel.
We do the dirty work so your dataset stays clean. 😈
Top comments (0)