Quick answer
SEC EDGAR has a full-text search index over every filing since 2001 — 10-Ks, 8-Ks, S-1s, comment letters, all of it — and the backend behind the official search page is a public, keyless JSON endpoint at efts.sec.gov.
Wrapping it is not hard. Wrapping it correctly comes down to one field most clients ignore, and it is the difference between "we returned everything" and "we returned the first page and stopped."
That is the SEC EDGAR Full-Text Search Scraper. Give it a phrase; get one row per matching filing.
total.relation is the whole ballgame 🎯
Every response carries a hit total shaped like this:
{"hits": {"total": {"value": 10000, "relation": "gte"}, "hits": [...]}}
That is Elasticsearch underneath, and relation has two values:
-
"eq"— the total is exact. 4,312 means 4,312. -
"gte"— the total is a floor. Elasticsearch stopped counting at 10,000 and is telling you "at least this many."
The obvious pagination loop is while offset + page_size < total: fetch(). It is correct under "eq" and quietly wrong under "gte", because the floor is not the total. Depending on which way you round, you either stop early on a broad query or you page into a void.
The rule we settled on: a short page is the only proof there is nothing left. An exact total may end pagination early as an optimisation. A "gte" total may never be used to stop, ever. It is one branch, and it decides whether a broad search silently truncates.
The 10,000-row ceiling is a real wall, and it announces itself 🧱
Page deep enough and EDGAR stops returning results and starts returning an error envelope instead — its own pagination ceiling.
This is worth handling explicitly rather than letting it surface as a crash, because the honest thing to tell a customer is "you hit EDGAR's ceiling, narrow your query" — not a stack trace, and definitely not a silently short dataset that looks complete. We detect the overflow envelope, stop cleanly, and say so in the run status.
If you need more than 10,000 hits, the answer is not a cleverer client. It is slicing your query by date range and unioning the results.
The fair-access gate that looks like a block 🛡️
EDGAR requires a descriptive User-Agent with contact information. Send a generic one and you get 403s that look exactly like anti-bot defence. They are not — it is the SEC's stated fair-access policy, and it is entirely reasonable. Send a real identifying UA and the endpoint is well-behaved and generous.
One more, found live during the build: efts.sec.gov will occasionally answer 500 on rapid successive requests and then serve the identical request fine seconds later. That is a genuine transient on a non-adversarial host, so we retry it alongside the usual 408/429/503 — a distinction worth making deliberately, because on a hostile target retrying a 500 is how you get yourself banned.
What a row looks like
Accession number, form type and root form, filed date, period ending, filer names with CIKs, SIC codes, business locations, file numbers, relevance score, and a resolved link to the filing's index page.
We spot-check rows back against the SEC before shipping. One row came out as accession 0001104659-06-003856, form CORRESP, filed 2006-01-25, filer Vistula Communications Services. Opening that index page on sec.gov shows exactly that — same form, same date, same filer. Rows existing is not proof they are real; we check that they are.
Who this is for 🎯
- Compliance and legal research — every 8-K mentioning "material weakness" in a quarter, market-wide.
- Investigative journalism — trace a product name, an executive, or a lawsuit across every filer who mentioned it.
- Due diligence — did this company's own 10-Ks ever say "going concern"?
- Market research — count filers referencing a technology or regulation over a window.
The honest limitations 🚧
- The full-text index starts at 2001-01-01. That is a platform limit, not something a scraper can route around.
- 10,000 rows per query is EDGAR's ceiling. Slice by date to go deeper.
- You get the search hit and the index-page link — not the filing body, and not XBRL financials.
- Searching for one company's complete filing history is a different job; that is
sec-edgar-filings-scraper.
Pricing
$0.20 per run plus $0.003 per row — $3.20 per 1,000. No matches means no per-row charge.
→ SEC EDGAR Full-Text Search Scraper on Apify
Built by Devil Scrapes. We handle the fair-access headers, the ceiling that ends a query, and the total that is only a floor.
Top comments (0)