DEV Community

Cover image for Building a SEC EDGAR Filings Scraper: 10-K Risk Factors and 13F Holdings as Clean JSON
Oaida Adrian
Oaida Adrian

Posted on

Building a SEC EDGAR Filings Scraper: 10-K Risk Factors and 13F Holdings as Clean JSON

#python #webscraping #finance #tutorial

US public companies file everything with the SEC, and EDGAR is free to use. But the raw archive is hostile to analysis: filings are HTML documents with table-heavy layouts, and the full-text search UI is built for humans, not pipelines. I built an actor that turns a ticker into a clean corpus of JSON records — one per filing — ready for financial alt-data, risk analysis, RAG corpora, or LLM fine-tuning.

The source

No API key, no data vendor. Three official SEC endpoints:

  • company_tickers.json — ticker to CIK resolution
  • the EDGAR full-text search JSON API (efts.sec.gov/LATEST/search-index) — query by ticker, form type, and date window
  • the SEC Archives for the documents themselves

Two traps surfaced immediately. First, SEC fair-access enforcement is real: a bare curl or python-requests User-Agent gets a 403, while a descriptive UA (Darknezz Research admin@…) gets a 200. The actor sets a descriptive UA, paces at ~5 requests/second (well under the 10 rps limit), and backs off on 429s. Second, the ciks search parameter requires the 10-digit zero-padded CIK — 320193 returns zero hits, 0000320193 returns the filing. Padding is mandatory.

The structuring

Raw HTML is only the beginning. Each filing becomes:

  • Metadata — accession number, ticker, company, CIK, form type, filing date, period ending, 8-K event items, document and index URLs
  • Sections — split on Item headings (Item 1., Item 1A., Item 7., Item 2.02…), per-section capped so one enormous filing can't blow a record
  • Risk factors — the full 10-K/10-Q Item 1A text, verified on a real Apple 10-K at the 60k-character cap
  • Holdings — 13F-HR InfoTable XML parsed into issuer, CUSIP, value, shares, and voting authority. The XML is namespace-prefixed (<ns1:infoTable>), so every regex is namespace-tolerant
  • Summary — a deterministic, LLM-ready markdown digest: metadata, section map, risk excerpt, holdings preview. No external LLM call, no extra cost

The smoke test

  • AAPL 8-K ×3 — sections Item 2.02 and 9.01, summaries 476 chars each
  • AAPL 10-K ×1 — 10+ sections, Item 1A risk factors 60,038 chars, summary 3,267 chars
  • JPM 13F-HR — 34,064 positions parsed locally; the mega-manager cap (20,000 positions) proven, with holdingsCount and holdingsTruncated flags so the record stays under Apify's item size limit

All cloud runs succeeded with non-zero, well-formed output.

The honest bits

  • Each filing yields its primary document only; exhibits (press releases, contracts) stay one click away via the filing index URL.
  • The summary is deterministic, not generative — great for pipelines, not a substitute for an LLM pass.
  • Mega-manager 13Fs are truncated at 20,000 positions (flagged), because a 9 MB JSON record doesn't ship.

Try it

👉 SEC EDGAR Filings Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Top comments (0)