Every time a company insider (an officer, a director, or a 10% owner) buys or sells their own company's stock, they have two business days to file a Form 4 with the SEC. The filings are public, free and machine-readable on EDGAR. So building an insider-trading alert service sounds like a weekend job: poll a feed, parse some XML, send an email.
It wasn't. Here is what one real day of data taught me: 334 filings, 1,220 transactions, 151 companies.
1. The real-time feed quietly drops filings
EDGAR has a real-time Atom feed of new filings. It only ever shows the most recent batch, roughly the last 100 entries. On a busy afternoon, filings scroll off before your next poll.
I found out the hard way. For one day, my poller had captured 44 of 334 Form 4s. It had started mid-Sunday, but even while it runs continuously, busy windows can overflow the feed.
The fix is to treat the feed as a fast path, not the source of truth. EDGAR also publishes a daily full index (form.YYYYMMDD.idx) that lists every filing for the day. A reconciliation job diffs yesterday's index against what the poller caught and backfills the gap. That backfill recovered the other 290 filings, each tagged source='reconciliation' so capture rate stays measurable. The job exits non-zero when capture drops below target, so a silent regression turns into a failed systemd unit I actually see.
2. You can't guess the XML filename
Each filing has an index page, and somewhere behind it is the XML document you actually want. I assumed it would be called form4.xml. In real data I saw form4.xml, primary_doc.xml, ownership.xml, doc4.xml and assorted filer-specific names.
The only reliable approach is to fetch the -index.htm page and resolve the primary document from it. That costs one extra request per filing, which matters because of point 5.
3. A $1.6 quadrillion purchase
My first parse of that day included a single insider purchase worth about $1.6 quadrillion.
The culprit was a MetLife filing for $40M of notes. Debt securities report <valueOwnedFollowingTransaction> (a dollar amount) where stock reports <sharesOwnedFollowingTransaction>. My code multiplied "shares" by price and got nonsense.
The fix was a schema-driven is_value_denominated flag rather than guessing from the security title text. The real filing is now a permanent regression fixture. Lesson: when a field has two schema variants, branch on the schema, never on a string heuristic.
4. Most "insider selling" isn't a decision
This was the big one. Form 4 transactions carry one of 20 SEC transaction codes. Here is how those 1,220 transactions broke down:
| Category | Share |
|---|---|
| Open-market sale | 53.4% |
| Compensation award | 16.6% |
| Option exercise | 11.6% |
| Other (needs review) | 7.5% |
| Open-market buy | 3.9% |
| Tax withholding | 3.0% |
| Everything else | 4.0% |
Only two of those categories are an insider choosing to trade at market: open-market buys and sales. Everything else is mechanical: shares vesting, shares withheld for tax, options exercised on schedule, gifts.
Even the sales need a second filter. Of 652 open-market sales, 109 (17%) carried the 10b5-1 checkbox, meaning they were pre-scheduled months earlier under a trading plan. That's a floor, not a ceiling: some filers only mention the plan in a footnote.
Once you strip all that out, the signal is small, and buys are rare. A tool that alerts on "insider selling" without this step mostly alerts on payroll.
I also stopped trying to be clever with code J ("other"). I spot-checked a couple by hand. One was a venture fund's pro-rata distribution to its partners, another a dividend-equivalent RSU accrual. Neither is a market signal, and no rule would have guessed that from the code. They get routed to review instead of guessed.
5. EDGAR has manners, and enforces them
The SEC asks for at most 10 requests per second and a User-Agent with your app name and a contact email. Break either and you get throttled or blocked.
A token-bucket limiter kept well under the cap, plus backoff on 429/403. Over a 14-hour soak (838 poll cycles) it recorded 0 rate-limit violations.
The stack
Nothing exotic: Python 3.12, httpx, Postgres, FastAPI with Jinja2 and htmx for the web app, Auth0 for login, and systemd timers on a single small VM. Polling runs continuously. Parsing, classification and alerting run every 5 minutes. Cluster detection runs every 30 minutes and flags 3+ insiders at the same company trading the same direction within a few days. Reconciliation runs daily.
What I built with it
All of this became InsiderTradingRadar: Form 4 alerts minutes after filing, with the mechanical noise and 10b5-1 plan trades filtered out, cluster detection, and historical context on each alert. The free plan covers one ticker by email. If you trade or just find this data interesting, I'd love feedback, especially on filings you think I'm misclassifying.
Happy to answer questions about EDGAR's quirks in the comments.
Top comments (0)