DEV Community: Mustapha

How to track insider buying with Python (SEC Form 4, free EDGAR data)

Mustapha — Wed, 28 Oct 2026 19:01:40 +0000

Disclosure: I built the Apify actor used in the second half of this post. It's a paid tool (pay per trade). The first half is free and uses only SEC EDGAR. Nothing here is investment advice.

When a CEO buys $500k of their own company's stock on the open market, the trade has to be public within two business days. It shows up on SEC Form 4, and anyone can read it on EDGAR for free.

Getting that data into code is harder than it should be:

OpenInsider is a great screener, but it has no API. People end up scraping HTML or using IMPORTXML in Google Sheets.
Paid insider-trading APIs typically cost $20–200 a month.
EDGAR itself is free, but each Form 4 is a separate XML file, the filing index links to a rendered HTML version, and SEC blocks scripts that don't identify themselves.

This post covers the free route first, then a hosted option for when you want the whole market every day with alerts.

What a Form 4 row tells you

Each Form 4 has one line per transaction. The field that matters most is the transaction code:

Code	Meaning	Signal?
`P`	Open-market or private purchase	The one people watch
`S`	Open-market or private sale	Often planned or for diversification
`A`	Grant or award	Compensation, not a decision
`M`	Option exercise	Mechanical
`F`	Shares withheld for tax	Mechanical
`G`	Gift	Usually estate planning

Since April 2023, Form 4 also has a Rule 10b5-1 checkbox for trades under a pre-arranged plan. Planned sales say much less about the insider's view than discretionary ones.

The free way: SEC EDGAR in ~35 lines

SEC's fair-access rules ask automated tools to declare a User-Agent with your name and e-mail and to stay under 10 requests per second. Without a declared User-Agent you get a 403 "Undeclared Automated Tool" page. Watch the classic bug: requests.get(url, headers) passes your dict as query parameters. It has to be headers=headers.

import time
import requests
import xml.etree.ElementTree as ET

HEADERS = {"User-Agent": "Jane Doe jane.doe@example.com"}  # your name + e-mail

def get(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    time.sleep(0.2)  # well under SEC's 10 requests/second
    return r

def val(node, path):
    el = node.find(path)
    return el.text.strip() if el is not None and el.text else None

TICKER = "NVDA"
tickers = get("https://www.sec.gov/files/company_tickers.json").json()
cik = next(v["cik_str"] for v in tickers.values() if v["ticker"] == TICKER)

recent = get(f"https://data.sec.gov/submissions/CIK{cik:010d}.json").json()["filings"]["recent"]
form4s = [(acc, doc, filed) for acc, doc, filed, form in zip(
    recent["accessionNumber"], recent["primaryDocument"], recent["filingDate"], recent["form"])
    if form == "4"][:20]

rows = []
for acc, doc, filed in form4s:
    # primaryDocument points at the rendered HTML ("xslF345X05/..."): drop the folder for raw XML
    url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{acc.replace('-', '')}/{doc.split('/')[-1]}"
    root = ET.fromstring(get(url).content)
    if int(val(root, "issuer/issuerCik")) != cik:
        continue  # the company itself was the insider at another issuer
    owner = val(root, "reportingOwner/reportingOwnerId/rptOwnerName")
    for t in root.findall("nonDerivativeTable/nonDerivativeTransaction"):
        shares = float(val(t, "transactionAmounts/transactionShares/value") or 0)
        price = float(val(t, "transactionAmounts/transactionPricePerShare/value") or 0)
        rows.append({"filed": filed, "date": val(t, "transactionDate/value"), "insider": owner,
                     "code": val(t, "transactionCoding/transactionCode"),
                     "shares": shares, "price": price, "value": round(shares * price)})

for r in rows:
    if r["code"] in ("P", "S"):
        print(r)

I ran this against NVIDIA's recent filings. It printed open-market sales by named executives, tax-withholding (F) lines and gifts (G), which is exactly why you filter on the code.

Things you'll hit next if you keep going:

Whole market: https://www.sec.gov/Archives/edgar/daily-index/2026/QTR4/form.20261027.idx lists every filing of the day by form type. Each Form 4 appears twice, once under the company and once under the insider, so dedupe on the accession number.
Derivatives: options, RSUs and warrants live in derivativeTable, with their own fields.
Footnotes: prices are often weighted averages explained in a footnote, and 10b5-1 plans are usually mentioned there too.
Amendments: a Form 4/A corrects an earlier filing. It doesn't replace it in the index.
State: for alerts you have to remember which filings you've already seen.

None of this is hard. It just adds up to a small project.

The hosted way: one call for the whole market

Insider Trading Tracker is the Apify actor I built that does the above at scale. It reads the same public Form 4 filings, returns one row per transaction (common stock and derivatives, with footnote text), and adds the 10b5-1 flag, holdings before and after, and cluster signals. It has three modes: latest (the whole market), tickers and insiders (by CIK or name).

With a free Apify account and your API token, this returns yesterday's discretionary open-market buys of $100k or more, market-wide:

import requests

TOKEN = "YOUR_APIFY_TOKEN"
url = ("https://api.apify.com/v2/acts/groupoject~sec-insider-trading-tracker"
       f"/run-sync-get-dataset-items?token={TOKEN}")

trades = requests.post(url, json={
    "mode": "latest",
    "lastNDays": 1,
    "transactionCodes": ["P"],
    "minTransactionValueUsd": 100000,
    "excludePlannedTrades": True,
    "maxResults": 500,
}, timeout=320).json()

for t in trades:
    print(t["issuerTicker"], t["insiderName"], t["insiderRole"],
          t["transactionDate"], t["shares"], t["pricePerShare"], t["valueUsd"])

For longer runs, such as a year of trades for a watchlist, use the client library (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("groupoject/sec-insider-trading-tracker").call(run_input={
    "mode": "tickers",
    "tickers": ["AAPL", "NVDA", "JPM"],
    "lastNDays": 365,
    "openMarketOnly": True,
    "includeDerivative": False,
})

for t in client.dataset(run["defaultDatasetId"]).iterate_items():
    plan = "10b5-1 plan" if t["is10b5_1Plan"] else "discretionary"
    print(t["transactionDate"], t["issuerTicker"], t["transactionCode"], t["valueUsd"], plan)

Each row also has transactionLabel (the code in plain English), sharesOwnedAfter, ownershipChangePct, isNewPosition, footnotes, isAmendment, acceptedAt (EDGAR acceptance time) and filingUrl.

A daily insider-buying alert in Slack

Save this input as a task in Apify Console:

{
  "mode": "latest",
  "lastNDays": 3,
  "transactionCodes": ["P"],
  "minTransactionValueUsd": 250000,
  "roles": ["officer", "director"],
  "excludePlannedTrades": true,
  "onlyNew": true,
  "stateKey": "daily-insider-buys"
}

Schedules → Create: run the task once a day in the evening (US Eastern). Most Form 4s are filed late in the afternoon.
On the task's Integrations tab, add Slack and post a message when a run succeeds.

onlyNew + stateKey make each run remember what earlier runs returned. A 3-day window catches late filers, but you only get, and only pay for, trades you haven't seen. A run with nothing new saves no rows. Use a different stateKey for each watchlist.

Cluster buys

One director buying can mean anything. Several insiders buying the same stock within days is the pattern people look for. Each row carries clusterBuy and clusterInsiderCount. By default that means 3 or more distinct insiders with open-market purchases within 10 days (clusterMinInsiders, clusterWindowDays). Use a date range at least as long as the window:

run = client.actor("groupoject/sec-insider-trading-tracker").call(run_input={
    "mode": "latest", "lastNDays": 14, "transactionCodes": ["P"],
    "clusterMinInsiders": 3, "clusterWindowDays": 10,
})

clusters = {}
for t in client.dataset(run["defaultDatasetId"]).iterate_items():
    if t["clusterBuy"]:
        c = clusters.setdefault(t["issuerTicker"], {"insiders": set(), "usd": 0})
        c["insiders"].add(t["insiderName"])
        c["usd"] += t["valueUsd"] or 0

for ticker, c in sorted(clusters.items(), key=lambda kv: -kv[1]["usd"]):
    print(ticker, len(c["insiders"]), "insiders", round(c["usd"]), "USD")

The run's SUMMARY record (free) also lists cluster buys by company and the top buys and sells by value.

What it costs

$2.50 per 1,000 trades on Apify's free plan, and lower on paid plans. Errors, non-matching filings and the SUMMARY record aren't charged. A typical trading day has a few hundred open-market buy and sell rows, so a full daily scan costs well under $1. A filtered buy alert usually costs cents.

Honest limits

Filing delay. Insiders have up to two business days to file. transactionDate can be a day or two before filingDate, and late filings do happen.
Amendments. A Form 4/A comes back as its own rows with isAmendment: true. It doesn't delete the original rows, so compare them yourself.
Derivatives. Option exercises, RSU vesting and conversions are included by default. Set includeDerivative: false or openMarketOnly: true to see only stock bought or sold with the insider's own money.
Counts differ from OpenInsider. The actor returns every transaction line. OpenInsider merges a filing into one line with an average price.
No price data. It reports what was filed. It doesn't tell you what the stock did afterwards.
Not investment advice. Insiders buy and sell for many reasons, including taxes, diversification, planned sales and option expiries. A cluster buy is a reason to read the filing, not a reason to trade.

If you only follow a few tickers, the free script above is all you need. If you want the whole market every evening, with derivatives, 10b5-1 flags and clusters already worked out, try the actor: apify.com/groupoject/sec-insider-trading-tracker. If something parses wrong, open an issue with the accession number. I read them daily.

How to get YouTube transcripts in bulk with Python (no API key, works on cloud servers)

Mustapha — Wed, 28 Oct 2026 16:55:21 +0000

Disclosure: I built the Apify actor used in this post. It is a paid tool (pay per transcript), and I say below where it doesn't fit.

If you have tried to get YouTube transcripts from code, you have probably hit one of these walls:

The official YouTube Data API doesn't help. Its captions.download endpoint only works for videos you own. For anyone else's video you get a 403.
youtube-transcript-api works on your laptop and then dies on a server. YouTube answers requests from cloud IP ranges (AWS, GCP, Hetzner, most VPSes) with a bot check, so the same code that ran fine locally starts throwing RequestBlocked / IpBlocked in production. Developers on the library's issue tracker discuss exactly this.
There is no "whole channel" button. You need to list the channel's videos first, then fetch each transcript, then handle the videos that have no captions.

This post shows a way around all three from Python, in about 20 lines.

The short version: one HTTP call

YouTube Transcript Scraper runs on Apify's infrastructure. It tries a datacenter proxy first and automatically retries through a residential IP when YouTube shows a bot check, so you don't manage proxies yourself. You send video, playlist or channel URLs and get one JSON row per video.

You need a free Apify account and your API token (Console → Settings → Integrations). Then:

import requests

TOKEN = "YOUR_APIFY_TOKEN"
url = ("https://api.apify.com/v2/acts/groupoject~youtube-transcript-scraper"
       f"/run-sync-get-dataset-items?token={TOKEN}")

rows = requests.post(url, json={
    "urls": ["https://www.youtube.com/watch?v=iG9CE55wbtY"],
    "outputFormats": ["text"],
}, timeout=320).json()

for row in rows:
    print(row["title"], "-", row["wordCount"], "words")
    print(row["text"][:300])

run-sync-get-dataset-items starts the run, waits for it to finish and returns the rows in one response. Apify caps synchronous calls at about 5 minutes, so use it for a handful of videos. For a whole channel, use the client library below.

Bulk: a whole channel or playlist

pip install apify-client, then:

from pathlib import Path
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("groupoject/youtube-transcript-scraper").call(run_input={
    "urls": [
        "@lexfridman",                                          # a channel handle
        "https://www.youtube.com/playlist?list=PLxxxxxxxxxxxx",  # a playlist
    ],
    "maxVideosPerSource": 50,          # newest 50 per channel / playlist
    "channelContentTypes": ["videos"], # or add "shorts", "streams"
    "languages": ["en"],               # preference order, e.g. ["es", "en"]
    "outputFormats": ["text", "segments", "srt"],
})

out = Path("transcripts")
out.mkdir(exist_ok=True)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    (out / f"{item['videoId']}.txt").write_text(item["text"], encoding="utf-8")
    (out / f"{item['videoId']}.srt").write_text(item["srt"], encoding="utf-8")
    print(item["videoId"], item["languageCode"],
          "auto" if item["isAutoGenerated"] else "manual", item["title"])

A few input details worth knowing (these are the real field names from the actor's input schema):

urls accepts watch?v= links, youtu.be links, Shorts, /live/ URLs, bare 11-character IDs, playlist URLs, channel URLs and @handles, all mixed in one list.
maxVideosPerSource caps each playlist and each channel content type (newest first, up to 5,000). Single video URLs are not affected.
captionType is any (default: prefer creator-uploaded, fall back to auto-generated), manual or auto.
fallbackToAnyLanguage (default true) returns the original-language captions when none of your languages exist. Set it to false to skip those videos instead.
includeVideoMetadata (default true) adds title, channel, publish date, views, likes, duration, description and tags to each row.

Which videos had no transcript?

Videos without captions are not pushed to the dataset and are not charged. They are listed in the run's key-value store:

store = client.key_value_store(run["defaultKeyValueStoreId"])
missing = (store.get_record("NO_TRANSCRIPT") or {}).get("value") or []
for m in missing:
    print(m["videoId"], m["reason"], m.get("availableLanguages"))

reason is NO_CAPTIONS, LANGUAGE_NOT_AVAILABLE or EMPTY_CAPTIONS. Private, deleted or region-blocked videos go to the ERRORS record instead.

Feeding transcripts to an LLM or a RAG pipeline

For summarising, the text field is all you need. For retrieval with citations, use segments: each segment has start, duration and text in seconds, so you can chunk by time and keep a deep link back to the moment in the video.

def chunks(item, seconds=90):
    buf, t0 = [], None
    for seg in item["transcript"]:
        t0 = seg["start"] if t0 is None else t0
        buf.append(seg["text"])
        if seg["start"] - t0 >= seconds:
            yield {"text": " ".join(buf),
                   "source": f"{item['url']}&t={int(t0)}s",
                   "title": item["title"]}
            buf, t0 = [], None
    if buf:
        yield {"text": " ".join(buf), "source": f"{item['url']}&t={int(t0)}s",
               "title": item["title"]}

Each chunk can go straight into your vector store with source as metadata, so answers can link to the exact timestamp.

No-code: n8n, Make, Zapier

If you'd rather not run Python, the actor works with Apify's integrations. In n8n, add the Apify node, choose Run an Actor and Get Dataset, set the actor to groupoject~youtube-transcript-scraper and paste the JSON input above. Connect a Google Sheets or Notion node after it. In Make and Zapier the Apify app has the same "run actor" and "get dataset items" steps. You can also call it from Claude, Cursor or another MCP client through https://mcp.apify.com?tools=groupoject/youtube-transcript-scraper.

Monitor a channel for new transcripts

Turn on onlyNewVideos and give the monitor a stateKey (any name, e.g. ai-podcasts). Each run then saves only videos it hasn't saved before for that key, so you're never charged twice for the same video. Schedule it daily in Apify (Schedules → Create) and connect a Slack, email or Google Sheets integration to get each new transcript as it lands:

run = client.actor("groupoject/youtube-transcript-scraper").call(run_input={
    "urls": ["https://www.youtube.com/@lexfridman"],
    "onlyNewVideos": True,
    "stateKey": "lex-podcast",
    "publishedAfter": "7 days",   # optional: ignore older uploads on the first run
})

Brand-new uploads sometimes get auto-captions a few hours late; the actor re-checks those for 24 hours instead of skipping them.

What it costs

$2.50 per 1,000 transcripts on the free plan ($0.0025 per video), with proxy traffic included. Paid Apify plans get a lower per-result rate automatically. Videos with no captions, and failed, private or deleted videos, are not charged. A 100-video playlist where every video has captions costs $0.25. You can set a maximum charge per run, and the actor stops cleanly before it goes over.

Honest limits

It returns YouTube's own captions. It does not transcribe audio. If a video has no captions at all (music clips, some Shorts), you get nothing for it. For those you need Whisper or a speech-to-text API.
Auto-generated captions are only as good as YouTube's speech recognition. Names and jargon are often wrong. isAutoGenerated tells you which kind you got.
Translation (translateTo) is best effort. YouTube rate-limits its auto-translate endpoint heavily. When it refuses, you still get the original-language transcript and translationError explains why.
No login, by design. Age-restricted, members-only and private videos are reported in ERRORS, not scraped.
If you need a few videos a day from your laptop, you don't need this. youtube-transcript-api or yt-dlp --write-auto-subs is free and works fine from a home connection. This tool is for when you need volume, channels, or a server.
Respect copyright. Transcripts are the creators' work. Summarising for research is one thing. Republishing them is another.

DEV Community: Mustapha

How to track insider buying with Python (SEC Form 4, free EDGAR data)

What a Form 4 row tells you

The free way: SEC EDGAR in ~35 lines

The hosted way: one call for the whole market

A daily insider-buying alert in Slack

Cluster buys

What it costs

Honest limits

How to get YouTube transcripts in bulk with Python (no API key, works on cloud servers)

The short version: one HTTP call

Bulk: a whole channel or playlist

Which videos had no transcript?

Feeding transcripts to an LLM or a RAG pipeline

No-code: n8n, Make, Zapier

Monitor a channel for new transcripts

What it costs

Honest limits

Links